Privacy and Cooperation in Peer-To-Peer Systems

Home , Dat (software), EDonkey network, GNUnet, NAT traversal, Retroshare

Privacy and Cooperation in Peer-to-Peer Systems

Nicolaas Simon Marinus Zeilemaker

Privacy and Cooperation in Peer-to-Peer Systems

Proefschrift

ter verkrijging van de graad van doctor aan de Technische Universiteit Delft, op gezag van de Rector Magniﬁcus prof. ir. K. C. A. M. Luyben, voorzitter van het College voor Promoties, in het openbaar te verdedigen op vrijdag 22 mei 2015 om 15:00 uur

door

Nicolaas Simon Marinus Zeilemaker

ingenieur in de technische informatica geboren te Harmelen, Nederland. Dit proefschrift is goedgekeurd door de promotor: Prof. dr. ir. H. J. Sips

Samenstelling promotiecommissie: Rector Magniﬁcus, voorzitter Prof. dr. ir. H. J. Sips, Technische Universiteit Delft, promotor Dr. ir. J. A. Pouwelse, Technische Universiteit Delft, copromotor Prof. dr. T. Strufe, Technische Universitat Darmstadt Prof. dr. ir. A. K. Pras, Universiteit Twente Prof. dr. A. Hanjalic, Technische Universiteit Delft Prof. dr. C. Witteveen, Technische Universiteit Delft Prof. dr. M. J. G. van Eeten, Technische Universiteit Delft

The work described in this thesis has been carried out in the ASCI graduate school. ASCI dissertation series number 327. This work was supported by the Future and Emerging Technologies programme FP7-COSI-ICT of the European Commission through the QLectives project (grant no.: 231200).

Printed by: Proefschriftmaken.nl || Uitgeverij BOXPress Published by: Uitgeverij BOXPress, ’s-Hertogenbosch Front & Back: Janna Alberts

c 2015 Nicolaas Simon Marinus Zeilemaker ISBN 978-94-6295-195-2 An electronic version of this dissertation is available at http://repository.tudelft.nl/. Acknowledgments

Although, I never had any problems implementing/building stuff, I needed quite some help to be able sell my ideas to the research community. This however, never seemed to bother either my promotor or co-promotor which were confident that a suitable venue for my ideas would be found. A clear vote of confidence, which helped me to see the upsides of yet another reject. Johan, talking with you remains an adventure until this day. Before starting a discussion, I would have a general idea or direction of where I wanted to go to with a paper. However, afterwards the direction usually completely changed. Not because you convinced me, or tricked me into going into another direction, but by seemingly random remarks you made during the discussion. This is a skill which I have yet to discover in anyone else, but was a great help to me during my time as a PhD-student. Henk, without your advice regarding structure and layout wrt organising a paper, this thesis would not have been possible. I know now that all details matter while presenting an idea in a paper, and that without proper structure not a single reader/reviewer will be grasp the gems hidden beneath. Janna, although you did not make any technical contributions to my work, you acted as a sparring partner helping me to understand the core of my ideas. Moreover, you (almost) never complained when I started running a new experiment during dinner, while on holiday, or when we were almost going to sleep. Finally, when travelling to conferences you were a great help, supporting me in preparing my presentation, and not being focussed on shopping at all... This helped me to get some perspective on doing a PhD, its probably not going to change the world.

Contents

1 Introduction1 1.1 Collaboration in P2P...... 2 1.1.1 File Sharing...... 3 1.1.2 Reputation...... 4 1.1.3 Database Synchronization...... 5 1.2 Privacy in P2P...... 6 1.2.1 Hiding your Location...... 6 1.2.2 Hiding your Search Queries...... 6 1.2.3 Hiding your Social Network...... 7 1.3 Popular Attacks on P2P systems...... 7 1.4 Research Questions...... 8 1.5 Contribution and thesis outline...... 10 2 Tribler: Peer-to-Peer Media Search and Sharing 13 2.1 Introduction...... 14 2.2 Design and Features...... 15 2.2.1 Architecture...... 15 2.2.2 Torrent Collecting...... 16 2.2.3 Remote Search...... 17 2.2.4 Video on Demand...... 17 2.2.5 Live Streaming...... 17 2.2.6 Reputation...... 17 2.2.7 Channels...... 18 2.2.8 GUI...... 18 2.2.9 Other deployments: TV and Browser...... 19 2.3 Usage...... 20 2.3.1 Users...... 20 2.3.2 Scientists...... 20 2.4 Conclusion...... 20 3 Large-Scale Message Synchronization in Challenged Networks 23 3.1 Introduction...... 24 3.1.1 Contribution...... 24 3.1.2 Background and Related Work...... 25 3.2 System Design...... 26 3.2.1 Overlay Deﬁnition...... 26 3.2.2 Peer Selection...... 27 3.2.3 Peer Discovery and NAT Traversal...... 27 3.2.4 Bloom Filter Creation...... 28

vii viii Contents

3.3 Evaluation...... 30 3.3.1 Overlay...... 30 3.3.2 Propagation...... 31 3.3.3 Synchronization...... 31 3.3.4 Bandwidth Requirements...... 33 3.3.5 Bloom Filter Optimizations...... 34 3.3.6 NAT Traversal...... 35 3.3.7 Churn Resilience...... 36 3.3.8 Throughput...... 36 3.4 Deployment Results...... 39 3.5 Conclusion...... 39 4 Open2Edit: a Peer-to-Peer platform for collaboration 41 4.1 Introduction...... 42 4.2 Related work...... 43 4.3 Motivation...... 44 4.4 Dispersy...... 44 4.4.1 Overview...... 45 4.4.2 Permissions...... 45 4.5 Open2Edit design...... 46 4.5.1 CommunityOverlay...... 47 4.5.2 CommunityDiscoveryOverlay...... 47 4.5.3 Authentication...... 48 4.5.4 Flexibility...... 48 4.5.5 Detecting conﬂicts...... 49 4.6 Tribler deployment...... 49 4.6.1 Tribler overview...... 49 4.6.2 Implementation details...... 50 4.6.3 User interface...... 51 4.7 Experiments...... 51 4.7.1 DAS-4 emulation...... 51 4.7.2 Internet deployment...... 54 4.8 Discussion...... 55 4.9 Conclusion...... 57 5 Building a Privacy-Preserving Semantic Overlay for Peer-to-Peer Networks 59 5.1 Introduction...... 60 5.2 Related Work...... 61 5.3 Preliminaries...... 62 5.3.1 RSA...... 62 5.3.2 Paillier...... 63 Contents ix

5.4 Privacy-Preserving Protocols...... 63 5.4.1 Private Set Intersection Protocol I...... 63 5.4.2 Private Set Intersection Protocol II...... 64 5.4.3 Private Set Intersection Protocol III...... 65 5.4.4 Security Discussion...... 67 5.4.5 Improving Discovery Speed...... 68 5.5 Experimental Results...... 68 5.5.1 Dataset...... 68 5.5.2 Experimental setup...... 69 5.5.3 Results...... 70 5.6 Conclusion...... 72 6 4P: Performant Private Peer-to-Peer File Sharing 75 6.1 Introduction...... 76 6.2 Related Work...... 77 6.3 Cost and Limitations of Current Systems...... 79 6.3.1 Bandwidth Cost...... 80 6.3.2 Limitations...... 82 6.4 4P Design...... 82 6.5 4P Implementation...... 83 6.5.1 Semantic Overlay...... 83 6.5.2 Search Messages...... 84 6.5.3 Downloading Content...... 85 6.5.4 Choosing the initial TTL, IEP, and FEP...... 86 6.6 Security...... 87 6.7 Experiments...... 88 6.7.1 Dataset...... 88 6.7.2 Emulation setup...... 89 6.7.3 Evaluated strategies...... 89 6.7.4 Results...... 90 6.8 Conclusion...... 93 7 ReClaim: a Privacy-Preserving Decentralized Social Network 95 7.1 Introduction...... 96 7.2 Related Work...... 97 7.3 Goals and Security model...... 98 7.4 Features...... 99 7.4.1 Establishing a new Friendship...... 99 7.4.2 Locating Friends...... 100 7.4.3 Posting a Message...... 101 7.4.4 Unfriending...... 101 7.5 Protocol Details...... 101 7.5.1 PSI protocol...... 101 7.5.2 Message Distribution...... 103 7.5.3 Message Body...... 104 7.5.4 Large Binary Download...... 104 x Contents

7.5.5 Peercache...... 105 7.6 Application Design...... 105 7.7 Evaluation...... 106 7.7.1 Dataset...... 106 7.7.2 Setup...... 107 7.7.3 Boostrapping...... 107 7.7.4 Message distribution...... 108 7.8 Conclusion...... 109 8 Conclusion 111 8.1 Conclusions...... 111 8.2 Suggestions for future work...... 112 Bibliography 115 Summary 125 Samenvatting 127 Curriculum Vitæ 129 List of Publications 131 1 Introduction

In a traditional centralized network architecture, the central server (or a group of servers) needs to be able to handle all requests issued by clients. This, as shown by popular websites such as Youtube1 and Facebook2, is not necessarily a problem but requires significant investments in infrastructure. E.g. Youtube uses a large network of servers distributed across different continents to be able to serve data to clients from a server which is geographically close to them [94]. In contrast, in a Peer-to-Peer network (P2P) the resources available at clients are employed. For instance, by using spare bandwidth at clients, a P2P network reduces the bandwidth requirements at the server. This lowers the initial investment, as less servers need to be bought. However, the protocols which allow clients to collaborate need to be carefully designed, as clients in a P2P network are highly susceptible to attacks. Attackers can try exploit their willingness to collaborate, in order to gain knowledge, perform synchronized attacks on others, or free-ride the system. Moreover, attackers can more easily target and influence a single client in a P2P network, hence a need arises for collaboration between honest clients to detect possible malicious acts. A considerable amount of research focuses on collaboration and pooling of resources, improving efficiency, reducing the vulnerability to attacks, etc. However, most research efforts investigate solutions which overcome bandwidth bottlenecks, in order to efficiently share files, or creating backup solutions. Overlooking that collaboration in a P2P network can improve upon other forms of centralized solutions as well. In this thesis, we focus on collaboration between clients to distribute messages, improve privacy, and implement decentralized search. Choosing a decentralized over a centralized solution will initially cause trivial problems (such as making a piece of information available to all clients) to be substantially more difficult. However, after overcoming those initial problems, a decentralized solution requires a smaller

1www.youtube.com 2www.facebook.com

1 2 1. Introduction

Centralized Unstructured

Structured Hybrid

Figure 1.1: Diﬀerent overlay structures

investment in servers, has less running costs, improves privacy due to not having to store all data in the same place, and can be made more robust against attacks as there is no central point of failure. Nevertheless, in order to exploit those beneﬁts fully, care must be taken when designing the collaboration protocols. Poorly designed protocols can cause even greater privacy problems compared to a centralized solution, as an attacker would only be required to listen and record the messages sent by clients to construct a dataset consisting of potentially privacy sensitive information.

1.1. Collaboration in P2P In a P2P network, the roles of the participating peers (or clients) are equal. The term peer stands for “a person of the same age, status, or ability as another specified person” 3. Without any specific roles, a P2P network depends on collaboration between peers in order to achieve a working system. Figure 1.1 gives an overview of the different overlay (network) structures used in P2P networks. In the following subsections, we give an example of each of the overlay

3http://www.oxforddictionaries.com/definition/english/peer 1.1. Collaboration in P2P 3 structures. Morover, we will elaborate on current state-of-the-art P2P collaboration schemes, their goals, and their security against malicious peers. 1

1.1.1. File Sharing Using the available bandwidth at peers can substantially reduce the bandwidth required at a server. In a P2P file-sharing network, the server could be compared to the first peer injecting a file into the network. This peer alone usually does not have the bandwidth to send a full copy of the file to all other peers requesting it. However, after the initial peer has uploaded one or more copies of the file to others, some of these other peers will assist the initial peer by uploading the file to others. One of the first P2P file-sharing systems, Napster, was cited to be the “destruc- tor” of copyright [53]. When introduced in 1999, it allowed its users to easily share files. Searching for files relied on central servers, but after receiving a list of files which were ready to be downloaded, a direct connection to another peer was made in order to download a file. Peers in the Napster network downloaded a file from a single source which would upload a complete file. However, a peer could choose between sources to manually to improve its download speed. Gnutella [82] removed the need for centralized search servers by implementing a decentralized search protocol. It used an unstructured overlay to connect peers with each other in a more or less random manner. An unstructured overlay has low maintenance costs but efficiently searching such it can be hard. Initially, the developers opted for flooding. This is a technique in which a search message includes a TTL (time-to-live) property which is decremented whenever a peer receives the message. All peers forward a search message to all its neighbors (active connections) while the TTL is larger than 0. The TTL value is set by the initial peer and determines how many other peers it reaches. However, even small values for the TTL will result in a massive amounts of search messages to be sent. eMule [54] started off as open-source client to the eDonkey network, however the developers quickly extended its functionality by integrating a Kademlia based DHT. A DHT (Distributed Hash Table), is a popular method to create a structured overlay. In contrast to an unstructured overlay, maintaining it costs more, but searching or looking up values in a DHT requires less messages to be sent. Every peer in a DHT initially generates an identifier, usually by hashing its own IP-address. The identifiers position peers in the DHT ring, in which the peer with identifier 0 is positioned at the top, peer 1 next to him clockwise, etc. Next, peers maintain a connection to their neighbors both before and after them, and peers which are located further away in the ring with increasingly larger distances. Distances between two identifiers are computed by their XOR distance. When searching a DHT, a peer first converts the keyword/keywords into an identifier by applying the same hash function. A peer then forwards the search message to the peer of whose identifier is the closest to this hashed keyword. This peer will do the same, until the message arrives at the peer responsible for the keyword, and replies. This causes a search message to be forwarded O (log N) times, as Kademlia [63] provably to contacts this many peers during a lookup. Reducing both the latency and number of messages sent, makes it a efficient method for looking up search 4 1. Introduction

results. However, in order to allow the responsible peer to reply with a list of 1 results, every peer in the network needs to announce what it is sharing. Therefore, each peer sends a message per keyword it has matching files for to the responsible peer. Without optimizations, this approach results in a similar overhead compared to flooding an unstructured overlay. Kazaa [58] attempted to combine the best of both worlds. Its overlay, at its core, is an unstructured one, but in this unstructured overlay some peers are more important than others. These peers are called superpeers, and they index “ordinary” peers connected to them. Superpeers are peers which have a fast Internet connection, big CPU, no firewall, and have been online for a long time. Amongst superpeers a separate superpeer network is constructed, which consists of much fewer peers. Whenever an ordinary peer wants to search the network, it sends the query to the superpeer it is connected to, which then floods the superpeer network and replies with the result list. Because the superpeer network contains fewer peers, and the peers it does contain have above average performance, flooding this network can achieve excellent results. Creating a network consisting of ordinary and super peers is often referred to as a hybrid overlay. BitTorrent [74] focuses at improving the speed at which peers in a P2P network can download files. It splits a file into pieces, and uses a descriptor called a torrent to store their checksums. Moreover, it uses an identifier called the infohash to distinguish files. The infohash is computed by hashing the info part of the torrent, which contains amongst others the filename and the checksums of the pieces. Using the checksums, a peer can download multiple pieces in parallel and verify each piece individually. This overcomes a substantial bandwidth bottleneck, as a peer can now download from multiple peers at the same time. The group of peers which are downloading/uploading the same file is often referred to as a swarm. Another improvement BitTorrent incorporated was rarest-first-downloading, this improves the overall health of the swarm wherein peers are downloading/uploading. Each peer using this policy maintains a list of pieces available at its directly connected neighbors, and then downloads the pieces which are the “rarest” first. Down- loading the rarest pieces first helps the overall piece availability, and improves the ability to barter (trade) for pieces with other peers [57]. Finally, BitTorrent incorporated an incentive mechanism to promote uploading. Without an incentive mechanism, there is no reason for peers to upload while downloading a file. But because of tit-for-tat, the incentive mechanism implemented by BitTorrent, each peer will only upload to the peers which have given it the most in return. These peers will be unchocked, which allows them to send requests for pieces. Additionally, each peer has an optimistic unchocke slot to allow a random peer to prove that its willing to reciprocate. Although, tit-for-tat does not solve the incentive problem (peers can still leave before uploading as much as they have downloaded), it improves upon all existing systems.

1.1.2. Reputation Being able to determine the reputation of a peer, which reflects its long-term behaviour, can improve both the performance and the security of a P2P network. As 1.1. Collaboration in P2P 5 described above, BitTorrent lacks an incentive mechanism to prevents peers from leaving before they upload as much as they downloaded. Similarly, peers in Gnutella 1 should ignore, or neglect those which do not share. As shown by Adar et al. [4], almost 70% of peers in Gnutella do not share any files (and hence only download) and roughly 50% of replies to search queries are from the top 1% of sharing hosts (overloading those in the process). A reputation system, capable of recording/representing this behaviour in a reputation score, will allow peers to avoid such free-riders and improve the download speed of peers which upload at least as much as they download. Generally, a reputation system warns honest peers about maliciously behaving others, thereby allowing them to be more vigilant while interacting with them. However, creating a distributed reputation system has proved to be very hard, as it must be able to resist attackers trying to exploit it. Having a reputation system which is easily exploitable might even cause more harm to a P2P network than good, as honest peers will trust malicious peers instead being cautious with everyone. An example of a deployed reputation system is BarterCast [67]. In this system peers collect BarterCast records describing up and download traffic between two peers. Both peers sign this record, indicating that both agree on the content. By collecting BarterCast records, a peer can estimate the reputation score for the peers its interacting with based on their previous download/upload behaviour. However, a peer cannot collect all BarterCast records as there are simply too many, and hence it needs to decide which to collect/keep. Therein lies the difficulty of a reputation system, a peer needs to predict which peers it needs to collect the BarterCast records from, and which peers it can ignore. Moreover, it needs to do so, without being susceptible to attacks. A recent improvement to BarterCast uses similarity between peers to determine which records are interesting and which are not [34].

1.1.3. Database Synchronization Synchronizing a database between two or more peers is a well defined problem. Demers et al. [35] outlined, in 1987, methods which created consistency across many nodes linked together using a large heterogeneous, slightly unreliable and slowly changing network. Moreover, he stated that these nodes can only be consistent after all updating activity has stopped. Compared to file sharing, database synchronization is a slightly different problem as databases can change while synchronizing. Synchronizing a dynamic/changing file prevents peers from computing checksums in advance and hence prevents other peers from verifying those after receiving part of it. Moreover, a method is required which is able to detect which parts of a database are missing/updated. A common approach is to use Merkle Trees [66] as implemented in Amazon’s Dynamo [32] distributed key-value store. Each peer builds a Merkle Tree of its current database by first hashing each row, and next building a tree by inserting parents which are the hash of its children. The root(hash) of a Merkle Tree can be used to compare if there are any differences between two databases, and if so by traversing the tree peers can efficiently determine where the changes are located. 6 1. Introduction

1 1.2. Privacy in P2P A P2P network can also yield substantial privacy improvements over a solution which uses a central server. By removing the central component, a P2P network gets rid of the single point of storage of privacy sensitive information highly susceptible to attacks. Moreover, a peer in a P2P network has more control over what is revealed and to whom than in a typical client-server setup wherein a client can only decide to either trust a server or not. In the following subsections, we will elaborate on current state-of-the-art P2P privacy preserving protocols, their goals and their security against malicious peers.

1.2.1. Hiding your Location Tor4 is an open-source project aiming to improve the privacy and security of its users on the Internet. Tor uses virtual tunnels to hide the location, or IP-address, of the original user requesting a website. All data sent over the tunnels is encrypted, only the source and endpoint of a tunnel can read the messages. The endpoint of a tunnel needs to be able to read the message, as it will act as if its connecting to the website. Tor employs onion routing in order to create the virtual tunnels. Users sending a message over a virtual tunnel encrypt it multiple times, with a separate key known only to each hop in the tunnel. After receiving a message, each hop in the tunnel decrypts it before forwarding it to the next hop, peeling the layers of the onion. The endpoint does the same, and ends up with an unencrypted request. When the endpoint wants to send a reply, it encrypts with its own key, and each hop in the virtual tunnel does the same. Finally, when the source receives the encrypted reply, only it can decrypt it as it has all the keys from each hop in the tunnel. After some time, Tor destroys its current tunnels and builds new ones. This prevents endpoints from being able to collect too much information. However, the protocols a users runs on top of the virtual tunnels can still leak privacy sensitive information, and the user itself is still vulnerable to attacks. Recently this vulnerability was exploited by the FBI, which injected javascript into a website to break out of the Tor tunnels and make the user exposes its location5.

1.2.2. Hiding your Search Queries Searching could also be made more privacy preserving using a P2P network. By hiding the user issuing a query, it can be made more diﬃcult to create individual proﬁles of users as is commonly performed by centralized websites. Popular examples of P2P networks which aim to provide its users with privacy preserving search are Freenet [22], Gnunet [13], OneSwarm [46], and RetroShare [79]. These networks employ well known techniques from cryptography in order to protect the privacy of their users. Examples of these are path length obfuscation, wherein peers construct paths of a unknown length between the user searching and the user replying to a query. End-to-end encryption, where all messages sent between

4https://www.torproject.org/ 5http://www.wired.com/2013/09/freedom-hosting-fbi/ 1.3. Popular Attacks on P2P systems 7 the user searching and the user replying are encrypted, and ﬁnally the latter two networks use a Friend-to-Friend network, wherein users have to manually connect 1 to others by establishing friendships with them. Most networks achieve a certain level of anonymity against collaborating peers in the network, however none are able to resist a local eavesdropper. Such an attacker is able to observe all network traﬃc received by and sent from a node, hence can detect when a peer creates a new query, replies to a query etc. This does not necessarily mean that the attacker can read the queries, as those could be send in encrypted from, but simply that it can prove that an user is issuing them.

1.2.3. Hiding your Social Network Social Networks are another prime example of a field wherein a P2P network could potentially improve the privacy of its users. Instead of relying on the security of centralized websites, peers can limit the exposure of their data by storing it in encrypted form and deciding whom can decrypt it. We distiguish two approaches to hiding your social network. First, approaches similar to Scramble [10] use an existing social network to relay encrypted updates to friends. It uses broadcast encryption to send multiple friends an encrypted update which only needs to be stored once at an external storage provider. Broadcast encryption is a technique which allows multiple users using different decryption keys to decrypt the same file. Using a browser plugin, Scarmble hides the decryption process from its users letting them communicate more securely with ease. However, due to freeriding on an existing network, these approaches can be easily blocked. Alternatively, another approach is to implement a new social network which does not rely on existing centralized social networks. Herein systems such as Diaspora (a federated approach) [1], SafeBook [31], and Cachet [71] reside. Although Di- aspora is the most well known, it does not have a truly decentralized design, but allows users to setup their own server which then can be integrated into the current system. SafeBook and Cachet both rely on a DHT to exchange updates between users. However, Safebook additionally requires a centralized identity server which hands out identities after verifying an existing one. Cachet does away with that requirement, but exposes the social graph in order to optimize performance.

1.3. Popular Attacks on P2P systems Without a reputation system, peers in a P2P network rely on an incentive mechanism designed to be robust to strategic manipulation. Such incentive mechanisms are often derived from game theory, wherein the behaviour of users is modelled when confronted with strategic decisions [77]. An example from game theory is the public goods game, wherein each player has a private pool of tokens. In each round a player has to decide how many of his private tokens he/she wants to put in a public pot. The tokens in the public pot are multiplied by a factor, and then divided equally over all players. In this game, the proﬁt is maximized if all players put all of their tokens in the public pot. However, a player that does not contribute will still receive a payout. Hence, a rational player 8 1. Introduction

would not contribute at all, and only profit from the contributions of others. A very 1 successful strategy which allows a single player to freeride the system. However, if all players employ this strategy, no-one will benefit from the multiplication factor of the public pool. The incentive mechanism of BitTorrent prevents others from freeriding. How- ever, without initial risk taking (the optimistic unchocke slot), the protocol cannot bootstrap as no preexisting trust relationships exist. Locher et al. [59] have shown that this mechanism can be easily exploited, if an attacker simply connects to many peers, its probability to be selected as the lucky randomly selected unchocke peer increases and it receives bandwidth for free. Similar to the public good game, this is a successful strategy if not employed by all peers in the network. A more general attack which do not target a specific P2P protocol, but is appli- cable to most, is called the sybil attack [37]. Herein an attacker poses as multiple peers by creating multiple identities. Using these identities an attacker can, for instance, gain reputation by creating fake upload records between them. This results in a single identity having a great reputation, as it seems that it uploaded to many other peers, and numerous identities which have a bad reputation. By simply throwing away the identities with a bad reputation, the attacker now has a great reputation without needing to upload to anyone. Somewhat similar is the eclipse attack [88] wherein, an attacker joins the P2P network more than once, and uses those peers to control the view of others. Eclipsing a peer consists of controlling all/most of its connections to the network, and by doing this the attacker can greatly influence the view of a peer on the network. Moreover, after eclipsing a peer, an attacker can gain much more knowledge on the behaviour of it, greatly impacting the privacy of a peer, e.g. it can detect which search queries a peer issues, which files it is uploading, etc. Both the sybil and eclipse attacks have been well researched, and numerous papers have been published on evaluating their impact [52, 62], reducing it [85, 100], or even preventing the attacks from happening [60, 88]. However, most “solutions” either rely on a trusted third party (TTP) in charge of handing out identities, or using another network (such as a social network) wherein edges between peers indicate friendship instead of being created at random. The idea being that attackers have a lot of difficulty in creating edges, as users in the social network will not accept friend requests from strangers.

1.4. Research Questions Improve collaboration while maintaining privacy has many challenges, and as such not many solutions have been developed which are ready for deployment w.r.t. efficiency and practicality. In this thesis we aim to address the following research questions: How can we efficiently synchronize messages in a P2P network? A lot of research effort was put into analyzing and optimizing file sharing in a P2P network, much less is known about efficiently synchronizing messages. Although seemingly a trivial task, a P2P network suffers from intermittent connectivity between peers, large delays on links, and the complete absence of end-to-end paths. To overcome 1.4. Research Questions 9 these problems and still be able to synchronize messages between all peers in a P2P network, the need for a stateless, NAT-puncturing, synchronization method arises. 1 Moreover, we envision a scenario in which all peers must be able to create messages, without knowing beforehand which of them will. This prevents peers from being able to detect when they have received/synchronized all messages, as new messages are created at irregular intervals. How can we use a P2P network to build a collaborative platform that does not depend on central servers? Websites such as Wikipedia and StackExchange have shown that it is possible to build a system in which user knowledge, judgement, and expertise is donated without any apparent remuneration in return. Additionally, P2P networks have shown to scale to accommodate substantial amounts of concurrent users, without the need for servers. By combining a P2P network, with the methods used by collaboration websites to encourage user participation, a hybrid system will arise which in contrast to the centralized collaboration websites does not need large initial investments, or monthly costs to maintain an existing infrastructure. However, the latency introduced by the P2P network might cause collaboration to fail, as users spent effort on fixing/improving older versions of an article which has already been fixed. How can we cluster peers by taste without exposing their privacy? It has been shown by Voulgaris et al. [97] that clustering peers by taste, or preferences, can result in a P2P network in which peers can efficiently perform keywords search. However, in order to allow a peer to find other peers which are similar to it taste- wise, protocol developers often resorted to letting peers broadcast their preferences in plain-text. This results in a peer fully exposing its preferences for items or content, and therefore revealing potentially privacy sensitive information. Can we employ methods from the cryptographic domain to improve upon this? And what are the costs associated to using those methods? How can we query a P2P network efficiently while hiding the identity of the issuer? A semantic overlay, a P2P network in which we cluster peers by taste, holds a great promise if we manage to build it in a private manner. However, even after constructing a semantic overlay, care must be taken in order to not reveal the source of a search query. Typically, a peer in a semantic overlay would send a query to its neighbors which then reply with a result list. Hence, any peer receiving a query will know that the peer which sent the query is the source. Can we improve upon this? And what are the costs associated to hiding the source of a query? Can a semantic overlay reduce the bandwidth costs compared to currently deployed networks? How can we enable peers to communicate with their friends privately, without requiring them to be online at the same time? In an online social network (OSN), friends communicate with each other using a central server storing all their messages, likes, images, etc. Companies like Facebook provide its users with such a service, however in return they use private data of users to attract advertisers in order to be able to run and maintain the infrastructure required to be able to handle a network consisting of more than 200bn friendships. A P2P network could be viable alternative as it does not require the same amount of infrastructure to 10 1. Introduction

1 keep running, and hence does not require a similar investment in infrastructure. 1.5. Contribution and thesis outline The contributions of this thesis are as follows: Introducing Tribler, the P2P client used as a testbed for our P2P experiments (Chapter2) Tribler is an open-source software project initiated in 2005. It was funded by multiple European research grants, which over time helped it to become a viable testbed for testing new P2P protocols. Currently, Tribler has more than 1500 users daily. In this thesis, Tribler was used to evaluate the performance of Dispersy (Chapter3) and Open2Edit (Chapter4). This chapter is largely based on our work published in ACM MM 2011 [102]: N. Zeilemaker, M. Capotă, A. Bakker, and J. Pouwelse. Tribler: P2P media search and sharing. In Proceedings of the 19th ACM international conference on Multimedia, 2011. Synchronizing messages in a P2P network with Dispersy (Chapter3) Synchronizing messages in a P2P network, or in a network which isn’t necessarily client-server, has been well researched. Even going back as far as Demers et al. [35], which in 1987 described a theoretical model outlining the synchronization performance of a single message being created by a peer in a random network. Us- ing this theoretical model, we design and implement a fully decentralized database build around a semi-random network. Additionally, we integrate NAT-traversal, to allow the database to continue to operate even when deployed in a challenged network such as the Internet. Finally, we extensively evaluate the performance of Dispersy, measuring propagation and synchronization speed, its bandwidth requirements, churn resilience, and throughput. This chapter is largely based on our work published in ACM SAC 2014 [107]: N. Zeilemaker, B. Schoon, and J. Pouwelse. Large-scale message synchronization in challenged networks. In Proceedings of the 29th Annual ACM Symposiumon Applied Computing (SAC), 2014. Improving decentralized collaboration between users with Open2Edit (Chapter4) P2P file-sharing networks owe much of their success to the efficiency by which they can combine user contributed resources. Similarly, websites such as Wikipedia and StackExchange, connect experts with novices in such a manner that experts donate their knowledge without any remuneration in return. In this chapter, we introduce Open2Edit, a platform which enables developers to create decentralized versions of collaboration websites. In contrast to related works, Open2Edit does not implement a decentralized Wikipedia, but aims to be a flexible platform upon which collaborative systems can be developed. Using Open2Edit, we implement a YouTube-like media sharing system, integrate this system into Tribler, and show that users collaborated in discovering the best channels by casting over 100,000 votes. This chapter is largely based on our work published in IFIP Networking 2013 [101]: N. Zeilemaker, M. Capotă, and J. Pouwelse. Open2edit: A peer-to-peer platform for collaboration. In IFIP Networking Conference, 2013. Building a semantic network while preserving the privacy of peers 1.5. Contribution and thesis outline 11

(Chapter5) In this chapter, we evaluate three methods which are able to compute the cardinality of the intersection of two sets without disclosing the sets themselves. 1 This problem is referred to as Private Set Intersection Cardinality (PSI-C) problem, and we modify three existing well known solutions. We implement a semantic overlay wherein peers compute their similarity to others using the three methods, and show their performance differences; comparing the CPU time, bandwidth, and speed of discovering the most similar peers. This chapter is largely based on our work published in IEEE WIFS 2013 [103]: N. Zeilemaker, Z. Erkin, P. Palmieri, and J. Pouwelse. Building a privacy- preserving semantic overlay for peer-to-peer networks. In Proceedings of the IEEE International Workshop on Information Forensics and Security, 2013. Performant private file-sharing using 4P (Chapter6) A semantic overlay holds great promise in creating scalable decentralized search. However, searching on top of a semantic overlay will typically still expose the initiator of a query. In this chapter, we show that by combining a semantic overlay with existing privacy enhancing features we can substantially reduce the number of messages sent per query while still achieve a high recall. As we show in Chapter5 it is possible to create a semantic overlay in a private manner, and by combining this with probabilistic query forwarding, path uncertainty, and encrypted links, 4P implements a performant private P2P file sharing system ready for deployment. We extend a PSI-C method described in Chapter5, to allow two peers to agree upon a session- key which is kept safe from a man-in-the-middle attack. During the evaluation, we compare the performance of 4P to Gnutella, OneSwarm, and RetroShare. This chapter is largely based on our work published in IEEE P2P 2014 [105]: N. Zeilemaker, J. Pouwelse, and H. Sips. 4P: Performant Private Peer-to-Peer File Sharing. In Proceedings of the IEEE International Conference on Peer-to-Peer Computing (P2P), 2014. Creating a Privacy Preserving online social network (Chapter7) Us- ing a P2P network to create a more privacy preserving online social network has been widely researched. However, most papers focus on either storing the data in encrypted form while letting users still access it, or are build around a DHT wherein a user relies on strangers to store its data. A semantic overlay, wherein peers only connect to their friends and friends-of-friends, might be better suited as these peers have a greater incentive to store your data. In this chapter, we introduce ReClaim, a privacy preserving decentralized social network which builds upon our previous work of Chapter5 to construct a semantic overlay which connects peers to their friends and friends-of-friends. Using those connections, we create a social network which uses friends to store messages. Every wallpost a user creates, gets encrypted with the public key of each destination, and then stored at the friends of each destination. This allows two friends to continue to communicate even when they are not online at the same time. Using Bloom filters, peers can detect which of the messages destined for your friends you did not stored locally. This chapter is largely based on our work published in USENIX FOCI 2014 [104]: N. Zeilemaker and J. Pouwelse. ReClaim: a Privacy-Preserving Decentralized Social Network. In Proceedings of the USENIX Workshop on Free and Open Com- 12 1. Introduction

1 munications on the Internet, 2014. 2 Tribler: Peer-to-Peer Media Search and Sharing

Tribler is an open-source software project facilitating search, streaming and sharing content using P2P technology. Over 1,200,000 people have used Tribler since the project started in 2005. The Tribler P2P core supports BitTorrent-compatible downloading, video on demand and live streaming. Aside from a regular desktop GUI that runs on multiple OSes, it can be installed as a browser plug-in, currently used by Wikipedia. Additionally, it runs on a 450 MHz processor, showcasing future TV support. We continuously work on extensions and test out novel research ideas within our user base, resulting in sub-second content search, a reputation system for rewarding upload, and channels for content publishing and spam prevention. Presently, over 2000 channels have been created, enabling rich multimedia communities without requiring any server.

Parts of this chapter have been published in ACM MM 2011 [102].

13 14 2. Tribler: Peer-to-Peer Media Search and Sharing

2.1. Introduction As of 2001 BitTorrent has transformed the manner in which files are transfered over the Internet. While not the first P2P file-sharing application, it became the most successful because of its unprecedented efficiency. However, BitTorrent fails to 2 properly address important issues, such as search, video on demand and long-term contribution incentives. In this chapter, we introduce Tribler, our BitTorrent-based open-source project that adds missing features and new use-cases, while remaining fully backwards compatible. BitTorrent introduced a new mechanism for transferring files that maximizes the resource contribution of peers. In BitTorrent, before the transfer begins, files are split into small pieces which can then be individually downloaded from different peers. This way, downloaders (called leechers) can upload completed pieces to other leechers, without the need to have the whole file first. Furthermore, this uploading is encouraged by the tit-for-tat incentive mechanism built into BitTorrent: a leecher will rank its peers by upload speed and will upload in return only to the fastest uploaders. Peers which have completed the file can help others by sending them pieces for free, these peers are called seeders. To protect against transfer errors and malicious modifications, each file piece is accompanied by its SHA-1 cryptographic hash. The peer that initiates the transfer of a file will first record these hashes in a metadata file called a torrent. This torrent has to be distributed out-of-band to the users interested in downloading the file, e.g., through email or the Web. Only after getting hold of the torrent can a BitTorrent client start downloading the files. This is one of the problems of the original BitTorrent protocol. Searching for torrents is also not part of the BitTorrent protocol and is normally done through centralized websites. The original BitTorrent protocol had another important design flaw. Peer discovery was done through centralized servers called trackers. The address of a tracker is recorded in the torrent and every interested peer will announce itself to the tracker. In turn, the tracker will provide the peer a list of other peers currently uploading or downloading the file. Whenever the tracker is offline, new peers cannot join the swarm (the group of peers sharing the file) because there is no other way to discover it. Initiated in 2005, Tribler [3]1 is an open-source research project funded by multiple European research grants, which addresses the problems of the original Bit- Torrent protocol and extends it beyond file sharing to new areas such as video on demand and live streaming. The existence of Tribler was made possible by the open nature of BitTorrent. The protocol specification is freely available [23] and the original BitTorrent software used an open-source license [14]. After extensive research and numerous incremental improvements, Tribler has implemented and deployed fully decentralized remote search, channels, reputation, video on demand, and live streaming.

1Sources and documentation can be found at https://github.com/Tribler/tribler 2.2. Design and Features 15

GUI

Tribler SwarmPlayer

2 libTribler “glue”

Download Protocols Remote Search Video On Demand Engines

Channels Live Streaming

Torrent Collecting

Interact with

BarterCast Reputation

Run on top of

Overlay/Dispersy

Figure 2.1: Tribler architecture

2.2. Design and Features Tribler consist of two base components, the core which implements all communication protocols, and the GUI which presents the features of the software to the user. As part of the project we have implemented a secondary GUI by repackaging the core as a browser plug-in called SwarmPlayer. This section ﬁrst describes the overall structure of the core, then the Tribler GUI, and ﬁnally the SwarmPlayer.

2.2.1. Architecture The core of Tribler can be further divided into three parts: (1) the overlay which allows for communication between Tribler peers, (2) the protocols implementing remote search, torrent collecting etc., and (3) the download engine, which allows users to download files in BitTorrent swarms. One of the missing features of BitTorrent is cross-swarm identification of peers. Tribler addresses this by using public key cryptography to generate an identifier called PermID for each peer. This PermID is stored separately from the Tribler installation files, so that upgrading or reinstalling Tribler do not cause it to change. 16 2. Tribler: Peer-to-Peer Media Search and Sharing

The PermID allows us to identify and authenticate messages from users, allowing for a secure method of communication. A key feature of Tribler is its custom overlay network. Initially implemented as a BitTorrent swarm with a predefined identifier, it allowed Tribler peers to exchange information using custom overlay communication protocols. At the same time, the 2 overlay does not require more TCP sockets to be opened in addition to regular BitTorrent. More recently however, Tribler switched to a new method which uses a separate overlay per protocol, and UDP instead of TCP to be able to puncture NAT-firewalls. In the search overlay, Tribler peers are exchanging preferences lists containing infohashes of downloaded torrents. Infohashes are a commonly used as an identifier for BitTorrent swarms. By comparing a peer’s own downloads to those of others, it can calculate a similarity between them. Peers with a high similarity, those which have downloaded the same torrents, are called taste buddies, and connections to those peers are kept open. During the BuddyCast handshake, in addition to the preference list, peers also exchange which torrent files they have collected recently. Additionally, instead of sending the preference lists in plain-text, peers in the search overlay construct a Bloom filter containing their preferences. Bloom filters are compact hash-representation which allow for efficient membership testing. A peer receiving this Bloom filter can use it to check if it has any overlapping preferences, but determining the preferences of a peer by testing all possible infohashes will result in a lot of false positives, due to the compression of the Bloom filter. Providing some degree protection from attackers trying to determining the of the preferences of a peer.

2.2.2. Torrent Collecting After discovering new infohashes using the search overlay, a peer will start collecting the associated torrents. An algorithm determines which torrents are the most interesting, from this list, and should be collected. It does so by sorting the torrents based on its popularity or if it exists in a channel or not. Popular torrents will thus be requested first. Torrents are downloaded using PPSP. PPSP can be compared to BitTorrent as it splits files into smaller pieces which then can be downloaded in parallel from multiple sources. However, in contrast to BitTorrent, PPSP uses a Merkle tree to both identify and verify pieces of the data. A peer can therefore verify pieces of a PPSP swarm using only the roothash of the Merkle tree. In the search overlay, peers specify a roothash for each torrent file they recently downloaded. When requesting a torrent, peers use the associated roothash to download the file from the sending peer. While searching, a torrent is fetched when the user wants to see its details, e.g., file name, number of peers in the swarm. The first 25 torrents are prefetched to improve the speed of showing the user details of these highly relevant torrents. If no peer is willing to send the torrent, it is downloaded using the BitTorrent extension for sending torrents [43]. After receiving a torrent, a peer will store its information in a embedded database called the MegaCache. This is a local SQlite database, in which we store information 2.2. Design and Features 17 from up to 50,000 torrents, is used during search.

2.2.3. Remote Search A Tribler peer will, by default, maintain connections to 10 random peers and 10 taste buddies. By doing so, it will create a semantic overlay consisting of peers which are 2 very similar it. Having a connection to similar peer allows us to implement remote search in a very eﬃcient manner. By only using these connected peers, instead of ﬂooding the network as is usually done, we can show very relevant results in less than a second. In addition, due to each Tribler peer storing information for up to 50,000 relevant torrents, local results can be shown to the user even quicker, resulting in a very responsive search. The databases of the connected peers, can store up to 1,000,000 relevant torrents improving overall recall of the system.

2.2.4. Video on Demand Another novel feature in Tribler is our video on demand (VOD) implementation. BitTorrent, by default, uses a piece picking policy which downloads the rarest piece. As a result, a peer maximizes the demand for the pieces it downloads, allowing the peer to trade them with others. By modifying the piece picking policy of BitTorrent, we can allow users to start playing the video even before completing the download. Tribler uses an internal HTTP server to determine the playback position of the video player and adjust the piece priorities accordingly. This allows users to not only start playing the video, but also seek through the ﬁle before the download is complete. Pieces of a ﬁle which are before the playback position are only downloaded after all pieces after the playback position have been downloaded.

2.2.5. Live Streaming A slightly diﬀerent technique, compared to VOD, is used to provide users with live streaming using PPSP. Since the pieces are not known in advance (the video content is generated live), it is impossible to construct the Merkle tree and hence compute its roothash. The solution is to let the seeder sign all pieces with a public/private key pair, and the leechers verify them using the roothash the public key. This way, all peers that receive the torrent can verify the authenticity of the pieces using the public key. Live streaming can last an indeﬁnitely long time. Hence, piece availability cannot be expected to be appropriate for old pieces.

2.2.6. Reputation Currently, we are experimenting with a global reputation system for Tribler peers. Having such a system will allow peers to identify others who are behaving well over a long time, not only in a single swarm, as is the case with normal BitTorrent. Lazy freeriders, peers who after downloading a torrent leave immediately, will have a lower reputation then peers who stay and seed. Cooperative behavior is thus encouraged, leading to better performance. 18 2. Tribler: Peer-to-Peer Media Search and Sharing

The system we implemented, called BarterCast [67], creates records stating how much data has been exchanged between two peers. By collecting these BarterCast records, each peer can create a subjective graph of the whole Tribler population, whose edges describe the data exchanged between peers. Using this graph, a peer can estimate the contribution another peer has made to the system. To protect from 2 malicious peers that are misreporting data exchanges, a peer computes the actual reputation of other peers using a maximum ﬂow algorithm starting from itself. This reputation can then be used to order the peers which are requesting pieces, or even decide to not send any pieces to peers with a low reputation. The manner in which we collect the BarterCast records, choosing the peers we synchronize with is where most research eﬀort is put into. Peers select those to synchronize with according to heuristics, which have a substantial impact on the speed of synchronization and the accuracy.

2.2.7. Channels Since the release of Tribler 5.2 we have implemented fully decentralized channels that each user can utilize to publish torrents they like. Other users can mark a channel as favorite, causing it to become more popular. Furthermore, channels can be marked as spam when users determine that the torrents provided are fake. Using both the total number of users marking a channel as a favorite or spam, a popularity metric is calculated and used to sort the channels. ChannelCast, the overlay protocol used to discover and update channels, exploits this popularity to ensure the right channels are discovered faster. Channels allow for co-operation and communication. Moving away from a single injector publishing all content, users will be able to post comments and iteratively improve the metadata (title, description, etc.). A distributed permission system will allow for granting users speciﬁc permissions within a channel. Currently over 2000 channels have been created.

2.2.8. GUI As part of the standalone Tribler client, we have implemented a GUI using the open-source wxPython toolkit, which allows us to implement a GUI once and run it on multiple platforms (Windows, Mac OS X, Linux). A big effort has been put into creating a GUI which is fully resizable and not dependent on a specific platform. Native GUI elements are used, which are positioned relative to other elements. Thus specifying that a button is positioned below another element, or to the right of it. This allows us to provide the same functionality on all platforms, while still having a slightly layout due to differences in the native elements. The Tribler GUI includes search, channels and a library view. VOD is available for torrents with video content; the embedded version of the open-source VLC media player is used for playback. We implemented some additional features, such as the “Network Buzz” which shows the user what popular content is currently available in the Tribler network. Furthermore, the GUI provides several hints to users suggesting a specific action or promoting the use of a feature of the program. In Figure 2.2, a screenshot of the search results view is shown. This view consists 2.2. Design and Features 19

Figure 2.2: Tribler GUI showing search results

of a list of results, which expand after being selected by the user to show the torrent details. This view includes a short overview, but additionally provides speciﬁc information using tabs. Hiding these details was a design choice as we expect most users to be intimidated by them.

2.2.9. Other deployments: TV and Browser As part of the European funded P2P-Next project, we developed two other deployments based on the same Tribler core. Being able to run Python, Pioneer has created a set-top box called NextShare TV which allows users to do BitTorrent live streaming and VOD on their TV. The set-top box is a 450 MHz machine, with a custom GUI designed to be used on TVs [91]. Another deployment of the Tribler core is called SwarmPlayer and provides users with the same functionality as NextShare TV, but is packaged as a browser plugin. It uses the combination of the HTML 5 video tag and a custom tribe:// protocol to download and stream the video data in the browser. After a torrent has been completely downloaded, it will be automatically seeded. Videos that have not been completely downloaded, due to the user browsing away from the Web page are considered unwanted videos and are removed. Seeding is supported through a limited cache which is normally transparent to the user. Manual control over the seeding is available using a separate HTML GUI, where users can stop and start torrents, as well as modify advanced settings like the upload speed limit. 20 2. Tribler: Peer-to-Peer Media Search and Sharing

2.3. Usage Tribler serves two purposes; it is a BitTorrent client with additional features and, as such, it is used by people all over the world; on the other hand, it is also a tool for experimenting with and evaluating protocols and algorithms. In this section, we 2 present both these use-cases. 2.3.1. Users Since it’s release in 2006, Tribler has been downloaded over 1.2 million times. All big releases result in a temporary increase in usage. Tribler 5.3 was downloaded 10,000 times within a three day period, Tribler 6.0 roughly 15,000 times. These temporary increases of usage are usually caused by articles on popular websites such as Slashdot, Ars Technica, and BBC News. After the dust settles, our loyal user base continues to use Tribler and participates in the forums to help us improve the client. In March 2011, we had more than 5000 unique Tribler users downloading more than 6500 torrents (of which 2500 using VOD). SwarmPlayer, our other deployment of the Tribler core, has been incorporated into Wikipedia and VODO2, where it has been downloaded by 6000 users and 13,000 users, respectively.

2.3.2. Scientists Any researcher can use Tribler to experiment with and evaluate algorithms they are developing. Tribler is very well suited for this since it has both a stable user base, allowing for long experiments, and it gets downloaded by many users at every major release. These spikes in usage provide researchers with large amounts of data which have been used to conduct numerous experiments. There are two methods implemented to collect data, with user consent; the first is simply using a Tribler instance as an “ordinary” peer and keeping a log of all received responses. In the search overlay, for instance, we can record search queries issued by peers, which can then be used to improve the relevance ranking of the search results by tuning them for queries our users issue. Secondly, we can periodically push information to a server, wherein participating peers send a file to a server which is predefined. After the initial release of our BarterCast reputation system, the data collected using a crawler resulted in a clear understanding that simply using one-hop gossip was not sufficient. Simulations based on the data collected resulted in a complete redesign using BarterCast records signed by both peers that are involved in the data exchange. Signed BarterCast records will allow for full gossip, due to the records being immutable. Using this scheme and other modifications resulted in a large improvement in the error estimation and coverage of the reputation system [33].

2.4. Conclusion With Tribler we have successfully deployed a BitTorrent client that pushes the boundaries of P2P technology. It allows a stable user base to beneﬁt from a set

2VODO is a website promoting and distributing free-to-share media www.vodo.net 2.4. Conclusion 21 of novel features, including remote search, VOD, live streaming, reputation, and channels. Testing research ideas with real users has always proven to be a significant challenge. Researchers therefore often rely on simulations instead of real systems with thousands of users. With Tribler we are providing scientists with an open platform to conduct experiments on. All features have been implemented using open-source tools, readily available 2 for all, while the Tribler source code is also available to be modified. Furthermore, SwarmPlayer and NextShare TV show how the implementation versatility of the Tribler core allows for different GUIs and use cases. Finally, due to indirect and direct support from European research programmes, the platform has been available for over nine years with steady improvements; twelve big releases have already been completed, with more scheduled in coming years.

3 Large-Scale Message Synchronization in Challenged Networks

In this chapter we introduce Dispersy, a message synchronization platform capable of running inside a challenged network. Dispersy uses Bloom filters to let peers advertise their local state and receive missing messages. However, in contrast to previous work, the efficiency and effectiveness of our design allows for deployment in networks of unprecedented scale. We show, through extensive experimental evidence that peers have no difficulties in synchronizing over 100,000 messages. We integrate in Dispersy a NAT traversal technique able to puncture 77% of NAT- firewalls. Not puncturing these firewalls would prevent up to 64% of peers from receiving any synchronization requests. Implementing a NAT traversal technique proved essential when we used Dispersy to extend the functionalities of a BitTor- rent client. To date, over 350,000 users used our modified BitTorrent client which included Dispersy, synchronizing in overlays consisting of more than 500,000 messages. We emulate an overlay consisting of 1000 unmodified Dispersy peers in order to show the propagation and synchronization speed, bandwidth requirements, churn resilience, and overall throughput of Dispersy. Dispersy is able to synchronize a single message to all peers within 19 synchronization steps, withstand an average session-time of 30 seconds, and achieve an average throughput of 10,000 messages/s.

Parts of this chapter have been published in ACM SAC 2014 [107].

23 24 3. Large-Scale Message Synchronization in Challenged Networks

3.1. Introduction With Dispersy we present a large-scale message synchronization platform which is deployed to over 350,000 users. Dispersy is designed to run inside a challenged network, and is able to deal with communication limitations such as intermittent connectivity, large delay between links, and absence of end-to-end paths. Messages are application dependent, however Dispersy adds optional headers describing if and to whom this message needs to be synchronized, the id and or signature of the creator, etc. 3 Challenging conditions can be found in a wide range of networks, i.e. vehicular ad-hoc networks (VANETs), Peer-to-Peer networks (P2P), and delay tolerant networks (DTNs). As a result of this, message synchronization or more generally data dissemination while dealing with those conditions is a large established field with researchers writing over 20,000 papers in 2013 alone1. However, in contrast to most of the proposed designs which never see an actual implementation, or are not even suitable to be implemented, we design, implement, emulate, and deploy Dispersy into the wild. We define message synchronization as creating consistency in a network of peers by resolving differences in messages received between two peers at a time. Peers regularly request missing messages by advertising their locally available messages using a Bloom filter. A Bloom filter is a compact hash-representation which allows for membership testing. Upon receiving such a request, a peer uses the Bloom filter to test if it has messages available the other has not. Although such an approach has been proposed before, we implement additional heuristics which allow a Dispersy overlay to scale to a much larger extend than previous works. Moreover, we implement a NAT traversal technique allowing peers to communicate in spite of NAT-firewalls. When designing a message synchronization platform for a challenged network we cannot assume the possibility of a connection between any two peers. Therefore, Dispersy does not require a central server in order to operate. Moreover, all peers in the network perform the same tasks, improving its robustness but more importantly removing the dependency on peers which might have more computational/infra- structural resources.

3.1.1. Contribution We propose, implement, evaluate, and deploy a message synchronization platform capable of running inside a challenged network. Whereas previous works have used Bloom ﬁlters to synchronize messages, we improve their approach to the point of ﬁnally being able to synchronize at a much larger scale. Heuristics allow us to synchronize a larger number of messages while reducing the CPU time required to do so considerably. We thoroughly test the performance of our platform by emulating several overlays measuring the propagation and synchronization speed, bandwidth requirements, churn resilience, and overall throughput. Finally, we show how we used our plat-

1http://scholar.google.com/scholar?as_ylo=2013&q=data+dissemination+network 3.1. Introduction 25 form to implement and deploy an application which was used by over 350,000 users creating more than 500,000 messages. In contrast, previous work [56] struggles to synchronize more than 10,000 messages due to very high false positive rates.

3.1.2. Background and Related Work More than 25 years ago, Demers et al. [35] outlined methods for creating consistency across many nodes using a large heterogeneous, slightly unreliable and slowly changing network. Demers introduced the concept by stating that nodes can only be consistent after all updating activity has stopped. Demers et al. describe three 3 different methods of distributing updates; direct mail, in which an update is sent from the initial node to all other nodes, anti-entropy, in which a node chooses another node at random and resolves the differences between the two, and finally, rumor mongering, in which a node after receiving a new update (a hot rumor), will keep forwarding this update until it goes cold (due to receiving it from many other nodes). Introduced by Bloom [15], Bloom filters have become popular due to being a very space efficient method for membership testing. Bloom filters use a hash area consisting of N bits, which are initially set to 0. For each item that needs to be “stored” in the Bloom filter, K distinct addresses are generated using a hash function. The bits addressed in the Bloom filter are set to 1. Upon checking if an item is part of a Bloom filter, the same addresses are generated, but now the bits addressed are checked to be 1. If not all bits are 1, then this item is not in the Bloom filter. When all bits are set to 1, this item might be part of the Bloom filter or (due to the compression of the Bloom filter) could be a false positive. Because Bloom filters do not have false negatives, they can be a drop in optimisations to reduce disk-io. E.g. if we put all files which are on disk in a Bloom filter, then if the Bloom filter returns false a file is not on disk. If it returns true, then a file might be on disk. An overview of the use of Bloom filters in a range of network applications can be found in [19]. VANETs, vehicular ad hoc networks are quickly getting more commercially relevant, because of recent advances in inter-vehicular communications and their associated costs. In many respects VANETs can be considered as a P2P system as they are not restricted by power requirements, suffer from excessive churn, and have to be able to scale across millions of peers without the use of central components. Churn is the rate by which peer join and leave a network. In VANETs churn is caused by the movement of vehicles, which causes them to have a very small connection window wherein communication between two nodes is possible [45]. Lee et al. [56] envisioned a vehicular sensing platform able to provide proactive urban monitoring services by using sensors found in vehicles to detect events from urban streets, e.g. recognizing license plates, etc. Their MobEyes middleware runs inside a vehicle and creates summaries of detected events. Those summaries are then pushed by the creator for a period of time to all one-hop neighbors, and are harvested in parallel using a Bloom filter. By building a Bloom filter of the already- harvested still-valid summary packets, all other peers within the broadcasting range can detect the missing summaries. Lee et al. are using Bloom filters which are 26 3. Large-Scale Message Synchronization in Challenged Networks

1. Peer selection

2. Bloomfilter creation 3. NAT traversal

3 4. Receive missing Messages

Figure 3.1: The four step Dispersy synchronization method.

1024 bytes in size combined with a periodically changing hash function to reduce the number of false positives. Fixing the size of the Bloom filter without performing a selection of summary packets limits the number of summaries which can be synchronized. Inserting 2000 summaries into this Bloom filter results in a false positive rate of 10%, inserting 10000 results in a false positve rate of roughly 70%. Incurring such a high false positive rate will greatly reduce the efficientcy of the synchronization.

3.2. System Design In Dispersy peers communicate in one or more overlays. An overlay is an additional network built on top of an existing one, most commonly the Internet. Within an overlay, peers synchronize by advertising their locally available messages to others.

3.2.1. Overlay Definition The design of Dispersy is modular and extensible and lets applications define their own overlay types for specific goals. By defining an overlay, an application can determine which messages are going to be used, and which need to be synchronized. The body of a message is application specific, but its headers are added by Dispersy and can contain to if and to whom this message needs to be synchronized, the id and or signature of the creator, etc. After defining an overlay type, a peer needs to instantiate it. Instantiating an overlay requires a peer to generate an identifier for it. This identifier has to be known to all peers who attempt to join this instance. Dispersy overlays are completely separate from each other, as messages contain the identifier of the overlay they are part of and thus are not shared between overlay instances. After joining an overlay, peers synchronize by advertising their locally available messages using a Bloom filter. At a fixed interval, each peer sends its Bloom filter to another peer in what we call a synchronization-step or step. This Bloom filter allows those two peers to compare their local messages and resolve the differences. 3.2. System Design 27

Active connection A ↔ B Peer B Peer A 1. Introduction-request C

↔

2a. Introduction-reply B

st n

e o u i eq t -r c

3 e e . ur Pu ct n

n n ct un 3 ure . P o b c 2 e v i t c A

Peer C

Figure 3.2: A synchronization step undertaken by Peer A.

Figure 3.1 gives an overview of a single synchronization step. In the following subsections, we will explain peer selection, peer discovery, NAT traversal, and Bloom ﬁlter creation in depth.

3.2.2. Peer Selection Dispersy maintains a peer list for each overlay, this list contains peers which are deemed to be connectable. Initially, the peer list is empty, requiring a peer to initiate the peer discovery algorithm, i.e. bootstrap. There are several mechanisms available to discover peers; if we assume an Inter- net connection then the most basic solution is to use trackers. Trackers are known to all peers and must be connectable, i.e. they must not be behind a NAT-ﬁrewall. Trackers maintain lists of peers for several overlays, and return these upon request. After a peer has populated its peer list with initial peers, it can use those to discover others. If we assume a non-empty peer list, then peer selects a destination peer randomly from this list to perform a synchronization step with.

3.2.3. Peer Discovery and NAT Traversal After selecting a destination peer, a peer sends an introduction-request message as shown in Figure 3.2. Upon receiving an introduction- request, Peer B selects another peer at random from its peer list (in this case Peer C) and sends an introduction-reply message back to Peer A containing the ip/port of this peer. Moreover, Peer B sends a puncture-request message to Peer C requesting it to send a puncture message to Peer A, thereby puncturing its NAT-ﬁrewall allowing Peer A to connect to it. Note that after sending this puncture message Peer A can connect to Peer C, but Peer C cannot connect to Peer A as the NAT-ﬁrewall of 28 3. Large-Scale Message Synchronization in Challenged Networks

Peer A is still blocking its messages. We need to puncture NAT-firewalls, as up to 64% of all peers connected to the Internet are behind a NAT [41]. Not puncturing those would cause the peers behind them to never receive any introduction-requests, because their NAT-firewall simply dropped the messages. Upon receiving the introduction-reply from Peer B, Peer A adds the ip/port of Peer C to its peer list. Peer B added the ip/port of Peer A to its peer list when it received the introduction-request from Peer A. When adding a peer to the peer list, peers additionally store the current time. These timestamps are used to remove 3 peers from the peer list for which the probability of the NAT-firewall having closed the port due to inactivity is high. Using the timeouts as observed by Halkes et al. [41], we remove peers which we have sent or received an introduction-request from within 60 seconds. If a peer replies to an introduction-request, we reset the timestamp to indicate this.

3.2.4. Bloom Filter Creation Included in the introduction-request messages is a Bloom filter which advertises the locally available messages of a peer. However, as described in Section 3.1.2, in contrast to previous work [56] we only include a subset of all messages locally available in the Bloom filter in order to keep the false positive rate low. Moreover, as Dispersy is based on UDP we limit the size of the message which includes the Bloom filter to the maximum transmission unit (MTU) of the link in order to avoid fragmentation. IP fragmentation is caused by a router in the path having a lower MTU than the packet size [87]. A typical MTU size on the Internet is 1500 bytes, for 802.11 it is 2304 bytes [25].

Message Selection Due to these constraints, Dispersy has to make a selection of messages to include in a Bloom filter. In order to do so, we implemented two heuristics; pivot and modulo. Both heuristics base their selection on the Lamport clock value of messages. Lamport’s logical clock [55] creates a partial causal ordering for messages, which we implemented by having each peer increment its local clock value when creating a new message and setting its local clock value to be maximum of all received messages. Although duplicate clock values are possible using a Lamport clock, we expect the distribution of clock values to be uniform, i.e. we don’t expect a period in time wherein many more duplicate clock values than average are created. If a peer has received many missing messages in the last couple synchronization steps, it uses the modulo heuristic as it assumes to be missing many more. The modulo heuristic selects every nth message starting at a random offset X. In order to create such a selection, a peer first counts its locally available messages. Next it uses this number to determine which modulo value to use, e.g. if a peer has 10,000 messages and the Bloom filter has a capacity of 1000, it uses a modulo value of 10. The random offset allows a peer to select any message to be included into the Bloom filter, e.g. using a random value of 1 and a modulo of 10 will select all messages with a Lamport clock equal to 1, 11, 21 etc. 3.2. System Design 29

Selected pivot d e v i e c e r lity s babi e t pro g Pivo a

s Left Right s e

M Missing messages

Lamport clock → 3 Figure 3.3: Message selection using the pivot heuristic.

The modulo heuristic allows a peer to retrieve many missing messages in each step, however if a peer is almost completely synchronized it introduces latency, as it could require up to modulo number of synchronization steps before a new message is received. The pivot heuristic aims to reduce this latency by synchronizing newer messages more often. A peer uses the pivot heuristic when it did not receive many messages in the last few synchronization steps. As shown in Figure 3.3, the pivot heuristic first chooses a pivot according to an exponential distribution. Hence, a pivot is more likely to be chosen in the higher ranges of the Lamport clock. Next, we select up to the Bloom filter capacity number of messages both to the left and right of the pivot and choose the range which has the largest difference in Lamport clock. The range with the largest difference contains the least amount of messages, and is thus the most likely candidate for missing messages. Depending on the state of the overlay, a peer can choose between using the pivot or the modulo heuristic. After selecting a range of messages, each message is first hashed and then inserted into the Bloom filter by computing its indexes. When sending a Bloom filter to another peer, we include the modulo, offset, and/or Lamport clock range to allow the other peer to use the same selection policy when checking for missing messages. Similar to Lee et al. [56] we use different hash- function configurations in order to reduce the effect of false positives. Upon receiving a Bloom filter, a peer looks at the included subset information to perform a similar selection on its locally available messages and then computes their hashes/ indexes to determine which messages the requesting peer is missing. Those messages are then send back as a reply to the requesting peer.

Reducing CPU Load Hashing and computing indexes for each message in a subset is inexpensive CPU wise, as it requires a single SHA-1 call and some bit shifting. However, as a Dispersy peer can join multiple overlays, wherein it periodically creates a new Bloom filter, those tasks combined do require a significant CPU load. E.g. synchronizing 5 overlays each with an interval of 5 seconds and a Bloom filter capacity of 1000 items requires a peer to hash/index 1000 items per second. In order to reduce this, we implemented an optimization which re-use a Bloom filter whenever a peer is missing many messages and another which does not include a Bloom filter whenever a peer 30 3. Large-Scale Message Synchronization in Challenged Networks

believes it is fully synchronized. Before creating a new Bloom filter a peer verifies if the previous synchronization step resulted in receiving some missing messages. If it did, then the previous Bloom filter is updated with these messages and re-used until no missing messages are received. Although adding more messages to a Bloom filter increases the false positive rate, we assume its still working if a peer can receive missing messages. If a peer did not receive any missing messages, then the Bloom filter must have been exhausted either due to the increased false positive rate, or because this given range is fully synchronized. Hence it creates a new one. 3 In contrast, if a peer determines that after sending a new Bloom filter in the previous synchronization step no missing messages were received, it will increase the probability of it not including a Bloom filter in the next synchronization step. Not including a Bloom filter has two advantages as it reduces the CPU load of both the peer sending and receiving it. E.g. the peer sending it did not have to create a Bloom filter and the peer receiving it did not have to check for missing messages. Peers continue to send introduction-request messages as these are additionally used to perform NAT traversal and to discover new peers.

3.3. Evaluation In order to evaluate Dispersy, we are going to perform several experiments on our DAS-4 supercomputer2. Unless stated otherwise, we deploy 1000 peers on 10 nodes emulating a small overlay. We do not modify Dispersy in any way, the nodes are running the same code as used during the deployment. Using the emulated overlays we extract information regarding its structure, propagation and synchronization speed, bandwidth usage, and the throughput performance of Dispersy.

3.3.1. Overlay Using an approach defined by Jelasity et al. [47] we compare an instance of a Dispersy overlay to a random overlay by comparing outdegree, clustering coefficient, path length, and diameter. As the overlay of Dispersy is continuously changing, we collected over 20 snapshots of all peer lists. Using the peers a peer connected to in the last 11 steps we created a directed graph and constructed a random overlay with the same amount of edges. Table 3.1 shows the results. As can be seen, the average outdegree of peers in the Dispersy overlay is marginally lower due to timeouts. As this experiment is run on a supercomputer this is expected, while timeouts in the random overlay cannot occur as there are no real connections. The clustering coefficient of the Dispersy overlay is higher, although this does not have a big impact on the average path length. Clustering in the Dispersy overlay is caused by peers introducing peers from their local peer list to others, thereby increasing the overlap between their two peer lists. The average path length is slightly higher, lowering the speed at which messages will propagate through the network. However, as described by Jelasity et al. [47], the propagation speed largely depends on the diameter which is very similar. By

2http://www.cs.vu.nl/das4/ 3.3. Evaluation 31

Random overlay Dispersy overlay Average outdegree 11 10.93 Average clustering coeﬃcient 0.010 0.046 Average path length 3.139 3.456 Average diameter 3.140 3.183 Maximum diameter 6 6

Table 3.1: Comparing a random overlay to our overlay 3 comparing the shortest-path distributions of the two overlays, we can see that the random overlay has more 2 and 3 hop connections, while the overlay constructed by Dispersy has more 3 and 4 hop connections. The maximum diameter for both overlays is the same.

3.3.2. Propagation Using the theoretical propagation speed as deﬁned by Demers et al. [35], we can show the actual diﬀerences in performance between our overlay and a random one. Demers et al. state that if implemented on top of a random peer selection, the probability of a peer not having received a message after i synchronization steps is:

2 Pi+1 = (Pi)

By recording the time at which a message is received by each peer we can determine its propagation speed. We therefore conduct two experiments; one in which a peer after creating a message solely relies on the synchronization algorithm (thus waits for incoming Bloom filters, allowing it to reply with this missing message), the other in which a peer after creating a message sends it to 10 peers from its peer list (thus jump starting the synchronization process). Figure 3.4 shows the results. Clearly, Dispersy is performing worse than would be expected from theory. However, as stated by Demers et al. their theory describes the performance if implemented on top of a random peer selection. As shown in the previous section, the overlay constructed by Dispersy has a higher clustering coefficient and greater average diameter compared to a random overlay. Moreover, the distribution of shortest paths differs between a Dispersy overlay and a random one. We feel that this is a likely cause for the drop in performance. However, Dispersy still manages to synchronize a single message created by a random peer in an overlay consisting of 1000 peers within 19 steps. If we were to assume a synchronization interval of 5 seconds, then a message would thus be synchronized to all peers in the overlay after 95 seconds. Jump starting the synchronization by sending it to 10 peers further reduces this to 75 seconds.

3.3.3. Synchronization Similar to the theoretical model we used to determine the propagation speed we want to have a model describing the optimal synchronization performance. Three 32 3. Large-Scale Message Synchronization in Challenged Networks

1.00 ● ● ●

●

0.75

●

0.50

●

0.25 3 Probability of peer ● ● ● having received the message received having ● 0.00 ● ● ● ● ● 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Synchronization step

● Demers 0 Demers 10 Dispersy 0 Dispersy 10

Figure 3.4: Propagation speed of a single message created by a random peer.

factors influence the speed of synchronization, first the capacity of the Bloom filter created by a peer (C), second the synchronization level of a peer (Si,A), and third the synchronization level of the peer the Bloom filter is send to (Si,B). The capacity of a Bloom filter depends on the number of bits available (B) for it and the false positive rate (P ). Using the formula as described by Almeida et al. [5] we can compute the capacity as follows:

(ln 2)2 C = B × |ln P |

In Dispersy, the number of bits available is determined the MTU and the size of the headers used in the message. Due to the synchronization level of a peer, its Bloom filter is actually describing a larger range. E.g. if a peer creates a Bloom filter with a capacity of 1000 messages, but has a synchronization level of 75%, then its Bloom filter actually describes range roughly consisting of 1333 messages. We refer to the range which a Bloom filter is actually describing as the detectable range (R). However, the false positive rate of a Bloom filter reduces the detectable range, as it prevents the other peer from detecting the missing message. Therefore we can compute R as follows:

1 R = × C × (1 − P ) Si,A

Finally, the synchronization level of the peer receiving the Bloom ﬁlter determines how many messages are returned. The following equation allows us to compute the synchronization level of a peer after performing a synchronization step, 3.3. Evaluation 33

1.00

0.95

0.90

0.85 3 Synchronization Level

0.80 0 100 200 300 Number of synchronization steps required

False positive rate 1% 5% 10% 25% 50%

Figure 3.5: Synchronization performance for various false positive rates. wherein N deﬁnes the total number of messages in an overlay. (R − C) × S S = S + i,B i+1,A i,A N Note that this assumes that all messages are distributed randomly and hence any peer can reply with missing messages, which in practice will not be the case. Using the formulas allows us to visualise the impact of using diﬀerent false positive rates on the number of synchronization steps required as shown in Figure 3.5. Increasing the false positive rate increases the capacity and hence the detectable range which reduces the number of steps required for synchronization. However, increasing the capacity will additionally cause the creation and testing for missing messages to be more computationally intensive, as more messages need to be hashed and converted into indexes.

3.3.4. Bandwidth Requirements The results of the previous two experiments allow us to compute the bandwidth requirements of Dispersy. A step performed by a single peer requires four messages to be send; an introduction-request, an introduction-reply, a puncture-request, and ﬁnally a puncture. If we assume a 1500 bytes MTU, then these messages require 1858 bytes of bandwidth (this includes the typical IP and UDP headers). The total amount of bandwidth required to propagate a single message to all peers in a 1000 peer network and the total amount a single peer spends to synchronize 5000 messages is shown in Table 3.2. We calculated the total amount of bandwidth to propagate a single message as follows: |nodes| × (message size + 1858 bytes × steps required) 34 3. Large-Scale Message Synchronization in Challenged Networks

in Kbytes Perform a single synchronization step 1.81 Propagate a single message to all peers 29.98 Let a single peer synchronize 5000 messages 341.01

Table 3.2: Bandwidth Required by Dispersy.

3 3

1 Bloom filter creation CPU wall time spend on CPU wall

0 0 50 100 150 200 250 300 350 400 450 500 550 600 650 700 750 Time since start of experiment (Seconds)

No optimizations Re−use Not include Both

Figure 3.6: Cumulative CPU time spent on creating Bloom ﬁlters.

The bandwidth a single peer spends was calculated by multiplying the amount of bandwidth required for a single step (1858 bytes) by the expected number of steps as shown in the previous sections. We assumed a 10% false positive rate. As Dispersy is optimized to allow for multiple messages to be synchronized per step, the bandwidth for propagating a single message case is considerable. However, this overhead will not change depending on the number of new messages in an overlay, as the bandwidth is caused by peers performing steps, not by the number of new messages created.

3.3.5. Bloom Filter Optimizations As introduced in Section 3.2.4, we implement two optimizations which aim to reduce the CPU time spent on creating Bloom filters. In order to show the impact of these optimizations we run an experiment outlining their impact on performance. In this experiment one peer creates 500 messages which are then synchronized to all other peers in the first 300 seconds. Without any optimizations, peers will create a new Bloom filter every 5 seconds, by re-using and skipping Bloom filters we aim reduce this. 3.3. Evaluation 35

Type Remarks Success rate APDM New port for every destination ip:port 43% ADM New port for every destination ip 75% EIM,APDF Accepts if packet was sent to destination ip:port 81% EIM,ADF Accepts if packet was sent to destination ip 100% EIM,EIF Accepts if opened before 100% No NAT 100%

Table 3.3: NAT-experiment performed with 17 nodes connected by 15 diﬀerent NAT-ﬁrewalls. 3

Figure 3.6 on the facing page shows the results. As can be seen, in the first 250 seconds the creation of the Bloom filters is becoming increasingly expensive. This is due to peers receiving more messages and hence needing to hash and create indexes for more messages when creating a new Bloom filter. When we comparing the reuse and skip optimizations, we can observe that re-using Bloom filters decreases the CPU time required when many missing messages are being synchronized (the first 300-350 seconds of the experiment), while skipping Bloom filter creation helps whenever the overlay is (almost) idle (after 400 seconds). By combining the two optimizations we reduce the CPU time spent on the creation of Bloom filters from 2.9 seconds to 1.0 seconds.

3.3.6. NAT Traversal In order to test the NAT traversal performance of Dispersy, we bought 15 different NAT-routers from various vendors and use them to create a small overlay consisting of 17 peers (2 peers have a direct connection). We use a synchronization interval of 5 minutes to allow the NAT-firewalls to close the opened ports due to inactivity. A short description on the behaviour of NAT-firewalls types can be read in Table 3.3, for more information we refer to the BEHAVE workgroup [6]. We used the NAT identification script by Andreas et al. [70] to determine the type of the NAT-firewalls we bought. By recording the success rate of connections between nodes, we created Ta- ble 3.3. Dispersy achieves an overall success rate of 77%, with peers behind a APDM NAT-firewall having the lowest success rate. However, the low success rate can be explained by the fact that an APDF NAT-firewall does not accept a packet if no packet was send to that ip/port beforehand. An APDM NAT-firewall opens a new port whenever they connect to a new ip/port combination. As this new port is not known to the APDF NAT-firewall beforehand, it drops the packet lowering the overall successrate for peers behind an APDM NAT-firewall. According to Halkes et al. [41] 11% of peers with a NAT use an A(P)DM NAT- firewall, 52% of NATted peers are behind an APDF NAT-firewall. Detecting if a peer is behind an APDF NAT-firewall is difficult, as we can only observe the lack of incoming messages from A(P)DM peers. Currently, this feature is not yet implemented. 36 3. Large-Scale Message Synchronization in Challenged Networks

3.3.7. Churn Resilience Being designed to run in a challenged network, Dispersy must be able to handle excessive churn. Churn is sometimes referred to as average session-time, which denotes the length a peer stays online before leaving. A network with a low average session-time is considered to have high churn, a high session-time is considered to be low churn as not many peers join and leave the network over time. Rhea et al. [81] state in their paper that session-times of peers in a P2P network can range from 50% of the peers having a session-time of less than 60 minutes to 50% of the peers having a session-time of less than 1 minute. Using an approach similar as described 3 in their paper, we modify the average session-times of peers in order to find out how well Dispersy handles churn during a 30 hour experiment. In our experiment peers choose a random session-time between 50% less and 50% more than the average session-time. After staying online for this duration, it will stay offline at least 2 minutes before joining again. Every 6 hours, we lower the average session-time and monitor the success rate of the sychronization steps. Figure 3.7 shows the results. Note that by lowering the average session-time, less peers are online at any given moment and hence it is expected that a lower number of requests is made. In an overlay which has an average session-time of 30 seconds, Dispersy is able to successfully for 60% of its attempts. E.g. a peer staying online for 30 seconds successfully synchronizes roughly 3 times (if we assume a 5 second synchronization interval). In VANETs churn has a different cause, as its due to the movement of vehicles. Huang et al. [45] reported that session-times can be as low as 50% of all connections having a session-time lower than 7 seconds. If Dispersy were to be used in such an environment, then simply decreasing the synchronization interval combined with a fast peer discovery algorithm can allow peer to perform many synchronization steps within this short connection window.

3.3.8. Throughput Finally, we run an experiment which shows the throughput performance of Dispersy. Using the Yahoo Cloud Serving Benchmark (YCSB), we test the performance of Dispersy as a key-value store. In order to do so, we implemented a basic key-value store using Dispersy able to perform three actions; reading, inserting, and updating a key. Messages which are inserted or updating a key are stored and synchronized by Dispersy, a reading message causes a peer to return the current value for that key. Using an additional Dispersy peer on the headnode of the supercomputer, we convert the requests of YCSB into Dispersy messages. As YCSB was written in Java and Dispersy in Python, we use Pyro3 to expose the read, insert, and update methods to YCSB. Pyro uses a separate socket per thread to allow an RPC-like exposure of Python methods to Java. We conﬁgure YCSB to use 120 threads in parallel. We use workload Scenario A which is included in YCSB to insert 2,500,000 keys into the key-value store, for more information regarding this scenario we refer to Cooper et al. [26].

3http://packages.python.org/Pyro4/ 3.3. Evaluation 37

1000 ●●● ● 3 ● Requests/Timeouts 750 ●

500 ●

● 250 ●● ● ● ● ● ● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●● ●●●●●● ● ●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ● ●● ● ● ●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ●●●● ●● ●●●● ●●●●●● ●●●● ● ●● ● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● 0 ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ●●● ● 1.00

0.75 Success rate

0.50

0.25 30m 10m 3m 1m 30s 0.00 6 12 18 24 30 Time since start of experiment (Hours)

● Requests per step ● Success rate ● Timeouts per step

Figure 3.7: Churn resilience of Dispersy, 30 hour experiment wherein average session-time is lowered from 30 minutes to 30 seconds. 38 3. Large-Scale Message Synchronization in Challenged Networks

3 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 10000 ● Operations/s

● ● 5000

●

0 ● ● ● ●

●

● Latency (ms)

40 ●

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

0 ● 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 Time since start of experiment (Seconds)

Figure 3.8: YCSB benchmarking results, inserting 2,500,000 keys into a key-value store implemented using Dispersy. 3.4. Deployment Results 39

Figure 3.8 shows the number of operations per second and latency achieved by Dispersy. As shown, Dispersy is able to achieve more than 10,000 operations per second, while having a latency of roughly 10 ms. 200 seconds into the experiment, the achieved operations per seconds drop due to some of the YCSB threads having inserted their subset of keys. After 270 seconds YCSB has successfully inserted all keys, achieving 9259 operations/s on average.

3.4. Deployment Results We integrate Dispersy into Tribler [102], an open-source BitTorrent client, in order 3 to evaluate its real world performance. Tribler is able to download .torrents and provides users with remote search, video-on-demand, live streaming, and channels. Initiated in 2005, Tribler is the product of a research effort funded by multiple European research grants. Tribler has been used by more than 1 Million users and more than 350,000 users used the version of Tribler including Dispersy. Currently, our version of Tribler has roughly 2000 active daily users. We use Dispersy to improve an existing feature of Tribler called channels. Before using Dispersy, channels were created by a single user and used to promote .torrents they liked. By using Dispersy we extended channels to allow other users to collaborate in adding, modifying, commenting on, and categorizing .torrents. We convert each channel into a separate overlay in which Dispersy peers are synchronizing. An- other overlay is used to discover new and popular channels. During deployment more than 90,000 users collaborated in over 2,000 channels, in total more than 500,000 messages were created. By instrumenting one Tribler client, we show the number of synchronization steps required in order to receive 100,000 messages. These messages are votes for channels, allowing other Tribler users to see which channels are popular and which are not. Comparing Figures 3.5 and 3.9 shows a big increase in number of steps required to synchronize all messages. This has three causes; first in this experiment we are synchronizing 100,000 instead of 5,000 messages. Second, in this deployed setting we cannot control/measure the synchronization level of other peers, greatly influencing if peers can reply with missing messages or not. Third, in Tribler we implemented an upper bound by limiting the number of messages a peer will reply with after receiving a Bloom filter. Using Wifi, Dispersy required roughly 30 Mbytes of bandwidth to fetch all 100,000 messages. Using 3G clearly reduces the performance, but we believe this is caused by lost UDP packets. Due to the sheer growth in mobile Internet use, we expect providers to drop UDP packets whenever congestion occurs.

3.5. Conclusion In this chapter we present Dispersy, a message synchronization platform designed to continue to function in a challenged network. Using Bloom ﬁlters, peers advertise their locally available messages and receive messages they have not received before. Thereby synchronizing their locally available messages. Integrated into this syn- 40 3. Large-Scale Message Synchronization in Challenged Networks

1.00

0.75

0.50

3 0.25 Synchronization Level

0.00 0 2000 4000 6000 8000 Number of synchronization steps required

Wifi 1% FPR Wifi 10% FPR 3G 1% FPR

Figure 3.9: Synchronizing 100,000 messages in a deployed overlay, relying solely on the donated resources of other peers.

chronization step is a peer discovery mechanism additionally able to puncture 77% of NAT-firewalls. Without puncturing these NATs, 64% of peers connected to the Internet would not receive synchronization requests. During extensive experiments we show the speed at which messages in a Dispersy overlay propagate, a theoretical model for the synchronization speed, the bandwidth requirements, effect of implemented Bloom filter optimizations, churn resilience, and throughput. More specifically, Dispersy is able to propagate messages within 19 steps, requires only 1.81 Kbytes of bandwidth to perform a single synchronization step, is able to synchronize even when faced with an average session-time of 30 seconds, and can average 10,000 operations/s. Finally, we deploy Dispersy into the wild by modifying an open-source BitTor- rent client called Tribler. More than 350,000 users used our modified version and created/synchronized over 500,000 messages while joining multiple overlays able to synchronize in parallel. 4 Open2Edit: a Peer-to-Peer platform for collaboration

Peer-to-peer systems owe much of their success to user contributed resources like storage space and bandwidth. At the same time, popular collaborative systems like Wikipedia or StackExchange are built around user-contributed knowledge, judgement, and expertise. In this chapter, we combine peer-to-peer and collaborative systems to create Open2Edit, a peer-to-peer platform for collaboration. Open2Edit provides the means for users to publish, modify, comment on, and categorize content. Content-centric collaboration in Open2Edit takes place within communities. A ﬂexible permission system allows each community to customize the privileges users receive for interacting with content. Any user can create a new community, and Open2Edit facilitates their discovery. Furthermore, Open2Edit allows users to vote for communities, and keeps track of community popularity. We deploy Open2Edit in Tribler, a BitTorrent-based peer-to-peer application, in order to create a YouTube-like media sharing system. Through Open2Edit, users in Tribler can collaboratively improve media metadata using a wiki-like approach. We present results from 7.5 months of usage that show Open2Edit is a viable, sustainable platform that leads to the emergence of collaboration.

Parts of this chapter have been published in IFIP Networking 2013 [101].

41 42 4. Open2Edit: a Peer-to-Peer platform for collaboration

4.1. Introduction Peer-to-peer (P2P) systems such as BitTorrent have proved that it is possible to create a scalable, fault-tolerant file sharing system which depends only on infrastructure resources donated by users. Similarly, collaborative websites such as Wikipedia and StackExchange show that it is possible to build a repository of free knowledge and expertise using intellectual resources donated by world-wide communities of users. However, these collaborative websites, while providing a free service, require significant investments in infrastructure to keep running. We believe that if both types of user contributions are combined, a system will emerge which can provide users with access to free knowledge and expertise, while requiring little investment in infrastructure. In this chapter, we introduce Open2Edit, a platform capable of combining do- 4 nated bandwidth, storage space, and CPU cycles with knowledge, judgement, and expertise. Open2Edit is a P2P system that allow users to create communities in which they can collaborate without relying on centralized servers. Furthermore, the flexibility of Open2Edit allows for a wide range of deployment scenarios, ranging from a closed traditional news website in which users can only comment on articles, to a community very similar to Wikipedia in which users can modify almost anything. Within a community, the users of Open2Edit engage in content-centric collaboration: adding new content, editing, commenting, and categorizing. Any user can create a new community and grant other users a diverse range of permissions, including the ability to perform the content-centric actions mentioned above, but also the permission to grant permissions. Open2Edit allows users to vote on the quality of communities and aggregates the votes to promote the best communities. Data synchronization among peers is an important issue in designing a P2P collaboration platform. In Open2Edit, we build upon Dispersy (see Chapter 33 for details) to address this issue. Dispersy provides low-level functionality for synchronizing messages between peers, has a permission system, and NAT-traversal. Moreover, peers use Bloom filters to efficiently replicate peer data. With extensive emulation, we show that Open2Edit can successfully synchronize data between 1000 peers. Furthermore, the experiment proves that Open2Edit load balances user contributed bandwidth, and requires a minimal overhead when community activity is low. We show the maturity of the Open2Edit platform through our Tribler [3] deployment. Using Open2Edit we created a YouTube-like media sharing system based on Tribler, a BitTorrent-based P2P application. Besides file sharing, Tribler implements remote search, video-on-demand, and a reputation system. In addition to these features, using Open2Edit communities, we implement channels as a means for users to publish, modify, comment on, and categorize media items. Channel creators can make full use of the Open2Edit permission system to make their channels more or less open. By monitoring Tribler, we can state that Open2Edit is a viable platform for collaboration proven to work in an Internet deployment. During 7.5 months of data collection, over 2000 channels were created by almost 90,000 users who cast over 4.2. Related work 43

100,000 votes expressing their preference for channels. The rest of the paper is organized as follows. In Section 4.2, we review related work. We present the motivation for building Open2Edit in Section 4.3. In Sec- tion 4.4, we describe the Dispersy data replication engine. The design of Open2Edit is described in Section 4.5. In Section 4.6, we present the Tribler deployment. The results of our experiments are discussed in Section 4.7.

4.2. Related work P2P systems such as BitTorrent [24] have shown that a scalable and fault tolerant file sharing system can be created even by depending only on user contributed resources. In BitTorrent, a separate overlay is created for each file being shared. Within an overlay, peers use an incentive mechanism called tit-for-tat which rewards user contributions with faster download speed. 4 The largest example of a knowledge-oriented collaboration platform is Wikipedia. Using Internet volunteers it has become an open alternative to traditionally edited encyclopedias. Wikipedia currently has more than 4 million articles written in English and over 40 other languages with more than 100,000 articles. When a user modifies an article, a new revision is saved. This new revision of the article supersedes the previous one, but no revisions are removed from the history. A small army of bots (automated scripts) monitor new revisions and detects vandalism – intentional errors introduced by users which corrupt articles. Wikipedia articles are text based, but users can upload images and movies to enrich articles. Furthermore, a special markup language is employed to allow users to create sections, lists, add references, and link to other articles. Wikipedia uses donations to pay for website hosting and its employees, and currently requires roughly $30 million per year 1. Alternatively, Wikia allows users to create many wikis in order to address different topics. It is currently hosting over 280,000 wikis2. Every wiki is allowed to have its own look-and-feel, improving the sense of community. Furthermore, links between communities can be established. Finally, in contrast to Wikipedia, Wikia uses ads to generate the income necessary to pay for hosting. Stack Exchange represents a different type of collaboration platform, one upon which Question & Answer websites can be build. These Q&A websites allow users to pose a question to which other users can provide answers. Together, users vote for the best answer to a question. Moderators can modify, close, and delete posts and user profiles. Similar to Wikipedia, moderators are elected by users themselves. An example of a deployed Q&A website using the Stack Exchange platform is Stack Overflow. Mamykina et al. [61] state that in Stack Overflow more than 90% of questions asked are answered, and that the median answer time is only 11 minutes. Currently (January 2013), Stack Overflow has 4.2 million questions and 8.1 million answers3. In contrast to the centralized collaboration systems described above, there have

1https://blog.wikimedia.org/2012/01/02/wikimedia-fundraiser-concludes-with-record- breaking-donations/ 2http://www.wikia.com/ 3http://data.stackexchange.com/ 44 4. Open2Edit: a Peer-to-Peer platform for collaboration

been attempts at creating decentralized collaboration systems. These can be divided into two categories. The first focus on providing a decentralized wiki and try to automatically merge conflicting copies of an article. Examples of such systems are the XWiki Concerto [20], and its extensions Wooki [98] and Swooki [76]. These systems provide collaboration by storing modifications made by users as either ins or del operations. Operations made on a single article, can be merged at a later stage allowing for decentralized collaboration. However, no explicit details are given on how modifications of an article are transferred between peers besides stating that they use anti-entropy. The second category of decentralized collaboration systems focus on distributing the load of a system like Wikipedia over a number of peers. These systems usually replicate articles in a DHT ring, and push articles to the peer closest to where the 4 article should be stored in the ring. One example is Piki [69], which is build upon Pastry. However, the authors do not address the scalability issues which arise due to implementing search using the DHT, nor the potential problems caused by peer churn in the system.

4.3. Motivation In this chapter, we design and deploy a peer-to-peer platform for collaboration. Our aim is to provide the flexibility needed for a range of use cases. At the same time, we intend to create a uniform architecture that is content centric and supports fine-grained permissions. As described in the related work section, there have been attempts at creating decentralized collaboration systems. However, most papers focus on creating a decentralized version of Wikipedia. Other types of collaboration websites, such as Stack Exchange, are not considered while building these systems. Moreover, the systems implementing a decentralized Wikipedia, such as XWiki Concerto, cannot be easily modified into a general collaboration platform as they only support the Wikipedia use case. In a nutshell, we address the problem of building a P2P platform for collaboration by starting with the Dispersy data replication engine and adding the features necessary to accommodate user collaboration. The main feature we add is the possibility for users to create communities, in which collaboration takes place. In addition, we define permission sets that users can easily apply to their communities to customize their level of openness. Finally, we enable users to discover popular communities through a voting system.

4.4. Dispersy Dispersy is a general purpose P2P message synchronization engine, created as part of the Tribler project. In this section, we give an overview of Dispersy, and introduce the basics of its permission system. For more details, we refer to Chapter3 and our technical report [106]. 4.4. Dispersy 45

1. Peer 3. Send the Bloom filter to the selected peer selection

2. Bloom filter creation NAT traversal

accept 4. Check Permissions Upon receiving data items

Data item storage 4a. Requesting proof of permission Figure 4.1: Dispersy data replication procedure. 4

4.4.1. Overview Each peer replicating messages in Dispersy is running a five step synchronization procedure, as shown in Figure 4.1: 1. Select a peer from the candidate list. This list contains all locally known peers with which a peer can freely communicate (without being hindered by NAT-firewalls). 2. Create a compact hash representation (Bloom filter) containing the data the peer has stored in its local database. 3. Send the Bloom filter to the peer which was selected in step one. 4. Wait for a fixed interval and while waiting process the data items which are received from other peers. For each data item, a peer checks if the peer who created the item had the permissions to do so. If it was not known that a peer had this permission, the proof is requested from the peer who sent the data item. 5. Goto step 1 and repeat. In Chapter3 we introduced these basic synchronization steps. However, in Open2Edit we employ the build-in permission system of Dispersy, which is not previously discussed and therefore introduced below.

4.4.2. Permissions The permission system is an important feature of Dispersy. Permissions are implemented at the overlay level and they refer to creating messages, revoking messages, and delegating to other peers the possibility to grant permissions. The private key of an overlay is used to grant and revoke permissions to or from peers in an overlay. When deﬁning the messages in a new overlay type, a developer can specify whether messages are protected by permissions or accessible to all peers. By default, all peers can create all messages in an overlay. However, when message creation is 46 4. Open2Edit: a Peer-to-Peer platform for collaboration

CategoryContent ContentMessage TagMessage Message Title, Sections, Article Tag content, etc.

CategoryMessage ModificationMessage Name, Description Update, Extend a ContentMessage CommentMessage CommunityMessage RevertMessage Comment body Name, Description 4 Reason, Issue warning

Figure 4.2: Open2Edit CommunityOverlay messages.

restricted, peers creating messages need to provide other peers a proof of being granted the permission to create messages of that type. These permission messages are also replicated by Dispersy, which allows peers to detect when a permission is revoked. Granting and revoking permissions is based on time. As there is no real concept of time in a decentralized system, Dispersy uses an implementation of Lamport’s logical clock [55]. The unique identifier of a message in Dispersy is the combination of the clock and the user id of the user which created the message. The identifier, together with a sequence number (messages restricted by a permission require all peers to include a sequence number in these messages) provides the platform with the tools to revoke a permission from a peer. A revoke message concerning a peer specifies the time and sequence number of the last valid message created by the peer. After this time, all messages created by the peer are dropped by peers who received the revoke message. If a peer receives a revoke message after already accepting an invalid message, then the incorrectly accepted messages are rolled back. If a peer attempts to spoof messages by re-using valid old time/sequence number combinations, all messages of this peer will be removed from the system, as messages with the same sequence number are not allowed to exist in an overlay.

4.5. Open2Edit design

In this section, we describe the design of Open2Edit using the framework provided by Dispersy. Collaboration between peers takes place in a new type of overlay that we create, called CommunityOverlay, while community creation and discovery takes place in an overlay of type CommunityDiscoveryOverlay. 4.5. Open2Edit design 47

4.5.1. CommunityOverlay An Open2Edit community is an instance of a Dispersy overlay type called Commu- nityOverlay. In order to implement the CommunityOverlay, we define eight message types, depicted in Figure 4.2. A community has a name and description, which are set by a community moderator through a CommunityMessage. By default, the peer creating the Communi- tyOverlay instance is the only moderator, but can grant the moderator permissions to other peers to have a group of moderators controlling the community. To create content, peers use the ContentMessage. This message is used to replicate an initial copy of the content, e.g., in a Wiki, it can be used to publish the first revision of an article; in a forum, it can be the opening-post of a thread; in a news-website it can be the news article, etc. There is no predefined structure in the ContentMessage, it is implemented as a dictionary (a list of key-value pairs). Modifications can be made to a ContentMessage, using the ModificationMessage. 4 A ModificationMessage contains a reference to the ContentMessage, a key-value pair that modifies or extends the dictionary of the ContentMessage, and optionally, a reference to the previous ModificationMessage it is replacing. Peers can post comments using the CommentMessage, which has a comment field, a reference to the ContentMessage it is a comment on, and two references to previous comments. The first reference is used for replying to another comment, while the second reference points to the latest received comment and is used for ordering comments. Open2Edit also allows peers to post tags on a ContentMessage. Using the TagMessage, a peer can tag the message using free-form text. The tags from all users can be used, for example, to create tag clouds for each ContentMessage. Community moderators can also create well-defined categories using the Cate- goryMessage. A category has a name, and a description. Any peer can assign a ContentMessage to a category using a CategoryContentMessage, but a ContentMes- sage can be part of multiple categories. The CommentMessage and ModificationMessage can also be used for commenting on and modifying other types of messages, not only ContentMessages. If a Cat- egoryMessage is referenced, peers can comment on, and modify a global category instead of a ContentMessage. Similarly, the CommunityMessage can be commented on, and modified. Finally, in order to have some tools for moderation in a community, we have defined a RevertMessage. This message has a reference to another message (e.g., a ModificationMessage), a reason for reverting it, and, optionally, a warning. Issuing a warning, will allow all other peers in the community to detect that a peer has intentionally done something wrong.

4.5.2. CommunityDiscoveryOverlay In order to discover CommunityOverlays, peers join a CommunityDiscoveryOverlay using a locally known identiﬁer. When deploying Open2Edit, a developer must create a new instance of the CommunityDiscoveryOverlay and include its identiﬁer in the source code of the deployment. If two peers join CommunityDiscoveryOverlays 48 4. Open2Edit: a Peer-to-Peer platform for collaboration

with different identifiers, they will never meet, and hence will never discover the same CommunityOverlays. For the CommunityDiscoveryOverlay, we have defined three message types. A CommunityCastMessage is used to send other peers information on the communities a peer knows. Every peer creates a CommunityCastMessage at regular intervals (15 seconds), and sends it to a random peer. The CommunityCastMessage contains a list of community identifiers divided into three sets: a) communities the peer has created; b) communities the peer is a part of; and c) communities we know the peer has interacted with. This allows new peers to quickly determine which communities are popular. Furthermore, we have defined a VoteMessage which allows peers to cast a vote on communities. A vote can be either positive or negative. A peer can only cast one vote per community, i.e., a newer vote replaces an older one. Using the votes, 4 peers can quickly discover popular communities. After discovering a new community identifier, either through CommunityCastMes- sages or VoteMessages, a peer will request a snapshot of the community. A Snap- shotMessage is sent to the peer which advertised the new community, requesting a small collection of data items from the community. A snapshot will allow a peer to get a general feel for the community. If, after having received the snapshot, a peer is interested in more content, it must join the community. Open2Edit will then start looking for other peers in the community and contact them to replicate the data in the community. It will also create a positive VoteMessage, indicating to other peers that it appreciates the community.

4.5.3. Authentication In order to prevent users from impersonating other users, we conﬁgure Dispersy to add a signature to each message created. Furthermore, each message contains the ID of the user who created it. The user ID is implemented as a SHA-1 hash of the public key. If the public key of a user is not locally known, Dispersy requests this public key from the peer who sent the message. As this peer has accepted the message, it should be able to also provide the public key. After receiving the public key, Dispersy can verify the signature and conﬁrm the user ID of the message creator.

4.5.4. Flexibility In total Open2Edit defines 11 different message types, some of them providing only minor functions. This however, is intentional as it allows the community-creator (the peer who created a community) to define which peers can send what messages. An example, if Open2Edit was used to create a distributed news-website, then articles (ContentMessages) must only be created by a limited set of peers (the journal- ists). Modifications must be disabled, and commenting can be enabled for all peers, mimicking a traditional news-website. However, if we would like to employ crowdsourcing it could be useful to let other peers categorize the articles, but to not let them create categories. If so, then the community-creator can allow all peers to create CategoryContentMessages, but prevent them from creating CategoryMessages. 4.6. Tribler deployment 49

Peer A Peer B Peer A Time: 1 Time: 5 Time: 7 PMR: 0 PMR: 1 PMR: 5

Peer C Peer D Time: 100 Time: 70 PMR: 1 PMR: 5 Message Ordering 4 Figure 4.3: Example of the global ordering of ModiﬁcationMessages using the prior modiﬁcation reference (PMR).

Opening up, or closing down functionalities uses the DynamicPermissionMessage as described in the previous section.

4.5.5. Detecting conflicts Conflicts are inherent to decentralized systems. In contrast to Xwiki concerto, Wooki and Swooki, Open2Edit does not implement conflict resolution algorithms. As Open2Edit is designed as a general collaboration platform, and not specifically as a decentralized Wiki, the additional complexity of such algorithms would hinder the flexibility as described in the previous section. However, we do implement a new method for providing a consistent ordering of ModificationMessages across peers. Using the prior modification reference (PMR) in the ModificationMessages, we build a tree connecting all modifications. If a node in the tree is referenced by more than one child, a conflict occurred. When choosing between two children, we sort messages based on time (oldest first), and finally alphabetically based on the user IDs of the message creators. By default, Open2Edit uses the latest ModificationMessages to construct a representation of a ContentMessage. Figure 4.3 shows an example of an ordering by PMR. The message created by Peer A at time 7 is selected as the latest modification.

4.6. Tribler deployment We deploy Open2Edit in Tribler in order to create a YouTube-like media sharing system. In this section, we give an overview of Tribler and describe the features we add as a result of the Open2Edit deployment.

4.6.1. Tribler overview Tribler is a BitTorrent-based ﬁle-sharing application that provides remote search, video-on-demand, live-streaming, and a reputation system, in addition to torrent 50 4. Open2Edit: a Peer-to-Peer platform for collaboration

Figure 4.4: Open2Edit deployed in Tribler, with a media item being modiﬁed in the edit tab, a Creative Commons licensed TV show from the VODO channel.

downloads. Initiated in 2005, Tribler is designed to advance research in P2P technology. It has been downloaded more than one million times and it currently has a community of approximately 3000 daily users. One of the reasons BitTorrent became a success is the large number of communities that were created around it. Before the Open2Edit deployment, Tribler was lacking this social aspect, because it purposely removed the BitTorrent dependency on centralized websites by implementing decentralized search.

4.6.2. Implementation details We add channels in Tribler as a means for user to collaborate in adding, modifying, commenting on, and categorizing media items like videos or photos. Channels are implemented as CommunityOverlays and media items are added to a channel using the ContentMessage. Media items, can then be improved by other users by modifying the name or description. Users can also create categories, and use them categorize the media items. Finally, users can comment on media items in a channel. Combined, all these features implement a decentralized YouTube-like system. Furthermore, as we want to cater to the needs of diﬀerent community creators, we implemented three levels of the openness of a channel: closed, semi-open, and open. When setting the level to open, all users can add, modify, comment-on, and categorize media items. The moderators can create categories, and remove comments and media items. The channel creator can grant moderator permissions to any user in the channel. The semi-open permission level for channels only allows users to comment on and categorize media items. Finally, in a closed channel only moderators can interact with the media items, while normal users can only download 4.7. Experiments 51 them.

4.6.3. User interface While designing the user interface of Open2Edit in Tribler, we focused on exposing the channel activity to users. An open channel allows peers to perform almost all Open2Edit actions, and hence requires a method for users to monitor the community to prevent vandalism. By providing users with activity lists, similar to the recent changes in Wikipedia, users can monitor each other. This allows users to detect when vandalism occurs and easily revert it. This technique works well in Wikipedia, where 42% of damage to articles is reverted almost immediately [75]. A small sample of frequent Wikipedia users are actively monitoring the recent changes pages to prevent vandalism making an impact [95]. In Tribler, we also allow users to monitor the media items they have shown interest in by providing filters for the activity and moderation lists. 4 To encourage user participation, we implement methods to lower the barrier to entry for engaging in collaboration. This is very important as casual users can make significant contributions. Kanefsky et al. [48] study a NASA crowdsourcing website where volunteers identify craters on Mars and show that 37% of all craters were identified by users which only visited the website once. The main user interface feature we implement to encourage collaboration is an edit mode for all editable entities, e.g., channels or item descriptions. Figure 4.4 shows the user interface as implemented and deployed in Tribler. The figure shows the editing of a media item called Deadside.2012.720p.x246-VODO (note the red highlighting for “Edit” in the lower part). The media item is part of a channel called VODO. Furthermore, the comments, activity, and moderations tabs for the channel are visible. These lists allow users to monitor the channel. Note also the star next to the channel name, signifying it is on the user’s list of favourite channels. Marking a channel as favorite subscribes the user to the channel, thus initiating the replication process. Peers in Tribler do not automatically replicate all channels they discover, as that would result in significant overhead. The user interface has other subtle features that help foster collaboration, like drag-and-drop categorization of media items and a “Thank you” button which allows users to quickly show their appreciation for people posting media items.

4.7. Experiments We run two experiments outlining the performance of Open2Edit. The ﬁrst experiment is an emulation of Open2Edit on the DAS-4 supercomputer. The second concerns user activity in Tribler during the Internet deployment of Open2Edit.

4.7.1. DAS-4 emulation Using the DAS-4 supercomputer4, we deploy 1000 peers onto 20 computing nodes. Peers run Open2Edit without any modiﬁcations. Although, we would like to perform an emulation of Open2Edit with more than 1000 peers, the nodes on the DAS-4

4http://www.cs.vu.nl/das4/ 52 4. Open2Edit: a Peer-to-Peer platform for collaboration

8000

6000

4000

2000

4 peer by ContentMessages received 0

0 1000 2000 3000

Time into experiment (Seconds)

Peer type Creator Normal Late joining Late arriving

Figure 4.5: Community experiment, number of ContentMessages received by peers.

cannot cope with running more than 50 Open2Edit processes simultaneously. The experiment uses an activity trace from the Tribler Internet deployment. We assign one of four different roles to each peer. One peer is the community creator; 980 normal peers are online at the start of the experiment and join the community immediately; ten peers are late-joining peers that are online at the start of the experiment, but only join the community after 1250 seconds; and finally nine late- arriving peers come online and join the community after 1500 seconds. Using an actual trace from a channel created by a user in Tribler, we assign the community creator with the task of creating ContentMessages. Each second in the experiment translates to approximately 2 hours in the trace. Discovering the CommunityOverlay instance can be achieved using two methods. The first method involves the CommunityCastMessage. A peer that has discovered a community can advertise it using CommunityCastMessages sent to other peers. The second method involves VoteMessages. All peers in the CommunityDiscoveryOver- lay are replicating VoteMessages. After receiving a VoteMessage for an unknown community, a peer request a snapshot and thus discovers the community. In this instance of Open2Edit, we configure peers to return at most 25 KB worth of data in response to a replication request. This causes to maximum upload rate per peer to be 5 KB/s as we are using a 5 second replication interval. Bloom filters were configured to accommodate at most 1000 data items. Figures 4.5 and 4.6 depict the results of the emulation experiment. Note that we use different line types to differentiate between the four user roles. In Figure 4.5, 4.7. Experiments 53

● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ●● 15 ● ● ● ●● ● ● ● ●● ● ●●●● ● ● ● ● ● ● ●● ● ●● ●● ●● ● ● ● ●●●●● ● ●● ● ●● ● ● ● ● ● ● ●● ● ● ● ●● ●●● ●● ● ●● ●●●● ● ● ● ●● ● ●● ●● ● ●● ● ●● ● ● ● ● ●● ● ●●●●●●●●● ●●●● ● ● ● ●●●●●●●● ● ●● ●● ●● ●● ● ● ● ●●●●●● ● ● ● ●● ● ● ●●●●● ● ● ●●● ●●●●●● ● ● ● ● ●●●●●●●● ●●●●●●● ● ●● ● ● ●● ● ●●●●●●●●● ● ●●●●●●●● ● ● ●● ● ● ● ●●●●●●● ●● ●●●●●● ●● ● ●● ● ●●● ●●●●●●●● ● ●●●●● ●● ● ●● ● ● ● ● ●● ● ● ●● ●●●●●●●●●● ●●●●●●●● ●● ● ● ●● ● ●●●●●●●● ● ● ●●●●●●● ●●●●●●● ● ● ● ● ● ●●●● ●●●●●●●● ● ●● ●●●●●●●●●● ●●●● ●●● ● ●● ●● ●● ●●●●●●●●●●● ●● ● ●●●●● ●●●●● ● ● 10 ●● ● ● ● ● ●● ●●●●●●●●● ● ●●●●●●●●●●●●●● ● ●●● ●● ● ● ●● ● ●●●●●●●●●● ● ● ●●●●●●●●●●●●●●●●●●● ● ● ● ●●●●● ● ●●●●●●●●●●●●●● ● ●● ●●●●●●●●●●●●●●● ●●● ● ●● ● ●● ● ●●●● ● ●● ●●●●●●●●● ●● ● ●●●●●●●●●●●●●●● ●● ●●● ● ● ●● ●●● ●●●●● ●● ●●●●●●●●●●●● ● ●●●●●●●●●●●●●●●●●●●●●● ● ● ● ● ● ● ●● ●●● ●●● ●●●●●●●●●●●●●●●● ● ●●●●●●●●●●●●●● ●●● ●● ● ● ● ● ●●●● ●● ● ●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●● ●●●● ●●● ● ● ●● ● ●●● ● ●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●● ●●●●●● ●● ●●● ● ●● ●● ●●●●●●●●●●●●●●●● ●●●●●●● ●● ●●●●●●●●●●●●●●●●●● ●●●● ● ●●●●●● ●● ● ● ●●● ●●●●●●●●●●●●●●●●●●●●●●● ●●● ●●●●●●●●●●●●●●●●●●●●●●●● ● ●●● ● ● ● ● ● ●● ●●●●●● ●●●●●●●●●●●●●●●●●●● ●●●● ●●●●●●●●●●●●●●●●●●●● ●●●●● ● ●●●● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●● ●●●●● ● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ● ●● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●● ●●●●●● ●●●●●●●●●● ● ●● ● ●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●● ●●●● ●●● ●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●● ●●●●●● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●● ● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●● ●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●● ● ●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●● 5 ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ● ● ●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●● ●●● ●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●● ●●● ●●●●●●●●●●●●●● ●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●● ●●● ●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●● ●●● ●●●●●●●●●● ●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●● ● Upload bandwidth per peer (KiB/s) ●●● ●●●●● ●●●●● ●●● ●● ●●●●●● ● ● ● ●● ● ●●● ●●●●●● ●●●●●●●●● ●●●●●●●●●●●●●●●●● ● ●●●●●●●●●●●●●●●●●●●● ● ●● ●●●●●●●●●● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●● ● ●●●●●●●●●●●●●● ●●●● ● ●●●● ●●●● ●●●●● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ● ● ●●● ●●●●●●●●●● ●●●● ●●●●●●● ● ●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ●●●● ● ●●●● ●●●●●●●●●●●●●●● ●●●●●●● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ● ● ●●●●● ● ● ● ● 4 ●●● ●●●●●●●●● ●●●● ●● ● ●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●● ●●●●●●●●●●●●●● ● ●●● ●●●●● ● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●●●●●●●●●●●●●● ●●●● ●●●●●●●● ●●●●●●● ● ● ● ●●●● ● ● ●●● ● ● ● ●● ●●● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ●●● ●● ● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●● ●

0 1000 2000 3000

Time into experiment (Seconds)

Peer type ●● Normal Late joining Late arriving

Figure 4.6: Community experiment, upload bandwidth used by peers. the community creator, that is also creating the data items, always has more data items than the other peers. As time passes, the other peers synchronize their local databases and receive more and more data items until they have all of them, around 3000 seconds into the experiment. Between 500 and 1000 seconds into the experiment, the community creator is creating new data items at a rate higher than 5 KB/s. This causes the normal peers to not be able to keep up due to their configured maximum upload rate. After 1000 seconds, they are starting to catch up, and at 1500 seconds most data items are again replicated to all peers in the community at that time. In practice, the upload rate will not affect the replication speed, as in this experiment 1 second represents 2 hours in real life. Furthermore, both the replication interval and the upload limit are configurable when deploying Open2Edit. The late-joining and late-arriving peers are showing the 25 KB upload limit as well. Their slope is defined by this limit. Furthermore, from these two groups of peers we can see that the subset selection process is working very well. The peers were configured to create Bloom filters which can only accommodate 1000 data items. In this experiment, we have more than 8000 data items, hence the peers are selecting subsets of data items in order to replicate. The difference between late- joining and late-arriving peers can be seen in the figure as well, as the late-joining peers replicate a small number (100) of data items while online before joining. This is due to the snapshots they collect. The late-arriving peers do not collect snapshots, and hence do not have any data items. 54 4. Open2Edit: a Peer-to-Peer platform for collaboration

Collection period 2011-12-09 to 2012-07-23 Users 89,819 Channels 2,038 Votes 103,189 Media items (total) 503,461 Media items (mean per channel) 247

Table 4.1: Internet deployment data summary

Figure 4.6 shows the upload bandwidth usage per peer. We left out the community creator as this peer is sending each ContentMessage it created to 10 neighbors, 4 and thus is incurring a much higher bandwidth usage. From the ﬁgure, we can see that normal peers have bandwidth spikes at the 600 second and 1200 second marks. This is when the community-creator is creating many new ContentMessages, and hence you would expect an increase in bandwidth usage. The late-joining and late- arriving peers only start participating in the replication process after they join the community (at 1250, and 1500 seconds respectively). This is clearly visible from their upload bandwidth. After 3000 seconds, the community-creator has stopped creating ContentMes- sages and all other peers replicated all the data items. The remaining bandwidth usage is the overhead Dispersy generates while trying to replicate two communities. Peers are part of both the CommunityDiscoveryOverlay and the CommunityOver- lay and are trying to discover data items to replicate in both. At 3200 seconds, the mean bandwidth usage per community for each peer is 0.56 KB/s. To conclude, we can observe that Open2Edit achieves a good load distribution, as no single peer is consuming more than 20 KB/s. Moreover, we note that Open2Edit has a low bandwidth overhead at the end of the experiment when the load is low.

4.7.2. Internet deployment In order to monitor the deployment of Open2Edit in Tribler we modiﬁed one client to, in contrast to normal peers, automatically subscribe to all channels it discovers. This client was started on a server and collected data over a 7.5 month period. During this period, we discovered over 2000 channels and observed that more than 100,000 votes were cast by almost 90,000 peers. The data collected is shown in Table 4.1. After releasing a new version, many new users install Tribler and use Open2Edit to discover and join channels. In Figure 4.7 we show the eﬀect of new votes per channel being cast by users after such a successful launch. For clarity, we only show channels which received at least 100 votes during this period. Also, note the logarithmic vertical scale. The most successful channel received more than 7000 votes in this launch window. After this period, because the new Tribler users have already discovered their favorite channels, voting for new channels is less frequent. Collaboration now moves 4.8. Discussion 55

8000 5000

2500

1000

500

250

100

25 10 4 Votes cast by users per channel cast by Votes 5

07 09 11 13 15 17 19

Day in February 2012

Figure 4.7: Votes cast by users for the 37 channels with more than 100 votes in a two-week period around a Tribler release. away from the CommunityDiscoveryOverlay into the speciﬁc CommunityOverlay instances. Figure 4.8 shows a trace of all activity within a single channel. In this channel, 66 peers collaborated in improving the metadata of 280 media items. Tagging media items is the most popular method for collaboration, which is expected as it requires less eﬀort than commenting on, modifying, or categorizing media items. Initially, we can observe that one peer is inserting new content into the channel. However, thereafter other peers continue improving the metadata of these content items by tagging and modifying them. To conclude, we can observe that Open2Edit is able to serve a sizeable group of users for a long time. Furthermore, it keeps functioning under stress, during a successful launch window, when channels are quickly discovered and voted upon by many new users.

4.8. Discussion During the deployment of Open2Edit in Tribler we have shown that Dispersy can synchronize over 100,000 data items. While this is suﬃcient for smaller communities, such as those found in Wikia, it is not suﬃcient if Open2Edit is used to implement a community of a similar size as Wikipedia. In such a community every peer will attempt to synchronize millions or tens-of-millions of data items in order to get all 56 4. Open2Edit: a Peer-to-Peer platform for collaboration ContentMessages

200

100

0 25

20 Contributions

Messages created by peer Messages created by 10 4 5 0 0 250 500 750 1000

Time since creation of Channel (Hours)

Message type Content Tags Modifications Comments

Figure 4.8: History of activity in a channel.

revisions of all articles, their comments etc. We propose two possible solutions which will allow for larger decentralized communities in the future. The first is based on the concept of decay, which means specifying when a data item is no longer relevant and can be removed from a community. In Open2Edit, decay can be used to remove older revisions of an article, thus significantly reducing the number of data items in a community. A revision is only considered to be old if it is both old time wise, and has newer revisions. The second solution is based on replication management, which defines a subset of items to be stored at each peer. Replication management is a well researched topic, however most papers only consider a system in which backup is the primary goal and item availability latency is expressed in hours. For our use case, latency should be a few minutes at most, therefore we first need to find a new approach to replication management.

Finally, we want to elaborate on our experience deploying Open2Edit into Tri- bler. For us, a big lesson learned is that implementing Open2Edit is only half the challenge. The other half is creating a user interface which is easy to use, does not hide the activity in a community, and entices users to start modifying the content within a community. Furthermore, as we were deploying Open2Edit into Tribler we had a diﬃcult time explaining the concept of communities to our users as its not common to be able to create your own community in a ﬁle-sharing application. 4.9. Conclusion 57

4.9. Conclusion The main idea behind Open2Edit is combining user contributed resources to create a self-sustaining P2P collaboration platform. We use Dispersy as a sound technical basis for our work and implement communities on top of it. At the same time, we preserve the ﬂexibility of the system, allowing for a wide range of deployments, from wikis to forums and news sites. We deploy Open2Edit in Tribler to create a media sharing system with YouTube- like functionality. The thousands of Tribler users can now create their own channels or join others’ channels to publish, modify, comment on, and categorize media items. The robust permission system allows channel creators to choose their preferred level of openness and designate other users as moderators. To test Open2Edit we create an emulation environment on the DAS-4 supercomputer using a trace of real-world Tribler usage as input. We observe that Open2Edit 4 distributes the load in the system in a balanced way, which is a key feature for a P2P system. Conversely, when the load is low, Open2Edit has a very low overhead for the peers. While the emulation experiment proves that Open2Edit functions in a controlled environment, only an Internet deployment can prove that the system also works in an uncontrolled possibly-hostile environment. The data we collected from Tribler unequivocally shows that Open2Edit can serve Internet-scale user communities. Al- most 90,000 users actively collaborated in over 2000 channels during the 7.5 months of monitoring.

5 Building a Privacy-Preserving Semantic Overlay for Peer-to-Peer Networks

Searching a Peer-to-Peer (P2P) network without using a central index has been widely investigated but proved to be very difficult. Various strategies have been proposed, however no practical solution to date also addresses privacy concerns. By clustering peers which have similar interests, a semantic overlay provides a method for achieving scalable search. Traditionally, in order to find similar peers, a peer is required to fully expose its preferences for items or content, therefore disclosing this private information. However, in a hostile environment, such as a P2P system, a peer can not know the true identity or intentions of fellow peers. In this chapter, we propose three protocols for building a semantic overlay in a privacy-preserving manner by modifying existing solutions to the Private Set Inter- section (PSI) problem. Peers in our overlay compute their similarity to other peers in the encrypted domain, allowing them to find similar peers. Using homomorphic encryption, peers can carrying out computations on encrypted values, without needing to decrypt them first. We propose three protocols, one based on RSA, the other based on the inner product of vectors, and the third based on multivariate polynomial evaluation. All three are able to compute a similarity value between two peers, and all three protocols are implemented on top of an existing P2P platform and are designed for actual deployment. Using a supercomputer and a dataset extracted from a real world instance

Parts of this chapter have been published in IEEE WIFS 2013 [103].

59 60 5. Building a Privacy-Preserving Semantic Overlay for Peer-to-Peer Networks

of a semantic overlay, we emulate our protocols in a network consisting of a thousand peers. Finally, we show the actual computational and bandwidth usage of the protocols as recorded during those experiments.

5.1. Introduction Over recent years numerous papers have been published on designing systems which provide fully decentralized search in Peer-to-Peer (P2P) networks. However, searching a vast network of peers sharing numerous items has proved to be difficult. This is, to a large extent, determined by the manner in which peers are organized. Traditionally, P2P networks can be divided into four classes: structured, unstructured, hybrid, and semantic networks. Structured networks allow peers to efficiently locate items, but keeping information stored in the network up to date is costly. Unstructured networks do not suffer from this overhead, but can not provide a method for efficiently locating items. Hybrid networks can provide both, by electing superpeers to index and locate items of ordinary peers. Lastly, semantic networks allow peers to locate items by connecting them to peers with similar in- 5 terests. This greatly reduces the number of peers contacted during a search (as a message is not forwarded), but at the same time reduces the number of items that can be found. Even though the aforementioned methods are well researched, privacy consider- ations are not addressed. Moreover, only two out of the four approaches mentioned can be considered for practical purposes: a hybrid or a semantic network, as both structured and unstructured networks suffer from substantial overheads. Concerning privacy, a hybrid network, such as the one used by Kazaa [58], has a significant drawback as it requires peers to fully expose the items they are sharing with the superpeers. This violates the privacy of the peers, as the superpeers are trusted with this information without knowing true identities or intentions of these peers. Furthermore, by sending a query to a superpeer, and thus trusting it, a query is then forwarded to a large number of unknown peers in the superpeer-network. All of them are implicitly trusted with the privacy sensitive material which is present in the search queries [44]. Current deployed semantic networks have similar drawbacks [97, 102], as peers exchange their preferences in plain text with randomly selected peers, allowing peers to compute the similarity between them. We define preferences as the items for which we want to compute similarity, e.g. a list of favorite movies, jokes, etc. Exchanging these lists also exposes the preferences of a peer to all. In this chapter, however, we improve upon this by extending existing cryptographic protocols, which allow us to find common elements in two private sets without exposing the sets themselves. This problem is known in cryptography as the Private Set Intersection (PSI) problem [39]. In this chapter, we focus on the PSI problem within the P2P context to find similar peers in a distributed network. There are a number of existing PSI protocols for different privacy requirements and security assumptions. Unfor- tunately, while previous work provably protects private data, the underlying cryptographic approaches have not seen implementation in a P2P environment, due to 5.2. Related Work 61 their high overhead, either in computation, bandwidth, or both. Contribution In this chapter, we propose the first practical, fully implemented method for building a semantic overlay in a privacy-preserving manner. We do so by designing two protocols that allow for the computation of similarity between peers while preserving the privacy of each peer’s preferences. The protocols adapt solutions from the PSI problem in cryptography to fit the needs of large P2P networks. We thoroughly test the performance of the implementations in a real network by emulating an instance with a thousand peers. Using a dataset extracted from an existing semantic overlay, we show that our solutions are scalable and efficient in terms of both computation and bandwidth. Most importantly, they fit nicely within the real world limitations of P2P networks, since they do not require any trusted third party or assume the possibility of a connection between any two peers.

5.2. Related Work Structured networks, such as Chord [90], Kademlia [63], or Pastry [86], allow peers to efficiently locate information by predefining where items should be stored and 5 how items can be found. In their approaches, keywords are converted into hashes and then used to route a query to the responsible peer. This responsible peer, maintains a list of peers which have matching items for the hashed keywords. However, the costs associated with keeping such a network up to date are prohibitively high. For each keyword by which a peer wants its items to be found, it needs to “announce” this to the responsible peer, i.e., in the case of a file-sharing system, a peer needs to split the name of a file into separate keywords and send an announce-message to each peer responsible for that keyword. Hence, multiple announces are required for each shared file causing a substantial portion of the bandwidth available at a peer to be used. Moreover, as peers in P2P networks leave and join the system at a high rate (churn), peers need to re-announce their items on a regular basis. Unstructured networks require much less maintenance. However, in contrast to structured networks, locating items is much more difficult in an unstructured network. Naive approaches, such as implemented by Gnutella in 2000 [82], caused each search query to be sent to almost all the peers in the network (flooding), resulting in a substantial overhead. Later in 2001, Kazaa improved upon this concept by creating a hybrid network [58]. Their network consists of two types of peers; superpeers and ordinary peers. Superpeers index the ordinary peers connected to them and thereby dramat- ically reduce the number of peers contacted during each search. In semantic networks, peers are clustered by their interest for the same content. When issuing a search query, a peer sends a message to the peers that have similar interests. In this way, the number of peers contacted during a search is greatly reduced (as a message is not forwarded), but also causing that not all items can be found (as scope is limited). Various clustering approaches have been suggested, which usually either group peers into a fixed number of clusters, or let peers independently find their most similar neighbors. 62 5. Building a Privacy-Preserving Semantic Overlay for Peer-to-Peer Networks

The challenge, as described in the introduction, of finding common elements in two sets without revealing the elements of the sets, falls into the more general category of comparing private information from two parties without leaking the information itself. It can therefore be formulated and solved in terms of two-party secure function evaluation. The PSI problem has been specifically discussed in various recent works, and is generally solved by protocols based on the homomorphic properties of an encryption scheme (most notably, ElGamal or Paillier). In 2004, Freedman et al. presented a cryptographic protocol based on oblivious polynomial evaluation and the Paillier cryptosystem [39]. The work by Kissner and Song also focuses on polynomial evaluation, but the authors propose protocols for a number of set operations, including unions, in a multi-party environment [51]. The most efficient set of polynomial protocols to date is instead the one proposed by Kiayias and Mitrofanova in [50], and is based on an ElGamal encryption scheme. Hazay and Lindell proposed a different approach to the PSI problem [42], which they solve using oblivious pseudo-random functions [38]. Another useful cryptographic tool in secure two-party evaluation protocols is 5 identity based encryption [17]. It has been used for the first time for PSI by De Cristofaro et al. in [29]. Their assumption is that the first party receives an autho- rization from a trusted entity before being able to make queries to the other party’s set using identity based encryption. Later, this line of research further progressed in terms of efficiency [27, 28, 30]. In this chapter, however, we do not assume any trusted third party, and therefore all the communication in the protocols we propose happens between the two set owners, namely Alice and Bob.

5.3. Preliminaries In this section, we brieﬂy introduce the building blocks that are used in the protocols while building a privacy preserving semantic overlay.

5.3.1. RSA The ﬁrst protocol is based on a public-key cryptosystem, in this case the RSA scheme [84]. The scheme works as follows. Let n be a product of two large prime numbers p and q. A random integer e is chosen such that 1 < e < φ(n) and gcd(e, φ(n)) = 1, where φ is the Euler’s totient function deﬁned as φ(n) = (p − 1)(q − 1). The public key to be used for encryption is (e, n) A message m ∈ Zn is encrypted as follows, e Epk (m) = c = m mod n (5.1) To decrypt ciphertext c, the value d is determined such that e · d = 1 mod φ(n). As the computation of d requires the knowledge of p and q, they are kept as the secret key. Having d in hand, decryption works as follows.

d Dsk (c) = m = c mod n (5.2) The security of the scheme depends on the diﬃculty of factorization of n, thus p and q should be large primes. For a modest security level, p and q are chosen of size 5.4. Privacy-Preserving Protocols 63

512 bits.

5.3.2. Paillier The second and third protocol are based on additively homomorphic encryption, namely the Paillier cryptosystem [73], that allows to add two plaintext messages by operating on their encryptions:

Dsk (Epk (m1) × Epk (m2)) = m1 + m2 (5.3) where m1 and m2 are plaintexts and Epk and Dsk the encryption and decryption functions, respectively. It follows that, given a constant c, the following also holds: c Dsk (Epk (m) ) = m × c (5.4) The Paillier cryptosystem is probabilistic, in the sense that a fresh random param- eter is introduced as input in each encryption operation.

5.4. Privacy-Preserving Protocols We propose three protocols that allow a peer in a P2P network to calculate its 5 similarity to other peers. In this section, we assume a setting in which the ﬁrst peer, Alice, has a set of items IA. Alice queries a second peer, Bob (which has a set of items IB), in order to compute the number of common items in the two sets without disclosing the sets themselves. Since we are interested in ﬁnding the similarity between two peers, we use the cardinality of the intersection between the two sets as a similarity score simA,B.

5.4.1. Private Set Intersection Protocol I We first present our RSA-based protocol. This protocol is a modified version of the protocol given in [30]. We assume that like every other node, Alice has a RSA key pair (EpkA , DskA ). Protocol 1: The cryptographic protocol that enables Alice to find the number of common items with Bob’s set works as follows:

1. Alice encrypts her items IA,i for 1 ≤ i ≤ I using her public key and sends

EpkA (IA,i) to Bob.

2. Bob chooses a random integer eB such that eB < nA/2 and for 1 ≤ i ≤ I, he eB computes (EpkA (IA,i)) mod nA, which can be formulated as EeB (EpkA (IA,i)).

3. Bob also uses eB and nA to encrypt his items IB,k for 1 ≤ k ≤ K. Then, Bob

computes the hash values of these encryptions, H(EeB (IB,k)), and sends them

all to Alice along with EeB (EpkA (IA,i)) after applying a random permutation to change their order.

4. Upon receiving EeB (EpkA (IA,i)), Alice decrypts the ciphertexts using her se-

cret key and obtains EeB (IA,i) for 1 ≤ i ≤ I. Next, she computes the hash of these values and compares each of them with the hash values she received from Bob. 64 5. Building a Privacy-Preserving Semantic Overlay for Peer-to-Peer Networks

5. By counting the number of overlapping hash values between her list and the list received from Bob she can calculate the similarity score.

The above protocol uses different keys for encryption, namely eA and eB, which is different from the original protocol in [30] that uses a common e and uses a random value. Note that using new eB values each time will provide a similar freshness as is achieved by using a random value in [30]. It is clear from the protocol that Alice can find the number of common elements in two sets. However, she cannot determine what items Bob has in his set and which are the common items, since she only receives the hash of encrypted items with a random ordering after permutation. As an malicious act, she can run the protocol repeatedly with a single item, which is different in each run. To make this kind of attacks more difficult, we propose to use hashcash to force Alice to compute a suitable random value to pad her initial message. In this way, each run of the protocol will take considerable amount of time and performing an exhaustive search 5 on Bob’s set will be expensive. More precisely, during step 1 of the protocol, Alice has to extend her message with a proof-of-work which counters the attack as described above. The proof-of- work consists of 3 elements; 1. the public key of Bob: eB, 2. the current time: ti, 3. a random string s. By including the public key of Bob and the current time, this proof-of-work can only be used once. The current time, and the random string are included in the message, which is sent to Bob. Next, in order for this proof-of-work to be valid, the hash should have ` leading zero bits. In order to generate such a proof-of-work, Alice has to find a random string that satisfies this criteria. For the above protocol to work, Alice needs to perform I encryptions, I decryptions and K hash computations. Bob performs I + K encryptions and K hash computations.

5.4.2. Private Set Intersection Protocol II This protocol is based on the inner product of vectors. In order for the algorithm to work, we make the assumption that there is a global list of all possible items, IG = {IG,1,..., IG,M }, of size M and this list is available to all peers. Based on that, each peer N creates a binary vector ΓN = hγ1, . . . , γM i, for the items in the global list. If node N has item IG,i in his set IN , then γN,i = 1 else γN,i = 0.

Protocol 2: Alice selects the secret key parameters for a Paillier encryption scheme, and publishes the respective public key and parameters. The communication proceeds as follows:

1. Alice encrypts each term in her list using her public key: EpkA (γA,i) for 1 ≤ i ≤ M, and sends them to Bob.

2. Bob computes the similarity score by using the homomorphic property of the 5.4. Privacy-Preserving Protocols 65

encryption scheme as follows:

M ! X EpkA (simA,B) = EpkA γA,i · γB,i i=1 M Y γB,i := EpkA (γA,i) (5.5) i=1

Bob then sends EpkA (simA,B) to Alice.

3. Alice obtains the similarity score simA,B after decryption.

It could be argued that step 2 is computationally expensive due to modular expo- nentiation. However, in practice this computation can be performed with minimum eﬀort. In fact, the values of γB,i are either 1 or 0. Therefore, Bob only needs to multiply Alice’s encrypted values for i values where γB,i = 1 (this corresponds to adding in the plain text domain) and re-randomize the ﬁnal encryption. Thus, the 5 computation becomes: M Y EpkA (γA,i) (5.6)

i=1,γB,i=1 This simpliﬁcation results in a very eﬃcient protocol in terms of computation: Alice performs M encryptions, Bob performs M modular multiplications in the worst case, and Alice performs 1 decryption. The bandwidth requirement, on the other hand, is linear in the number M as Alice sends M encryptions to Bob.

5.4.3. Private Set Intersection Protocol III Contrary to the previous protocol, this approach does not assume the existence of a global list of possible items, and is therefore suited for settings in which such an assumption is unrealistic. Here we compute the cardinality of the intersection of two sets using multivariate polynomial evaluation. In the following we propose a modiﬁcation of the protocol introduced by Freedman et al.[39] that follows this technique.

Protocol 3: Alice selects the secret key parameters for a Paillier encryption scheme, and publishes the respective public key and parameters. Communication between the parties proceeds as follows:

1. Alice builds a polynomial having roots in each of the items contained in her set IA. That is, she computes the n + 1 coeﬃcients α0, . . . , αn of the polynomial

2 n f (x) = α0 + α1x + α2x + ... + αnx (5.7)

for which f (IA,i) = 0 for any item in her set.

2. Next, Alice encrypts the coeﬃcients and sends them to Bob EpkA (f (x)). 66 5. Building a Privacy-Preserving Semantic Overlay for Peer-to-Peer Networks

3. Bob uses the homomorphic properties of the encryption scheme to evaluate the polynomial for each item in his set ΓB, and multiplies each result by a

fresh random number ri, obtaining EpkA (ri · f (IB,i)).

4. Bob adds all evaluated polynomials to a list, permutates the order, and sends it back to Alice.

5. After receiving the list of evaluated polynomials from Bob, Alice decrypts each ciphertext and counts the number of 0’s she received. The ciphertexts decrypt to 0 in case of an item in the intersection IA ∩ IB, or to a random value 1 otherwise. Therefore, simA,B is the number of 0’s decrypted by Alice.

The polynomial required in step 1 can be obtained by simple multiplication of the factors (x − IA,i). This way, the resulting polynomial will have the most significant coefficient take the value 1 and a degree equal to the number of items in Alice’s set. Computationally this protocol is more expensive than Protocol1. Alice needs 5 to perform |IA| encryptions, Bob needs to perform |IB| · |IA| modular multiplications, and Alice needs to perform |IB| decryptions. The required bandwidth for the protocol is linear in the number of items both Alice and Bob have in their sets. However, the bandwidth efficiency of this protocol can be improved by adopting a partitioning scheme. By reducing the degree of the polynomial, we increase the bandwidth efficiency as we can use a smaller keysize and increase the size of the universe at the same time. We do so by partitioning the universe (or domain) of possible items into a number of subsets. Each polynomial will then be relative to the items contained in one specific partition only, thus reducing on average the values of the coefficients and therefore the communication cost. Without a partitioning scheme, a universe in the order of magnitude of millions of items or more (for instance, 224) and a relatively small set counting 100 items, would require a keysize of 1200 bits. A polynomial with a degree of 100, would in this case have a least significant coefficient with a maximum possible value close to 22400. Encrypting such a large coefficient would require a 1200 bit key, as Pallier encryption works over modulo n2. Partitioning the universe inserts a step in the protocol before step 1, in which Alice needs to divide her set of items IA into multiple subsets. For each subset, or partition, she needs to compute a polynomial and send it to Bob together with her partitioning choice. Note that the number of coefficients Alice sends to Bob does not change. The resulting improvement allows us to accommodate an increase in the size of the universe depending on the expected maximum number of items per partition. E.g. if we expect a maximum number of items per partition of 25, then we can use an universe of size 280 and a 1024 bit key. However, if Alice has more

1 Let us observe that, if Bob also adds an encryption of the value of his input IB,i, obtaining therefore EpkA ri · f IB,i + IB,i , the result of the computation is the value of the element IB,i if IB,i ∈ IA ∩ IB , and a random value otherwise. This would allow us to identify the elements in the intersection set and therefore obtain a private set matching protocol, as done in [39]. 5.4. Privacy-Preserving Protocols 67 than the maximum number of items for a given partition, she is required to make a selection. A similar reduction in communication cost can be observed from Bob, as he can remove items from his set IB which do not corresponding to the received partitions. In order to maximize the beneﬁts of this approach without compromising the security of the protocol, the number of distinct partitions should be small enough compared to the number of elements in the universe, but comparable to the number of elements in IA.

5.4.4. Security Discussion All three protocols protect Alice’s privacy by hiding her item set from Bob. In protocols two and three, the set is first encoded (as a vector in Protocol2, as the coefficients of a polynomial in Protocol3), and then transmitted to Bob in encrypted form. In protocol one, the item set of Alice is only encrypted. Due to the homomorphic properties of the chosen encryption schemes, Bob can not decrypt, but can perform simple operations on the received values. In three algorithms, he computes a function that takes Alice’s encrypted values and his own values as 5 inputs, and sends the result to Alice, who decrypts using her secret key. Bobs inputs are hidden thanks to shuffling the lists in the first protocol, the properties of the powers in the case of the second protocol, and to multiplication by random number in the third. Since the Paillier encryption scheme is probabilistic, the same plaintext encrypted twice results in two different outputs. This prevents parties from directly comparing the encrypted values, when using the second and third protocol. We note that in Protcols1 and3, Alice discloses the number of items that are being compared (by counting the number of encrypted items, or by counting the number of coefficients minus the number of polynomials). This allows Bob to decide a minimum number of items below which he will not compute the intersection, thus preventing Alice to run the protocol for a set composed of one item only and learning whether or not Bob possesses that specific item. All three protocols achieve security in the semi-honest setting [39]. This setting, also known as honest but curious, assumes that the parties do not deviate from the protocol, and therefore includes the case of passive wiretappers, the most common threat to P2P networks. The protocols are however not secure against active adversaries. While an active eavesdropper deploying a man-in-the-middle attack by modifying the messages between the parties can be detected by adding signatures to the messages themselves, the case of a malicious Alice or Bob requires significant modifications to the protocols. In particular, a zero-knowledge proof protocol can be used to prevent Bob from pretending he always has a degree of similarity with Alice [16]. We assume Bob has no interest in pretending not to have a similarity with Alice, since in that case he could just not reply to Alice’s query. A zero knowledge protocol, together with a commitment scheme, is also useful in preventing malicious attacks by Alice [18]. We note however that Protocol3 already offers some degree of protection against a malicious Alice. In fact, besides improving bandwidth efficiency, another benefit of partitioning is preventing binary search attacks. Under such a scenario, Alice computes multiple polynomials with the 68 5. Building a Privacy-Preserving Semantic Overlay for Peer-to-Peer Networks

Full dataset Subset Peers 74 797 1 000 Items 296 358 7 635 Preferences 1 332 508 25 429 Average number of preferences per peer 18 25

Table 5.1: Datasets created for performance evaluation

same roots (for example 1 polynomial having a root in the ﬁrst item, 2 polynomials for the second item, 4 for the third and so on). This allows her to identify exactly which items Bob has by counting the number 0’s returned by Bob. However, since Bob evaluates a polynomial only at elements in the corresponding partition and replies to only one polynomial for each partition, this attack can not be carried out.

5.4.5. Improving Discovery Speed 5 As introduced in the previous sections, all three protocols allow Alice to compute a similarity score simA,B between her and Bob. However, in order to improve the speed of discovering peers which are similar to Alice we extend these protocols with forwarding. When sending a request to Bob, Bob will not only execute his steps of the protocol, but as well forward the request of Alice to his most similar neighbors. Bob will combine all replies from his neighbors with his own and send it back to Alice, allowing her to discover the similarity not only between her and Bob, but also between her and Bob’s neighbors. The reasoning behind this is that if Alice and Bob are found to be similar, then we can expect Alice to be similar to one of Bob’s most similar peers as well. Moreover, using Bob as a proxy is required, as we assume an active connection between her and Bob, but the same can not be assumed between her and Bob’s neighbors in a P2P environment, due to ﬁrewalls, NAT-routers, etc.

5.5. Experimental Results To evaluate the performance of the three outlined protocols, we have deployed 1000 peers on a supercomputer and monitor the time required to ﬁnd similar peers. In the following paragraphs, we elaborate on both the dataset and experimental setup before discussing the results.

5.5.1. Dataset Our dataset is constructed based on collected information from a real-world P2P ﬁle sharing network, called Tribler [102]. This ensures that our experimental results are close as possible to an actual implementation. Peers in the Tribler P2P network construct a semantic overlay in a non-privacy preserving manner. Using a BuddyCast message, peers send a list of their preferences to others in order to compute a similarity among them. We deployed instrumented clients into the Tribler network, which collected these lists. Using the generated ﬁles, we then constructed a dataset 5.5. Experimental Results 69 containing the preferences of the peers in the Tribler network. In total, we collected the preferences of almost 75 000 peers. Those peers preferred 1 332 508 items, with an average of 17.81 items being preferred per peer. For the experiment, we made a subset and selected 1000 peers at random. Fur- thermore, we made sure that each peer had at least 10 items, and each item was in the preference lists of at least two peers. This requirement causes every item in the preference list of a peer to overlap with at least one other peer. Full details of both datasets are shown in Table 5.1. Using the data from a deployed semantic overlay allows us to compare the performance of both protocols without having to resort to a synthetic dataset.

5.5.2. Experimental setup To evaluate the time required to ﬁnd similar peers we implemented a semantic overlay, in which peers connect to another peer every 5 seconds in a semi-random fashion. After connecting to a new peer, one of the protocols will be used to compute a similarity score. Similarity scores are used to create a list containing the 10 peers, which are most similar to it. A peer then uses this list to maintain an open 5 connection to those 10 neighbors. Related work has shown that maintaining a connection to 10 neighbors yields a good trade-oﬀ between the cost of discovering those and the expected hit ratio when performing keyword search [97, 102].

Monitoring Each peer is provided with a list of his preferences (as defined by the subset), and a list of his most similar neighbors. Using this similar neighbor list, we can determine if a peer has found its most similar peers or not. The list of most similar neighbors is generated using the complete subset, i.e. by computing the similarities between all peers. During the experiments, we monitor the peers by comparing the neighbors of each peer to the static list of its most similar neighbors. Moreover, in our emulations we want to see the differences in terms of performance between an “empty” overlay, which has no connections, and an overlay with existing connections also known as a bootstrapped overlay. We analyze effects of peers in the network being bootstrapped on the speed of discovering similar peers of the non-bootstrapped peers.

Infrastructure During the experiments we use 20 computing-nodes, which all run 50 instances (an instance is an emulation of a peer trying to ﬁnd similar peers). In this way, we create an actual semantic P2P network on the supercomputer consisting of 1000 peers.

Parameters Table 5.2 shows the used parameters during the experiments. All three PSI-C protocols are implemented in Python using GMPY2 which exposes GMP3 to Python.

2http://code.google.com/p/gmpy/ 3The GNU Multiple Precision Arithmetic Library http://gmplib.org/ 70 5. Building a Privacy-Preserving Semantic Overlay for Peer-to-Peer Networks

RSA Paillier Size of p and q 512 bits 512 bits Size of an encrypted value 1024 bits 2048 bits Number of partitions - 256 (1 byte) Connection interval 5 seconds Similar neighbors to ﬁnd 10 Length of preference lists 100

Table 5.2: Parameters used in experiments

Protocol I Protocol II Protocol III 1.00

0.75

5 0.50

0.25

0.00 0 20 40 60 0 20 40 60 0 20 40 60 Connected to most similar neighbors Time into experiment (Minutes)

Bootstrapped 100% 90% 50% 10% 0%

Figure 5.1: Speed of discovering the 10 most similar neighbors in a network of 1000 peers. When the network is not completely bootstrapped, the performance of the non-boostrapped peers is shown.

5.5.3. Results Discovery speed Figure 5.1 shows the speed of finding the most similar neighbors for each protocol in an overlay bootstrapped at different percentages. By comparing the three protocols we can see that the performance difference is minimal. This is to be expected, as all three protocols should compute the same cardinality. Moreover, all three protocols let Bob forward the request of Alice to its neighbors, thus hence the speed of discovery should be the same. After 10 minutes, all three protocols find more than 50% of all similar neighbors regardless of the level of bootstrapping of the overlay. After 20 minutes, roughly 75% of similar peers are found. However, the remaining 25% of peers are discovered at a much slower pace. We believe that this is caused by clustering in the overlay. Although we promote 5.5. Experimental Results 71

Protocol I Protocol II Protocol III 400

300

200

100 in the 60 Minute runin the 60 Minute (Seconds) 0

CPU time spent for cryptography operations cryptography CPU time spent for Send Receive Decrypt Send Receive Decrypt Send Receive Decrypt

Type of operation while executing the protocol 5 Figure 5.2: CPU time spent at the encryption/decryption phases of each protocol. clustering, as we are building a semantic overlay, it is also working against us in discovering peers with a clearly diﬀerent set of preferences than our current similar neighbors (peers are stuck in a local optimum).

CPU consumption Figure 5.2 shows the mean CPU time peers spend during the encryption/decryption phases of the protocols. We use CPU time as the wall time it took to perform the encryption/decryption steps during the 60 minute run as shown above. From the figure, we can see that Protocol1 requires much more CPU time compared to Protocol2 and3, especially while decrypting the received responses from Bob. This is expected, as in Protocol1, Bob has to first create a compatible RSA key with Alice’s, then encrypt all preferences of Alice with this key, encrypt all his preferences with this key, and finally hash those values. Both Protocol2 and3 can skip some of these computations, as they know if a preference is Alice is overlapping, or might be overlapping. The differences between Protocol2 and3 are much smaller. In Protocol3 Bob replies with multiple encrypted evaluated polynomials while in Protocol2 he only returns one encrypted sum, hence decrypting is cheaper. Moreover, we can see that receiving a request from Alice requires Bob to spend more CPU time in Protocol3 then in Protocol2, as Bob needs to evaluate the multiple polynomials for each item for which he has matching a partition. In Protocol2, Bob only has to perform computations whenever γB, i = 1 as explained in Section 5.4.2. In all three protocols, it is possible to reuse a similarity request for multiple peers, since the list of preferences of a peer usually does not change frequently (in our emulations, not at all). However, it has to be noted that in a situation, in 72 5. Building a Privacy-Preserving Semantic Overlay for Peer-to-Peer Networks

Protocol I Protocol II Protocol III 400

300

200

Bandwidth used 100 in the 60 Minute runin the 60 Minute (KBytes)

0 Send Forward Reply Send Forward Reply Send Forward Reply

Type of operation while executing the protocol 5 Figure 5.3: Bandwidth usage of all three protocols.

which the preferences of a peer may vary more frequently, creating a request using Protocol2 is most expensive.

Bandwidth usage With respect to the bandwidth usage of all three protocols, Protocol1 requires the least amount of bandwidth. The most important factor inﬂuencing the bandwidth costs is the dynamic size of the messages in Protocols1 and3. In contrast, in Protocol2 Alice always sends a preference vector of a ﬁxed length; the other two protocols send as many encrypted preferences Alice has. As in our subset, peers have 18 items on average, the expected bandwidth usage of these 18% of that of Protocol2 as it sends vectors with a length of 100 items (for the sending and forwarding steps). However, the reply of Protocols1 and2 requires more bandwidth as these protocols do not sum all values computed at Bob, but return a value per preference.

5.6. Conclusion In this chapter, we proposed, implemented, and evaluated three protocols that solve the problem of building a privacy preserving semantic overlay. Our solution extends existing protocols for the Private Set Intersection problem, in order to allow two peers to calculate their respective similarity. In order to show that all three protocols are indeed ready for widespread deployment, we emulated our privacy-preserving semantic overlay using data from an existing semantic overlay. Moreover, we showed that neither protocol required a trusted third party or assumed the possibility of a connection between any two peers. Therefore all three ﬁt nicely within the real world limitations of a Peer-to-Peer environment. Each of the proposed protocols introduces a trade-oﬀ between bandwidth and 5.6. Conclusion 73 computational overhead. Protocol2 requires more bandwidth, but the least amount of CPU time. Protocol1 is the opposite, requiring the least amount bandwidth and more CPU time. Protocol3 is somewhere in the middle, not requiring as much bandwidth as Protocol2 and also not requiring as much CPU time as Protocol1. And therefore, our recommendation for future work.

6 4P: Performant Private Peer-to-Peer File Sharing

In recent years fully decentralized file sharing systems were developed aimed at improving anonymity among their users. These systems provide typical file sharing features such as searching for and downloading files. However, elaborate schemes originally aimed at improving anonymity cause partial keyword matching to be virtually impossible, or introduce a substantial bandwidth overhead. In this chapter we introduce 4P, a system that provides users with anonymous search on top of a semantic overlay. The semantic overlay allows users to efficiently locate files using partial keyword matching, without having to resort to an expensive flooding operation. Included into 4P are a number of privacy enhancing features such as probabilistic query forwarding, path uncertainty, caching, and encrypted links. Moreover, we integrate a content retrieval channel into our protocol allowing users to start downloading a file from multiple sources immediately without requiring all intermediate nodes to cache a complete copy. Using a trace-based dataset, we mimic a real-world query workload and show the cost and performance of search using six overlay configurations, comparing random, semantic, Gnutella, RetroShare, and OneSwarm to 4P. The state-of-the-art flooding based alternatives required approximately 10,000 messages to be sent per query, in contrast 4P only required 313. Showing that while flooding can achieve a high recall (more than 85% in our experiments) it is prohibitively expensive. With 4P we achieve a recall of 76% at a considerable reduction in messages sent.

Parts of this chapter have been published in IEEE P2P 2014 [105].

75 76 6. 4P: Performant Private Peer-to-Peer File Sharing

6.1. Introduction In recent years, a number of fully decentralized file sharing systems were developed aimed at providing users with improved anonymity while searching and downloading files. Examples of such systems are Freenet [22] and Gnunet [13] which were released in the early 2000s, while RetroShare [79] and OneSwarm [46] are more recent solutions. These systems rely heavily on encryption techniques in order to provide users with the best possible preservation of publisher, sender, and receiver anonymity. However, the increase in complexity of these solutions has given rise to substantial drawbacks, either due to their P2P overlay structure or the approach used to search for files. E.g. in order to route search queries both Gnunet and Freenet hash the keywords, thereby unintentionally prohibiting partial keyword matching. Similarly, flooding induced search overhead found in Gnunet, RetroShare, and OneSwarm prohibits those systems to scale as their costs rise exponentially compared to the network size. In 4P, we build upon a semantic overlay which has shown to yield a high recall at a fraction of the cost [97]. Additionally, we combine the semantic overlay with privacy enhancing features which protect publisher, sender, and receiver anonymity from collaborating peers. 6 Contribution In this chapter we present 4P, a fully decentralized private file sharing system which introduces a trade-off between overhead and recall. 4P requires substantially fewer messages to be sent per query compared to flooding based alternatives, at a marginally lower recall rate. Our main contributions in this chapter are:

1. Leveraging semantic similarity among users, to query for ﬁles without resorting to ﬂooding.

2. We introduce a method which is able to negotiate a session-key between two peers without the need of a PKI, while still being able to thrump a man-in- the-middle.

3. Employing probabilistic query forwarding, path uncertainty, caching, and encrypted links we preserve publisher, sender, and receiver anonymity from collaborating peers.

4. Finally, we present the design, implementation, and evaluation of 4P using a trace-based dataset mimicking a real-world query workload.

4P is built on top of previous work [103] which allows us to construct a semantic overlay in a privacy preserving manner. By employing homomorphic encryption techniques, peers in our network exchange preferences and compute their overlap in the encrypted domain. This results in both peers knowing how many preferences overlap, without actually knowing which. Moreover, a peer replying to a request to compute the overlap cannot guess the contents of the request, and therefore cannot 6.2. Related Work 77 fake similarity. After ﬁnding and connecting to semantically similar peers, 4P uses the semantic overlay to achieve a high recall while incurring a low overhead. Our experiments show that peers achieve an average recall of 76% while requiring only 313 messages to be sent, substantially less than the approximately 10,000 messages sent by current systems.

6.2. Related Work Searching in a Peer-to-Peer (P2P) network without the use of centralized components has been widely researched over the last decade. However, it remains a very difficult problem to this day. Adding privacy preserving features to the mix has increased the complexity of the proposed systems, while at the same time substantially decreased their efficiency. In this section we will describe the current state-of-the-art in privacy preserving P2P file sharing. One of the first fully decentralized and unstructured P2P network providing search is Gnutella. In the Gnutella network [2, 82], peers connect to each other in an unstructured manner. At the same time no single peer is more important than another one. Search is implemented by creating a query message, which a peer sends to all of its neighbors. This query message includes a time to live (TTL) value, which prevents a single query from being forwarded more than TTL times. Upon receiving a query message a peer reduces the TTL value by 1, and forwards it to all its neighbors while the updated TTL is greater than 0. Sending messages in 6 this manner is called flooding and causes all peers within TTL hops to receive the query. Freenet, introduced in 1999, aims at being a more privacy oriented alternative to then existing filesharing systems [22]. It uses an unstructured network in which every peer and every file is assigned to a key. Files can be found by generating a key, using one of three different key-schemes. The key-schemes provide peers with increasingly more features, such as claiming their own keyspace, and storing files in encrypted form. However, they also make finding files increasingly more difficult as peers need to know public keys and or passwords beforehand. In Freenet, peers forward a query to a single neighbor based on the XOR distance between the keyword hash and the keys of its neighbors. If a peer exhausts its neighbors and cannot forward a query, it will send a request-failed message telling the previous peer to forward the message to another peer. If a file was found, a SendData message is sent to the peer which claimed to be issuing the query. How- ever, as any peer in the search-path (the chain of peers a query traverses over) can probabilistically rewrite the source identifier of the requesting peer, the SendData message will be forwarded over some hops. At each hop, the content of the Send- Data message is cached locally. This causes popular content to be cached in the network while unpopular content is dropped. Sending a query in Freenet using a property called depth to record how many times a query is forwarded. However, this depth value is not initialized at 0, but is set probabilistically to a small value. Hence, the first peer receiving the query cannot automatically assume that the peer it received the message from is the initiator of the query. In contrast, in Gnutella a peer receiving a message with TTL set to 6 78 6. 4P: Performant Private Peer-to-Peer File Sharing

can immediately deduce that the predecessor created the query [99]. Additionally, instead of sending a reply, a peer will forward a query with a TTL value of 1 with a finite probability, extending the path and obscuring the total path-length from all peers. Gnunet aims at providing users with a reliable anonymous backup system [13]. However, in contrast to Freenet, Gnunet uses a flooding-variant similar to Gnutella. In their protocol, the initiator of a query has to specify a priority and TTL value, together with a triple hashed keyword using the RIPE160 hash algorithm. When receiving a query, a peer will use the priority value to charge for the service of handling the query by subtracting the priority from the credit of the sender. If a query is received with a priority value of 0, no charge for the service can be applied and thus the query should only be honored if a peer has excess bandwidth. Content that matches the hashed keyword will be sent back in a reply. If a peer does not have matching content, it will forward the query to n neighbors selected at random if the TTL value is higher than 0. The priority value for each of the forwarded p messages is reduced to n+1 . The number of neighbors selected (n) depends on the system load. A query in Gnunet contains a triple hashed keyword, data is stored at a double hashed keyword. This makes it is less likely that a peer can reply to a query without having the actual data, as the properties of the chosen hash function makes 6 it computationally intractable to deduce the double hashed keyword from the triple hashed keyword. Files in Gnunet are split into pieces, which can be downloaded by traversing a tree. A reply to a query consists of a 1Kbyte root-node containing a pointer to the actual data (the root of the tree), a description, a CRC checksum, and a optional pseudonym and signature. For each keyword by which a file should be able to be found, a root-node is created. Using the associated keyword, every root-node is encrypted, thus preventing a peer storing a root-node from knowing what it is storing. RetroShare is a friend-to-friend (F2F) network which aims at providing users with a private social network [79]. On top of this social network, in which users have to manually exchange their public keys in order to become friends, features such as instant messaging, group chat, voice over IP, and file sharing are built. Users can find and download files using the TurtleRouter [80]. This component of RetroShare provides users with anonymous multi-hop file transfer, performed over encrypted tunnels. Searching for a file is called digging for a tunnel, and peers dig a tunnel by sending a tunnel request to all their neighbors. This tunnel request message contains a file hash, a depth (comparable to the inverse of a TTL value), a randomly generated ID, and a half-tunnel ID. Requests are forwarded to all neighbors while the depth field is lower than 6, but cycles are avoided. After locating a file, an encrypted tunnel is established using the hash of the file hash, source peer ID, and destination peer ID. In 2010, OneSwarm [46] took an approach similar to RetroShare and extended BitTorrent with a F2F network. In OneSwarm, peers can decide locally what data is shared with whom. Using a combination of private and public files, they achieve 6.3. Cost and Limitations of Current Systems 79

Initial Fanout Successive Fanout

Successive forwarding property failed Search-Message Search-Message

Response-Message Response-Message

Figure 6.1: Query being forwarded across a search-path, highlighting the search properties.

Approach Initial/Successive Initial Fwd Successive Fwd fanout property property Freenet Routing 1/1 TTL = 18 - TTL > 0 + ex- small random tend TTL = 1 value Gnutella Flooding 10/10 TTL = 7 TTL > 0 RetroShare Flooding 10/10 TTL = 6 TTL > 0 OneSwarm Flooding 0.5 * #Friends - Not seen yet + System load Gnunet Flooding System load TTL = 7 TTL > 0 6

Table 6.1: Search approaches implemented by Current Systems a higher download speed compared to Tor and Freenet. Exchanging public keys with friends is somewhat easier in OneSwarm, as they piggy-back on existing social networks. All traffic between peers is encrypted using SSLv3. Searching for files uses flooding, which does not have a TTL value and hence is unconstrained. Only a Bloom filter prevents peers from forwarding a single query more than once. If a peer has found enough sources, it may cancel a query by sending a cancel-message. OneSwarm delays forwarding a query for 150 ms at each hop, to allow for the cancel-message to catch up if necessary. Messages forwarded to untrusted peers are delayed with an additional 150-300 ms to obfuscate the path-length. After a file is found, a BitTor- rent transfer can be initiated over the same path as the search. To allow for this search results include a path-id variable.

6.3. Cost and Limitations of Current Systems All five related works implement a file sharing system on top of an unstructured network. In order to compare the approaches used, we show a generalization of their search properties in Figure 6.1. The initial fanout and successive fanout properties determine how many neighbors the query is send/forwarded to. The successive forwarding property defines when a message should stop being forwarded. Table 6.1 80 6. 4P: Performant Private Peer-to-Peer File Sharing

shows the search approaches of the previously mentioned systems using the properties introduced in Figure 6.1.

6.3.1. Bandwidth Cost We can distinguish between two different approaches to search in an unstructured network. First, a P2P network can employ routing to find the peer which is the most likely to have matching files. This approach is implemented by Freenet, but is more commonly found in DHTs. Second, a P2P network can employ (constrained) flooding. Here each peer forwards a query until the forwarding condition is not satisfied anymore. This approach is implemented by Gnutella, Gnunet, RetroShare, and OneSwarm. A big drawback of routing approaches is that partial keyword matching becomes almost impossible to implement. A search query is routed to a particular peer based on the hash of the keyword, therefore partial keyword matching requires multiple queries as they must be routed to different peers. Moreover, for each keyword (or partial keyword), a peer inserting a file must announce its availability. These announces often consume large amounts of bandwidth in DHTs, as they have to be repeated frequently due to churn [89]. Churn denotes the frequent leaving and joining of new peers in the overlay, which causes information stored in the DHT to be stale. In Freenet, the situation is worse as complete files have to be stored at the 6 peer which is responsible for a keyword, e.g. in order to implement partial keyword matching multiple copies of a single file have to be stored in the network. However, flooding a network can cause even greater bandwidth usage. In 2000, Gnutella used a default TTL value of 7 for queries, causing 95% of the peers in the network to receive each query message. Ripeanu et al. [83] discovered that flooding the network with query messages caused Gnutella to consume roughly 330 TBytes per month excluding file-transfers in a relatively small network consisting of 50,000 peers. Using a similar flooding approach to Gnutella, RetroShare employs a TTL of 6 when digging for a tunnel, leading us to believe that it has comparable bandwidth usage. OneSwarm constrains its flooding in a different manner. It does not incorporate a TTL value, but delays query messages at each hop, allowing cancel messages sent by the issuing peer to catch up with the flood and stop it after enough sources are found. Peers cancel their query after 10 sources have been found. For simplicity, we call the peer which sends this 10th reply the cancel trigger (CT). Figure 6.2 shows the time it takes for a query and cancel message to arrive at a certain hop, depending on which hop sends the CT. From the figure we can see that if hop 2 causes the CT, the reply will arrive at the initial peer after 750 ms (using 150 ms for RTT, and delaying 150 ms at each hop before forwarding). When the issuing peer then immediately sends the cancel message, it catches up with the flood at the 6th hop (1650 ms after the initial query). This is similar to using a TTL of 6 and hence not a scalable solution. Please note that a higher RTT will increase the time it takes for the cancel message to catch up, and a lower RTT will the reduce it. The number of sources required to cancel a search, and forward delay are specified in the OneSwarm paper [46] (10, 6.3. Cost and Limitations of Current Systems 81

Search Query Hop 1 causes CT Hop 2 causes CT Hop 3 causes CT

10 9 8 7 6 5 4 3

Arrived at Hop Arrived 2 1 0 0 500 1000 1500 2000 2500 3000 Time since intial request (ms)

Figure 6.2: OneSwarm cancel mechanism, if a peer in hop 2 causes the CT, the cancel message catches up with the query at the 6th hop.

150ms). However in the current implementation, the number of sources required to cancel a search is increased to 401, greatly increasing the risk of flooding when peers issue queries which don’t have many results. Out of all of the introduced related works, Gnunet has constrained its flooding 6 the most; reducing the number of neighbors contacted based on load, charging peers for forwarding a query, and using a TTL value to limit the number of times a query is forwarded. However, Gnunet requires many queries to be performed in order to download a file, as it needs to traverse the tree. The actual number of request can be computed using the following equation wherein P equals the number of initial pieces (filesize divided by 1024 bytes), as described in [13].

blog (P )c X51 RequiredQueries = P + 51i i=0

If a peer wants to download a 1 GByte file, it is required to perform 1,000,052 queries. In order to have an estimate of the bandwidth required to perform this many queries, we assume that a single query requires 28 bytes to be send over the network (20 bytes hash, and two integers: TTL and priority). If we additionally assume that, by constraining its flooding, Gnunet manages to send a query to only 5% of all peers, sending 1,000,052 queries would require 65 GBytes of overhead traffic in a network consisting of 50,000 peers. In “version 2.0” of their protocol, Gnunet uses 32 Kbyte pieces resulting in 32,770 queries and 2 GBytes of overhead traffic [12].

1https://github.com/CSEMike/OneSwarm/blob/88556421a7572565c882c6fcc7c5858cf55eabed/ oneswarm_az_mods/mods_constants/org/gudy/azureus2/core3/config/impl/ ConfigurationDefaults.java (f2f_search_max_paths) 82 6. 4P: Performant Private Peer-to-Peer File Sharing

6.3.2. Limitations Besides bandwidth cost, both Gnunet and Freenet are unable to provide partial keyword matching. Gnunet uses triple hashed keywords in order to prevent intermediate nodes from being able to decipher the query. Thereby making it impossible to perform partial keyword matching, as partial hashes cannot be constructed. Moreover, all three key-schemes implemented by Freenet suffer from the same problem, but two of those schemes (SSK and CSK) additionally require users to know both the public key of the user inserting a file and its keywords before being able to locate them. Although this solves the spam problem which renders the KSK scheme useless, it also causes locating files to be virtually impossible without an additional index.

6.4. 4P Design Another approach to search in an unstructured network is to cluster peers according to their preferences. Preferences can be defined as the files a user downloaded, ratings for movies, etc. A network which is clustering peers according to their preferences is called a semantic network or overlay, and previous work [97] has shown that by organising peers in such a manner, recall can be high without flooding the network. Typical values for a semantic overlay include an initial fanout of 10 and a successive fanout of 0. Therefore in a semantic overlay, as opposed to routing 6 or flooding, a peer searching for a file sends a query to its similar neighbors which then reply, i.e. a query is not forwarded limiting the privacy exposure of a peer. However, similar to Gnutella, neighbors in a semantic overlay still know from which peer the query originated. 4P improves upon a traditional semantic overlay by implementing three features which allow us to preserve the anonymity of both the sender and the receiver of a query. Using a similar approach as Freenet, peers probabilistically forward query messages. Initially, a TTL value is specified by the initiator of the query. This TTL value is decremented at each hop, and when it reaches 0 a reply message is sent back. Additionally, the TTL value of a message is used to determine to how many peers the message is forwarded. If a peer receives a message with a TTL value of 5, it decrements it to 4 and then forwards it to 4 peers. We define the max/initial TTL value as the agreed TTL value used by all peers when creating a new query message. Moreover, a peer can probabilistically decide not to decrement the TTL of a message with the agreed upon max/initial TTL value. This prevents peers receiving a message with the max/initial TTL value from knowing if the previous peer has created or forwarded the query. We define this probability as the initial extension probability or IEP . Finally, a peer can probabilistically decide to keep forwarding a query with a TTL value of 1, preventing an attacker from being able to query a single peer. We define this probability as the final extension probability or FEP . To protect the privacy of the peers replying to a query we use caching. When receiving a reply-message from another peer all results within it are cached locally. This achieves two goals, first the cache of a peer now contains not only its own data, but also cached results from others. Consequently, if after enabling caching, 6.5. 4P Implementation 83 an attacker tries to determine the contents of a peer’s local cache he will not be able to distinguish between cached and local results. Second, as we have built 4P on top of a semantic overlay the cached results will likely be relevant to a peer and its neighbors, thus improving overall recall of the system. However, in contrast to Freenet, peers in 4P do not cache complete files by default. In 4P, we split searching for and downloading files into two separate actions. First, the peer issuing a query will receive a message containing all meta-data found by peers on the search-path. Second, after deciding upon which content to download, a peer can use the established content-path to start downloading it. The content-path is a tunnel from the initiator of a query to a peer which has the content. A content- path is constructed for each result a peer back-propagates to the original query initiator. During the search, each peer in the search-path will rewrite the source IP protecting the identity of the peer it received the query from, and hence a tunnel is required to download a file. While a content-path is active, the metadata associated to it is cached. Content-paths are kept active by sending ping/pong messages to the next hop. As long as this peer replies, the content-path is marked as active. An active content-path, allows peers to use the locally cached metadata while replying to other queries. If a content-path breaks, a peer removes the metadata from its cache as it is unable to provide other peers with the actual contents of the file.

6.5. 4P Implementation 6 Based on our design, we need to build three components in order to implement 4P. First, we provide a method which is able to construct a semantic overlay in a privacy preserving manner. Second, we define the messages used when sending a query. Third, we describe how a peer can download files after finding them. In this section we elaborate on the actual implementation of these components.

6.5.1. Semantic Overlay In order to construct a semantic overlay in a privacy preserving manner we have modiﬁed our earlier work [103]. In this protocol, peers are able to discover and connect to similar peers without having to reveal their preferences/items. By applying homomorphic encryption, two peers can compute the overlap between their preferences sets without disclosing which preferences do overlap, and which don’t. Our extension in this work is a method which allows Bob to reply with a session-key which can be used to encrypt the link without being vulnerable to a man-in-the- middle attack.

Steps required to determine overlap between two peers

1. Alice builds a polynomial having roots in each of the items contained in her set IA. That is, she computes the n + 1 coeﬃcients α0, . . . , αn of the polynomial

2 n f (x) = α0 + α1x + α2x + ... + αnx (6.1)

for which f(IA,i) = 0 for any item in her set. 84 6. 4P: Performant Private Peer-to-Peer File Sharing

2. Next, Alice encrypts the coeﬃcients and sends them to Bob, e.g. sending

EpkA (f (x)). 3. Bob uses the homomorphic properties of the encryption scheme to evaluate the polynomial for each item in his set ΓB. He multiplies each result by a fresh random number ri, and adds a session-key generated for Alice obtaining

EpkA (ri · f(IB,i) + SA,B). 4. Bob adds all evaluated polynomials to a list, permutes the order, and sends it back to Alice. 5. After receiving the list of evaluated polynomials from Bob, Alice decrypts each ciphertext. The ciphertexts decrypt to SA,B in case of an item in the intersection IA ∩ IB, or to a random value otherwise.

Alice can detect SA,B, if more than one item overlap between her and Bob. Moreover, it is very likely that the random values are larger than the 128 bit key we’re using for SA,B. Hence, if Alice finds a single 128 bit value in Bob’s decrypted list, she assumes it is the session key. By counting the occurrences of the session-key, Alice can compute simA,B The session-key generated by Bob cannot be intercepted by any man-in-the- middle, as its encrypted with the key of Alice. Moreover, it is intractable to decrypt 6 either the encrypted coefficients of the polynomial, or the encrypted evaluated polynomials, as they are encrypted using the Paillier scheme [73]. In step4, Bob permutes the order of the list to hide the index of the items which are overlapping. Without this permutation, the index of the overlapping items can reveal information on when Bob added this particular item to his preference list, or even worse if his preference list is sorted it could reveal the distribution of the values in the list. Furthermore, please note that Bob cannot simply reply with all preferences, as the list of possible preferences is very big (in our implementation we use a hash function to limit the size to 240), and Alice can easily detect this as Bob will reply with many evaluated polynomials. Finally, in order to prevent an attack in which Alice will attempt to discover the preferences of Bob by repeatedly running the protocol with only one preference, Alice is required to append a proof-of-work to each request. This proof-of-work can only be used once, and requires Alice to spend a considerable amount of time for each request. An example of such a proof-of- work is HashCash [8], wherein a user sending an email is required to compute a hash based on the current time, destination address, and a random seed. This hash needs to start with at least 20 zero’s, and can only be computed brute forcing different random seeds.

6.5.2. Search Messages In the semantic overlay, peers use two messages while searching for files. Initially, a peer sends a Search-Message consisting of 3 fields. First an identifier, this is a randomly chosen 2 byte unsigned short. The identifier is used to detect cycles in the search-path. Next, a 1 byte unsigned TTL value used by peers to determine 6.5. 4P Implementation 85 if this message needs to be forwarded. Finally, the message contains the keywords specified by the user, represented as a string with a dynamic size of at most 256 characters. Upon receiving a Search-Message, a peer updates the TTL value. If the TTL value is greater than 0, the peer creates a new Search-Message using the same identifier and the updated TTL. This message is then forwarded to the next TTL peers, which are randomly chosen from the list of most similar peers. If the updated TTL value equals 0, a peer replies with a Response-Message instead. To create this message, a peer will perform a local search and constructs a Response-Message with the same identifier. We only include the metadata of each result, e.g. the name of the file, a description, the size, etc. to reduce the size of the Response-Message. Additionally, each result has a unique identifier used by peers in order to download it. Upon receiving a Response-Message, each peer in the search-path appends its local results and creates a new Response-Message which it sends back to the peer it originally received the Search-Message from. This will eventually cause the Response-Message to back propagate to the initiating peer2. Moreover, each peer in the search-path sorts the results after appending its local results using the same relevance-ranking and only includes the Top-K most relevant results in the Response-Message. This aims to prevent peers from guessing the length of the search-path by counting the number of results. Choosing a K value is application dependent, setting it low will make guessing the path-length harder but 6 also reduces the number of results per query. If a peer receives a Search-Message with an identifier it has seen before (indicating a cycle), it still forwards the Search-Message. However, a peer will make sure that it does not forward a Search-Message with the same identifier to the same neighbor twice. When a peer exhausts all possible neighbors for a particular identifier, it aborts the search-path and replies with a Response-Message.

6.5.3. Downloading Content As searching for and downloading a file are two separate actions, a peer has to be able to request to download a file after receiving a Response-Message. However, as described in the previous section, a Search-Message is sent to a random peer at each hop. Therefore, in order to satisfy a download request, the path to the peer which has the content has to be kept open, as attempting to send a request for the file will probably not end up at the source. This content-path is established by keeping open a connection to the peer it received the result from when back-propagating a Response-Message. When the initiating peer attempts to download the file its request is forwarded along this content-path, and the data is back-propagated. In order to support downloading files from multiple sources we have modified the Peer-to-Peer Streaming Peer protocol (PPSP) [9]. At its core, PPSP can be compared to BitTorrent. It splits data into smaller pieces which then can be down-

2Note that only the initiating peer knows that it was the initiator, all other peers cannot tell whether they are sending the Response-Message back to a forwarding peer or the initiator of the query. 86 6. 4P: Performant Private Peer-to-Peer File Sharing

DataSource

Search-Path Previously established

Content-Path Content-Path

Figure 6.3: Example of a search-path and an content-path. Note, that the DataSource is not actually queried during the search.

loaded in parallel from multiple sources. However, in contrast to BitTorrent it uses UDP, a Merkle tree to both identify and verify pieces of the data, and multiplexes parallel downloads over a single socket. We modify PPSP such that intermediate nodes in the content-path will forward/back-propagate all PPSP messages. This tunnel allows the PPSP protocol to operate as normal. If a peer requires more sources for a download, it can perform an exact search using his remaining similar neighbors. If a hit is found, the new content-path is added to the PPSP download. Moreover, the content-path of a file does not necessarily have to overlap with the 6 search-path. If a peer has cached metadata from another request, then this cached metadata is kept while the content-path is active. Cached metadata is used while replying to queries, and hence successive searches matching cached metadata are used to boost recall. Additionally, because we are using PPSP, we can keep multiple content-paths alive without incurring high costs as PPSP multiplexes traffic over a single socket (re-uses), and thus only one open socket is required for each of our similar neighbors. Figure 6.3 gives an example of a search-path and the established content-path. While performing this search, the DataSource of the matched file did not actually receive the query. However, a peer caching the metadata of the file used the previously established content-path in a reply. Downloading a file in this manner causes each peer in the path to download/upload part of a file. This is clear drawback, but it will preserve the privacy of both the downloader and uploader. If privacy is less of a concern, then by enabling peer exchange (PEX) in the PPSP protocol a peer can decide to (partially) collapse the path and thereby reducing the number of hops between it and the DataSource. Not rewriting the source address is implemented in both Freenet and Gnunet. However it has been shown to be exploitable by a shortcut attack in Gnunet allowing an attacker to determine the initiator of a download request [49].

6.5.4. Choosing the initial TTL, IEP, and FEP Searching for files in 4P, very much resembles (constrained) flooding. As flooding is something we definitely want to avoid, the initial TTL, IEP , and FEP need to be chosen carefully as they might cause 4P to behave similar to flooding, i.e. a 6.6. Security 87 combination of these values needs to be found which limits the number of messages sent. Please note that we can constrain the flooding much more because of the semantic overlay we’re searching on top of, the semantic overlay does not need many messages to be sent as the peers which have the best results are nearby. Whether 4P will flood the network is determined by the combination of IEP and the chosen initial TTL. These two variables determine the probability that one of the peers extended the initial path. If this probability is too high, larger than 0.5, the initial path extension is unlikely to stop. Hence causing flooding of the network. We can compute the probability of at least one peer extending the path (PIE) as follows: PIE = 1 − (1 − IEP )initial TTL Empirically we found that if we choose the initial TTL and IEP such that the PIE stays below 0.5, then no flooding occurs. However, by lowering the IEP , and thereby being able to increase the initial TTL, the initial path extension is reduced. Causing less uncertainty on whether or not the previous peer created the query. The expected value of the initial extension can be computed using:

∞ X IEV = i × PIEi i=1 In our experiments, we have set the initial TTL to 4, IEP to 0.15, and the FEP 6 to 0.5. This results in a PIE of 0.45, and a IEV of 1.5.

6.6. Security In this section we briefly discuss possible attacks, and the extend to which 4P protects the anonymity of peers. In the original Freenet paper [22], the authors evaluate their protocol by considering sender and receiver anonymity. Sender anonymity defines the extent to which the identity of an initiator of a query is protected, receiver anonymity as the extent to which the identity of a peer which replied to a query is protected. The levels of protecting anonymity were described in a paper by Reiter et. al. [78], and range from absolute privacy in which communication cannot be observed to provably exposed in which an adversary can prove that a peer was the initiator. Please note that being able to prove that a peer was the initiator does not necessarily mean that the attacker can read the query. The anonymity of both sender and receiver is exposed in Freenet, when attacked by either a local eavesdropper or collaborating peers. However, sender anonymity is preserved beyond suspicion for collaborating peers. Table 6.2 shows a summary of the anonymity of 4P. Against a local eavesdropper 4P cannot provide any anonymity. We defined a local eavesdropper, as an ISP which can observe all communication in which the computer of a peer participates. Therefore, it can detect that a peer has created a new query, as it can observe that no query was sent to the peer beforehand, and detect that a peer replied to a query, as it might be able to observe new results in a Response-Message due to the increase in size. Encrypting, padding, and combining messages can improve anonymity while 88 6. 4P: Performant Private Peer-to-Peer File Sharing

Attacker Sender Anonymity Receiver Anonymity Local eavesdropper Provably exposed Provably exposed Collaborating peers Probably innocent Probably innocent

Table 6.2: Anonymity evaluation

dealing with a local eavesdropper as it can prevent the eavesdropper from detecting new queries or new responses. Against collaborating peers, 4P is better equipped. The sender anonymity of a peer is protected as a query with the initial TTL value is probabilistically forwarded. Additionally, receiver anonymity is protected as every peer in the search-path will always forward a query until TTL = 0. Therefore, no peer can determine if a reply originated from the peer it just received the Response-Message from or if it originated from a peer higher up the path. Moreover, as peers may forward a query with TTL = 1, an attacker is not able to use TTL = 1 queries in order to determine the actual ﬁles being shared by a peer. Finally, all links are encrypted using the session- keys peers exchanged during the construction of the semantic overlay, ensuring that no man-in-the-middle can listen-in on the queries/message sent by a peer.

6 6.7. Experiments In order to evaluate the performance of 4P, we have designed several experiments which compare the recall performance and associated cost to several other overlay conﬁgurations and their search approach. In the following subsections we describe the dataset, emulation setup, and the metrics we used during the evaluation.

6.7.1. Dataset During the experiments we used a dataset extracted from a deployed semantic overlay network [102]. In this P2P network peers create a semantic overlay by exchanging plaintext messages containing all preferences of a peer. By comparing the preferences of it and other peers, a peer is able to compute which other peers are similar to it. By deploying instrumented clients into the semantic overlay we have created dataset consisting of the preferences of more than 75,000 peers. From this dataset, we selected 1000 peers at random and made sure that each peer has at least 10 preferences. Additionally, we require that each item was preferred by more than one peer (all other items were removed from the dataset). Next, we have split the preferences of each peer in a training/testing set using a 80%-20% split. Dividing a dataset in this manner allows each peer to use its training set to find similar peers, and then use the testing set to check if those peers are similar enough to provide it with search results. E.g. we issue a query for each item in the testset, and see if we can find it using the search algorithm being tested. A 80%- 20% split of training and testing set is common practice in collaborative filtering and used to asses the quality of the discovered similar neighbors [36, 40]. Finally, we computed the 10 most similar peers for each peer using its training set. 6.7. Experiments 89

Overlay Initial Successive Forwarding fanout fanout condition Random Random 10 0 - Semantic Semantic 10 0 - Gnutella Random 10 10 TTL > 0 RetroShare Semantic 10 10 TTL > 0 OneSwarm Semantic 0.5 × #Friends 0.5 × #Friends Not seen yet 4P Semantic TTL TTL TTL > 0

Table 6.3: Search strategies, as evaluated

6.7.2. Emulation setup In order to emulate the semantic overlay we implemented the protocol as described in Section 6.5.1 on top of Dispersy [107]. Dispersy is a P2P data replication platform which implements a semi-random peer selection algorithm, NAT-traversal, and fully decentralized permissions. Building the overlay on top of Dispersy allows us to easily deploy and test the system both on our supercomputer and in the future, deploy it to the Internet. We create a process for each peer and configure it with its own separate database containing its own training and testing set, and the 10 most similar peers it must 6 connect to. In each scenario we deploy 1000 peers onto 10 nodes of our supercomputer, and force them to connect to their most similar peers3. Next, 200 peers use their test set (20% of their data) to send queries. Using the random identifiers of all queries we track to which peers a query is sent, how many peers in total receive a query, at what point in time they receive it, and when the initial peer receives a reply. If two peers during the experiment use the same identifier we remove all data regarding those queries, to prevent possible misinterpretation of the results. During the experiments each peer writes its recall to a file, allowing us to compute the overall mean recall of those peers and thus the search-performance. Recall is computed as follows: #found items Recall = #queries sent

6.7.3. Evaluated strategies In order to compare the performance of 4P, we implemented and evaluated several other strategies derived from the related work as shown in Table 6.3. We have chosen to evaluate a random strategy to show the gain of using a semantic overlay. Furthermore, we implemented the ﬂooding strategy of Gnutella, in order to show the performance gain and higher bandwidth cost of such an approach. Note that we

3By doing this we achieve a performance which does not reflect the actual performance when deployed, as the speed of discovering similar peers, dealing with churn, etc. are not taken into account. However, these factors influence all systems we compare negatively, and both RetroShare and OneSwarm additionally rely on specific friends being online to provide a peer with search results (not depending only on those peers which have a similar preferences). 90 6. 4P: Performant Private Peer-to-Peer File Sharing

0.99 1.00 0.98 0.86 0.76 0.75 Recall

0.50 0.4

0.25 0.09

0.00 15000 13379

10316 Messages send 10000 9108 (note the logarithmic scale) Mean messages sent per query

5000

20 20 313 0 6 Random Semantic 4P RetroShare Gnutella OneSwarm

Strategy used

Figure 6.4: Recall and Messages send of evaluated strategies

actually limit the flooding scenario as we “only” forward a query to 10 neighbors. Finally, we compare the approaches of RetroShare and OneSwarm to 4P using the same semantic overlay (i.e. each peer connects to its 10 most similar neighbors). RetroShare floods the network similar to Gnutella, however it uses a slightly lower TTL. For OneSwarm, we converted their latest published Java code to Python and integrated it into our experiment setup. Moreover, we used all their defaults in order to constrain the flooding of the network. We configured 4P using the values as described in Section 6.5.4.

6.7.4. Results Figure 6.4 shows the mean recall of the evaluated strategies. From the figure it is clear that having a semantic overlay greatly improves recall over simply using a random overlay. A random overlay only obtains a recall of 9%, clustering peers according to their preferences results in 40% recall. However, we note that building and maintaining a semantic overlay is not free. During our experiment a single handshake (used to find your most similar peers) using our RSA-based algorithm costs roughly 79.5 Kbytes on average. Next, using 4P improves recall to 76%. This improvement can be explained 6.7. Experiments 91 by the increased number of messages being sent due to fowarding the request. We believe that the combination of a semantic overlay and our very transparent method for caching yields the increase in performance. The flooding strategies of RetroShare, Gnutella, and OneSwarm show that indeed if only recall is considered, flooding a network results in the best performance. RetroShare achieves a recall of 86%, Gnutella 98% and even more im- pressive OneSwarm achieves a recall of 99%. Altough this is somewhat expected as OneSwarm does not incorporate any constraints on the depth of the flooding. However, if we look at the number of messages send we can see that flooding has one very significant drawback: it requires many messages to be sent in order to achieve the recall performance. During the experiment, all three flooding strategies required roughly 10,000 messages to be sent. RetroShare requires less messages than Gnutella (due to having a TTL of 6, instead of 7), but not less than OneSwarm. Although OneSwarm does not use a TTL value to constrain the flooding, it does manage to require less messages per query. We believe this is mostly caused by a peer forwarding a query to only half of its friends, and the limit on outgoing searches (max 300 per minute). We note that in this configuration, OneSwarm never cancels its requests as a peer does not receive enough replies. After we lowered the number of replies received to 1, the number of messages sent was reduced to 6,200. 4P requires only 313 messages to be sent. These numbers include the replies of all peers, hence only 157 Search-Messages are sent. We do not expect more messages to 6 be sent whenever the size and/or topology of the network changes. This is because the number of messages sent is influenced by the initial TTL value, the IEP and the FEP , and not by the size of the network. In contrast, highly connected friends in the RetroShare and OneSwarm networks influence to how many peers a query will be forwarded, as they both do not limit the number of friends a request is forwarded to (but limit it to 0.5 of friends). The semantic strategies only require 20 messages to be sent (10 requests and 10 replies). If we consider both the recall and bandwidth cost of the evaluated strategies, we believe that 4P clearly improves upon current state of the art flooding based private file sharing systems. Figure 6.5 shows the time it takes for the initial peer to receive the first result, and when its Search-Message was last forwarded. Please note that the RTT of our supercomputer is very low, hence we get very low response times for all protocols and searches for which we do not get a response are not used when computing the mean. Considering the first received result random, semantic, and OneSwarm average below 50 ms, which for the first two is not surprising. Those strategies consist of a single request/reply which should be very quick. However, the performance of OneSwarm is surprising as we expected it to perform equal to RetroShare. As these protocols reply immediately whenever a result is found, and both run on top of the same semantic overlay. We believe that the difference is caused by the larger fanout of RetroShare, resulting in a higher load on the nodes as they receive more messages in a smaller time frame. Looking at the CPU consumption of both protocols during these runs seem to be consistent with this, RetroShare requiring roughly twice as 92 6. 4P: Performant Private Peer-to-Peer File Sharing

4000

3000 First result

2000

1000 539 259 310 48 40 45 0

6 4000 3811 Last Request forwarded

Time in milliseconds 3000

2000 1597 1455

1000 522 48 44 0 Random Semantic 4P RetroShare Gnutella OneSwarm

Strategy used

Figure 6.5: First response received, and last request forwarded during a search. 6.8. Conclusion 93 much CPU-time than OneSwarm. The CPU-time required by Gnutella is even higher (almost three times as much as OneSwarm) and hence is causing the first result to arrive after 310 ms. 4P performs the worst, but this is due to all peers waiting for a reply of their neighbors before back-propagating all results to the initial peer. Considering the last request forwarded durations, 4P performs as expected. Peers only add their replies when back-propagating a request, and hence slightly lower numbers for when the last request was forwarded are expected. Furthermore, both random and semantic perform as expected showing roughly the same numbers as when the first reply was received. For the flooding approaches, all three strategies continue to forward a request even after the first reply has arrived. This is as expected, however RetroShare is on average still forwarding a request almost 4 seconds after the initial peer created it. We expect that this is caused by two factors: first the higher fanout is overloading the nodes, second due to the semantic overlay some peers get much more requests than others. These nodes are similar to many peers (due to downloading popular files) and hence get more requests compared to other peers do not download these. Overloading the popular peers creates a back-log which slows down the forwarding process. Although Gnutella causes an even higher average CPU load, its random overlay does not cause some peers to be overloaded (as we expect all peers to have a similar in-degree), and hence the last request is forwarded after 1.6 seconds. OneSwarm performs similarly to Gnutella, and is helped by the fact that 6 it causes a lower CPU load. Moreover, OneSwarm has a lower fanout which reduces the load on the popular peers and hence does not overload them as we have seen in RetroShare.

6.8. Conclusion In this chapter we have shown that it is possible to create a privacy preserving and performant file sharing system which in contrast to current systems does not flood the entire network. Moreover, since we do not hash keywords, peers in the network can perform partial keyword matching, a feature that does not exists current private file sharing system such as Freenet and Gnunet. After issuing a query, we provide peers with a list of results from which they can decide which file to download. Using the pre-established tunnels to one or more datasources of a file, downloading does not require new queries to be sent. If more than one datasource, or multiple paths to a single datasource are found then those paths can be used in parallel. This reduces the load imposed on intermediate nodes which otherwise are required transfer a complete file. Moreover, multiple paths to datasources help us bypass possible bottlenecks in the network, and thus improve overall download performance. Thanks to extensive experimental evidence we have shown that 4P requires substantially fewer messages to be sent compared to a (constrained) flooding approach, while still achieving a mean recall of 76%. Moreover, in contrast to flooding the number of messages used by 4P during search is constant and does not scale exponentially with the network size.

7 ReClaim: a Privacy-Preserving Decentralized Social Network

The privacy concerns associated with the popularity of online social networks have given rise to numerous research papers which focus on either storing the data in encrypted form, or decentralizing the network. However, without the incentive of being able to read private data, a centralized system cannot be sustained. A decentralized network does not rely on such incentives and can continue to operate based solely on user contributed resources. Unfortunately, current proposals either still rely on centralized components, a pre-existing friend-to-friend network, or incur substantial bandwidth overhead. This renders them useless for actual deployment. ReClaim employs an existing technique to the PSI problem to build a semantic network wherein peers have connections to friends and friends-of-friends. This is achieved by a handshake which uses homomorphic encryption to privately determine which friends are shared amongst two peers. Afterwards, peers synchronize messages destined for friends, providing them with easy access to replicas. Moreover, storing those messages allows ReClaim to handle excessive churn (peers joining and leaving the network at a high rate), overcome NAT-ﬁrewalls (by allowing for indirect communication), and allow friends to communicate without being online at the same time. After describing the design of ReClaim, we implement a proof-of-concept application and show during extensive emulation that peers can connect to more than 60% of their friends within 10 minutes in a network consisting of 4000 peers. Moreover, within 10 minutes, peers using ReClaim can establish an indirect connection to 95% of their friends.

Parts of this chapter have been published in USENIX FOCI 2014 [104].

95 96 7. ReClaim: a Privacy-Preserving Decentralized Social Network

7.1. Introduction Online social networks (OSNs) have become hugely popular over the last ten years. Facebook, with more than 1.23bn monthly users, is the largest. Its network consists of more than 200bn friendships, 400bn photographs, and generates more than 6bn “likes” every day1. However, the success of Facebook has also given rise to some substantial privacy concerns, as a single company is now responsible for managing a wealth of privacy sensitive data. Although Facebook is committed to protect our privacy2 it can be questioned if such a big collection of data can be secured. Without questioning the level of expertise of Facebook’s employees, the Lavabit case has revealed that any company can be asked to hand over its private keys in order to comply with a court order, thereby exposing all data stored in its cloud. Centralized storage can therefore be considered compromised, even if stored in encrypted form using a key managed by the storage provider. Therefore, managing keys locally is a more sensible solution. However, as social networks depend on being able to read your private data, there is a strong incentive for them against implementing this. Moreover, freeriding on top of an existing social network and using it to send your friends pointers to encrypted data stored elsewhere is likely to be actively blocked and although your data will be protected in this case, the social network graph is still accessible. In this chapter we present a fully decentralized solution for online social networks which does not depend on any centralized component. We use friends of friends to replicate data, and therefore do not need any server for storage. The resulting running costs are negligible, as there are no costs associated to storing data. Friendships 7 are established by exchanging public keys out-of-band, allowing all messages to be encrypted, making them unreadable for everyone but the user who has the private key. Discovering friends which are currently online is achieved by employing an existing approach to the Private-Set-Intersection (PSI) problem. This allows two peers to discover which friends they have in common without disclosing those they do not. After discovering common friends, peers synchronize missing messages destined for those friends using Bloom filters (compact hash representations of their local message database). Bloom filters allow peers to efficiently receive all messages destined for themselves and their friends, as they prevent duplicate message transfers.

Contribution

1. The design and implementation of a protocol which allows peers to construct a decentralized social network.

2. The means of ﬁnding friends which are online without exposing all friends of a peer.

1http://blogs.ft.com/tech-blog/2014/02/facebook-turns-ten-likes-and-lows 2https://www.facebook.com/notes/10150378701937131 7.2. Related Work 97

3. A method to synchronize with and store messages at friends, allowing the network to operate without centralized storage servers.

4. An extensive evaluation, testing both the speed with which friends can be found and the synchronization performance under churn.

7.2. Related Work Sun et al. [93] describe an efficient revocation mechanism for a private OSN. Their approach is based on broadcast encryption. Broadcast encryption is a technique in which you encrypt a file/message once with a symmetric key, and let a subset of privileged users decrypt this symmetric key with their private key. Hence, a file can be stored on a central server once, but can be decrypted by multiple users. Revoking the privileges of a single user does not require any messages to be send, it will only cause the next encrypted message to not be decryptable by the revoked user. Additionally, Sun et al. discuss how to query for encrypted files by employing public key encryption with keyword search (PEKS). PEKS allows a storage site, which is only storing encrypted files, to return only those files matching a query. The storage site cannot determine content of the query or the contents of the returned files. A secure index is used to allow for this, which, although a costly to build enables storage sites to efficiently respond to such queries. Beato et al. [10] describe a browser plugin which employs broadcast encryption to be able to post and retrieve content stored in encrypted form at a central server. In contrast to Sun et al., their solution (Scramble) uses existing social networks to send links to encrypted content and hence does not require a secure index. Users receiving such a link can decrypt the content by first decrypting the symmetric key 7 using their private key, then decrypt the content, and finally verify the signature. Scramble encrypts/decrypts content in a transparent manner, hiding the complexity from its end users, e.g. users of Scamble do not see any difference between encrypted and non-encrypted files. Safebook is a decentralized online social network (DOSN) [31] based around the concept of matryoshkas. Using the trust relationships of the social graph, ma- troyshkas relay messages from a peer in their center. The matryoshkas directly connected to the peer they are protecting are called mirrors and store encrypted data destined for the peer. The outermost matryoshkas are called entrypoints, and register themselves in a DHT as the innermost peer. By varying the pathlength between the inner and outer matryoshkas, the actual identity/location of the peer is hidden from the DHT. In order to join Safebook, a peer is required to go to the TIS. This centralized server ensures that each user only gets a single identity, thereby “solving” Sybil and impersonation attacks. However, a user is required to provide proof of his identity using an out-of-band process that relies on real-life mechanisms such as face-to-face meetings, or existing tamper-proof schemes such as a passport of ID card. After receiving an identifier from the TIS, a peer can bootstrap into the network and create/connect to his matryoshkas. Cachet [71] is also built around a DHT. However, it does not incorporate ma- 98 7. ReClaim: a Privacy-Preserving Decentralized Social Network

tryoshkas, but uses the DHT as a resilient data store. Users of Cachet store their latest update in the DHT, and point towards this update on their wall. All updates are encrypted using attribute-based encryption, which allows users to target groups of users with each update. Additionally, as a performance enhancement, friends are used to cache decrypted updates in order to reduce the number of lookups in the DHT/decryptions. Mega et al. [65] describe how they envision eﬃcient dissemination of messages in a DOSN. Instead of relying on a DHT, they build a friend-to-friend network in which messages are distributed using rumor mongering. Herein, peers proactively push a message to others while it is hot. After getting one or more replies from peers that they already received this rumor, it goes cold and a peer stops forwarding it. By assuming a prebuilt F2F network and incorporating history in the rumors, they claim that their protocol signiﬁcantly improves over the mainstream gossip protocols and direct mailing.

7.3. Goals and Security model In this chapter we design, implement, and evaluate a decentralized online social network which does not rely on a central server. We employ an existing approach to the PSI problem to locate friends and friends-of-friends which are online in a privacy preserving manner. This results in a system that does not rely on any existing network, and one which runs without any centralized component/oversight. Messages between peers are synchronized in a push-pull manner, and can contain wallposts, likes, images, etc. After creating a new message, it is pushed to all connected peers which have the destination peer in common. Older or missed 7 messages are synchronized using Bloom filters, allowing peers to request messages which were sent when it was offline. All peers in our DOSN, named ReClaim, store encrypted messages destined for their friends. This allows ReClaim to continue to operate reliably in the presence of NAT-firewalls, intermittent network connectivity, and significant link delays. In contrast to related work, we do not employ broadcast encryption. While broadcast encryption will reduce both bandwidth and storage costs, it requires us to either annotate each encrypted message with the destination peers, or implement a secure indexing scheme. The former will expose the social graph to our friends (something we want to prevent), the latter will require peers to build and maintain costly secure indexes. Moreover, the use of attribute based encryption, as in Cachet, is non-trivial. Managing the access policies of cached objects in a decentralized setting is hard, and the latency introduced by the ABDecryption [71] could only be overcome in Cachet by adding unencrypted social caching. Therefore, in order to simplify our system and the resulting protocol, a ReClaim peer creates an encrypted message for each peer it wants to send an post to. We assume a semi-honest setting [39] also known as honest but curious. This setting assumes that peers do not deviate from the protocol, but might attempt to analyze messages in order to infer additional information. In this setting, ReClaim will protect the social graph from peers which do not have any friends in common, i.e., any two peers which do not share friends will never be able to determine the 7.4. Features 99 friends of the other. Moreover, such peers will never receive an encrypted message destined for the other. Peers which do have overlap in friends are able to detect which friends overlap, and hence construct a partial social graph. However, if these two peers are not friends themselves they will not be able to discover their true identities, as these are kept hidden. Finally, the two peers will still never receive a message destined for the other, as peers will only be able to receive messages destined for their friends. Peers which are friends will receive and store messages destined for each other. However, it is intractable to read these messages as they are encrypted using the public key of their friend. Moreover, friends are not able to read the sender-id of a message as this is encrypted as well. Finally, peers with overlapping friends will expose which messages they create. This is due to the push based synchronization. Whenever a peer receives an update message without sending a synchronization request, it can assume that the sender created it. Disabling the push based synchronization will resolve this. However, at a latency cost, as synchronization requests are send in a periodic manner. As ReClaim aims to be an alternative to a centralized OSN, we choose latency over privacy as this only impacts privacy when dealing with peers which share at least one friend.

7.4. Features Our privacy preserving decentralized social network will provide users with all the functionality found in current centralized solutions with the added beneﬁt of remaining in control of your private data and hiding the social graph. However, this requires a method that allows a user to determine if any of its friends are online, 7 allow him make new friends, and unfriend friends after a relationship has gone sour.

7.4.1. Establishing a new Friendship In order to establish a new friendship, two users are required to exchange their identities. We use a public key to identify a peer, which is only shared with friends whom we want to communicate with. As there is no central public key infrastructure (PKI), we rely on out-of-band communication in order to send a friend-request. The public/private keypair is generated locally, but both parts of the key are kept secret.

Steps required to establish a new friendship 1. Using out-of-band communication Alice obtains the public key of Bob. E.g. Bob sends her an email with his public key inviting her to befriend him. 2. Next, Alice adds Bob’s public key to her handshake message and starts looking for a peer which has this key in common. 3. After ﬁnding Bob or one of his friends, Alice encrypts her public key using Bob’s public key and sends it to him/stores this message at his friend. 4. Finally, after receiving the public key of Alice, Bob can start sending her messages. 100 7. ReClaim: a Privacy-Preserving Decentralized Social Network

Alice Bob Charlie Dean Friends with Bob, Alice, Alice, Alice Charlie, Charlie Bob Dean

Table 7.1: An example social network

Alice A,B,C Bob

A,D A,B,C A,B,C

Dean A Charlie

Figure 7.1: Resulting FSF handshake

7.4.2. Locating Friends Without a centralized lookup service, peers need another method to locate friends 7 which are currently online. In ReClaim we achieve this with a method which determines if the peer we have just connected to is a friend or has a friend with us in common. We call this method FSFA,B. However, as we are trying to locate friends in a privacy preserving manner, we require that the method must be able to function without disclosing any of the friends Alice and Bob do not share. Moreover, it should be possible for Bob to forward the request of Alice, and should not be able to determine if any of his neighbors share friends with Alice. The input of the FSF method is a friendset F which contains the identifiers of all friends of a user and its own identifier. We implement the FSF method using an existing approach to the PSI problem from cryptography. The PSI-problem defines a setting wherein two parties to find out which items in their sets overlap without disclosing the sets themselves. By using the identities of friends as items, we can compute which friends or shared friends Alice and Bob have in common. In Table 7.1 and Figure 7.1 we show an example of the FSF handshakes. When Alice performs the handshake with Bob, the resulting overlap will be the identities of Alice, Bob, and Charlie. Connections to peers which have friends in common are kept active by regularly sending a keep-alive messages. We rely on a peer sampling service to provide us with new peers to run the FSF method with. 7.5. Protocol Details 101

7.4.3. Posting a Message When posting a message, Alice first signs the message using her private key. Next, Alice uses the public keys of each friend, to create separate encrypted copies. This allows for a very flexible manner of distributing messages, as Alice can decide per post whom to send this particular update to, e.g. it allows Alice to define groups whom to send a particular post to. After creating the encrypted messages, Alice determines which of her current connections accepts messages for the friends she is sending the post to, and pushes the messages. Referring to Fig. 7.1, if Alice posts an update to Charlie she will send an encrypted message to both Bob and Charlie. Although this message in encrypted, both peers will know that Alice is sending an update to Charlie (as the destination is not encrypted). As Bob already knew that Charlie was a shared friend we do not consider this harmful, moreover Dean does not receive the message and hence does not learn about the friendship between Alice and Charlie. Peers additionally store the encrypted updates locally in order to reply to synchronization requests. Hence, if Alice does not have active connection for a particular friend, she will synchronize the message at a later time when replying to a synchronization request. In Section 7.5.2 we will describe this pull based synchronization mechanism in depth.

7.4.4. Unfriending If Alice unfriends Bob, she can simply decide locally to stop sending messages to him. However, this will still allow Bob to locate Alice and her friends, as her public key (her identity) has not changed. Bob will not receive any new messages as Alice 7 stopped encrypting messages with his public key. In order to permanently unfriend Bob, Alice changes her identity (public/private keypair), and sends all her friends (which now does not include Bob) her new public key, and she can switch identities. However, as changing identities can cause Alice to miss messages addressed to her old identity, we implement a time period in which Alice uses both her new and old identity in order to minimize this risk.

7.5. Protocol Details In the following paragraphs we will introduce the FSF method and the synchronization protocol we developed and implemented for ReClaim.

7.5.1. PSI protocol As mentioned in Section 7.4.2, ReClaim requires a FSFA,B method which allows two peers to determine which of their friends overlap. In our implementation we use an existing approach to the PSI problem as deﬁned by Freedman et al. [39]. We deﬁne the friendset of Alice as FA, and the friendset of Bob as FB.

Protocol 4: PSI protocol, communication between Alice and Bob proceeds as follows: 102 7. ReClaim: a Privacy-Preserving Decentralized Social Network

1. Alice builds a polynomial having roots in each of her friends contained in her set FA. That is, she computes the n + 1 coeﬃcients α0, . . . , αn of the polynomial 2 n f (x) = α0 + α1x + α2x + ... + αnx (7.1)

for which f(FA,i) = 0 for any friend in her set. 2. Next, Alice encrypts the coeﬃcients and sends them to Bob, e.g. sending him

EpkA (f (x)). 3. Bob uses the homomorphic properties of the encryption scheme to evaluate

the polynomial for each item in his set FB, obtaining EpkA (f (FB,i)).

4. Bob multiplies each result with a fresh random number ri, and then adds the

original input to the result EpkA (f (FB,i) × ri + FB,i) 5. Bob appends all evaluated polynomials to a list, permutes the order, and sends it back to Alice.

6. Alice decrypts each ciphertext and veriﬁes if any of the items occur in her friendset FA. The items which overlap are the shared friends between her and Bob.

We encrypt the coefficients of the polynomial with the Paillier cryptosystem [73]. This allows Bob to compute the output of the polynomial, but prevents him from knowing if the output is 0 or a random value. After sending the value back to Alice, 7 she can decrypt it, allowing her to see which friends are shared between her and Bob. However, as Bob is multiplying the value of each evaluated polynomial with a fresh random value, Alice cannot know which friends are in his set and not in hers. Moreover, after performing an handshake Alice cannot differentiate between Bob being a friend, or Bob having some friends in common due to him permutating the order of the evaluated polynomials. We can obtain the polynomial required in step 1 by simple multiplication of the factors (x − FA,i). This way, the resulting polynomial will have the most significant coefficient take the value 1 and a degree equal to the number of items in Alice’s set. Finally, we use hashing and partitioning to reduce the bandwidth required for the FSF handshake. This reduces the size of the Paillier key required to encrypt the polynomial as that depends on its degree and the size of the identifiers. The expected maximum value of the largest coefficient is 2 hashsize×degree, which determines the minimum value of the Paillier key. For more information we refer to Chapter5 wherein an additionally reduction in CPU overhead is described. By hashing the public keys which identify friends, we can reduce the size of the coefficients, and hence use a smaller Paillier keysize. Partitioning reduces the degree of the polynomial, by splitting it into multiple smaller ones. We group the identifies of our friends by the first byte, and creating a polynomial for each. Without using partitioning, a peer creating a polynomial consisting of 100 friend identities, each being 100 bits requires a Paillier keysize of 10000 bits. If by partitioning we can reduce the number 7.5. Protocol Details 103

Alice Bob Charlie

FSF request Forward FSF request

FSF reply FSF reply

Bloom filter + overlap + intro Charlie NAT Puncture request Missing messages

NAT Puncture

Figure 7.2: Synchronizing messages in the social network of friends per polynomial to 10, we can switch to a keysize of 1000 bits resulting in substantial bandwidth savings. For more information we refer to [103] wherein additionally a reduction in CPU overhead is described.

7.5.2. Message Distribution In our implementation we build upon a fully decentralized message synchronization platform which provides us with semi-random peer selection and a synchronization 7 technique based upon Bloom filters. Bloom filters are compact hash-representations which allow for membership testing [15]. A peer uses Bloom filters to tell another peers which messages it already received without having to send a complete copy of its database. Upon receiving a request, a peer will reply with all messages it has locally but the requesting peer has not. In our system, we modify synchronization approach, as we only want to synchronize messages of friends overlapping between us and the peer sending the request. Referring to Figure 7.1, if Alice performs an handshake with Bob and she will discover that the identifies of Alice, Bob, and Charlie intersect. Next, Alice creates a Bloom filter containing the messages destined for this group of users she has locally available and sends it to Bob. After receiving this Bloom filter, Bob will respond with all messages he has destined to Alice, Bob, and Charlie but were missing in the Bloom filter of Alice. Figure 7.2 gives an overview of the synchronization steps. As can be seen Bob forwards the FSF request of Alice to Charlie (and others), and waits for their replies before replying to Alice. This allows Alice to request Bob to send a message to one of Bob’s neighbors which is most interesting to Alice (in this case Charlie) and instructing him to puncture/prepare his NAT-firewall. Puncturing a NAT-firewall will allow Alice to connect to Charlie in a next handshake round. 104 7. ReClaim: a Privacy-Preserving Decentralized Social Network

This synchronization technique will eventually cause all peers to store all messages addressed to their friends as well as those addressed to themselves. E.g. adding an identity of a friend to your friendset, implies that you as a peer are willing to store all messages addressed to this friend in order to spread them to him. However, as all messages are encrypted with the public key of the destination, it is intractable for a friend to read messages except those which are addresses to him.

7.5.3. Message Body Messages in our system allow users to perform a range of actions in its social network. In order to allow for easy extendability we use a language similar to that of irc. All messages are plain-text (after decrypting), and start with a “/”. The list of implemented actions include:

• /post header message

• /like message-id

• /comment message-id header message

• /me new-thought

• /thumbnail binary-blob

• /binary identiﬁer encryption-key header message

• /small-binary binary-blob header message 7 Additionally, each message contains a message-id. This message-id is a combination of the public key of a peer with an autoincrement integer and is used to reference a specific message when commenting-on, liking, etc. Using this basic format most of the action which can be performed in a centralized social network are supported in ReClaim. When sending a binary file, we don’t include the complete file (as it would be replicated to all friends) but include an identifier to let a peer download the file from the peer it received the update from. E.g. an application could send a BitTorrent infohash to allow a peer to download the file from multiple peers concurrently. Moreover, we additionally encrypt the files to allow peers attempt to download a file without revealing its content. Smaller files such as thumbnails, are replicated to all friends as they require less storage and bandwidth. Similarly, the small-binary action can be used to send generic small files.

7.5.4. Large Binary Download After receiving and decrypting a message which contains a binary identifier, a peer will optimistically assume that the peer from whom it received this message has the file. This assumption holds whenever the message is received from a peer which received a similar message only addressed to it, or if the message was received from the peer who created it. If the assumption does not hold, a peer will try other peers 7.6. Application Design 105 it has an active connection to which have the friend which created the message in common or will wait for a peer which have this friend in common to be found. Attempting to download the file from a peer exposes the identifier of a swarm. However, without the key which is also included in the “/binary” message a friend isn’t able to decrypt a file. Files themselves are encrypted with a random AES key. However, when sending a large file to multiple peers, we encrypt the file once and send the same binary identifier to all peers.

7.5.5. Peercache In ReClaim, peers themselves are responsible for establishing connections to their friends, and friends of friends. However, by reducing the time it takes to establish those connections, we will improve the user experience substantially. Therefore, a ReClaim peer will maintain a peercache containing the ip/port addresses of peers which share at least one friend and are directly connected to the Internet. These peers can assist in reconnecting to friends, as messages sent these peers are not blocked by a NAT-ﬁrewall, and hence we can easily verify if they are online. Halkes et al. [41] measured that up to 64% of peers in a P2P network are behind a NAT-ﬁrewall, hence we expect to be able to create a peercache consisting of 36% of peers which share at least one friend. In a social network, this results in a substantial amount of peers, as users typically have a high number of friends, and hence an even larger number of friends of friends.

7.6. Application Design We implement ReClaim using the core of Tribler [102]. Tribler is the product of 7 a research project funded by multiple European research grants and provides its users with the ability to download .torrents, remote search, video-on-demand, live streaming, etc. Using only the core allows us to cherrypick some of its components for our social network. PPSP can be compared to BitTorrent as it splits files into smaller pieces which then can be downloaded in parallel from multiple sources. However, in contrast to BitTorrent, PPSP uses a Merkle tree to both identify and verify pieces of the data. A peer can therefore verify pieces of a PPSP swarm using only the roothash of the Merkle tree. In contrast, in BitTorrent a peer is required to first download the .torrent file in order to do this. We include the roothash of the Merkle tree in the “/binary” message, allowing a peer to immediately download a file after receiving the message. We implement our FSF method using Dispersy [107] and modify its core to allow for subsets of messages to be synchronized based on the shared friends between two peers. Moreover, in the network peers maintain a list of active connections to peers which share friends. After detecting the NAT-type of these peers, we create a peercache containing peers which share friends and have a direct connection to the Internet. These peers are used to quickly bootstrap into the social network after starting the application. A screenshot of the proof-of-concept ReClaim application is shown in Figure 7.3. 106 7. ReClaim: a Privacy-Preserving Decentralized Social Network

Figure 7.3: ReClaim proof-of-concept application

In the next section, we will evaluate the performance of this proof-of-concept application when emulating a network consisting of 1000, 2000, and 4000 peers, running this exact code. 7 7.7. Evaluation In this section we will evaluate the performance of ReClaim in its typical use-case scenarios. Using two datasets we show the speed of ﬁnding friends, a process called bootstrapping, the speed of distributing messages in the network, and the impact of churn on the network.

7.7.1. Dataset In order to show the performance of ReClaim we will use two datasets. First, we use a Facebook dataset collected by McAuley et al. [64]. This dataset contains just over 4000 users and their relationships. We use this dataset to show the speed of bootstrapping the P2P network (the speed at which you connect to your friends). In order to show the impact of the network size, we create two subsets consisting of 1000 and 2000 users by selecting random edges. Second, we use the much larger Facebook New Orleans dataset collected by Viswanath et al. [96]. This dataset, in contrast to the previous dataset, contains over 800,000 user interactions (e.g. wallposts) between 60,000 users over time. However, as we cannot emulate a network of this scale, we select all 80,000 interactions between 4000 users. Please note that although the complete dataset contains over 60,000 users, only 39,985 users actually posted something on their wall. 7.7. Evaluation 107

1.00 Direct connection 0.75

0.50

0.25

0.00 1.00 Indirect connection 0.75 Connected to friends 0.50

0.25

0.00 0 10 20 30 40 50 60 70 80 90

Time into experiment (Minutes)

Number of peers 1000 2000 4000 Without Cache With Cache

Figure 7.4: Bootstrapping speed of peers in a ReClaim overlay 7 7.7.2. Setup To emulate ReClaim using these datasets, we deploy a separate process per peer using 20 nodes of our supercomputer. Each process is running the code from the proof of concept application introduced in the previous section with some additional instrumentation to be able to monitor the connections of each peer.

7.7.3. Boostrapping Using the first dataset we show the speed of locating friends in an ReClaim network. In this experiment 80% of the peers are online, and 20% of peers join/start locating their friends. We perform the experiments with and without the peercache, showing the speedup of being able to connect to 36% of your friends at startup. Moreover, we show the speed at which peers find an indirect connection to their friends. These indirect connections are used to store replicas of messages destined for your friends. We expect any peer to be able to connect to 36% of its friends because of the NAT-firewall is preventing a direct connection to the other 64% of your friends (Section 7.5.5). Figure 7.4 shows the result of the experiments. In total we have run six experiments by varying the number of peers and enabling/disabling the peercache. Each experiment took 1.5 hours. From the graph we can observe three things, first using the peercache substan- 108 7. ReClaim: a Privacy-Preserving Decentralized Social Network

300 s 500 s 700 s Daily Online time (seconds) 287 460 625 32 Oﬄine time (seconds) 285 470 625 324 Nr Sessions 12.5 7.7 5.6 20.9

Table 7.2: Characteristics of Churn Models

tially improves bootstrapping into the network as we can connect to 36% of our online friends. Second, doubling or quadrupling the size of the overlay does not have a substantial impact on the speed of bootstrapping. Third, although the initial speed of bootstrapping is quite rapid (connecting to 60% of online friends in 10 minutes) it drops off and slows down. The slowdown is an indication that the process of locating friends is being hindered by clustering. Although, a social graph has a high clustering coefficient, peers are stuck in a local optima which slows down the speed of locating friends which are not currently connected. Adding more randomness while forwarding the similarity request created by Alice to the selection of neighbors by Bob will improve this. However as the degree of randomness depends on the structure of the social graph this optimization is out the scope of this paper. Moreover, ReClaim does not require a direct connection to a friend to be able to send messages to him. If no direct connection is available, it will use the friends of a user to make sure that a message will still arrive. This is visualized in the Figure ?? using the indirect connection lines. Peers are able to connect to more than 95% of 7 their friends indirectly within 10 minutes. 7.7.4. Message distribution Using friends of a friend to replicate messages, ReClaim is able to cope with intermittent connections (churn) often found in a P2P network. In the following experiment we will demonstrate this using the second dataset which includes interactions between peers. We implement two different churn models, first using an approach as described by Zhonghong et al. [72] we emulate a network in a stable state. Herein, we keep the number of peers which are online/offline roughly the same by defining an mean online time and setting the mean offline time to the same value. We compute the values for the online/offline time using an exponential distribution combined with a minimum online time of 30s. Although, Stutzbach et al. [92] have shown that using an exponential function to model churn isn’t the most exact method, Zhonghong et al. showed that this doesn’t effect the performance of the emulated system. Second, we emulate the performance of the system using the daily online pattern of users found in Yahoo! Messenger as described by Chiluka [21]. In this model, roughly 15% of all users are online between 9 AM and 9 PM, from 9 PM to 4 AM this drops to 4%, and then picks up again to reach 15% at 9 AM. We converted each hour in to a 30 second timeslot, and used the probabilities to determine if a peer should be online/offline. 7.8. Conclusion 109

1.00

0.75

0.50 CDF

0.25

0.00 0 10 20 30 40 50 60 Delay between creating and receiving a message (Minutes)

Churn None 300 s 500 s 700 s Daily

Figure 7.5: Delay between sending and receiving a message

The characteristics of the two churn models are shown in Table 7.2. We used a 300, 500, and 700 second mean for the exponential churn model. These values are typical for P2P applications as they suffer from very low mean online times. In Gnutella average online times of less than 1 minute have been observed [81]. However, we expect a higher mean for ReClaim as its usecase differs from the typical 7 file sharing usecase common in P2P. Figure 7.5 shows the delay between sending and receiving a message from a friend. In order to show the differences in performance we have run the experiment without churn, with an exponential churn model using a mean of 300, 500, and 700 seconds, and with the daily churn pattern. Without churn, messages are received within seconds (having a mean of 0.34 seconds), clearly shown in the CDF. The churn model based on the daily pattern of Yahoo! users causes peers to be offline for most of the day, this result in 75% of messages being delayed. How- ever, as shown in the CDF, eventhough less peers are online at the same time, the synchronization protocol makes up for this difference in by using friends of friends to deliver messages. Additionally, as shown in Table 7.2, peers are online for more than 20 times during the experiment, allowing them to still synchronize the missing messages effectively. The daily churn pattern causes 75% of message to be received 22 minutes.

7.8. Conclusion In this chapter we have shown that it is possible to build a fully decentralized social network which does away with the requirement of a centralized server. Using a realistic workload and churn model, we have evaluated the performance of ReClaim 110 7. ReClaim: a Privacy-Preserving Decentralized Social Network

and shown that it can eﬀectively synchronize messages in a network consisting of 4000 peers. Thereby, removing the need of storing privacy sensitive information in a single compromised location, and instead making users themselves are responsible for their data. ReClaim is built on top of existing cryptographic primitives which allow us to limit the exposure to peers which have at least one friend in common. We have shown that using the FSF handshake a peer can connect to more than 60% of its friends in a system consisting of 4000 peers within 10 minutes. Moreover, it can create an indirect connection to 95% of its friends within 10 minutes. Finally, during emulation we have shown that churn (peers rapidly leaving and joining the system) does slows down the speed of receiving messages, but does not prevent them. Aside from experimental evaluation, we build a proof-of-concept application which showcases ReClaims basic functionality. The application allows users to communicate with their friends, exchange pictures, post likes, etc. in a fully decentralized manner. All traﬃc is encrypted, and so are the messages stored at friends.

7 8 Conclusion

In this thesis we have researched collaboration in P2P networks, its impact on privacy, and shown how to reduce these. First, we have shown that using Dispersy, peers can collaborate in large-scale message synchronization without the need for a central server. Second, using Open2Edit, we showed that users can collaborate using a P2P network as a backbone, eliminating the cost associated to hosting a website. Finally, we showed in three diﬀerent chapters that by redesigning the collaboration protocols, the privacy of peers in the P2P network can be increased substantially, making it much harder to track the actions of a single peer. In the remainder of this chapter, we will present our conclusions and suggest possible future work.

8.1. Conclusions Our main conclusions of this thesis are:

1. In Chapter3 we have shown that by combining Bloom filters and a NAT traversal technique, with two message selection heuristics, and two Bloom filter optimizations we can effectively synchronize more than 500,000 messages created by more than 300,000 peers. Going well beyond the current state- of-the-art, which cannot realistically synchronize more than 10,000 messages. Moreover, we designed Dispersy such that it has a minimal bandwidth overhead, and can handle churn very well. Finally, during deployment we found that peers can synchronize more than 100,000 messages in a single overlay with ease, requiring only 30 Mbytes of bandwidth.

2. Using Dispersy, we built a collaboration platform which allows for many different deployment scenarios. Using its built-in permission system Open2Edit allows community creators to deﬁne which messages can be created by autho- rized peers, or if all peers can create a message of a given type. In contrast to previous work, Open2Edit does not focus on creating the next decentralized Wikipedia, but targets collaboration platforms in general, hence making

111 112 8. Conclusion

it more flexible. We defined 8 different types of messages which allow users to publish new content, modify, categorize, comment on, etc. Finally, using Open2Edit, we implemented and deployed a new feature for Tribler. This feature allows users of Tribler to create their own community build around files they like, this combined with already existing features in Tribler resulted in a YouTube-like media sharing system (Chapter4).

3. We implemented and evaluated three private set intersection (PSI) algorithms which allow peers in a P2P network to compute how many preferences overlap between them. This allows us to construct a semantic overlay in a privacy preserving manner. The algorithms are based on existing solutions to the PSI-C problem, but are extended to improve the speed at which similar peers are found. We showed that while all three protocols are ready for deployment, as no assumptions were made regarding the possibility of a connection between any two peers or required a trusted third party. However, one protocol requires substantially more CPU time, another consumes more bandwidth, while the third does not require as much bandwidth as the latter and not as much CPU time as the former (Chapter5).

4. Using a semantic overlay, we have shown that it is possible to create a privacy preserving performant file sharing system. In contrast to previous work, 4P does not flood the entire network in order to search it. Instead, the semantic overlay causes relevant results to be close, and hence 4P only contacts a limited amount of peers. We use probabilistic path extension to prevent peers from determining who created a search query. Linked the TTL to the number of peers to forward a query to, in order to constrain flooding as much as possible. Moreover, we modified the PSI-C protocol to include a session-key exchange which cannot be observed by a man-in-the-middle (Chapter6). 8 5. Similarity, we have shown that a semantic overlay can also be used to implement a decentralized online social network (DOSN). By building on top of previous work, we implemented a DOSN which is resilient to churn, can efficiently synchronize missing updates, and uses your friends to replicate data destined for you. Moreover, peers in ReClaim create a separate copy for each friend, greatly increasing the flexibility of targeting specific groups of friends without the need of implemented attribute based encryption. (Chapter7).

8.2. Suggestions for future work In the following section we list a number of promising directions for future work related to the work in this thesis:

1. We’ve shown that Dispersy is able to synchronize more than 100,000 messages eﬃciently. However, in some cases it might not be practical to synchronize all messages to all peers. Therefore, in order to improve Dispersy in these scenarios, we suggest looking into reducing the number of replica’s per message. A reduction from synchronizing a message to everybody down to synchronizing 8.2. Suggestions for future work 113

it to 50% of peers is easily achievable and will yield substantial bandwidth, storage, and cpu savings. Subsequently, in order to reduce the number of replicas further, the Bloom ﬁlters could be used as a source to estimate the number of replica’s for each message.

2. Using Open2Edit, we can experiment with the replication policies. Initially, a user should be presented with subset of the complete community, e.g. a global overview. However, after a user has expressed interest in some messages, the replication strategy of Dispersy should adapt to synchronize similar ones. Allowing users to discover content they like, without the requirement of them needing to synchronize all messages. Determining which messages are similar, and hence which messages to synchronize is non-trivial, and therefore future work. 3. In Chapter6 we designed, and implemented a privacy preserving ﬁle sharing system. In our system we construct encrypted links, use probabilistic forwarding, etc. Yet, keywords are send in plaintext, allowing for censoring by malicious peers. In order to improve upon this, we could hash the keywords, as is done by related work. However, creating a hashed list of censored keywords is equally trivial. Moreover, it prevents partial keyword matching. Constructing secure indexes, as suggested in Chapter6 is a possible solution, but those are often expensive to construct/maintain. Alternatively, using encrypted Bloom ﬁlters as suggested by Bellovin et al. [11] is a promising approach.

4. Building a fully decentralized alternative to a public key infrastructure (PKI) is hard, and usually requires a preexisting trust model [7, 68]. Therefore, solutions often rely on out-of-band communication of public keys in order to distribute them. In Chapter7 we use out-of-band communication of public keys to create friends. However, Identity-Based Encryption (IBE) as described by Boneh et al. [17] can potentially improve upon this, as it allows peers to compute the public key of others based on an allready known identity. An example of such an identity is the email address of an user. This email address can be used as an input to generate the public key associated to it, and hence to encrypt messages destined to this user. IBE currently requires a PKG, a centralized instance, to generate private keys for users. This PKG is also in charge of authenticating a user, verifying if it is the owner of the identity, i.e. the email address.

Bibliography

[1] Diaspora. https://www.joindiaspora.com/. [2] Gnutella Protocol Speciﬁcation 0.6: http://rfc-gnutella.sourceforge.net .

[3] Tribler website. http://tribler.org. [4] E. Adar and B. A. Huberman. Free riding on gnutella. First Monday, 5(10), 2000.

[5] P. S. Almeida, C. Baquero, N. Preguiça, and D. Hutchison. Scalable bloom ﬁlters. Inf. Process. Lett., 101(6):255–261, Mar. 2007.

[6] C. Audet, F., Jennings. Network Address Translation (NAT) Behavioral Require- ments for Unicast UDP. RFC 4787 (Best Current Practice), 2007.

[7] A. Avramidis, P. Kotzanikolaou, and C. Douligeris. Chord-pki: Embedding a public key infrastructure into the chord overlay network. In J. Lopez, P. Sama- rati, and J. Ferrer, editors, Public Key Infrastructure, volume 4582 of Lecture Notes in Computer Science, pages 354–361. Springer Berlin Heidelberg, 2007.

[8] A. Back. Hashcash - a denial of service counter-measure. Technical report, 2002. available at http://www.cypherspace.org/hashcash/hashcash.pdf. [9] A. Bakker, R. Petrocco, and V. Grishchenko. Peer-to-Peer Streaming Peer Protocol (PPSPP), Feb. 2013. available at http://datatracker.ietf.org/ doc/draft-ietf-ppsp-peer-protocol. [10] F. Beato, M. Kohlweiss, and K. Wouters. Scramble! your social network data. In S. Fischer-Hbner and N. Hopper, editors, Privacy Enhancing Technologies, volume 6794 of Lecture Notes in Computer Science, pages 211–225. Springer Berlin Heidelberg, 2011.

[11] S. M. Bellovin and W. R. Cheswick. Privacy-enhanced searches using encrypted bloom ﬁlters. 2007.

[12] K. Bennett, C. Grothoﬀ, T. Horozov, and J. T. Lindgren. An encoding for censorship-resistant sharing, 2003.

[13] K. Bennett, T. Stef, C. Grothoﬀ, T. Horozov, and I. Patrascu. The gnet whitepaper. 06/2002 2002.

[14] BitTorrent, Inc. BitTorrent downloads website. http://download. bittorrent.com/dl/.

115 116 Bibliography

[15] B. H. Bloom. Space/time trade-offs in hash coding with allowable errors. Communications of the ACM, 13(7):422–426, July 1970. [16] M. Blum, P. Feldman, and S. Micali. Non-interactive zero-knowledge and its applications. In Proceedings of the twentieth annual ACM symposium on Theory of computing, STOC ’88, pages 103–112, New York, NY, USA, 1988. ACM. [17] D. Boneh and M. Franklin. Identity-based encryption from the weil pairing. In J. Kilian, editor, Advances in Cryptology CRYPTO 2001, volume 2139 of Lec- ture Notes in Computer Science, pages 213–229. Springer Berlin Heidelberg, 2001. [18] G. Brassard, D. Chaum, and C. Crépeau. Minimum disclosure proofs of knowledge. J. Comput. Syst. Sci., 37(2):156–189, Oct. 1988. [19] A. Broder and M. Mitzenmacher. Network applications of bloom filters: A survey. In Internet Mathematics, pages 636–646, 2002. [20] G. Canals, P. Molli, J. Maire, S. Lauri, E. Pacitti, and M. Tlili. XWiki Concerto: A P2P Wiki System Supporting Disconnected Work. In Y. Luo, editor, Cooperative Design, Visualization, and Engineering, volume 5220 of Lecture Notes in Computer Science, pages 98–106. Springer Berlin Heidelberg, 2008. [21] N. Chiluka. Securing Social Media: A Network Structure Approach. PhD thesis, Delft University of Technology, 6 2013. [22] I. Clarke, O. Sandberg, B. Wiley, and T. Hong. Freenet: A distributed anonymous information storage and retrieval system. In Designing Privacy En- hancing Technologies, volume 2009, pages 46–66. Springer Berlin Heidelberg, 2001. [23] B. Cohen. The BitTorrent Protocol Specification. http://www.bittorrent. org/beps/bep_0003.html. [24] B. Cohen. Incentives build robustness in BitTorrent. In Workshop on Eco- nomics of Peer-to-Peer systems, volume 6, pages 68–72, 2003. [25] P. Congdon, B. Aboba, A. Smith, G. Zorn, and J. Roese. Rfc 3580, ieee 802.1x remote authentication dial in user service, 2003. [26] B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears. Bench- marking cloud serving systems with ycsb. In Proceedings of the 1st ACM symposium on Cloud computing, SoCC ’10, pages 143–154, New York, NY, USA, 2010. ACM. [27] E. Cristofaro, P. Gasti, and G. Tsudik. Fast and private computation of cardinality of set intersection and union. In Cryptology and Network Security, volume 7712 of LNCS, pages 218–231. Springer, 2012. Bibliography 117

[28] E. Cristofaro, J. Kim, and G. Tsudik. Linear-complexity private set intersection protocols secure in malicious model. In ASIACRYPT ’10, volume 6477 of LNCS, pages 213–231. Springer, 2010.

[29] E. D. Cristofaro, S. Jarecki, J. Kim, and G. Tsudik. Privacy-preserving policy- based information transfer. In PETS ’09, pages 164–184, 2009.

[30] E. D. Cristofaro and G. Tsudik. Practical private set intersection protocols with linear complexity. In Financial Cryptography, pages 143–159, 2010.

[31] L. Cutillo, R. Molva, and T. Strufe. Safebook: A privacy-preserving online social network leveraging on real-life trust. Communications Magazine, IEEE, 47(12):94–101, 2009.

[32] G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels. Dynamo: Ama- zon’s highly available key-value store. SIGOPS Oper. Syst. Rev., 41(6):205– 220, Oct. 2007.

[33] R. Delaviz, N. Andrade, and J. Pouwelse. Improving accuracy and coverage in an internet-deployed reputation mechanism. In Peer-to-Peer Computing (P2P), 2010 IEEE Tenth International Conference on, pages 1 –9, aug. 2010.

[34] R. Delaviz, J. Pouwelse, and D. Epema. Targeted and scalable information dissemination in a distributed reputation mechanism. In Proceedings of the Seventh ACM Workshop on Scalable Trusted Computing, STC ’12, pages 55– 66, New York, NY, USA, 2012. ACM.

[35] A. Demers, D. Greene, C. Hauser, W. Irish, J. Larson, S. Shenker, H. Stur- gis, D. Swinehart, and D. Terry. Epidemic algorithms for replicated database maintenance. In Proceedings of the sixth annual ACM Symposium on Princi- ples of distributed computing, PODC ’87, pages 1–12, New York, NY, USA, 1987. ACM.

[36] A. Demiriz. Enhancing product recommender systems on sparse binary data. Data Mining and Knowledge Discovery, 9(2):147–170, 2004.

[37] J. Douceur. The sybil attack. In P. Druschel, F. Kaashoek, and A. Rowstron, editors, Peer-to-Peer Systems, volume 2429 of Lecture Notes in Computer Science, pages 251–260. Springer Berlin Heidelberg, 2002.

[38] M. J. Freedman, Y. Ishai, B. Pinkas, and O. Reingold. Keyword search and oblivious pseudorandom functions. In Second international conference on Theory of Cryptography, TCC’05, pages 303–324, Berlin, Heidelberg, 2005. Springer-Verlag.

[39] M. J. Freedman, K. Nissim, and B. Pinkas. Eﬃcient private matching and set intersection. In EUROCRYPT ’04, volume 3027 of LNCS, pages 1–19. Springer, 2004. 118 Bibliography

[40] K. Goldberg, T. Roeder, D. Gupta, and C. Perkins. Eigentaste: A constant time collaborative filtering algorithm. Information Retrieval, 4(2):133–151, 2001. [41] G. Halkes and J. Pouwelse. Udp nat and firewall puncturing in the wild. In Proceedings of the 10th international IFIP TC 6 conference on Networking - Volume Part II, NETWORKING’11, pages 1–12, Berlin, Heidelberg, 2011. Springer-Verlag. [42] C. Hazay and Y. Lindell. Efficient protocols for set intersection and pattern matching with security against malicious and covert adversaries. In 5th conference on Theory of cryptography, TCC’08, pages 155–175, Berlin, Heidelberg, 2008. Springer-Verlag. [43] G. Hazel and A. Norberg. Extension for Peers to Send Metadata Files. http: //www.bittorrent.org/beps/bep_0009.html. [44] D. C. Howe and H. Nissenbaum. Trackmenot: Resisting surveillance in web search. Lessons from the Identity Trail: Anonymity, Privacy, and Identity in a Networked Society, pages 417–436, 2009. [45] H.-Y. Huang, P.-E. Luo, M. Li, D. Li, X. Li, W. Shu, and M.-Y. Wu. Perfor- mance evaluation of suvnet with real-time traffic data. Vehicular Technology, IEEE Transactions on, 56(6):3381 –3396, nov. 2007. [46] T. Isdal, M. Piatek, A. Krishnamurthy, and T. Anderson. Privacy-preserving p2p data sharing with oneswarm. In Proceedings of the ACM SIGCOMM 2010 conference, SIGCOMM ’10, pages 111–122, New York, NY, USA, 2010. ACM. [47] M. Jelasity, S. Voulgaris, R. Guerraoui, A.-M. Kermarrec, and M. van Steen. Gossip-based peer sampling. ACM Trans. Comput. Syst., 25(3), Aug. 2007. [48] B. Kanefsky, N. Barlow, and V. Gulick. Can Distributed Volunteers Accom- plish Massive Data Analysis Tasks? In Lunar and Planetary Institute Science Conference Abstracts, volume 32, page 1272, 2001. [49] D. Kügler. An analysis of gnunet and the implications for anonymous, censorship-resistant networks. In Privacy Enhancing Technologies, volume 2760 of Lecture Notes in Computer Science, pages 161–176. Springer Berlin Heidelberg, 2003. [50] A. Kiayias and A. Mitrofanova. Testing disjointness of private datasets. In Financial Cryptography, volume 3570 of LNCS, pages 109–124. Springer, 2005. [51] L. Kissner and D. Song. Privacy-preserving set operations. In in Advances in Cryptology - CRYPTO 2005, LNCS, pages 241–257. Springer, 2005. [52] M. Konrath, M. Barcellos, and R. Mansilha. Attacking a swarm with a band of liars: evaluating the impact of attacks on bittorrent. In Peer-to-Peer Com- puting, 2007. P2P 2007. Seventh IEEE International Conference on, pages 37–44, Sept 2007. Bibliography 119

[53] R. S. R. Ku. The creative destruction of copyright: Napster and the new eco- nomics of digital technology. The University of Chicago Law Review, 69(1):pp. 263–324, 2002.

[54] Y. Kulbak, D. Bickson, et al. The emule protocol speciﬁcation. 2005.

[55] L. Lamport. Time, clocks, and the ordering of events in a distributed system. Commun. ACM, 21(7):558–565, July 1978.

[56] U. Lee, E. Magistretti, M. Gerla, P. Bellavista, and A. Corradi. Dissemination and harvesting of urban data using vehicular sensing platforms. Technical report, 2007.

[57] A. Legout, G. Urvoy-Keller, and P. Michiardi. Rarest ﬁrst and choke algorithms are enough. In Proceedings of the 6th ACM SIGCOMM Conference on Internet Measurement, IMC ’06, pages 203–216, New York, NY, USA, 2006. ACM.

[58] J. Liang, R. Kumar, and K. Ross. Understanding kazaa. Misc, 2004,.

[59] T. Locher, P. Moor, S. Schmid, and R. Wattenhofer. Free riding in bittorrent is cheap. In In HotNets, 2006.

[60] E. K. Lua. Securing peer-to-peer overlay networks from sybil attack. In Com- munications and Information Technologies, 2007. ISCIT ’07. International Symposium on, pages 1213–1218, Oct 2007.

[61] L. Mamykina, B. Manoim, M. Mittal, G. Hripcsak, and B. Hartmann. Design lessons from the fastest Q&A site in the west. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’11, pages 2857– 2866, New York, NY, USA, 2011. ACM.

[62] N. Margolin and B. Levine. Quantifying resistance to the sybil attack. In G. Tsudik, editor, Financial Cryptography and Data Security, volume 5143 of Lecture Notes in Computer Science, pages 1–15. Springer Berlin Heidelberg, 2008.

[63] P. Maymounkov and D. Mazis. Kademlia: A peer-to-peer information system based on the xor metric. In Peer-to-Peer Systems, volume 2429, pages 53–65. Springer Berlin Heidelberg, 2002.

[64] J. McAuley and J. Leskovec. Learning to discover social circles in ego networks. In Advances in Neural Information Processing Systems 25, pages 548–556, 2012.

[65] G. Mega, A. Montresor, and G. Picco. Eﬃcient dissemination in decentralized social networks. In Peer-to-Peer Computing (P2P), 2011 IEEE International Conference on, pages 338–347, 2011. 120 Bibliography

[66] R. Merkle. A digital signature based on a conventional encryption function. In C. Pomerance, editor, Advances in Cryptology — CRYPTO ’87, volume 293 of Lecture Notes in Computer Science, pages 369–378. Springer Berlin Heidelberg, 1988.

[67] M. Meulpolder, J. Pouwelse, D. H. J. Epema, and H. Sips. Bartercast: A practical approach to prevent lazy freeriding in p2p networks. In Parallel Distributed Processing, 2009. IPDPS 2009. IEEE International Symposium on, pages 1–8, May 2009.

[68] R. Morselli, B. Bhattacharjee, J. Katz, and M. A. Marsh. Keychains: A decentralized public-key infrastructure. 2006.

[69] P. Mukherjee, C. Leng, and A. Schurr. Piki - a peer-to-peer based wiki engine. In Peer-to-Peer Computing , 2008. P2P ’08. Eighth International Conference on, pages 185 –186, sept. 2008.

[70] A. Muller, N. Evans, C. Grothoﬀ, and S. Kamkar. Autonomous nat traversal. In Peer-to-Peer Computing (P2P), 2010 IEEE Tenth International Conference on, pages 1 –4, aug. 2010.

[71] S. Nilizadeh, S. Jahid, P. Mittal, N. Borisov, and A. Kapadia. Cachet: A decentralized architecture for privacy preserving social networking with caching. In Proceedings of the 8th International Conference on Emerging Networking Experiments and Technologies, CoNEXT ’12, pages 337–348, New York, NY, USA, 2012. ACM.

[72] Z. Ou, E. Harjula, and M. Ylianttila. Eﬀects of diﬀerent churn models on the performance of structured peer-to-peer networks. In Personal, Indoor and Mobile Radio Communications, 2009 IEEE 20th International Symposium on, pages 2856–2860, 2009.

[73] P. Paillier. Public-Key Cryptosystems Based on Composite Degree Residuos- ity Classes. In Advances in Cryptology — EUROCRYPT ’99, volume 1592 of LNCS, pages 223–238. Springer, May 2-6, 1999.

[74] J. Pouwelse, P. Garbacki, D. Epema, and H. Sips. The bittorrent p2p ﬁle- sharing system: Measurements and analysis. In M. Castro and R. van Renesse, editors, Peer-to-Peer Systems IV, volume 3640 of Lecture Notes in Computer Science, pages 205–216. Springer Berlin Heidelberg, 2005.

[75] R. Priedhorsky, J. Chen, S. T. K. Lam, K. Panciera, L. Terveen, and J. Riedl. Creating, destroying, and restoring value in wikipedia. Proceedings of the 2007 international ACM conference on Conference on supporting group work - GROUP ’07, page 259, 2007.

[76] C. Rahhal, H. Skaf-Molli, and P. Molli. SWOOKI: A Peer-to-peer Semantic Wiki. Rapport de recherche RR-6468, INRIA, 2008. Bibliography 121

[77] R. Rahman, T. Vinkó, D. Hales, J. Pouwelse, and H. Sips. Design space analysis for modeling incentives in distributed systems. SIGCOMM Comput. Commun. Rev., 41(4):182–193, Aug. 2011.

[78] M. K. Reiter and A. D. Rubin. Anonymous web transactions with crowds. Commun. ACM, 42(2):32–48, Feb. 1999.

[79] RetroShare. Retroshare aims to be a private f2f social network, May 2010. available at http://sourceforge.net/blog/retroshare-aims-to- be-a-private-f2f-social-network/.

[80] RetroShare. Turtle router - retroshare, Oktober 2010. available at http: //retroshare.sourceforge.net/wiki/index.php/Turtle_Router.

[81] S. Rhea, D. Geels, T. Roscoe, and J. Kubiatowicz. Handling churn in a dht. In In Proceedings of the USENIX Annual Technical Conference, 2004.

[82] M. Ripeanu. Peer-to-peer architecture case study: Gnutella network. In Peer-to-Peer Computing, 2001. Proceedings. First International Conference on, pages 99–100, 2001.

[83] M. Ripeanu, A. Iamnitchi, and I. Foster. Mapping the gnutella network. IEEE Internet Computing, 6(1):50–57, Jan. 2002.

[84] R. L. Rivest, A. Shamir, and L. M. Adleman. A method for obtaining digital signature and public-key cryptosystems. Communications of the Association for Computing Machinery, 21(2):120–126, 1978.

[85] H. Rowaihy, W. Enck, P. McDaniel, and T. La Porta. Limiting sybil attacks in structured p2p networks. In INFOCOM 2007. 26th IEEE International Con- ference on Computer Communications. IEEE, pages 2596–2600, May 2007.

[86] A. Rowstron and P. Druschel. Pastry: Scalable, decentralized object location, and routing for large-scale peer-to-peer systems. In Middleware 2001, volume 2218, pages 329–350. Springer Berlin Heidelberg, 2001.

[87] C. Shannon, D. Moore, and k. claﬀy. Characteristics of fragmented ip traﬃc on internet links. In Proceedings of the 1st ACM SIGCOMM Workshop on Internet Measurement, IMW ’01, pages 83–97, New York, NY, USA, 2001. ACM.

[88] A. Singh, T. wan “johnny Ngan, P. Druschel, and D. S. Wallach. Eclipse attacks on overlay networks: Threats and defenses. In In IEEE INFOCOM, 2006.

[89] M. Steiner, W. Eﬀelsberg, and T. En-najjary. Load reduction in the kad peer- to-peer system. In In Fifth International Workshop on Databases, Information Systems and Peer-to-Peer Computing (DBISP2P, 2007. 122 Bibliography

[90] I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H. Balakrishnan. Chord: A scalable peer-to-peer lookup service for internet applications. SIGCOMM ’01, pages 149–160, New York, NY, USA, 2001. ACM.

[91] M. Stuart. P2P-Next. http://www.dcia.info/activities/p2pmslv2009/ 1-7%20P2PMS%20P2P-Next.pdf.

[92] D. Stutzbach and R. Rejaie. Understanding churn in peer-to-peer networks. In Proceedings of the 6th ACM SIGCOMM Conference on Internet Measurement, IMC ’06, pages 189–202, New York, NY, USA, 2006. ACM.

[93] J. Sun, X. Zhu, and Y. Fang. A privacy-preserving scheme for online social networks with eﬃcient revocation. In INFOCOM, 2010 Proceedings IEEE, pages 1–9, 2010.

[94] R. Torres, A. Finamore, J. R. Kim, M. Mellia, M. Munafo, and S. Rao. Dis- secting video server selection strategies in the youtube cdn. In Distributed Computing Systems (ICDCS), 2011 31st International Conference on, pages 248–257, June 2011.

[95] F. Viegas, M. Wattenberg, J. Kriss, and F. Ham. Talk Before You Type: Co- ordination in Wikipedia. 2007 40th Annual Hawaii International Conference on System Sciences (HICSS’07), pages 78–78, Jan. 2007.

[96] B. Viswanath, A. Mislove, M. Cha, and K. P. Gummadi. On the evolution of user interaction in facebook. In Proceedings of the 2nd ACM SIGCOMM Workshop on Social Networks (WOSN’09), August 2009.

[97] S. Voulgaris and M. Steen. Epidemic-style management of semantic overlays for content-based searching. In Euro-Par 2005 Parallel Processing, volume 3648, pages 1143–1152. Springer Berlin Heidelberg, 2005.

[98] S. Weiss, P. Urso, and P. Molli. Wooki: A P2P Wiki-Based Collaborative Writing Tool. In B. Benatallah, F. Casati, D. Georgakopoulos, C. Bartolini, W. Sadiq, and C. Godart, editors, Web Information Systems Engineering WISE 2007, volume 4831 of Lecture Notes in Computer Science, pages 503– 512. Springer Berlin Heidelberg, 2007.

[99] R.-Y. Xiao. Survey on anonymity in unstructured peer-to-peer systems. Jour- nal of Computer Science and Technology, 23:660–671, 2008.

[100] H. Yu, M. Kaminsky, P. B. Gibbons, and A. Flaxman. Sybilguard: Defending against sybil attacks via social networks. In Proceedings of the 2006 Confer- ence on Applications, Technologies, Architectures, and Protocols for Computer Communications, SIGCOMM ’06, pages 267–278, New York, NY, USA, 2006. ACM.

[101] N. Zeilemaker, M. Capota, and J. Pouwelse. Open2edit: A peer-to-peer platform for collaboration. In IFIP Networking Conference, pages 1–9, May 2013. Bibliography 123

[102] N. Zeilemaker, M. Capotă, A. Bakker, and J. Pouwelse. Tribler: P2p media search and sharing. In Proceedings of the 19th ACM international conference on Multimedia, MM ’11, pages 739–742, New York, NY, USA, 2011. ACM. [103] N. Zeilemaker, Z. Erkin, P. Palmieri, and J. Pouwelse. Building a privacy- preserving semantic overlay for peer-to-peer networks. In Proceedings of the IEEE International Workshop on Information Forensics and Security (WIFS 2013), Guangzhou, China, November 2013. [104] N. Zeilemaker and J. Pouwelse. ReClaim: a Privacy-Preserving Decentralized Social Network. In Proceedings of the USENIX Workshop on Free and Open Communications on the Internet, 2014. [105] N. Zeilemaker, J. Pouwelse, and H. Sips. 4P: Performant Private Peer-to- Peer File Sharing. In Proceedings of the IEEE International Conference on Peer-to-Peer Computing (P2P), 2014. [106] N. Zeilemaker, B. Schoon, and J. Pouwelse. Dispersy bundle synchronization. Technical report, TU Delft, 2013. [Online] http://www.pds.ewi.tudelft. nl/fileadmin/pds/reports/2013/PDS-2013-002.pdf. [107] N. Zeilemaker, B. Schoon, and J. Pouwelse. Large-scale message synchronization in challenged networks. In Proceedings of the 29th Annual ACM Symposium on Applied Computing (SAC), Gyeongju, Korea, March 2014.

Summary

Privacy and Cooperation in Peer-to-Peer Systems P2P networks employ the resources available at peers to reduce the load at, or eliminate the need for a server. In order to achieve these goals, protocols are implemented which aim to allow peers to collaborate efficiently. However, these same protocols can make peers an easy target, as their willingness to collaborate can be susceptible to attackers. While a considerable body of research exists on reducing the vulnerability of peers to attackers, most only consider a file sharing or backup solution context. In this thesis, we look at other forms of collaboration wherein a P2P system can overcome some of the other downsides of centralized solutions, e.g. the privacy concerns. In Chapter2, we introduce Tribler our research testbed which allows us evaluate the performance of new protocols in the wild. Tribler is an open-source BitTorrent client, which was funded by multiple European research grants. Besides allowing its users to download files, Tribler adds remote search, channels, and video-on-demand. In Chapter3, we design, implement and evaluate a platform that allows peers to sychronize their databases in a challenged network. E.g. our platform can successfully synchronize when confronted with intermittent network connectivity, NAT- firewalls, and substantial churn. Peers synchronize using Bloom filters, which allow them to express the current state of their database in a single 1500 byte UDP message. After receiving such a synchronization request, a peer will reply with the rows not present in the Bloom filter. Building upon the platform introduced in the previous chapter, we introduce a Open2Edit in Chapter4. Open2Edit aims to improve collaboration between peers in a decentralized manner. It builds upon lessons learned from centralized websites such as Wikipedia and StackExchange, to in contrast to related works, not introduce another decentralized Wikipedia, but a flexible collaboration platform able to cope with a wide range of usage scenarios. By combining Open2Edit with Tribler, we implement and evaluate a collaborative Youtube-like media sharing system in which users can create channels consisting of content of their liking. Others peers can mark a channel as a favorite, and by combining those votes the most popular channels can be discovered. During the integration into Tribler over 100,000 votes were cast by users. Then in Chapter5, we leverage existing solutions to the Private Set Intersection problem to construct a semantic network in a privacy preserving manner. A semantic overlay yields substantial performance improvements while searching for information in a P2P system, as relevant information is close to the peer searching for it. However the evaluated PSI algorithms have never been evaluated/compared with eachother to see their performance with respect to CPU time, bandwidth, and their influence in determining overlap.

125 126 Summary

Leveraging one of the evaluated PSI algorithms allows us to implement a privacy preserving file sharing P2P system in Chapter6. In contrast to related work, 4P does not require flooding in order to search for files. It leverages the semantic overlay it builds upon to reduce the amount of messages send per query. We apply common anonymization techniques such a path obfuscation, probabilistic query forwarding, and encrypted links. By modifying the PSI algorithm slightly, we incorporate a mechanism which allows two peers to negotiate a session-key to be used to encrypt the connection. This session-key cannot be observed, or modified by a man-in-the- middle as it is encrypted with the key of the initiating peer. During the evaluation, we compare 4P against five other techniques and show that it requires substantially less messages (300 instead of 10,000). Finally, in Chapter7, we design and evaluate a fully decentralized social network which employs the same PSI algorithm to locate friends currently online. After establishing a direct connection to those friends, or to peers which have at least one friend in common, peer synchronize messages using slightly modified approach introduced in Chapter3. Each update is encrypted with the public key of the final destination, and is replicated to all of the friends of the destination. This results in a co-operative social network in which friends store updates destined for their friends and vice versa. Alliviating the system from the need of replication management strategies, in order to provide peers with enough replicas. Samenvatting

P2P-netwerken maken gebruik van de beschikbare middelen van peers voor het redu- ceren, of elimineren van de noodzaak van een server. Om dit mogelijk te maken zijn protocollen ontwikkeld die tot doel hebben om peers efficient samen te werken. Toch kunnen dezelfde protocollen peers een gemakkelijk doelwit maken, omdat hun be- reidheid om samen te werken gevoelig voor aanvallen is. Er bestaat een aanzienlijke hoeveelheid onderzoek naar het verminderen van kwetsbaarheden van peers, echter de meeste onderzoeken zijn gedaan in de context van file-sharing of in een back-up oplossingen. In dit proefschrift kijken we naar andere vormen van samenwerking in een P2P-systeem waarbij nadelen van gecentraliseerde oplossingen, bijvoorbeeld het gebrek aan privacy, worden verminderd. In Hoofdstuk2 introduceren we Tribler, een programma dat in ons onderzoek gebruikt is als testbed en ons in staat stelt de prestaties van nieuwe protocollen in het wild te evalueren. Tribler is een open-source BitTorrent-client, die is ontwikkeld met behulp van meerdere Europese projecten. Naast het downloaden van BitTorrent bestanden, voegt Tribler decentraal zoeken, kanalen en video-on-demand toe. In Hoofdstuk3 ontwerpen, implementeren en evalueren wij een platform waarmee peers hun databases kunnen sychronizeren in een netwerk. Met ons platform kunnen peers successvol synchroniseren wanneer ze worden blootgesteld aan wegvallende netwerkverbindingen, NAT-firewalls, en substantiele churn. Peers synchroniseren met elkaar mbv Bloom filters, die hun in staat stellen om de huidige stand van hun database uit te drukken in een 1500 byte UDP bericht. Na ontvangst van een dergelijk synchronisatie verzoek, zal een peer antwoorden met de rijen niet in de Bloom filter zitten. Voortbouwend op het in het vorige hoofdstuk geintroduceerde platform, introduceren we Open2Edit in Hoofdstuk4. Open2Edit is bedoeld om de samenwerking tussen peers in een gedecentraliseerde manier te verbeteren. Het bouwt voort op ideeen gebruikt door gecentraliseerde websites zoals Wikipedia en StackExchange. Echter in tegenstelling tot vergelijkbare papers, implementeren we niet nog een de- centrale versie van Wikipedia, maar een flexibel samenwerkingsplatform dat in staat is om in een breed scala aan gebruiksscenarios toegepast te worden. Door Open2Edit combineren met Tribler, implementeren en evalueren we een Youtube-achtig media sharing systeem waarin gebruikers kanalen kunnen aanmaken, bestaande uit de in- houd die zij zelf uit kiezen. Andere peers kunnen een kanaal markeren als favoriet, en door het combineren van deze stemmen ontdekken ze de populairste kanalen. Tijdens de integratie in Tribler werden meer dan 100.000 stemmen uitgebracht door gebruikers. Vervolgens benutten we in Hoofdstuk5 bestaande oplossingen voor het Private Set Intersection probleem om een semantisch netwerk te bouwen met behoud van privacy. Een semantisch netwerk levert aanzienlijke prestatieverbeteringen op tij-

127 128 Samenvatting dens het zoeken naar informatie in een P2P-systeem, omdat relevante informatie zich dichter bij de peer die de zoekopdracht uitvoert bevind. Echter de zijn de prestaties van de PSI algoritmen zijn nooit vergeleken met elkaar met betrekking tot processortijd, bandbreedte, en hun invloed bij het bepalen overlap. Dat doen wij in dit hoofdstuk. In Hoofdstuk6, gebruikmakend van een van de geevalueerde PSI algoritmen, implementeren we een ﬁle sharing system dat de privacy van zijn gebruikers waar- borgt. In tegenstelling tot vergelijkbare papers, gebruikt 4P geen ﬂooding om het P2P systeem te doorzoeken. Het maakt gebruik van de semantisch netwwerk, waar- door 4P het aantal berichten wat wordt verzonden per query kan verminderen. In dit hoofdstuk passen wij anonimisering technieken toe om het zoek pad te verhullen. Daarnaast sturen peers probabilistich berichten door, en gebruiken zij versleutelde verbindingen. Door een kleine wijziging in het PSI algoritme, voegen we een mecha- nisme toe waarmee twee peers een sessie-sleutel kunnen afspreken die wordt gebruikt om de verbinding te versleutelen. Deze sessie-sleutel kan niet worden onderschept, of aangepast door een man-in-the-middle omdat hij versleuteld is met de sleutel van de initierende peer. Tijdens de evaluatie vergelijken we 4P met andere technieken en laten zien dat het aanzienlijk minder berichten (300 in plaats van 10.000) vereist per zoek opdracht. Tenslotte, in Hoofdstuk7, ontwerpen en evalueren wij een volledig gedecentra- liseerd sociaal netwerk dat hetzelfde PSI algoritme gebruikt om vrienden te vinden die nu online zijn. Na het opbouwen van een directe verbinding met die vrienden, of peers die ten minste een vriend gemeen hebben, synchronizeren deze peers berichten met behulp van een licht aangepast algoritme zoals geintroduceerd in Hoofdstuk3. Elke update wordt versleuteld met de publieke sleutel van de uiteindelijke bestemming, en wordt gerepliceert naar alle van de vrienden van de bestemming. Dit resulteert in een cooperatief sociaal netwerk waarin vrienden berichten opslaan be- stemt voor hun vrienden en vice versa. Dit heeft als voordeel dat het systeem geen replicatie strategieen nodig heeft, omdat hun vrienden voldoende replicas bieden. Curriculum Vitæ

Nicolaas Simon Marinus Zeilemaker

07-06-1985 Born in Harmelen, The Netherlands.

Education 2002–2006 BSc Computer Science University of Applied Sciences Utrecht, Utrecht, The Netherlands

2006–2010 MSc Computer Science Master thesis title: Improving P2P keyword search by combining .torrent metadata and user preference in a semantic overlay Delft University of Technology, Delft, The Netherlands

2010–2014 PhD Computer Science Delft University of Technology, Delft, The Netherlands

Work Experience 2004–2009 Software Developer Logic8 BV, worked on improving and extending an existing Java web application.

2010–2014 PhD Student Parallel and Distributed Systems group, Delft University of Technology, Delft, The Netherlands

2014–present Big Data Hacker GoDataDriven BV, Hilversum, The Netherlands.

129

List of Publications

Publications 10. Niels Zeilemaker and Johan Pouwelse. ReClaim: a Privacy-Preserving Decentral- ized Social Network. In Proceedings of the USENIX Workshop on Free and Open Communications on the Internet (FOCI), 2014, San Diego, USA. 9. Niels Zeilemaker, Johan Pouwelse, and Henk Sips. 4P: Performant Private Peer-to- Peer File Sharing. In Proceedings of the IEEE International Conference on Peer-to- Peer Computing (P2P), 2014, London, UK. 8. Niels Zeilemaker and Johan Pouwelse. 100 Million DHT replies. In Proceedings of the IEEE International Conference on Peer-to-Peer Computing (P2P), 2014, London, UK. 7. Niels Zeilemaker, Boudewijn Schoon, and Johan Pouwelse. Large-scale message synchronization in challenged networks. In Proceedings of the 29th An nual ACM Symposium on Applied Computing (SAC), 2014, Gyeongju, Korea. 6. Niels Zeilemaker, Zekeriya Erkin, Paolo Palmieri, and Johan Pouwelse. Building a privacy-preserving semantic overlay for peer-to-peer networks. In Pro ceedings of the IEEE International Workshop on Information Forensics and Security (WIFS), 2013, Guangzhou, China. 5. Niels Zeilemaker, Mihai Capotă, and Johan Pouwelse. Open2edit: A peer-to-peer platform for collaboration. In IFIP Networking Conference, 2013, Brooklyn, New York, USA. 4. Rahim Delaviz, Niels Zeilemaker, Johan Pouwelse, and Dick Epema. A network science perspective of a distributed reputation mechanism. In IFIP Networking Con- ference, 2013, Brooklyn, New York, USA. 3. Niels Zeilemaker and Johan Pouwelse. Open Source Column: Tribler: P2P Search, Share and Stream. ACM SIGMultimedia Records Volume 4 Issue 1, March 2012. 2. Niels Zeilemaker, Mihai Capotă, Arno Bakker, and Johan Pouwelse. Tribler: P2P Media Search and Sharing. In Proceedings of the 19th ACM international conference on Multimedia (MM), 2011, Scottsdale, Arizona, USA 1. Niels Zeilemaker, Mihai Capotă, Arno Bakker, and Johan Pouwelse. Tribler: Search and Stream. In Proceedings of IEEE International Conference on Peer-to-Peer Com- puting (P2P), 2011, Kyoto, Japan.

Technical Reports 1. Niels Zeilemaker, Boudewijn Schoon, and Johan Pouwelse. Dispersy Bundle Syn- chronization. Technical Report PDH-2013-002, Delft University of Technology.

131