Overview

• Web 15-441 • Protocol interactions 15-441 Computer Networking 15-641 • Caching • Cookies • Consistent hashing Lecture 16-17: Delivering Content • Peer-to-peer Part 1 • CDN Peter Steenkiste • Video

Fall 2013 www.cs.cmu.edu/~prs/15-441-F13

2

Web history Web history (cont)

• 1945: Vannevar Bush, “As we may think”, Atlantic • 1992 Monthly, July, 1945. • NCSA server released • 26 WWW servers worldwide • Describes the idea of a distributed hypertext system. • 1993 • A “memex” that mimics the “web of trails” in our minds. • Marc Andreessen releases first version of NCSA Mosaic Mosaic • 1989: Tim Berners-Lee (CERN) writes internal proposal version released for (Windows, Mac, Unix). to develop a distributed hypertext system • Web (port 80) traffic at 1% of NSFNET backbone traffic. • Connects “a web of notes with links”. • Over 200 WWW servers worldwide. • Intended to help CERN physicists in large projects share and • 1994 manage information • Andreessen and colleagues leave NCSA to form "Mosaic • 1990: Tim BL writes graphical browser for Next Communications Corp" (Netscape). machines.

3 4

1 Typical Workload (Web Pages) HTTP 0.9/1.0

• Multiple (typically small) objects per page • One request/response per TCP connection • File sizes •Lots of small objects • Simple to implement • Heavy-tailed means & TCP • Disadvantages • Pareto distribution for tail • 3-way handshake • Multiple connection setups  three-way handshake each time • Lognormal for body of distribution • Lots of slow starts • Several extra round trips added to transfer • Embedded references • Extra connection state • Multiple slow starts • Number of embedded objects also Pareto -k Pr(X>x) = (x/xm) • This plays havoc with performance. Why? • Solutions?

5 6

Must Consider Network Layer - Single Transfer Example Example: TCP

1: TCP connections need to be set up Client Server 0 RTT SYN • “Three Way Handshake”: Client opens TCP SYN 1 RTT connection ACK Client sends HTTP request DAT Client Server ACK Server reads from for HTML DAT disk SYN (Synchronize) 2 RTT FIN ACK Client parses HTML FIN Client opens TCP ACK SYN/ACK (Synchronize + Acknowledgement) connection SYN SYN ACK 3 RTT ACK Client sends HTTP request DAT …Data… for image Server reads from ACK disk 4 RTT 2: TCP transfers start slowly and then ramp up the DAT bandwidth used – bandwidth is variable Image begins to arrive

7 Lecture 3: Applications 8

2 Performance Issues Netscape Solution

• Short transfers are hard on TCP • Mosaic (original popular Web browser) fetched one object • Stuck in slow start at a time! • Loss recovery is poor when windows are small • Netscape uses multiple concurrent connections to improve • Lots of extra connections response time • Increases server state/processing • Different parts of Web page arrive independently • Servers also hang on to connection state after the • Can grab more of the network bandwidth than other connection is closed users • Why must server keep these? • Does not necessarily improve response time • Tends to be an order of magnitude greater than # of active connections, why? • TCP loss recovery ends up being timeout dominated because windows are small

9 10

Persistent Connection Solution Persistent Connection Solution

• Multiplex multiple transfers onto one TCP Server connection Client • Also allow pipelined transfers 0 RTT DAT • How to identify requests/responses Client sends HTTP request ACK Server reads from for HTML DAT disk • Delimiter  Server must examine response for delimiter string 1 RTT • Content-length and delimiter  Must know size of transfer in ACK advance Client parses HTML DAT Server reads from Client sends HTTP request • Block-based transmission  send in multiple length delimited ACK DAT disk blocks for image • Store-and-forward  wait for entire response and then use 2 RTT content-length Image begins to arrive • Solution  use existing methods and close connection otherwise

11 12

3 Other Problems Web Proxy Caches

• Serialized transmission • Much of the useful information in first few bytes • User configures browser: Web origin • May be better to get the 1st 1/4 of all images than one complete accesses via cache server image (e.g., progressive JPEG) • Browser sends all HTTP Proxy • Can “packetize” transfer over TCP, e.g., range requests requests to cache server • Application specific solution to transport protocol • Object in cache: cache client problems. :( returns object • Could fix TCP so it works well with multiple • Else cache requests object simultaneous connections from origin server, then • More difficult to deploy returns object to client • Persistent connections can significantly increase state on the server client • Allow server to close HTTP session origin server • Client can no longer send requests

13 14

No Caching Example (1) No Caching Example (2)

Assumptions Possible solution • Average object size = 100,000 bits origin • Increase bandwidth of access link origin • Avg. request rate from institution’s servers to, say, 10 Mbps servers browser to origin servers = 15/sec • Often a costly upgrade public public • Delay from institutional router to Internet Internet any origin server and back to router Consequences = 2 sec • Utilization on LAN = 15% Consequences • Utilization on access link = 15% • Utilization on LAN = 15% 1.5 Mbps • Total delay = Internet delay + access 10 Mbps access link • Utilization on access link = 100% delay + LAN delay access link • Total delay = Internet delay + access institutional = 2 sec + msecs + msecs institutional delay + LAN delay network network 10 Mbps LAN 10 Mbps LAN = 2 sec + minutes + milliseconds

15 16

4 With Caching Example (3) HTTP Caching

Install cache origin • Clients often cache documents • Suppose hit rate is .4 servers • Challenge: update of documents Consequence • 40% requests will be satisfied almost public • If-Modified-Since requests to check immediately (say 10 msec) Internet • HTTP 0.9/1.0 used just date • 60% requests satisfied by origin server • HTTP 1.1 has an opaque “entity tag” (could be a file signature, • Utilization of access link reduced to 60%, etc.) as well resulting in negligible delays • Weighted average of delays 1.5 Mbps • When/how often should the original be checked for = .6*2 sec + .4*10msecs < 1.3 secs access link changes? institutional • Check every time? network 10 Mbps LAN • Check each session? Day? Etc? • Use Expires header • If no Expires, often use Last-Modified as estimate institutional cache

17 18

Example Cache Check Request Example Cache Check Response

GET / HTTP/1.1 HTTP/1.1 304 Not Modified Accept: */* Date: Tue, 27 Mar 2001 03:50:51 GMT Accept-Language: en-us Server: Apache/1.3.14 (Unix) (Red-Hat/Linux) mod_ssl/2.7.1 Accept-Encoding: gzip, deflate OpenSSL/0.9.5a DAV/1.0.2 PHP/4.0.1pl2 mod_perl/1.24 If-Modified-Since: Mon, 29 Jan 2001 17:54:18 GMT Connection: Keep-Alive If-None-Match: "7a11f-10ed-3a75ae4a" Keep-Alive: timeout=15, max=100 User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT ETag: "7a11f-10ed-3a75ae4a" 5.0) Host: www.intel-iris.net Connection: Keep-Alive

19 20

5 Problems Caching Proxies – Sources for Misses

• Well over 50% of all HTTP objects are not cacheable • Capacity • Why? • How large a cache is necessary or equivalent to infinite • Major exception? • On disk vs. in memory  typically on disk • This problem will not go away • Compulsory • Dynamic data  stock prices, scores, web cams • First time access to document • CGI scripts  results based on passed parameters • Non-cacheable documents • Other less obvious examples • CGI-scripts • SSL  encrypted data is not cacheable • Personalized documents (cookies, etc) • Most web clients don’t handle mixed pages well many generic • Encrypted data (SSL) objects transferred with SSL • Consistency • Cookies  results may be based on past data • Document has been updated/expired before reuse • Hit metering  owner wants to measure # of hits for revenue, etc. • Conflict • What will be the end result? • No such misses

21 22

Cookies: Keeping “state” Cookies: Keeping “State”

client Amazon server

Many major Web sites use Cookie file usual http request msg server cookies usual http response + creates ID Four components: Example: ebay: 8734 Set-cookie: 1678 1678 for user 1) Cookie header line in the • Susan accesses Internet HTTP response message always from the same PC Cookie file usual http request msg 2) Cookie header line in HTTP • She visits a specific e- amazon: 1678 cookie- request message cookie: 1678 commerce site for the first time ebay: 8734 specific 3) Cookie file kept on user’s • When initial HTTP requests usual http response msg action host and managed by user’s arrives at the site, the site one week later: browser creates a unique ID and creates 4) Back-end database at Web an entry in a backend database usual http request msg Cookie file cookie- site for that ID cookie: 1678 amazon: 1678 specific ebay: 8734 usual http response msg action

23 24

6 Overview Distributing Load across Servers

• Web • Given document XYZ, we need to choose a • Consistent hashing server to use • Peer-to-peer • Suppose we use simple hashing: modulo of a • CDN hash of the name of the document • Video • Number servers from 1…n • Place document XYZ on server (XYZ mod n) • What happens when a servers fails? n  n-1 • Same if different people have different measures of n • Why might this be bad?

25 26

Consistent Hash: Goals Consistent Hash – Example

• Construction • “view” = subset of all hash buckets that are visible 0 • Assign each of C hash buckets to 14 • Correspond to a real server random points on mod 2n circle, where, hash key size = n. Bucket • Desired features 12 4 • Map object to random position on • Smoothness – little impact on hash bucket unit interval contents when buckets are added/removed • Hash of object = closest bucket 8 • Spread – small set of hash buckets that may • Monotone  addition of bucket does not cause hold an object regardless of views movement between existing buckets • Load – all hash buckets have a similar number • Spread & Load  small set of buckets that lie of objects assigned to them near object • Balance  no bucket is responsible for large number of objects 27 28

7 Consistent Hashing Consistent Hashing

• Main idea: • The consistent hash function assigns each node and key • map both keys and nodes to the same (metric) identifier space an m-bit identifier using SHA-1 as a base hash function • find a “rule” how to assign keys to nodes • Node identifier: SHA-1 hash of IP address

• Key identifier: SHA-1 hash of key Ring is one option.

29 30

Identifiers Consistent Hashing Example

Rule: A key is stored at its successor: node with next higher or equal ID • m bit identifier space for both keys and nodes

• Key identifier: SHA-1(key) IP=“198.10.10.1” 0 K5

SHA-1 Key=“LetItBe” ID=60 N123 K20

• Node identifier: SHA-1(IP address) IP=“198.10.10.1” SHA-1 ID=123 Circular 7-bit K101 ID space N32

•How to map key IDs to node IDs? N90 Key=“LetItBe” K60

31 32

8 Consistent Hashing Properties Finer Grain Load Balance

• Load balance: all nodes receive roughly the same • Redirector knows all server IDs number of keys • It can also track approximate load (or delay) • For N nodes and K keys, with high probability • Each node holds at most (1+)K/N keys • To balance load: • Provided that K is large compared to N • Wi = Hash(URL, ip of si) for all i • When server is added, it receives its initial work load from • Sort W from high to low “neighbors” on the ring i • Find first server with low enough load • No other servers are affected • Similar property and a server is removed • Benefits?

• How should “load” be measured?

33 34

Consistent Hashing Overview Used in Many Contexts

• Distribute load across servers in a data center • Web • The redirector sits in data center • Consistent hashing • Peer-to-peer • Finding storage cluster for an object in a CDN uses centralized knowledge • Motivation • Architectures • Why? • TOR • Can use consistent hashing in the cluster • Skype • Consistent hashing can also be used in a • CDN distributed setting • Video • P2P systems can use it find files

35 36

9 Scaling Problem P2P System

• Millions of clients  server and network meltdown

• Leverage the resources of client machines (peers) • Computation, storage, bandwidth

37 38

Why p2p? Outline

• Harness lots of spare capacity • p2p file sharing techniques • 1 Big Fast Server: 1Gbit/s, $10k/month++ • Downloading: Whole-file vs. chunks • 2,000 cable modems: 1Gbit/s, $ ?? • Searching • 1M end-hosts: Uh, wow. • Centralized index (Napster, etc.) • Capacity grows with the number of users! • Flooding (Gnutella, etc.) • Smarter flooding (KaZaA, …) • Build self-managing systems / Deal with huge • Routing (Freenet, etc.) scale • Uses of p2p - what works well, what doesn’t? • Same techniques attractive for both companies / • Servers vs. arbitrary nodes servers / p2p • E.g., Akamai’s 14,000 nodes • Hard state (backups!) vs soft-state (caches) • Google’s 100,000+ nodes • Challenges • Fairness, freeloading, security, …

39 40

10 Framework Searching

• Common Primitives: • Join: how to I begin participating? N2 • Publish: how do I advertise my file? N1 N3 • Search: how to I find a file? • Fetch: how to I retrieve a file? Key=“title” Internet Value=MP3 data… ? Client Publisher Lookup(“title”)

N4 N6 N5

42 43

Searching – How? What’s out there?

• Needles vs. Haystacks • Searching for top 40, or an obscure punk track from Central Flood Super- Route 1981 that nobody ever heard of? node • Search expressiveness flood • Whole word? Regular expressions? File names? Whole Napster Gnutella Freenet Attributes? Whole-text search? File • E.g., p2p gnutella or p2p google? Chunk BitTorrent KaZaA DHTs Based (bytes, eDonkey not 2000 chunks)

44 45

11 The Solution Space Napster: History

• Centralized Database • 1999: Sean Fanning launches Napster • Napster • Query Flooding • Peaked at 1.5 million simultaneous users • Gnutella • Jul 2001: Napster shuts down • Intelligent Query Flooding • KaZaA • Swarming • BitTorrent • Structured Overlay Routing • Distributed Hash Tables • Unstructured Overlay Routing • Freenet

46 47

Napster: Overview Napster: Publish

• Based on Centralized Database: • Join: on startup, client contacts central server insert(X, • Publish: reports list of files to central 123.2.21.23) server ... • Search: query the server => return someone that stores the requested file • Fetch: get the file directly from peer Publish

I have X, Y, and Z! 123.2.21.23

48 49

12 Napster: Search Napster: Discussion

123.2.0.18 • Pros: • Simple search(A) • Search scope is O(1) --> • Controllable (pro or con?) Fetch 123.2.0.18 • Cons: • Server maintains O(N) State Query Reply • Server does all processing • Single point of failure

Where is file A?

50 51

Next Topic... Gnutella: History

• In 2000, J. Frankel and T. Pepper from Nullsoft • Centralized Database • Napster released Gnutella • Query Flooding • Soon many other clients: Bearshare, Morpheus, • Gnutella LimeWire, etc. • Intelligent Query Flooding • In 2001, many protocol enhancements including • KaZaA “ultrapeers” • Swarming • BitTorrent • Structured Overlay Routing • Distributed Hash Tables • Unstructured Overlay Routing • Freenet

52 53

13 Gnutella: Overview Gnutella: Search

• Query Flooding: I have file A. • Join: on startup, client contacts a few other nodes; these become its “neighbors” I have file A. • Publish: no need • Search: ask neighbors, who ask their neighbors, and so on... when/if found, reply to sender. Reply • TTL limits propagation • Fetch: get the file directly from peer

Query

Where is file A?

54 55

Gnutella: Discussion KaZaA: History

• Pros: • In 2001, KaZaA created by Dutch company Kazaa • Fully de-centralized BV • Search cost distributed • Single network called FastTrack used by other • Processing @ each node permits powerful search semantics clients as well: Morpheus, giFT, etc. • Cons: • Eventually protocol changed so other clients could • Search scope is O(N) no longer talk to it • Search time is O(???) • Very popular file sharing network today with >10 • Nodes leave often, network unstable million users (number varies) • TTL-limited search works well for haystacks. • For scalability, does NOT search every node. • May have to re-issue query later

56 57

14 KaZaA: Overview KaZaA: Network Design

• “Smart” Query Flooding: “Super Nodes” • Join: on startup, client contacts a “supernode” ... may at some point become one itself • Publish: send list of files to supernode • Search: send query to supernode, supernodes flood query amongst themselves. • Fetch: get the file directly from peer(s); can fetch simultaneously from multiple peers

58 59

KaZaA: File Insert KaZaA: File Search

search(A) --> 123.2.22.50 insert(X, 123.2.21.23) ... search(A) 123.2.22.50 --> Replies Publish Query 123.2.0.18

I have X! Where is file A? 123.2.21.23 123.2.0.18

60 61

15 KaZaA: Fetching KaZaA: Discussion

• More than one node may have requested file... • Pros: • How to tell? • Tries to take into account node heterogeneity: • Bandwidth • Must be able to distinguish identical files • Host Computational Resources • Not necessarily same filename • Host Availability (?) • Same filename not necessarily same file... • Rumored to take into account network locality • Use Hash of file • KaZaA uses UUHash: fast, but not secure • Cons: • Alternatives: MD5, SHA-1 • Mechanisms easy to circumvent • How to fetch? • Still no real guarantees on search scope or search time • Get bytes [0..1000] from A, [1001...2000] from B • Similar behavior to gnutella, but better. • Alternative: Erasure Codes

62 63

Stability and Superpeers BitTorrent: History

• Why superpeers? • In 2002, B. Cohen debuted BitTorrent • Query consolidation • Key Motivation: • Many connected nodes may have only a few files • Popularity exhibits temporal locality (Flash Crowds) • Propagating a query to a sub-node would take more b/w than • E.g., Slashdot effect, CNN on 9/11, new movie/game release answering it yourself • Focused on Efficient Fetching, not Searching: • Caching effect • Distribute the same file to all peers • Requires network stability • Single publisher, multiple downloaders • Superpeer selection is time-based • Has some “real” publishers: • How long you have been on is a good predictor of how • Blizzard Entertainment using it to distribute the beta of their new game long you will be around

64 65

16 BitTorrent: Overview BitTorrent: Publish/Join

• Swarming: Tracker • Join: contact centralized “tracker” server, get a list of peers. • Publish: Run a tracker server. • Search: Out-of-band. E.g., use Google to find a tracker for the file you want. • Fetch: Download chunks of the file from your peers. Upload chunks you have to them. • Big differences from Napster: • Chunk based downloading (sound familiar? :) • “few large files” focus • Anti-freeloading mechanisms

66 67

BitTorrent: Fetch BitTorrent: Sharing Strategy

• Employ “Tit-for-tat” sharing strategy • A is downloading from some other people • A will let the fastest N of those download from him • Be optimistic: occasionally let freeloaders download • Otherwise no one would ever start! • Also allows you to discover better peers to download from when they reciprocate • Goal: Pareto Efficiency • Game Theory: “No change can make anyone better off without making others worse off” • Does it work? (don’t know!)

68 69

17 BitTorrent: Summary Next Topic...

• Pros: • Centralized Database • Works reasonably well in practice • Napster • Gives peers incentive to share resources; avoids • Query Flooding freeloaders • Gnutella • Cons: • Intelligent Query Flooding • KaZaA • Pareto Efficiency relative weak condition • Swarming • Central tracker server needed to bootstrap swarm • BitTorrent • (Tracker is a design choice, not a requirement, as you • Structured Overlay Routing know from your projects. Could easily combine with • Distributed Hash Tables (DHT) other approaches.) • Unstructured Overlay Routing • Freenet

70 71

Distributed Hash Tables DHT: Overview

• Academic answer to p2p • Goals • Abstraction: a distributed “hash-table” (DHT) • Guaranteed lookup success data structure: • Provable bounds on search time • put(id, item); • Provable scalability • item = get(id); • Makes some things harder • Implementation: nodes in system form a • Fuzzy queries / full-text search / etc. distributed data structure • Read-write, not read-only • Can be Ring, Tree, Hypercube, Skip List, Butterfly • Hot Topic in networking since introduction in 2000- Network, ... 2005 time frame • Distributed version of consistent hashing concept

72 73

18 DHT Operations DHT: Example - Chord

• Join: On startup, contact a “bootstrap” node and integrate • Associate to each node and file a unique id in an yourself into the distributed data structure; get a node id uni-dimensional space (a Ring) • Publish: Route publication for file id toward a close node • E.g., pick from the range [0...2m] id along the data structure • Usually the hash of the file or IP address • Search: Route a query for file id toward a close node id. Data structure guarantees that query will meet the • Properties: publication. • Routing table size is O(log N) , where N is the total • Fetch: Two options: number of nodes • Publication contains actual file => fetch from where query stops • Guarantees that a file is found in O(log N) hops • Publication says “I have file X” => query tells you 128.2.1.3 has X, use IP routing to get X from 128.2.1.3

74 75

DHT: Consistent Hashing DHT: Chord Basic Lookup

N120 Key 5 K5 N10 Node 105 “Where is key 80?” N105 N105 K20

Circular ID space N32 “N90 has K80” N32

N90 K80 N90

K80 A key is stored at its successor: node with next higher ID N60 How do you speed up look up?

76 77

19 DHT: Chord “Finger Table” DHT: Chord Join

1/4 1/2 • Assume an identifier space [0..8]

• Node n1 joins Succ. Table 0 i id+2i succ 1/8 1 0 2 1 7 1 3 1 2 5 1 1/16 1/32 1/64 1/128 6 2 N80

• Entry i in the finger table of node n is the first node that 5 3 4 succeeds or equals n + 2i • In other words, the ith finger points 1/2n-i way around the ring

78 79

DHT: Chord Join DHT: Chord Join

Succ. Table i id+2i succ 0 1 1 1 2 2 2 4 0 • Node n2 joins Succ. Table • Nodes n0, n6 join Succ. Table 0 i id+2i succ 0 i id+2i succ 1 0 2 2 1 0 2 2 7 1 3 1 7 1 3 6 2 5 1 2 5 6 Succ. Table i id+2i succ 6 2 0 7 0 6 2 1 0 0 2 2 2 Succ. Table Succ. Table i id+2i succ i id+2i succ 5 3 5 3 4 0 3 1 4 0 3 6 1 4 1 1 4 6 2 6 1 2 6 6

80 81

20 DHT: Chord Join DHT: Chord Routing

Succ. Table Items Succ. Table Items i id+2i succ 7 i id+2i succ 7 • Nodes: 0 1 1 • Upon receiving a query for 0 1 1 1 2 2 item id, a node: 1 2 2 n1, n2, n0, n6 2 4 0 • Checks whether stores the 2 4 0 item locally • Items: 0 Succ. Table Items • If not, forwards the query to 0 Succ. Table Items 7 1 i id+2i succ 1 the largest node in its 7 1 i id+2i succ 1 f7, f2 0 2 2 0 2 2 successor table that does query(7) 1 3 6 not exceed id 1 3 6 2 5 6 2 5 6 Succ. Table 6 2 Succ. Table 6 2 i id+2i succ i id+2i succ 0 7 0 0 7 0 1 0 0 Succ. Table 1 0 0 Succ. Table 2 2 2 i id+2i succ 2 2 2 i id+2i succ 5 3 5 3 4 0 3 6 4 0 3 6 1 4 6 1 4 6 2 6 6 2 6 6

82 83

DHT: Chord Summary DHT: Discussion

• Pros: • Routing table size? • Guaranteed Lookup • Log N fingers • O(log N) per node state and search scope • Routing time? • Cons: • Each hop expects to 1/2 the distance to the • No one uses them? (only one file sharing app) desired id => expect O(log N) hops. • Supporting non-exact match search is hard

84 85

21 Freenet: History Freenet: Overview

• In 1999, I. Clarke started the Freenet project • Basic Idea: • Routed Queries: • Employ Internet-like routing on the overlay network to • Join: on startup, client contacts a few other publish and locate files nodes it knows about; gets a unique node id • Addition goals: • Publish: route file contents toward the file id. File • Provide anonymity and security is stored at node with id closest to file id • Make censorship difficult • Search: route query for file id toward the closest node id • Fetch: when query reaches a node containing file id, it returns the file to the sender

86 87

Freenet: Routing Tables Freenet: Routing

• id – file identifier (e.g., hash of file) query(10) • next_hop – another node that stores the file id • file – file identified by id being stored on the local node n1 n2 id next_hop file 4 n1 f4 1 9 n3 f9 12 n2 f12 4’ • Forwarding of query for file id … 5 n3 4 n4 n5 • If file id stored locally, then stop 14 n5 f14 4 n1 f4 2 5 • Forward data back to upstream requestor 13 n2 f13 10 n5 f10

• If not, search for the “closest” id in the table, and … n3 3 n6 8 n6 forward the message to the corresponding 3 n1 f3 next_hop 14 n4 f14 • If data is not found, failure is reported back 5 n3 • Requestor then tries next closest match in routing table

88 89

22 Freenet: Routing Properties Freenet: Anonymity & Security

• “Close” file ids tend to be stored on the same • Anonymity node • Randomly modify source of packet as it traverses the network • Why? Publications of similar file ids route toward the • Can use “mix-nets” or onion-routing same place • Security & Censorship resistance • Data is encrypted using a key that is derived from file • Network tend to be a “small world” • Two types of keys: • Small number of nodes have large number of neighbors • Content Hash Key: Hash of the (encrypted file) – easy to check correctness of a (i.e., ~ “six-degrees of separation”) returned file (by router or client) • Signed Subspace Key: documents are signed by inserter (based on public key • Consequence: cryptography) • Document can be modified by the inserter • Most queries only traverse a small number of hops to • Can create multiple ids find the file

90 91

Freenet: Discussion When are p2p / DHTs useful?

• Pros: • Caching and “soft-state” data • Intelligent routing makes queries relatively short • Works well! BitTorrent, KaZaA, etc., all use peers as • Search scope small (only nodes along search path caches for hot data involved); no flooding • Finding read-only data • Anonymity properties may give you “plausible • Limited flooding finds hay deniability” • DHTs find needles • Cons: • BUT • Still no provable guarantees! • Anonymity features make it hard to measure, debug

92 93

23 A Peer-to-peer Google? Writable, Persistent p2p

• Complex intersection queries (“the” + “who”) • Do you trust your data to 100,000 monkeys? • Billions of hits for each term alone • May be ok for “free” song, but not for information critical to a company • Sophisticated ranking • E.g., how about DNS based on a DHT? • Must compare many results before returning a subset • Node availability hurts to user • Ex: Store 5 copies of data on different nodes • Very, very hard for a DHT / p2p system • When someone goes away, you must replicate the data • Need high inter-node bandwidth they held • Hard drives are *huge*, but cable modem upload • (This is exactly what Google does - massive clusters) bandwidth is tiny - perhaps 10 Gbytes/day • Takes many days to upload contents of 200GB hard drive. Very expensive leave/replication situation!

94 95

24