SCALABLE CONTENT DELIVERY WITHOUT A
MIDDLEMAN
by
JUNBO XU
Submitted in partial fulfillment of the requirements for the degree of
Master of Science
Department of Electrical Engineering and Computer Science
CASE WESTERN RESERVE UNIVERSITY
August, 2017 CASE WESTERN RESERVE UNIVERSITY SCHOOL OF GRADUATE STUDIES
We hereby approve the thesis/dissertation of
Junbo Xu
candidate for the degree of
Master of Science*.
Committee Chair
Michael Rabinovich
Committee Member
Vincenzo Liberatore
Committee Member
Xusheng Xiao
Date of Defense
June 2, 2017
*We also certify that written approval has been obtained for any proprietary
material contained therein.
2 Table of Contents
Chapter 1 Introduction ...... 10
Chapter2 Related Work ...... 15
2.1 Decentralized content dissemination ...... 15
2.2 Peer-assisted content distribution platforms ...... 15
2.3 Browser-based P2P style content distribution ...... 17
Chapter3 Overview of our system design ...... 19
3.1 Terms definition ...... 19
3.2 Model of web objects in our system ...... 20
3.3 Workflow ...... 22
3.4 Functions of the wrapper page ...... 26
Chapter 4 The origin server ...... 27
4.1 The functions of the origin server ...... 27
4.2 The wrapper page’s generation process and databases in the origin server ..... 28
4.3 The origin server’s configuration details ...... 31
4.4 The process of handling a dynamically generated content page ...... 34
Chapter 5 The system component on the peer’s side ...... 35
5.1 The reverse proxy ...... 35
5.2 The network address translation issue ...... 40
Chapter 6 The implementation of user part ...... 50
6.1 The process of fetching and verifying web objects from the peer ...... 50
6.1.1 An introduction to data URI scheme ...... 51
3 6.1.2 The integration of web objects which forms a web page together ...... 53
6.1.3 The design and implementation of wrapper page ...... 59
6.2 The same origin policy issue and the wrapper page implementation technology
...... 62
6.2.1 The problem caused by the same origin policy ...... 62
6.2.2 The adopted solution in our implementation ...... 63
6.2.3 Another viable implementation ...... 64
6.2.4 About websocket ...... 67
6.2.5 About window.postMessage ...... 68
6.3 Why not integrate the wrapper page script into the first content object as an
inline script? ...... 71
Chapter 7 Reliable accounting mechanism ...... 76
7.1 The upper bound theory ...... 76
7.2 The records submitting mechanism ...... 77
7.3 The integrity of records ...... 78
Chapter 8 Fetching web objects from the peer in parallel ...... 82
8.1 The motivation for concurrency ...... 82
8.2 Theoretical concurrent DFS model ...... 84
8.3 The “Promise”-based implementation of ConcurrentDFS ...... 88
Chapter 9 Performance evaluation ...... 95
9.1 Performance of the user-side script ...... 95
9.1.1 The experiment setup and preliminary results ...... 95
9.1.2 Explanation on the overhead of the wrapper page ...... 96
9.2 End-to-end Performance evaluation ...... 100
4 9.2.1 Part 1 ...... 101
9.2.2 Part 2 ...... 107
9.3 How content pages structure affects the performance of the wrapper page ... 110
9.3.1 The experiment design ...... 110
9.3.2 The theoretical time complexity analysis ...... 113
9.3.2 The experiment results and the conclusion ...... 115
Conclusion ...... 120
Bibliography ...... 122
5 List of Figures
Figure 3-1 root.html ...... 21
Figure 3-2 child2.css ...... 21
Figure 3-3 child2.html ...... 22
Figure 3-4 the tree visualization of root.html ...... 22
Figure 3-5 The workflow of the system ...... 23
Figure 4-1 request rewrite configuration of an origin server ...... 31
Figure 4-2 the origin server’s configuration directives ...... 32
Figure 6-1 Data url scheme of redDot.png ...... 51
Figure 6-2 myFrame.html ...... 52
Figure 6-3 Data URI scheme of myFrame.html ...... 53
Figure 6-4 a web page rooted at myContent.html ...... 54
Figure 6-5 myFrame.html ...... 56
Figure 6-6 example.css ...... 56
Figure 6-7 redDot.png ...... 56
Figure 6-8 the “integrated web object” of the web page whose root is
“myFrame.html” ...... 57
Figure 6-9 blueDot.png ...... 58
Figure 6-10 myContent.html ...... 58
Figure 6-11 the “integrated web object” of the web page whose root is
“myContnet.html” ...... 59
Figure 6-12 wrapper page ...... 65
6 Figure 6-13 wrapper page of postMessage example ...... 69
Figure 6-14 postMessage2.html ...... 70
Figure 6-15 The integration of wrapper page and first content object ...... 73
Figure 8-1 A tree of web objects from the peer ...... 84
Figure 8-2 ConcurrentDFS ...... 85
Figure 8-3 The Java implementation of ConcurrentDFS ...... 87
Figure 8-4 An example of using xmlhttprequest ...... 89
Figure 8-5 The intuitive implementation ...... 90
Figure 8-6 ConcurrentDFS in JavaScript ...... 92
Figure 9-1 The flow chart in “cache miss” ...... 109
Figure 9-2 degenerate1.html ...... 111
Figure 9-3 degenerate2.html ...... 111
Figure 9-4 The degenerate tree structure ...... 112
Figure 9-5 ThreeLevel.html ...... 112
Figure 9-6 twoLevel1.html ...... 112
Figure 9-7 twoLevel2.html ...... 113
Figure 9-8 The three-level tree structure ...... 113
Figure 9-9 Component 2 Versus N in the Degenerate tree Structure ...... 116
Figure 9-10 Component 3 Versus N in the Degenerate tree Structure ...... 117
Figure 9-11 Component 2 Versus N in the Three-level tree Structure ...... 118
Figure 9-12 Component 3 Versus N in the Three-level tree Structure ...... 118
7 List of Tables
Table 4-1 region_id versus peer_id ...... 28
Table 4-2 peer_account_info ...... 29
Table 4-3 container_page_info ...... 30
Table 9-1 with wrapper page versus without wrapper page ...... 96
Table 9-2 the decomposition of without wrapper page results ...... 97
Table 9-3 the decomposition of with wrapper page results ...... 99
Table 9-4 Hardware environments 1 ...... 102
Table 9-5 Network bandwidth 1 ...... 102
Table 9-6 Network latency 1 ...... 103
Table 9-7 Part 1 results ...... 104
Table 9-8 Hardware information 2 ...... 107
Table 9-9 Network bandwidth 2 ...... 108
Table 9-10 Network latency 2 ...... 108
Table 9-11 Part 2 results ...... 108
Table 9-12 The Theoretical complexity analysis ...... 114
Table 9-13 Degenerate tree results ...... 116
Table 9-14 Three-level tree results ...... 117
8 Scalable Content Delivery Without a Middleman
Abstract
By
JUNBO XU
In this thesis, we introduce a new model for content delivery, where the content provider recruits Internet users directly to help deliver the provider’s content. User resources are then utilized to deliver the content to those requesting it. Presumably the content provider would compensate the users for delivering its content, but the costs would likely be lower as no costly middleware infrastructure is needed and no middleman operator is involved. Our approach resembles existing peer-to-peer CDN approaches, but these approaches usually augment the CDN while we eliminate it.
In the thesis we identify and address practical challenges in implementing our approach, mainly how to ensure content integrity, despite content being delivered by untrusted users, and billing integrity, despite the incentive for users involved in content delivery to inflate the amount of content they deliver. We further build a proof-of-concept prototype and conduct preliminary performance evaluation of our approach.
9 Chapter 1 Introduction
In the last three decades, the volume of Internet traffic has experienced nearly an exponential growth [7]. Various large-scale content providers, such as online video streaming sites, online commerce sites or large software distribution sites, have contributed significantly to this growth. To distribute their content (such as software, web pages, video) to their end users efficiently, large-scale content providers need to have their server clusters in multiple geographical locations and different ISPs. One reason for this is that their end users are often distributed across vast geographical areas. Delivering content from nearby content servers improves user experiences. A second reason is that, in case a server cluster in a region is overloaded or fails, server clusters in nearby regions can balance the load or mask the failure by taking over requests from the problematic server cluster, so that end users will still perceive acceptable quality of service. Third, these geographically distributed server clusters may cache content, thus saving bandwidth by avoiding the transmission of the same content repeatedly from a remote server. Therefore, by using such a geographically distributed network, content providers can ensure their contents can be accessed with high availability, high reliability and high performance.
Today’s content providers achieve the above goal of distributing content from widely distributed platform either by deploying such a platform themselves
(e.g., Google) or through a third-party content delivery network (CDN) such as
Akamai. Through CDNs, contents are delivered to end users with a stable and high quality of service. However, it is expensive for a CDN operator to expand the
10 capacity and geographical coverage of the CDN platform, as the CDN operator has to provision more computing resources and network bandwidth. To address this issue, peer-assisted CDN solutions have been recently proposed and implemented, such as
Akamai’s NetSession [5] or ChinaCache’s LiveSky [3]. Such hybrid systems are able to scale at a lower price than traditional CDNs. However, both traditional CDNs and peer-assisted CDNs still require a middleman (i.e. CDN operators) between content providers and content consumers (users), which gives rise to the third-party costs, including but not limited to computing resources, network bandwidths and human resource costs.
In this thesis, we explore the goal to remove the middleman between content providers and end users. We design and implement a prototype of a system to create a new market place (sharing economy not unlike “Uber”) where content providers hire end users who are willing to supply their computing and network resources directly (we refer to these participating end users as peers). In our system, which we call an infrastructurless CDN to highlight the absence of middleman’s infrastructure, an end user willing to help delivering web resources for a content providers signs up in the content provider’s website, thus becoming a peer for this content provider. The content provider is expected to reward its peers, in the form of cash or e-coupons, to attract more peers. The more bytes a peer serves to end-users, the more reward he will get from the content provider. An end user can sign up to become a peer for multiple content providers simultaneously.
There are significant challenges to implementing this vision, the most important ones of which are listed below.
11 • Peers are not expected to be sophisticated computer operators. Thus, any
software and system configuration must be automated to be similar to a
routine software install that users are accustomed to. In particular, this
involves automating configuration of Wi-Fi routers (that users commonly
employ in home networks) to allow the Wi-Fi routers admit incoming
connections from end users requesting content.
• In our systems, peers are individuals outside control of the content
providers. Thus, they cannot be trusted to always deliver unaltered content
to end users. A malicious peer might substitute content from the content
provider with content of his or her own, or with the competitor’s content.
Thus, a mechanism to ensure the content integrity is needed.
• Similarly, a peer cannot be counted to always report the count of bytes it has
served honestly; indeed, the peer may inflate the count of delivered bytes to
increase its reward. Therefore, an infrastructureless CDN needs reliable
accounting to prevent losses caused by potentially dishonest peers.
• Similar to today’s CDNs, our system must be completely transparent to end
users (content consumers). Thus there is no software or any browser plug-
ins installed on the end user side and our system must only rely on HTML
and JavaScript to instruct end user’s behaviors. The strict web browser
security model poses challenges not only to transparently implementing
content integrity validation and reliable accounting mentioned above, but
also to the end-user performance.
12 This thesis explores ways to address these challenges.
In summary, we make the following contributions:
1. We design and implement a prototype of a new content delivery system without a middleman.
2. We design a reliable accounting and content integrity validation mechanism for our system, which occurs transparently to the end users.
3. We design our system to be scalable at a low cost.
4. We evaluate the performance of our system using a three-prone experimental study. First, we explore how the structure of web objects embedded in a web page affects the performance of our system. Second, we measure the overhead of the user-side JavaScript code we introduce contributes to our system.
Third, we measure the overhead of our system compared to a traditional CDN system.
The remainder of this thesis is organized as follows: In chapter 2, we discuss related work. In chapter 3, we present an overview of our system design, the operations of our system along with terms definition. In chapter 4, we introduce the origin server in detail. In chapter 5, we give detailed implementation description for the peer’s part of the system. In chapter 6, we describe user side implementation and propose. In chapter 7, we introduce the reliable accounting and content integrity validation mechanism of the entire system. In chapter 8, we describe a different user’s side implementation, which improves the user side performance and is compatible with the reliable accounting and content integrity validation
13 scheme in chapter 6. In chapter 9, we evaluate our system from three different perspectives.
14 Chapter2 Related Work
2.1 Decentralized content dissemination
In our system, the content provider leverages resources of Internet users for scalable content delivery. In this aspect, it bares similarity to a large number of peer-to-peer file sharing systems such as BitTorrent [22]. This general approach has also been pursued for scalable delivery of Web content. Here is an example.
CoralCDN [14] is free publicly available content delivery network deployed on PlanetLab. CoralCDN open for use by any web site: any web sites can freely join
CoralCDN simply by following a certain URL format for its Web resources, and
CoralCDN will cache and distribute the resources to end users. The system is transparent to end users. One important feature of CoralCDN is that it avoids creating hot spots within its platform through Coral, a latency-optimized hierarchical indexing infrastructure based on a novel abstraction called a distributed sloppy hash table. As a result, if a CoralCDN server does not have a requested object in its cache, it may obtain this object from a peer CoralCDN server and not necessarily the origin site. Unlike CoralCDN, our system does not operate a distinct middleman infrastructure.
Other decentralized content dissemination system examples include flowerCDN [16] and webP2P [17].
2.2 Peer-assisted content distribution platforms
15 In recent years, there has been work exploring the possibility to assist a traditional CDN by a P2P system. In [2], researchers quantify the potential gains of a
CDN-P2P scheme. They find that hybrid CDN-P2P design can significantly reduce the cost of content distribution while being friendly to ISPs by restricting P2P sharing within the same ISP regions. There are also some commercial implementations of the CDN-P2P design, which we discuss next.
[5] gives a detailed description and evaluation of Akamai’s implementation of peer-assisted CDN system. Their system is predominantly designed for large software distribution, and they require all participating peers to install their software called “NetSession interface” so that pieces of contents can be stored in peers’ computers. In their design, peers will first get contents form a nearby edge server. Then a connection node, which always keeps connecting to the peer, plays the role of the tracker for P2P and returns a list of other peers who also have that content. Sharing among peers is confined within the region of that connection node so that P2P sharing is ISP friendly. Further, “NetSession” also has developed a reliable accounting mechanism, which is discussed in detail in [1]. Unlike netsession, our system requires no special software for end-user browser.
LiveSky [3], which is deployed by ChinaCache company, has a different design becasue Livesky is mainly designed for video streaming sharing. This system allows a peer to share content with another peer only if the source peer is watching the same content as the recipient peer. In their evaluation, they demonstrate that their implementation guarantees the stream’s quality of service and provides low startup latency while scaling can be achieved at a low cost.
16 AntFarm [20] is a featured system where central coordinator helps boost aggregate maximum bandwidths of multiple P2P swarms, by regulating bandwidths of infrastructure-based seeders among those swarms.
2.3 Browser-based P2P style content distribution
In recent years, there also has been work about turning the browser into a web cache in a simple manner and enabling web objects sharing among individual browsers directly.
FireCoral [19] enables web object sharing among browsers directly by requiring the browser to install their plug-ins. FireCoral focuses on security, privacy and other critical aspects of P2P system design.
Squirrel [15] is designed to enable browsers in a local area networks share contents in their local cache with each other. In Squirrel, a browser identifies and requests a cache copy of objects through Pastry [18], an implementation of DHT.
However, Squirrel requires a configuration or modification of the browser to share the browser cache with other browsers. A significant challenge our system overcomes is to allow browsers to fetch content from other users without modification of the browser.
In “Maygh” [6], researchers propose an economically scalable CDN for delivering web pages of a certain website. “Maygh” is a CDN where any end user is able to serve contents to other end users without need for any software or browser plug-ins. Their system is based on direct browser-to-browser communication, which is enabled by Adobe Flash’s RTMP or Google’s WebRTC. One limitation of
17 “Maygh” is that it requires both of the two parties are browsing the same web object to enable direct browser-to-browser communication between to end-users.
Stickler [21] is a framework to ensure content integrity on the end-user side when content providers publish contents through third-party CDN, without any end user browser’s plug-ins or software installation. Their framework is based on
JavaScript XHR, cryptographic hashing and asymmetric cryptography techniques.
Our system applies essentially the same techniques to achieve a different goal of ensuring content integrity in the infrastructure-less content delivery environment.
18 Chapter3 Overview of our system design
3.1 Terms definition
User: A person who uses a browser to fetch web objects, e.g., html file, images, script files.
Peer: A desktop, laptop or any smart device with storage behind a home router.
Peer machines are used in our system to accelerate content delivery on behalf of content providers. An http server will be installed on a peer machine.
Origin server: An http server helps direct the user to fetch contents from a peer nearby and also helps the user verify the integrity of those contents. Peers will register themselves here and user will fetch contents from the origin server directly in case he cannot obtain web objects from a peer or the web object he obtained cannot match corresponding cryptographic hash.
“web page” and “web object”: These two terms are common. However, as we use them heavily in the thesis, it is necessary to clarify them. In this thesis, a web object is the piece of content corresponding to a URL. A web page is what the user sees after clicking on a hyperlink or entering a URL on the address bar: the web page includes all the embedded images and other objects, so it’s a piece of content that aggregates many web objects.
“Content page” and “first content object”: The “content page” is the web page, which contains the entire content a user expects. A user fetches all encompassed web objects of the “content page” from a peer. We make “content page” a term to distinguish it from the “wrapper page” defined in next paragraph. The “first content
19 object” refers to the first web object a user fetches from a peer. As the name itself indicates, this web object is the first web object that contains content. Usually it is an html file web object.
“Wrapper page”: The “wrapper page” is a web page that a user fetches from the origin server. It does not encompass any contents that a user expects. Instead, it includes only an html web object. The html file object contains nothing but an inline
JavaScript file, which guides the user to fetch the “content page” from the peer.
ADNS: Authoritative DNS server for the origin server.
3.2 Model of web objects in our system
Except image objects, other types of web objects all could have URLs within.
We can view a web object as a tree node, and the web objects linked to this web object by corresponding URLs can be regarded as child nodes of this web object.
When we regard the first content object as the tree root, then all web objects that a user fetches from the peer forms a tree. This bears a resemblance to the Document
Object Model (DOM). However, the purpose is mighty different. DOM is used to parse a single html web object as a complex tree. Our purpose is to model all web objects transferred between a user and a peer as a tree. Let us use the example below to further illustrate. Assume an html web object “root.html” is the first content object that a user obtains from the peer. We also present children web objects linked to “root.html” so that we can see how they together compose a tree.
20
Figure 3-1 root.html body { background-image: url (“gc1.jpg”)
} p {
font-family: Arial, Helvetica, sans-serif;
}
Figure 3-2 child2.css
21 Figure 3-3 child2.html
We view this sequence of web objects from a peer as a three level tree like below:
root.html
child1.css child2.html child3.jpg
gc1.jpg gc2.jpg gc3.jpg gc4.jpg
Figure 3-4 the tree visualization of root.html
Note that there is an inline JavaScript in “root.html”. In our model, we do not regard any inline object as a web object so that an inline object is not a tree node in our model. Let us call the relationship between “child1.css” and “gc2.jpg” in Figure
2-4 “uncle-nephew”. If an uncle and nephew refer to the same web object, we regard them as separate web objects that happen to have the same content. Thus, we can avoid forming a cycle in our graph to break our tree structure.
3.3 Workflow
Below is the workflow that occurs when a user tries to fetch a web page from a nearby peer in our P2P_CDN system:
22
Origin ADNS server
5 content objects peer user 8 content objects
9 fetched bytes record
Figure 3-5 The workflow of the system
Step1 and 2: The local DNS of the user sends DNS queries for the content provider’s domain name and get an IP address of the origin server from corresponding ADNS of that content provider.
Step3 and 4: The user sends an http request (“Host” header is the content provider’s domain name e.g., www.cnn.com) for the first content object, e.g. http://www.cnn.com/news.html. Though the user is requesting for a specific content web object (like “news.html” here), the origin server will convert this request to a request for the wrapper page generator (a php file) internally and the generator will generate the corresponding wrapper page dynamically. The wrapper page returned includes the peer’s IP address, the peer’s port number, the first content object name (like news.html here), and a list of hashes (we will introduce it later). The origin server will assign a peer to serve contents based on geographical proximity of the user and the peer. Thus, the wrapper page is generated
23 dynamically based on information like what web page a user is requesting and the
IP address of the user.
In step 5, the user sends http requests for the first content object (e.g.
“news.html”) to the peer. The user requests in step 5 are using IP address of the assigned peer instead of the content provider’s domain name in step 3. Otherwise, if using the content provider’s domain name in step 5, the domain name will be resolved to the IP address of the origin server again. The port number of the http request in step 5 is not necessarily the default number 80. It can be any other port number, which is determined in the wrapper page. In our system, a peer could help deliver contents for multiple different content providers simultaneously. A peer achieves this by running multiple virtual hosts simultaneously. The virtual hosts are distinguished by different domain names of content providers. In this way, the peer is able to serve web objects of different content providers based on the port number on which a user request arrives. (We will give more details in chapter 5.) If the peer already has the requested object stored locally, i.e. cache hit, then it will respond web objects directly. In this scenario, the peer skips steps 6 and 7 and executes step
8 directly. On the other side, if the peer does not have the target web object, it then needs to fetch it from the nearby origin server, that is step 6 and 7. This situation poses the problem that how the origin server treats requests from a user or a peer differently. Our solution is that the peer side server is configured to retrieve web objects from a different port (not port 80) in the origin server. We will present the origin server and the peer proxy server’s configuration files in Chapter 4 and
Chapter 5 respectively.
24 Step 9 the user sends a signed record (an “acknowledgement”) to the peer.
This record contains count of bytes the users get from that peer. This count of bytes is serving for reliable accounting purposes.
If the user fails to fetch any web object from the peer, the user will fetch the web object from the origin server directly. However, the user has to retrieve it from a different port (same port used by the peer in step 6) on the origin server.
Otherwise, the origin server will return another wrapper page. In addition, the user sends a request to a fixed file (“user_fail_handler.php”) on the origin server (same domain name and default port number 80) through POST method after the user has received all content web objects either from the origin server or from the peer. This request carries the following parameters: “first_content_included_web_object” specifies the name of the first content web object; “peer_IP” and “peer_port” specifies the public IP address and port number of the peer, which failed to serve the web object; “user_IP” specifies the user’s public IP address; “count_of_bytes” specifies the real count of bytes that a user retrieves from the peer. This number does not count those web objects that the peer fails to serve.
It is important for the origin server to know this information to ensure the reliable accounting for billing purposes. After a user sends those parameters to
“user_fail_handling.php”, the origin server will add a log to indicate that the peer failed to deliver several web objects. We will talk about how this report helps ensure reliable accounting later. In case a user does not successfully retrieve all objects from the peer, the acknowledgements sent from a user to a peer in step 9 will record
25 the real count of bytes (same number as that in the report a user sent to
“user_fail_handling.php”) the a user obtained from a peer.
3.4 Functions of the wrapper page
The wrapper page plays four roles. The first role is to assign a peer to the user. Then the user will fetch the first content object and other subsequent web objects from the assigned peer. The second role is to help the user verify the integrity of every web object (all web objects that make up content page) from the peer. The third role is to help origin server summarize how many bytes of contents each peer helps deliver in a period (e.g., one month), so that the content provider can pay rewards according to this. If the user successfully fetched all requested web objects from the peer, then the wrapper page is the only web object the user fetched from the origin server. The last role of wrapper page is to help error handling in case the user did not successfully fetch all objects from the peer. Because the origin server cannot know which peer is close to the user until the user’s request arrives, the wrapper page is generated dynamically.
26 Chapter 4 The origin server
The web service technology stack we assume on the origin server side is the
LAMP (Linux operating system, apache http server, MySQL relational database server, the PHP programming language). The cryptographic hash algorithm we adopt is SHA-256. A peer needs to register himself before joining our sharing system. During the peer’s registration, the origin server records the peer’s public IP address and requires the peer to install software. We will describe the peer side software in the next chapter.
4.1 The functions of the origin server
In summary, the origin server integrates three functions.
First, it needs to dynamically generate a corresponding wrapper page according to the user’s request.
Second, the origin server takes on a regular http server responsibility, that is, to retrieve a file from a local system according to the request and return it to the requestor. This role is revealed when the peers fetch contents from the origin server in case of cache miss (step 6 in the section 3.3), or when the user fails to fetch an object from the peer and falls back to obtaining the page from the origin server.
Third, the origin server is the accounting manager of the entire system. This role is revealed in two scenarios. We mentioned in Section 3.3 that a peer receives an acknowledgement (a signed record indicating how many bytes received) from the user. The peer will keep records of a period (e.g. a month) and upload them to the origin server at the end of each period. Also in Section 3.3, we mentioned that
27 the user would send an accounting related update report to the origin server in case he fails to fetch any object from the peer.
4.2 The wrapper page’s generation process and databases in the origin server
Below we clarify the implementation details of how the origin server dynamically generates a wrapper page.
When a user’s request arrives, the origin server provides the user a peer’s IP address, port number and a list of hashes of every web object the user will fetch from the peer, i.e. all web objects that compose the content page. If we regard the html web object in Figure 3-1 (root.html) as the first content object from the peer, then the corresponding wrapper page that a user gets from the origin server should include a list of hashes of root.html, child1.html, child2.jpg, child3.jpg, gc1.jpg, gc2.jpg, gc3.jpg. To describe how the origin server decides the above three types of information, we first introduce the following relational tables stored in the original server. We use JSON type format to present them as instances:
1. region_id versus peer_id
[
{“region_id”:3, “peer_ids”:[1,5,7]},
{“region_id”:5,”peer_ids”:[2,8,9]}
]
Table 4-1 region_id versus peer_id
2.peer_account_info
28 [
{“peer_id”:0, “register_name”:”kitty”,”password”:”123456”,
“IP”:”121.23.23.12”,”port_number”:80,”region_id”:5}
{“peer_id”:1, “register_name”:”tommy”,”password”:”123456”,
“IP”:”76.23.93.1”,”port_number”:23456,”region_id”:6}
{“peer_id”:2, “register_name”:”sabastian”,”password”:”123456”,
“IP”:”11.87.23.87”,”port_number”:12345,”region_id”:8}
]
Table 4-2 peer_account_info
3. container_page_info
[
{
“firstRealContentPage”:”news.html” ,“firstRealContentPage_hash”:s3e90xwf,
”embedded_info”:[{“embedded_object_name”:”recipe.css”,”embedded_object_hash”:” a1so2nx”}, {“embedded_object_name”:”rotate.js”,
“embedded_object_hash”:”90slne23”}]
},
{
“firstRealContentPage”:”shows.html”,“firstRealContentPage_hash”:”a3e9s2wf”,
”embedded_info”:[{“embedded_object_name”:”abc.css”,”embedded_object_hash”:”21 cokmx”}, {“embedded_object_name”:”check.js”,
“embedded_object_hash”:”mu23ne23”}]
}
29 ]
Table 4-3 container_page_info
Table 4-1 and Table 4-2 above are updated when a peer registers at the origin server. Specifically, the origin server will first create an entry in Table 4-2 according to all information provided by the peer during registration. It then assigns a region id for the peer based on their IP address and records the region information in Table 4-1.
When a user sends an http request to the origin server, the origin server calculates the region id of the user based on the user’s IP, then looks up that region id in Table 4-1 and gets the ids of peer candidates. The origin server randomly picks a peer id from the candidates and looks it up in Table 4-2 to retrieve the corresponding IP address of that peer. If there are no candidates available, i.e., there is no peer located in the same region as the user, the origin server will pick a random peer from other regions.
The Table 4-3 stores those hashes of every web objects that the content provider is serving. These hashes are pre-computed according to SHA-256 algorithm and stored on the origin server of the content provider. The primary key of Table 4-3 is the file name of first content object. When we query a first content object in Table 4-3, the table will return all descents’ hashes of tree rooted at our query (recall the tree model we proposed in Chapter 3). Because the user’s request towards the origin server includes the file name of first content object, the origin server can retrieve all necessary hashes by querying Table 4-3.
30 4.3 The origin server’s configuration details
The origin server is an instance of apache http server. In this section, we present specific server configuration files to illustrate how these configurations help implement the three functions (mentioned in 4.1) of origin server.
There is a program to dynamically retrieve the above information from databases and generate the corresponding wrapper page according to the user’s request. It is named “container_handler.php”. The user does not send request to this program directly. Instead, a user sends a request for first content object so that content provider needs not change URLs of their web objects.
However, an internal user request rewriting in the origin server is required to convert native user’s request to the request for “container_handler.php” so that the origin server will trigger wrapper page generation process. For example, the request “http://www.cnn.com/news.html” should be converted to
“http://www.cnn.com/container_handler.php?reqFileName=container_news.html”.
Specifically, we want the user request’s file name to be passed to
“container_handler.php” as a parameter at the very early stage of request processing in the origin server. We utilize the “mod_rewrite” module provided by the Apache server to achieve this goal. The related configuration directives are listed below.
RewriteEngine on
RewriteRule "^/(.*\.html)$" "/container_handler.php?reqFileName=$1" [PT]
Figure 4-1 request rewrite configuration of an origin server
Only requests for html files will be rewritten and be passed as parameter to
“container_handler.php”.
31 In case the user fails to fetch objects from the peer or in case the peer does not have objects cached locally, the user or the peer sends a request for a web object and expects the origin server to return the content directly. That is the second role of the origin server. The question is how the origin server can distinguish this kind of requests from the requests for wrapper page described earlier, when the origin server needs to return the special dynamically generated page. To solve the problem, we configure the origin server to have multiple virtual hosts based on different port numbers. Below is part of the configuration file in our origin server.
ServerName www.xu.ipl.eecs.case.edu
DocumentRoot /var/www/html/xu
Header set Access-Control-Allow-Origin "*"
ServerName: www.xu.ipl.eecs.case.edu
DocumentRoot /var/www/html/handlers
RewriteEngine on
RewriteRule "^/(.*\.html)$" "/container_handler.php?reqFileName=$1" [PT]
Figure 4-2 the origin server’s configuration directives
The origin server chooses the correct virtual host according to the IP address and port number for which a user or a peer is requesting. If there are multiple
32 virtual hosts matches, then the origin server pick up one virtual host according to the domain name for which a user or a peer is requesting. In the configuration file above, both virtual hosts use “*” to represent any IP address and both virtual hosts have the same domain name www.xu.ipl.eecs.case.edu, so only port number takes effect in matching. In general, the first virtual host is used to handle normal http request (the second function of an origin server). The second virtual host is used to help implement the first function (generate a wrapper page) and third function
(reliable accounting). Any requests towards port 80 will be replied with a corresponding wrapper page. Any requests towards port 22422 will be replied with contents directly. Thus, in case a user fails to fetch web objects from a peer, he fetches the web object from port 22422 of the origin server. The directive “Header set Access-Control-Allow-Origin "*"” is intended to relax the same origin policy, which restrains the user from manipulating contents from a different domain or a same domain but different port. We will cover more about same origin policy in
Chapter 6. In case a cache miss happens on the peer side, the peer sends the request to port 22422 so that the peer will be able to fetch contents from the origin server directly.
Besides generating wrapper pages, virtual hosts on port 80 have files called
“user_fail_handler.php” and “peer_logs_handler.php”. “user_fail_handler” is used to report what object a user fails to fetch from the peer. It contributes to accuracy of accounting, which we have mentioned in section 3.3. “peer_logs_handler.php” is used to handle the user-signed records stored on the peer side. It verifies and
33 parses these user-signed records, and updates the database, which records how many bytes a peer claims to serve in a period.
4.4 The process of handling a dynamically generated content page
If the content web page that a user requests needs to be generated dynamically, the origin server will not only generate the content objects, but also calculate the hash values for each of the content objects. The origin server then updates Table 4-3 by inserting the newly generated web objects and corresponding hash values into the table. With the content web objects generated and hash values calculated, the origin server generates the corresponding wrapper page as described in section 4.2. The wrapper page will direct the user to fetch content objects from the peer. The peer does not have the content objects cached locally as the origin server generates the content objects dynamically. The peer has to fetch content objects from the origin server and transmit them back to the user.
34 Chapter 5 The system component on the peer’s side
5.1 The reverse proxy
In Chapter 3, we mention that the peer is an http server and will respond requests from users. Specifically, the peer needs to have the following three functions in addition to generating http responses by retrieving files from the local file system.
First, when the peer finds that it cannot find the user’s requested web object locally, the peer must send an http request for that object to the origin server on the user’s behalf. Once the peer fetches the user’s requested web object from the origin server, it caches it locally so that subsequent requests for the same web object can be retrieved from local file system.
Second, the peer needs to receive and store the acknowledgement from the user after each transaction (i.e. how many bytes the user has received from the peer) and periodically transfer a bundle of acknowledgements to the origin server. This allows the origin server to summarize how many bytes peers deliver and compensate them accordingly. The reward mechanism is an important incentive to attract peers to participate in content delivery, and we describe a reliable accounting mechanism that enables proper rewarding in next chapters.
Third, a peer should be able to serve contents for different content providers simultaneously. As the “Host” header of the user request is the IP address of the peer rather than the content provider’s domain name, the http server on the peer
35 side has to use different ports number to tell which content provider’s contents the user is requesting.
Though we have mentioned the reason why the user sends requests to a peer using the IP address directly in section 3.3, we’d like to elaborate on the reason why to use port numbers to recognize content providers on the peer side. When a user sends http requests through the IP address directly, the User Agent (the browser in our case) sets the “Host” header to that IP address instead of any domain name. It would seem that we could have made the JavaScript that downloads the content to change the “Host” header to corresponding content provider’s domain name.
However, this behavior is forbidden by the relevant technical standard due to web security concerns. That is, once the user initializes the “xmlhttprequst” object with the IP address of the peer, the “Host” header of any http request the user will send through this “xmlhttprequest” object will have the peer IP address in the host header, which cannot be overwritten by JavaScript. Here is a relevant statement from the standard specification of “xmlhttprequest” from W3C: “UAs (User Agent) must set the Host header appropriately (see open ()) and not allow it to be overridden” [9].
Thus, the “Host” header in the user request must be the IP address, leaving us with no other option but to use port numbers to distinguish different domain names.
There are various other security restrictions on the user side programming, and a lot of work is done to work around them while keeping the system safe and efficient.
We will present them in next chapters.
36 To implement the above functionality, we deploy the apache http server on the peer side as well. However, we configure it to act as a reverse proxy and further to host multiple domains based on different ports. The differences between a forward proxy and a reverse proxy are the following. A forward proxy needs browsers to be explicitly configured before they can use the proxy. The forward proxy can fetch web objects from any websites on the user’s behalf. A reverse proxy does not need any user side configuration and can deliver content to any user, but a reverse proxy delivers contents on behalf of only a preconfigured set of websites. In other words, a reverse proxy’s customer is a particular content provider (or a particular set of content providers) while a forward proxy’s customer is a particular user (or a particular set of users).
Below is part of the configuration file we put on the peer’s apache http server.
#ServerName www.xu.ipl.eecs.case.edu.
DocumentRoot F:/XAMPP/xamppfiles/htdoc/xu
ProxyPass / http://www.xu.ipl.eecs.case.edu:22422/
#RequestHeader set Host " www.xu.ipl.eecs.case.edu."
Header set Access-Control-Allow-Origin "*"
CacheRoot “F:/XAMPP/apache/cacheroot1”
CacheEnable disk "/"
CacheDirLevels 2
CacheDirLength 1
CacheDetailHeader on
37 CacheHeader on
#ServerName www.junbo.ipl.eecs.case.edu.
DocumentRoot F:/XAMPP/xamppfiles/htdoc/junbo
ProxyPass / http://www.junbo.ipl.eecs.case.edu:22422/
#RequestHeader set Host " www.junbo.ipl.eecs.case.edu."
Header set Access-Control-Allow-Origin "*"
CacheRoot “F:/XAMPP/apache/cacheroot2"
CacheEnable disk "/"
CacheDirLevels 2
CacheDirLength 1
CacheDetailHeader on
CacheHeader on
As on the origin server side, we use virtual host technology on the peer side again. The two virtual hosts serve contents for www.xu.ipl.eecs.case.edu and www.junbo.ipl.eecs.case.edu respectively. Recall from the previous chapter that the matching rule of virtual hosts is first by IP address matching and then by port number matching, and at last by domain-name matching. In the above configuration, virtual hosts both use “*” as IP address, but they have different ports. Thus any request arriving on port 34723 will be matched to the first virtual host and will be regarded as request for domain name www.xu.ipl.eecs.case.edu. Any request
38 arriving on port 34274 will be matched to the second virtual host and will be regarded as request for domain name www.junbo.ipl.eecs.case.edu.
Inside the “VirtualHost” directives, the two virtual hosts share the same group of directives but with different parameters. Let’s examine them one by one.
We do not set up any ServerName directive, because the Host header in user requests lists an IP address instead of a domain name. Thus we cannot use domain name based virtual host matching. The “ProxyPass” directive specifies how the proxy will fetch the objects form the origin servers on a cache miss. The first parameter is the local path name and the second parameter is URL of the remote origin server. In the first virtual host group, the first parameter of “ProxyPass” is “/” and the second is www.xu.ipl.eecs.case.edu:22422/. This means when a user requests any file under the root directory of the peer, the peer will fetch that object from the concatenation of http://www.xu.ipl.eecs.case.edu:22422/ and the local path in the user request, when the peer does not have the web object locally. Here is an example to illustrate. Assume the peer IP address is 75.210.23.12. When a user sends request http://75.210.23.12:34723/contents/somePicture.jpg to the peer, and if the peer doesn’t have somePicture.jpg locally, the peer will fetch this object from http://www.xu.ipl.eecs.case.edu:22422/contents/somePicture.jpg , return it to the user and cache it locally.
The “Header set Access-Control-Allow-Origin "*"“ directive tells the peer to add a header “Access-Control-Allow-Origin “*”” to any response to the user. This header is used to deal with the same-origin policy issue, which we will discuss in
39 detail in next sections. Finally, the directives staring with “Cache” configure the disk-based cache behavior of this reverse proxy.
5.2 The network address translation issue
Most residential users employ a network address translation (NAT) device that is situated between their home network and the Internet. Here is a definition of
NAT from Wikipedia [10]:
“Network address translation (NAT) is a method of remapping one IP address space into another by modifying network address information in Internet
Protocol (IP) datagram packet headers while they are in transit across a traffic routing device. The technique was originally used for ease of rerouting traffic in IP networks without readdressing every host. It has become a popular and essential tool in conserving global address space allocations in face of IPv4 address exhaustion by sharing one Internet-routable IP address of a NAT gateway for an entire private network”.
In our scenario, we assume the peer side machine sits behind the NAT- enabled router and thus it’s within a private network. We need to expose it in the
Internet so that external users can access the peer. We choose UPnP [23] to achieve our goal. Before we dig into details how we use UPnP for NAT traversal, we provide our rationale for adopting a UPnP solution.
To our knowledge, other hybrid CDN systems, including Livesky [3],
Netsession [1][5] and Maygh [6], which explicitly mention how they circumvent
NAT, adopt a popular solution called STUN, which has been standardized in
40 RFC5389. The basic idea of it is as follows. Assume two communication parties, named A and B, reside in their own respective private networks and both are behind a NAT. To allow B to reach A, we need deploy a STUN server in the public network.
Party A sends a UDP packet to the STUN server and the STUN server can know the external IP address and port now. Party B communicate with STUN server also, which tells Party B the current externally exposed IP address and port number of
Party A. In this way, Party B can reach Party A. However, this scenario relies on the assumption that Party A’s router allows Party B to communicate with A through that port mapping, even though the port mapping is created for communication between
Party A and the STUN server. If Party A’s router does not allow this behavior, then
Party B then needs a relay server in the public network to reach Party A. There are many other scenarios involving NAT traversal using STUN, which makes things complicated. According to the netsession paper [5], a large fraction of codes in that system is devoted to implementing NAT traversal by using STUN.
On the other side, if both A and B utilize UPnP compatible routers, the UPnP based NAT traversal solution is simple and effective. It does not need any third party server like the STUN server and avoids potential use of relay server. UPnP devices are widely adopted. Here is the statement from UPnP’s official website. “UPnP
Forum was originally formed in October 1999 as an industry initiative who gained more than 1000 leading companies in computing, printing and networking; consumer electronics; home appliances, automation, control and security; and mobile products.” Thus, even though our peer eligibility is limited to UPnP compatible device, this leaves a large pool of potential peers.
41 Now let us come back to the details of the UPnP-based solution to NAT traversal. The discussion below is based on the assumption that the home router of the peer is UPnP-enabled device.
Universal Plug and Play (UPnP) is a set of networking protocols that permits networked devices, such as personal computers, printers, Internet gateways, Wi-Fi access points and mobile devices to seamlessly discover each other's presence on the network and establish functional network services for data sharing, communications, and entertainment. Though UPnP is not especially designed for
NAT traversal, one can conveniently achieve this goal by utilizing the protocols implemented by all UPnP devices. The high level goal is to let the peer side machine automatically add, delete or modify a permanent port-mapping on the NAT-enabled gateway by running a program installed on the peer side machine. A port-mapping here refers to a key-value entry mapping from the external (Internet) IP address plus port number to a internal (private) IP address plus port number.
Generally speaking, this port-mapping configuration program executes the following steps. First, the peer broadcasts a message to discover the Internet gateway device (IGD). The IGD will reply to the peer with a URL, which points towards the device description of the IGD. Then the peer fetches the device description from the IGD. The device description contains a list of services it supports, service descriptions and corresponding control URLs for those corresponding services. The peer will choose the service it needs and access the corresponding control URL with appropriate arguments, so that the IGD will be
42 configured to add, delete or modify a port-mapping. Below we give a detailed implementation description of the above procedures.
To keep an eye on the big picture, we assume there are only two UPnP devices in the peer’s private network, the IGD and the peer side machine. In the discovery phase, the peer sends its search request as an http request but over UDP
(rather than TCP as normal) in a multicast manner (known as Simple Service
Discovery Protocol). The IGD will respond to the peer also by http over UDP but in a unicast manner. The multicast search message header is as follows, and there is no body for the search message.
M-SEARCH * HTTP/1.1
HOST : 239.255.255.250:1900
ST: urn:schemas-upnp-org:device:InternetGatewayDevice:1
MAN: “ssdp:discover”
MX: 2
“239.255.255.250” and “1900” are the fixed multicast IP address and port in the
UPnP discovery search message. ST stands for the search target, which can be a device or a service. If it’s a service, then we are searching any devices that provide that type of service. In the message above, we are searching for an IGD device. The
IGD will reply back with a URL that points to the its device description. The MAN header is required by http extension framework; its value in a discovery message must be “ssdp:discover”. MX header indicates the maximum wait time (in seconds) that the device has to response.
43 The IDG’s response message has no body either; listed below is the response message corresponding to the search message above. It ends with a blank line.
HTTP/1.1 200 OK
CACHE-CONTROL: max-age=100
DATE: Sun, 11 Dec 2016 04:32:40 GMT
EXT:
LOCATION: http://192.168.0.1:1900/igd.xml
SERVER: ipos/7.0 UPnP/1.0 TL-WR841N/9.0
ST: urn:schemas-upnp-org:device:InternetGatewayDevice:1
USN:uuid:060b7353-fca6-4070-85f4-1fbfb9add62c::urn:schemas-upnp- org:device:InternetGatewayDevice:1
The CACHE-CONTROL here specifies number of seconds this response is valid. That is the time period during which our peer could assume the device is available. DATE indicates when the response is generated. EXT is for backwards compatibility with old UPnP version. LOCATION is the URL of the device description for this device. The SERVER has three tokens. “ipos/7.0” is the router’s operating system name and “TL-WR841N/9.0” is router’s product name. ST here represents the search target, which corresponds to that in the request header. In the request, the peer requested an Internet Gateway Device (ST: urn:schemas-upnp- org:device:InternetGatewayDevice:1), and the response here indicates such a device.
USN stands for unique service name.
44 Then the peer just uses normal http (http over TCP) to request the device description using the URL in the LOCATION header above. The IDG sends back the description in xml format. As an example, below is part of the device description of the device used in the home network where one of our test peers has been deployed.
Type> 45 The peer’s goal here is to add a port mapping to the IGD. Therefore the only part we are interested is the block in the “ “ “ URL of device description we retrieved from LOCATION header (LOCATION: LOCATION: http://192.168.0.1:1900/igd.xml). SCPURL is the URL for service description. eventSubURL is to let peer listen for state changes in the IGD. In particular, controlURL is the one we use to send actions to a specific service in the device. To add a port mapping to our IGD, the peer sends an xml file defined by SOAP (Simple Object Access Protocol) over http to the URL in the “ 46 tag above. The peer has to use http POST method for sending this file. Below is the http post request for port mapping adding action we use in our implementation. POST /ipc HTTP/1.1 Content-Type: text/xml SOAPAction:"urn:schemas-upnp-org:service:WANIPConnection:1#AddPortMapping” Connection: Close User-Agent: Java/1.8.0_05 Host: 192.168.0.1:1900 Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 Content-Length: 627 (blank line here) xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"> xmlns:m="urn:schemas-upnp-org:service:WANIPConnection:1"> 47 Assuming the IGD’s the public IP address is “23.12.42.123”, after the IGD receives this file, the IGD will try to add a new port mapping from “23.12.43.123:80” to “192.168.0.105:80” and reply with a message to indicate whether or not this action has been successful. Let’s look at the above message again. There is a custom header called SOAPAction, with the value specifying the targeted service name and the peer’s intended action. All sub tags enclosed by “ IGD’s reply message. It contains no error information, so it indicates the IGD has successfully added the port mapping. HTTP/1.1 200 OK\r\n CONNECTION: close\r\n SERVER: ipos/7.0 UPnP/1.0 TL-WR841N/9.0\r\n CONTENT-LENGTH: 332\r\n CONTENT-TYPE: text/xml; charset="utf-8"\r\n 48 xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"> xmlns:u="urn:schemas-upnp-org:service:WANIPConnection:1"> 49 Chapter 6 The implementation of user part There are two goals of user part implementation. One is to ensure the integrity of each web object fetched from the peer, and ensure the user browser renders them as expected. Another goal is to support our reliable accounting system, which help the content provider know count of bytes that each peer help serve correctly. The two goals are challenging for our infrastructureless content delivery system. In a conventional CDN, the trusted CDN vender operates edge servers, so that it is safe to rely on edge servers to render unmodified contents and to report count and report the number of bytes they serve. In our scenario, a user fetches contents from an untrusted peer, who may serve modified contents and has an incentive to inflate the served byte count to increase its payment. The user may also not be reliable because he may collude with a malicious peer. In this chapter, we elaborate on the implementation of user side in our system. In the next chapter, we will elaborate on our system’s reliable accounting mechanism. 6.1 The process of fetching and verifying web objects from the peer As discussed in previous chapters, the wrapper page gives the user the IP address and port number of a nearby peer so that the user can fetch requested web objects from that peer. Further, there are verification hashes in the wrapper page to help user verify the contents from the peer. We also mentioned we model every web objects from the peer as a tree rooted at the first content object. In this section, we give details that how a user fetch and verify those web objects from the peer based 50 on that tree model and how our wrapper page finally renders all the fetched objects correctly to the user. 6.1.1 An introduction to data URI scheme We need to introduce the concept of data URI scheme first, before we enter further details of the whole process. The “data URI scheme” is an inline ASCII representation form of an arbitrary web object. If this object is an image or an object in another binary format, the binary object will be converted to an ASCII byte representation using based64 encoding. Here is the definition of “data URI scheme” from Wikipedia. “The data URI scheme is a uniform resource identifier (URI) scheme that provides a way to include data in-line in web pages as if they were external resources. It is a form of file literal or here document”. [11] We use the following two examples to illustrate with more details. Assume there is a tag . To fetch the binary data of “redDot.png”, we use the “xmlhttprequest” in a JavaScript file to send a request for it. After we have the binary data of that image, we calculate the corresponding base64 ASCII string for the binary data of “redDot.png”. Then we change value of the src property of that image to the based64 string of the content of “redDot.png”. The tag now looks like this: AAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO 9TXL0Y4OHwAAAABJRU5ErkJggg==" alt="Red dot" /> Figure 6-1 Data url scheme of redDot.png 51 The long string is the base64 representation of the binary data of the original image. When the browser tries to parse this tag and render contents represented by the long string in src property, the browser renders the exact same image as the one indicated by “”. There is one difference in processing the two tags by the browser. For the tag in Figure 6-1, after the browser parses the tag, the browser can render it without sending any request for “redDot.png” because the value of src property is data of the image itself. On the other side, after browser parses “”, the browser gets the link of “redDot.png” instead of real content. The browser has to send a request for content of “redDot.png”. We show another example, which is a tag referring to a web object of pure text data. Assume there is a tag Figure 6-3 Data URI scheme of myFrame.html This is simpler than handling binary data format and only involves putting the text content into the src property and adding the appropriate header. In Figure 6-3, the header is “data:text/html;charset=utf-8,”; In Figure 6-2, that header is “data:image/png;base64,” . The token after the colon in each header indicates the media type for the incoming data. 6.1.2 The integration of web objects which forms a web page together Recall that in Chapter 3, we introduce that a web page can be modeled as a tree of web objects. In this section, we show how we transform a tree of web objects that make up a web page together into a single web object, which can represent the same web page. We call this single web object the “integrated web object” of the corresponding web page. The “integrated web object” is still a web object, but it independently represents a web page. Below, we show the steps how we generate the “integrated web object” of a certain web page by leveraging the data URI scheme we introduced in section 6.1.1. In the next section we show the motivation of generating the “integrated web object”. 53 The tree below represents a web page whose root is the web object “myContent.html”. myContent.html myFrame.html blueDot.png example.css redDot.png Figure 6-4 a web page rooted at myContent.html Besides the node “myContent.html”, every other node (including leaf nodes) in the tree above is a root node of a sub-tree. As a tree, every sub-tree can represent a web page. The web page whose root is a leaf node is a base case, because such a web page consists only one web object. Therefore, the “integrated web object” of such a web page is the root web object itself without any transformation needed. To generate the “integrated web object” of a certain web page other than leaf-rooted web page, we run the function called “integrate()”. The input of this function is the root web object of a web page. The output of this function is the corresponding “integrated web object”. The “integrate()” consists of three steps: 1. Let us name the input web object as W. Parse W to retrieve URLs of external web objects in W. 2. If there is no URL retrieved, return W. 3. For each URL retrieved in step1, named Li (i ranges from 1 to n), fetch the corresponding web object. For each fetched web object, 54 named Ai (i ranges from 1 to n), use the web object as the input of function “integrate()”. Then each output, named Bi (i ranges from 1 to n), is the “integrated web object” of a web page whose root is the corresponding input object Ai. 4. Replace each external web object URL in W with corresponding Bi in data uri scheme (i.e. adding appropriate header before Bi ). Return the modified W. Note that we define “integrate()” recursively and we handle the base case in step 2. If the input web object has no URL referring to external web object, the web page whose root is the input is a tree with only one node. Then the input can independently represent a web page. It is an “integrated web object” itself by definition. Therefore, it is the base case for function “integrate()”. Now, we use the tree rooted in “myContent.html” in Figure 6-4 as an instance to show how the function “integrate()” generates the “integrated web object” of the web page whose root is “myContent.html”. According to step 3, to generate the “integrated web object” of the web page whose root is “myContent.html”, we need the “integrated web object” of the web page whose root is “myFrame.html” and the web page whose root is “blueDot.png”. The “blueDot.png” part is the base case, so this part’s “integrated web object” is “blueDot.png” itself. To retrieve the “integrated web object” of web page whose root is “myFrame.html”, we need to use “myFrame.html” as the input of “integrate()”. Below we show the content of “myFrame.html” in Figure 6-5. 55
Figure 6-5 myFrame.html
Figure 6-6 and Figure 6-7 show the contents of “example.css” and
“redDot.png” respectively. The content of “redDot.png” is represented by the base64 encoding string of its binary data. body {background-color: white} p {font-family: Arial, Helvetica, sans-serif;}
Figure 6-6 example.css iVBORw0KGgoAAAANSUhEUgAAAAUA
AAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO
9TXL0Y4OHwAAAABJRU5ErkJggg==
Figure 6-7 redDot.png
As “example.css” is leaf node in Figure 6-4, the “integrated web object” for the web page whose root is “example.css” is “example.css” itself. “redDot.png” has the same situation. With the “integrated web object” of the two sub trees of node
56 “myFrame.html”, we can obtain the “integrated web object” of sub tree whose root is
“myFrame.html”. The figure below shows the result.
Helvetica, sans-serif;}')” rel=”stylesheet”>
AAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO
9TXL0Y4OHwAAAABJRU5ErkJggg==" alt="Red dot" />
Figure 6-8 the “integrated web object” of the web page whose root is “myFrame.html”
As shown in Figure 6-8, we have got the “integrated web object” of the left sub tree of node “myContent.html”. The right sub tree of “myContent.html” consists of only a node called “blueDot.png”. Therefore, the “integrated web object” of the right sub tree is the web object “blueDot.png” itself. Below is the base64 string of
“blueDot.png”. iVKOxw0KGgoAVAFNSUhEFgACAAUA
AAAFCAYAAACNbCRlAAAAHElEQVQ8YNP4//5/w38GNBWDIBKEY7HxgljNBAAO
9TXL0K4OHeeAAACDEU5ErkJggg==
57 Figure 6-9 blueDot.png
We also show “myContent.html” below.
Figure 6-10 myContent.html
According to step 4 in “integrate()”, we replace “myFrame.html” with the corresponding “integrated web object”, that is Figure 6-8; We replace “blueDot.png” with corresponding “integrated web object”, that is Figure 6-9. We also adding appropriate header for each of the two replacements. The “integrated web object” for the web page whose root is “myContent.html” is shown in Figure 6-11.
AAAFCAYAAACNbCRlAAAAHElEQVQ8YNP4//5/w38GNBWDIBKEY7HxgljNBAAO
9TXL0K4OHeeAAACDEU5ErkJggg==" alt="Blue dot" />
Figure 6-11 the “integrated web object” of the web page whose root is “myContnet.html”
6.1.3 The design and implementation of wrapper page
As mentioned in Chapter 3, all web objects fetched from the peer make up a tree whose root is the first content object. In our design, the wrapper page will generate the “integrated web object” for the tree whose root is “first content included page”, after the wrapper page has finished fetching and verifying all web objects from the peer. Then let the user’s browser render this “integrated web
59 object” by using the browser’s web API “document.write()”. Before we show in detail the implementation of the wrapper page, we elaborate on the two relevant operations – fetch and verify, used in the wrapper page.
In Chapter 3, we mention that there is a list of hashes for each of web objects that user will obtain from the peer, when the origin server dynamically generates the wrapper page based on user’s request and user’s IP address. The list of hashes is embedded in the wrapper page, so that the script in the wrapper page can verify the integrity of each web object from the peer on the user’s behalf. Specifically, for each web object, the wrapper page calculates the SHA-256 hash of it and compares the calculated value with the given value. If the web object is not modified, the two values equal. Otherwise, this web object is forged.
In Chapter 4, we mention that the script in the wrapper page fetches each web object from the peer by invoking the web API “xmlhttprequest”. Additionally, the wrapper page fetches each web object one by one in a pre-order sequence, beginning from the first content object. The reason we have to adopt
“xmlhttprequest” to fetch web objects is explained in next section 6.2.
With the background above, we show the implementation of the wrapper page. The core function in the wrapper page is called “verifiedIntegrate()”, which is an augmented version of function “integrate()” we introduced in 6.1.2. The input of this function is a web object and the URL for this web object. The output of this function is the “integrated web object” of the web page whose root is input. In addition to generating “integrated web object”, “verifiedIntegrate()” verifies each web object before incorporating the web object into the final result. If any of the
60 web objects fails to pass the verification, the wrapper page will fetch it from the origin server directly. Below are the details of function “verifiedIntegrate()”. Let us name the input’s web object W. Name the URL of W “urlw”, which is also included in the input.
Step 1 Calculate the SHA-256 hash of the input web object.
Step 2 Verify the calculated hash value against the provided corresponding
hash value. If the two values do not match, retrieve the correct web
object from “urlw” directly. Assign the retrieved content to W, so that
W’s original content is erased. In the case that two values do not
match, the wrapper page will send a report to the origin server. This is
described in section 3.3.
Step 3 Parse W to retrieve URLs of external web objects in W.
Step 4 If there is no URL retrieved, go to step 7 directly.
Step 5 For each URL retrieved, named Li (i ranges from 1 to n), fetch the
corresponding web object. For each fetched web object, named Ai (i
ranges from 1 to n), use the web object and corresponding URL Li as
the input of function “verifiedIntegrate()”. Then each output, named Bi
(i ranges from 1 to n), is the “integrated web object” of a web page
whose root is the corresponding input’s web object Ai.
Step 6 Replace each external web object URL in W with corresponding Bi in
data uri scheme (i.e. adding appropriate header before Bi).
Step 7 Send a report about bytes counting to the peer. Return the modified W.
61 Details about the report sent to the peer in step 7 will be revealed in the next chapter. The replacing in step 6 is important because it help the user to avoiding sending duplicate request. To see where the duplicate request come from, assume we delete step 6 and change the output of “verifiedIntegrate()” to void. After execution of “verifiedIntegrate()” finishes, the browser will fetch the content of every embedded URL again during rendering. However, in step 5 of
“verifiedIntegrate()”, all web objects have already been fetched. That is why we utilize “integrated web object” method to store those already fetch web objects.
We now elaborate on the execution process of the wrapper page after it has loaded in user’s browser. The URL of first content object is already included in the wrapper page. The first thing of the wrapper page is to fetch and verify the first content object. Then the wrapper page invokes “verifiedIntegrate()” with the first content object and the corresponding URL as its input. Last, the wrapper page invokes the web API “document.write()” using the “integrated web object” generated before as its input.
6.2 The same origin policy issue and the wrapper page implementation technology
6.2.1 The problem caused by the same origin policy
When the script in the wrapper page tries to compute hashes of web objects from the peer, including the first content object, the wrapper page script is forbidden to access the data from these objects, preventing the script from calculating the hashes. This is because the wrapper page and content web objects
62 belong to different domains (the wrapper page’s domain name is the content provider’s domain name; the domain name of content web objects is, technically, the
IP address of the peer) and the same origin policy, which limits JavaScript included in a page to only access data from the domain of the page, will apply. We need to use some technique to relax this constrain. Listed below is an introduction of the same origin policy.
“In computing, the same-origin policy is an important concept in the web application security model. Under the policy, a web browser permits scripts contained in a first web page to access data in a second web page, but only if both web pages have the same origin. An origin is defined as a combination of URI scheme, hostname, and port number. This policy prevents a malicious script on one page from obtaining access to sensitive data on another web page through that page's Document Object Model.” [12]
6.2.2 The adopted solution in our implementation
In our wrapper page script implementation, we use Cross-Origin Resource
Sharing (abbreviated as CORS) method to relax same-origin policy and we use xmlhttprequest API to fetch an object from the peer.
Here is a description from CORS standard specification: “A response can include an Access-Control-Allow-Origin header, with the origin of where the request originated from as the value, to allow access to the resource's contents. The user agent validates that the value and origin of where the request originated match.” In our case, if the http server on the peer side explicitly specifies that it allows requests from the origin server’s domain name (the wrapper page belongs to the origin
63 server) to access its recourse, then the user’s browser will not enforce same origin policy, and the wrapper page will be able to access resources from the peer.
“xmlhttprequest” is an API provided by JavaScript to issue an http request to any remote party. Recall that in the step 5 of “verifiedIntegrate()” in section 6.1.3, the function needs to fetch each of the URLs parsed in step 3. To implement this, the script creates an “xmlhttprequest” for each of the URLs. We set the “xmlhttprequest” in synchronous mode. In this mode, the “xmlhttprequest” of a URL will not return until it has fetched the web object of the URL. Once the “xmlhttprequst” of a URL returns, the wrapper page script calculate the hash of the fetched object, as describe in step 1 in “verifiedIntegrate()”. The wrapper page script can access the web objects from the peer, as there is a CORS header in each of the web object from the peer now.
Usually the web objects from the peer refer to each other by relative URL.
The container needs to convert all URLs of web objects from the peer from relative to absolute by explicitly adding the peer’s IP address and port number to the host portion of the URL. Otherwise, the browser by default would set the domain name of these URLs to the domain name of the embedding page, which in our case is wrapper page having the domain name of the origin server.
6.2.3 Another viable implementation
The combination of CORS and “xmlhttprequest” is not the only option to implement “verifiedIntegrate()” and relax the same origin policy. Our initial intent was to use “iframe” to fetch objects from the peer and use the “document.domain property” method to relax the same-origin policy. Specifically, this method requires
64 the peer to have a domain name, and this domain name has to share a common suffix with the origin server’s domain name. When the peer’s page and origin server’s page both set document.domain property to that same suffix, the origin of both pages share the same domain name; The port number of the origin of both pages will be stored as null. Therefore, scripts in either page can access or manipulate DOM objects in the other’ page. We use an example to illustrate how the combination of “iframe” and “document.domain” can also achieve the goal that the user fetches and verifies each web object from the peer.
Assume the domain name of the origin server is “origin.example.com”, and the peer’s domain name is “peer.example.com”. Listed below is part of the wrapper page.
Line
1.
2.
Figure 6-12 wrapper page
65 In line 1 in Figure 6-12, the wrapper page utilizes “iframe” tag to fetch not only the first content web object “real_content.html” and all its embedded web objects from the peer. In line 4, the wrapper page defines the load event handler for the “iframe”. According to the DOM standard specification [25], the load event would fire after the web object itself and all dependable web objects load. In our case, when the “iframe” fires load event, not only iframe itself but also all the embedded web objects have loaded. Function “recursiveHashCheck” is expected to recursively calculate all web objects in the tree rooted at “real_content.html”. Based on the load event definition, all web objects from the peer have been fetched when
“recursiveHashCheck” start to run. The statement in line 5 changes the document.domain property of wrapper page to “example.com”. There should be a similar statement in the “real_content.html” so that the wrapper page and the peer’s web objects share the common document.domain value. In this way, when the
“recursiveHashCheck” in line 6 starts to run, the same origin policy has been relaxed and “recursiveHashCheck” is able to access “real_content.html” and its embedded web objects. Finally, after “recursiveHashCheck” finish executing, the wrapper page can determine whether any of the web objects from the peer does not own integrity.
Though the combination “iframe”+”document.domain” is effective, this method would make the browser fetch most of the web objects from the peer twice.
When “recursiveHashCheck” starts to run, all of the peer web objects have been fetched from the peer and should be stored in memory. It is ideal that
“recursiveHashCheck” calculates the hash values of these web objects from local copies. Though the wrapper page can access the first content web object
66 “real_content.html”, the wrapper page has no way to access most of the embedded web objects like: , or , even the wrapper page already fetched them. Element’s Attributes like innerHTML, textContent, text (only script has this attribute) can only access the inline contents between the open tag and close tag. According to DOM specification [25], there is no attribute or global method to access those fetched external object. Therefore, the only way to verify these objects is to use “xmlhttpreqeust”+”CORS” to fetch them again.
While the above situation is our dominant consideration to give up the
“iframe” based solution, another reason that we adopt the combination of
“xmlhttprequest” and “CORS” is we do not need to assign a domain name to the peer in this solution. Thus, we can refer to a peer using its IP address directly instead of using a domain name in the wrapper page. This will save a DNS resolving time for the peer domain name. In case a user sets a remote public DNS server as its local
DNS server, this resolving time could add significant latency.
6.2.4 About websocket
We also considered other ways of relaxing same-origin policy. The websocket protocol is exempt from same-origin policy by nature. It will also satisfy the requirement of hash check by working like
“websocket+relative_absolute_url_conversion”. The following considerations led us to prefer the “xml+CORS+ relative_absolute_url_conversion” method. On the one hand, we do not need the feature of websocket that allows the server to send content to the browser without being solicited by the client. On the other hand, we
67 use the apache http server at the peer side, and the apache http server does not have a module to parse a websocket directly. It only has a module to tunnel a websocket request from the browser to a backend websocket server.
6.2.5 About window.postMessage
We considered to relax the same origin policy by leveraging the web API
“window.postMessage()” as well. By invoking “window.postMessage()”, a web page in a window can transfer data to another web page in another window, even if the two web pages belong to different origins ( the concatenation of protocol, domain name plus port number). This method is designed for communication between two windows. Therefore, the wrapper page and content page need to reside in two different windows if we want to utilize this method to relax the same origin policy.
Specifically, the JavaScript in the wrapper page needs to use iframe to fetch web objects from the peer rather than to use “xmlhttprequest”. Below is an example to show how the wrapper page uses “window.postMessage()” plus ”iframe” to enable a user to fetch and verify web objects from a peer. We also show this solution’s security vulnerability, which leaves us “xmlhttpreqeust” plus “CORS” as the only viable option.
The wrapper page is as follows. Assume the origin (the concatenation of protocol, domain name plus port number) of the origin server is http://origin.example.com and the origin of the peer is http://peer.example.net.
Line
1
2
68 3
wrapper
4
5
17
18
Figure 6-13 wrapper page of postMessage example
In line 4, the wrapper page utilizes iframe tag to fetch web objects from the peer. The src property of that iframe tag is “postMessage2.html”, which is also the name of first content object. In line 15, the wrapper page adds an event listener for
“Message” event. That is, once the script in “postMessage2.html” dispatch an
Message event by invoking “window.postMessage()”, the wrapper page will handle it by invoking “receiveHandler” function. The “receiveHandler” is defined from line 6
69 to line 14 in Figure 6-13. The input of this function is an object called “event”. It contains the origin (the concatenation of protocol, domain name plus port number) of the peer and the data that a peer posts to the wrapper page. In line 7, the function verifies whether the origin of the peer is “peer.example.net”. To achieve our goal of verifying peer’s content, we need the peer to post the content of each web object.
Then the receiveHandler is responsible to verify the contents by invoking the function “hashCheck(text)” in line 13. In our example in Figure 6-10, we assume there is only one web object from the peer, that is the first content object. We now show the “postMessage2.html” (the only web object of “content page”) as follows.
Line
1
2
3
iframe
4
8
9
Figure 6-14 postMessage2.html
The inline script (line 4 - 7) is used for transferring the copy of whole content
(line 1 to line 9) to the wrapper page in Figure 6-13 so that wrapper page can verify the content. Specifically, in line 5, the whole content is indicated by variable
70 “entireContent”. In line 6, window.parent refers to the window object of the wrapper page in Figure 6-13. http://origin.example.com is the origin of wrapper page. After executing line 6, the peer’s web object transfer the content of itself to the wrapper page and also dispatch a Message event the wrapper page.
Until now, it seems this design “iframe” + “window.postMessage()” circumvent the same origin and let the wrapper page be able to verify contents from the peer. However, the effectiveness of this design depends on the peer’s honesty.
Suppose we regard texts from line 1 to line 9 in Figure 6-14 as a string, and we name it “fakeContent”. If we replace the first parameter in line 6 in Figure 6-14 with
“fakeContent”, and modify other part in Figure 6-14. For example, change line 3 from “
iframe
” to “modified content
”. The “fakeContent” is the same content as the original Figure 6-14. The script in Figure 6-13 will still think the content that the peer serves is intact. Actually, the content (“fakeContent”) that peers transfers to the wrapper page for verification is the correct content, same as the “entireContent” in line 5 in Figure 6-14. The thing is what the peer transfers to the wrapper page is not exactly the same as the current peer’s web object, where“
iframe
” is changed to ““modified content
”. Therefore, this design “iframe”+”window.postMessage()” cannot help the user effectively ensure the integrity of peer’s web object, though this design can help relax the same origin policy.6.3 Why not inline the wrapper page script into the first content object?
71 In our system, before the browser fetches any web objects from the peer, the browser first obtains a wrapper page from the origin server. The rest of the work, including fetching objects from the peer, verifying each object, building integrated web objects and accounting support, is done through the JavaScript in the wrapper page.
In most cases, the first content object embeds links to external CSS files, images, scripts or other web pages (e.g. using iframe tag). It is a common practice for a content provider to generate the first content object dynamically. It would seem better to let the peer only deliver web objects like CSS file, images and scripts instead of the first content object because other content objects usually has a long period before expiration while the first content object has a short expiration time and typically has to be fetched from the origin server regardless. Therefore, it would seem reasonable to serve the first content object from the origin server directly and integrate the wrapper page script into the first content object as an inline script.
As discussed later, the only way we find to implement the integration is to place the inline wrapper page script at the bottom of the content page. The code below shows the sketch of the possible implementation.
Line
1.
2. contents of first content object ….
3.
10.
Figure 6-15 The integration of wrapper page and first content object
The inline script from line 3 to line 9 is the inline wrapper page script. The xmlhttprequest used in the implementation of function verifiedIntegrate() has to be in synchronous mode – otherwise the content in line 2 will start executing, and an embedded script there could alter the rest of the page and remove any verification.
We will mention in Chapter 8 that synchronous mode xmlhttprequest is discouraged and deprecated by the xmlhttprequest standard. The synchronous mode xmlhttprequest will also affect the performance of web page loading severely.
Considering the usage of synchronous xmlhttprequest in this integration implementation and this is the only feasible integration implementation, we have to separate wrapper page from the first content object.
Now we explain the reason why the implementation sketched by Figure 6-15 works. On the one side, the inline script (from line 3 to line 9) in content page will immediately starts executing when the browser starts to parse line 3. As the xmlhttprequest is in a synchronous mode, the browser will dedicate itself in executing the script from line 3 to line 9 (without being interrupted by any asynchronous events generated by line 2). When the browser finishes the execution
73 of the inline wrapper page script, the asynchronous events dispatched by the inline wrapper page script just start to execute. On the other side, as the wrapper page script is placed on the bottom of the first content object, the browser will not start fetching any external script in the first content object in line 2 (referred to by external script tags) before the inline wrapper page script finishes executing. This is important. External scripts are not influenced by same origin policy because external scripts embedded in the first content object belong to the same domain as the first content object even if the src property of the external script is a URL referring to a different domain. Only the external script embedded in another iframe will be influenced by the same origin policy. If a malicious external script starts executing before the inline wrapper page script finish executing, the malicious script may alter content objects or redirect the user to another page. In the implementation sketched by Figure 6-15, even if the external script is not restricted by the same origin policy, it can do nothing, because the inline wrapper page script would fetch and verify the external script object and replace the URL in that tag our verified external script content.
It is important that the function verifiedIntegrate() is invoked directly in line 7.
If we instead use the function as the handler of “window.onload”, the integration would not work.
There may be external script tags (e.g. ) in the content of first content object (line 2). According to the definition of “window.onload”, not only first content object has been loaded but also all objects included into the first
74 content object have been loaded when the load event of window fires. Therefore, when the handler (i.e. “verifiedIntegrate()”) of "window.onload" executes, external scripts in line 2 have already finished executing. Therefore, a potential malicious external script can alter content page before the handler of "window.onload" starts executing. The problem cannot be solved no matter where we put the inline wrapper script in the content page.
Last, we explore the question that can we put the inline wrapper page script at the head of the content page. The answer is no. There are file names of each content object and hash values of each content object in the inline wrapper page script. The inline script can fetch and verify each of the content web object from the peer just like that in the function “verifiedIntegrate()”. However, the inline script does not know how to assemble them into the integrated web object identified by the first content object, as the inline script is at the top of the first content object.
Specifically, when the browser executes the inline script, the browser has not parsed any HTML tags below the inline script. In other words, the only thing in the DOM structure during the execution of the inline script is the inline script itself. Therefore, even if the inline script has fetched and verified all content objects, it cannot assemble them into the integrated object identified by the first content object.
75 Chapter 7 Reliable accounting mechanism
In our system, the peer who helps to deliver the content will get a reward from the content provider based on the volume of content it delivers. Thus it is important that the system gives a correct and accurate description of how many bytes each peer delivers for the content provider.
7.1 The upper bound theory
The origin server keeps a log of user requests. The log records the user IP address, the to-be-assigned peer IP address and port number, and the name of first content object requested by the user. By knowing the name of first content object, the origin server is able to know how many bytes this user is able to get from the peer at most for this request.
As mentioned in Chapter 3, in case the user does not obtain the desired object successfully, due to causes like an object verification failure or the peer not responding, the user will do two things. On the one side, it aborts that web object and fetches it from a different port on the origin server. On the other side, the user sends a record to the “user_fail_handler.php” on port 80 in the origin server. The record includes the IP address and port number of the peer, the IP address of the user, and the name of that web object. By combining these records with the corresponding logs mentioned in the previous paragraph, the origin server could get a tight upper bound of how many bytes a peer could serve users during a given
76 period1. It is still an upper bound, not an accurate number, because it is possible that the user browser may have cached some or all of the web objects that he previously obtained. In this case, the peer did not actually deliver the upper bound number of bytes. If the peer later claims that the number of bytes it helps deliver is even larger than this theoretical upper limit, the system has enough reason to suspect the peer is malicious.
7.2 The records submitting mechanism
As we mentioned earlier, the peer collects records from users and uploads these records to the origin server in order to get the reward. In this paragraph, we explain why we adopt this method. Due to the P2P nature of our system, we have to rely on the user and/or the peer to report their behavior directly to the origin server so that the origin sever operator could get accurate accounting data. For a user, the behavior comprises information on whether the user gets the objects correctly and how large every object it gets from the peer is. For a peer, the behavior includes the data on how many bytes it served totally within a time period.
If we consider the scenario that a user itself uploads his behavior to the origin server, there are two ways for implementing this functionality. One is the user uploads his behavior immediately after he fetches the content page and all embedded objects from the peer. However considering the scalability of the system,
1 Our prototype implementation includes the user-side functionality to support accounting but not the actual accounting that would be performed by the origin server. In particular, the origin server has a dummy “user_fail_handler.php”, which does nothing.
77 we want the burden on the origin server (including computing and network burden) to be as small as possible. Thus this is not a good choice.
Another way to implement user self-reporting is as follows. For every transaction (the process that encompasses everything from user sending request to the origin server to when the user obtains all objects from the peer), the user stores his behavior information in the local disk, and after a certain period, he uploads all behaviors accumulated during this time period to the origin server on a subsequent visit to the origin site. However, this method cannot guarantee the user will submit these records because the user may never visit our site again.
Considering these factors, our design is to let the user send his behavior to the peer immediately after the user has obtained objects successfully. The peer will periodically upload accumulated user behavior records to the origin server so that the origin server will be able to know how many bytes this peer has helped to serve.
In the following paragraph, we call the user behavior information that the user sends to the peer “user-generated record” or just “record”. This record includes
IP address of user and how many bytes the user gets from the corresponding peer.
7.3 The integrity of records
If the user just sends his record in clear text, a malicious peer could modify the records to inflate number of bytes he served. One simple method against this attack is to use message authentication code. That is the user appends a cryptographic hash after the clear text, and the hash is calculated based on the clear text of the record and also a secret key shared between the user and the origin
78 server (which the origin server would give the user during the initial request). In this way, the peer cannot modify the record unless he learns the secret key.
However, it is easy for the peer to get this secret key by creating a puppet user. In other words, the peer can just use his personal laptop to send a request to the origin server. In this way, the malicious peer will get the secret key.
To solve this vulnerability, the origin server can assign a different secret key for different users based on their public IP address. This method works if the user and the peer do not share the same public IP address. Based on this fact, our strategy is that origin server will only give the user the address of the peer whose public IP address is different from the user.
The origin server generates a secrete key based on the pair of the user and the peer rather than only the user. We show a security vulnerability the system would have if the origin server generated the secrete key only based on the user.
Assume the peer colludes with a user who has a different public IP. Specifically, the colluding user sends a request to the origin server. The origin server gives back a peer IP address, which is not the colluding peer, but this would not prevent collusion. The colluding user is only interested in the secret key and he will email this secret key to the colluding peer, which resides in a different network. Once the peer gets the secret key, it can forge user records and cause reward inflation.
In the extreme condition, the same person controls the user and the assigned peer. In this case, the peer can forge records to inflate count of bytes it serves.
However, the origin server can use the upper bound mentioned above to limit the possible inflation. That is, by checking records uploaded by the peer against
79 corresponding user access logs stored in the origin, the origin server can find that the total sizes of all objects the peer has served for a specific user IP address is larger than that theoretical upper bound of bytes that the peer can serve to the user.
If that is the case, the origin server has enough reason to suspect that the peer and the user are malicious. By further monitoring this peer and this user, the origin server can make decision whether the user and the peer should be blocked.
In summary, the record a user sends to a peer is like this:
IPuser+sizeofObject+nonce+md5(secretKey+IPuser+sizeOfObject+nonce). Nonce is a number which will be used only once in all records. It is used to prevent replay attack, in which the malicious peer resends a valid record to the origin server multiple times. “sizeOfObject” is the sum size of all web objects fetched from the peer. Note that it is the actual amount of downloaded content (which does not include objects obtained from the local cache) instead of theoretical maximum size.
Actually, we could further minimize the scale of the user-and-peer collusion problem by using a cookie to identify the browser of a user. Without the cookie, we may only identify the public IP address of a potentially colluding user. And once we make sure the user is malicious, we can only quarantine all users behind that public
IP. With cookie to identify the browser, we could narrow down the scale of quarantine.
Specifically, the origin server can keep a database to store a table mapping from the public IP address to a list of cookies associated with this IP (Recall that there may be many users behind the same public IPs). Though the pair of colluding user and peer can forge real records with any fake cookie, the origin server would
80 detect the fake cookie in their forged real records. In this case, the public IP addresses of malicious peer and user would expose immediately. But we could not narrow down the scale to a specific browser because the cookie is invalid. On the opposite, if the pair of malicious peer and user forges real records with real cookie, they may not expose them immediately. When the origin server detects the count of a peer’s reported bytes exceed the theoretical upper bound, the origin server can suspect a user with a certain cookie, rather than suspect all users behind a public IP address. Thus the new record format will be as follows:
IPuser+sizeofObject+nounce+cookie+md5(secretKey+IPuser+sizeOfObject+nounce
+cookie)
In summary, modification within the record can be prevented to the extent that the size does not exceed the upper bound.
81 Chapter 8 Fetching web objects from the peer in parallel
8.1 The motivation for concurrency
In 6.1.3, we describe a Depth First Search (DFS) algorithm (i.e. function
“verifiedIntegrate()”) to recursively fetch and verify web objects from the peer.
There are two obvious drawbacks in that algorithm.
For one thing, to implement it in JavaScript and in the browser environment, we have to use synchronous mode “xmlhttprequest” instead of the default asynchronous mode. However, the synchronous mode is discouraged and deprecated in the latest “xmlhttprequest” standard. Thus, browsers may forbid synchronous usage of “xmlhttprequest” in the future. To see why we have to use synchronous mode “xmlhttprequest”, let us look back on the step 5 of function
“verifiedIntegrate()” in section 6.1.3. Here is the related sentence:
“For each URL retrieved, named Li (i ranges from 1 to n), fetch the corresponding web object. For each fetched web object, named Ai (i ranges from 1 to n), use the web object and corresponding URL Li as the input of function verifiedIntegrate().”
If the wrapper page used asynchronous mode “xmlhttpreqeust” to fetch web object from Li, “fetch the corresponding web object” will proceed to “use the web object and corresponding URL Li as the input of function verifiedIntegrate()” immediately without returning the web object Ai. Then the input of
“verifiedIntegrate()” is null. This behavior results from the nature of asynchronous
82 mode. In asynchronous mode, “xmlhttprequest” will let execute remote fetching in the background and immediately return null. Later when the fetched object comes,
“xmlhttprequest” will trigger a load event.
Another drawback of “verifiedIntegrate()” is that it is a single-threaded algorithm. As it is safe to have part of the algorithm run in multi-thread, there will be a performance boost if we modify the algorithm by running part of it in multiple concurrent threads. We use an example to illustrate this.
The tree in Figure 8-1 shows a tree of web objects from the peer. Node0 is the first content object from the peer. Node1, Node2, Node3 and Node4 do not depend on each other. It is safe to start running DFS on four sub-trees rooted at these objects in parallel. In the following sections in this chapter, we introduce the asynchronous, concurrent version of DFS for our system. A downside of parallel fetching of the subtrees is that if they embed a common object, this object will be fetched anew in each subtree. However, this cost is likely to be acceptable because subsequent fetches will be typically satisfied from the browser cache, and the benefits of concurrent object fetching from the peer outweighs this cost.
Our concurrent version of “verifiedIntegrate()” uses asynchronous mode of
“xmlhttprequest”, so that it makes the system compatible with current recommendations. We also describe the difficulties we met and compromises we made due to the limitations of JavaScript and the computing environment within the browser.
83 0
1
2 3 4
9 10 5 6 7 8
11 12 13
Figure 8-1 A tree of web objects from the peer
8.2 Theoretical concurrent DFS model
Our goal is to retrieve and verify the content of every node, in depth-first- search (DFS) like traversal and in a concurrent way. To achieve this goal, we run the algorithm (described in pseudo code) below on a tree model like above. We assume the parameter passing mechanism of all the functions below is pass by reference.
ConcurrentDFS (URL rootURL, LinkedList listFromParent){ curNodeContent=Fetch_verify(rootURL); /* this linked list needs to have internal synchronization mechanism to prevent unexpected result during multithread access */ LinkedList listForChild; LinkedList URLList=parseURL(curNodeContent); /* the ConcurrentDFS’s return value will be put into results */ Create_thread(thread0,ConcurrentDFS(URLList[0],listForChild)); Create_thread(thread1,ConcurrentDFS(URLList[1], listForChild)); Create_thread(thread2,ConcurrentDFS(URLList[2], listForChild)); …….. Create_thread(threadN,ConcurrentDFS(URLList[N], listForChild)); /* main thread has to wait the termination of every child thread before continue executing */ ThreadJoin(thread0); ThreadJoin(thread1); ThreadJoin(thread2); …
84 ThreadJoin(threadN); integrated_web_object=LinkReplace(curNodeContent, listForChild); listFromParent.sychronizedAdd(pair{rootURL, integrated_web_object}) return integrated_web_object; } Figure 8-2 ConcurrentDFS
We first define an abbreviation. Assume there is a tree consisting of a group of web objects, and the tree’s root is the web object W. W’s URL is w. We abbreviate
“integrated web object” of the tree whose root is W for “integrated web object” identified by W or “integrated web object” identified by w.
“ConcurrentDFS(URL rootURL, LinkedList listFromParent)” returns the
“integrated web object” identified by rootURL. Just before it returns, it store the
“integrated web object” identified by rootURL and rootURL itself in a linked list called listFromParent, provided by the caller of this function.
“Fetch_verify (rootURL)” fetches and verifies the content of the argument root URL through an http connection, and it returns the content of this URL. The fetching process via http is synchronous, i.e. “fetch_verify” will not proceed to verify or return before the fetching process completes.
“parseURL (curNodeContent)” returns the embedded URLs in the web object referred to by rootURL and store those URLs in a list called “URLList”.
The function “ConcurrentDFS” also creates a local variable “listForChild” to store a set of “integrated web object”, each of which is identified by a corresponding
URL in “URLList”.
“LinkReplace(curNodeContent,listForChild)” is used to generate “integrated web object” identified by “rootURL”. It replaces each of the URLs in
85 “curNodeContent” with the “integrated web object” identified by the corresponding
URL. Those “integrated web object” are stored in “listForChild”.
The ThreadJoin(thread0) function pauses the caller’s thread execution until the thread identified in its argument (thread0) finishes executing. As we run this function in Figure 8-2 N (N is the number of URLs in “URLList”) times sequentially, the caller’s thread will resume after all its child threads (thread0, thread1, thread2…threadN) finish.
The code following is a Java implementation of the above algorithm. Instead of using a linked list (“listForChild”) to store a set of “integrated web object”, the java code below uses an independent self defined class Tuple for each of “integrated web object” identified by child URLs (those in “URLList”). class Tuple { String url=null; String combinedRst=null; public Tuple(){} } class ConcurrentDFS implements Runnable { String curUrl; Tuple rst; ConcurrentDFS(String url, Tuple rst) { this.curUrl = url; this.rst=rst; } /* return the urls embedded in the content */ String[] parseIntoUrls(String content){...} /* initialize an http connection and fetch the url in a synchronous way */ String getContent(String url){...} /* see LinkReplace the pseudo code*/ String urlsReplacer(Tuple[] cldnRst, String content){...} boolean verify(String content){...} public void run() { String content=getContent(curUrl); if(!verify(content)){ //fetch “curUrl” from origin server directly
86 //put it in the string content }
String[] urls=parseIntoUrls(content); Thread[] myPool=new Thread[urls.length]; int i=0; Tuple[] cldnRst=new Tuple[urls.length]; for(String url:urls) { Tuple cldRst=new Tuple(); cldnRst[i]=cldRst; Thread t=new Thread(new ConcurrentDFS(url,cldRst)); myPool[i]=t; i++; t.start(); } for(i=0;i Theoretically, the above solution is valid for a user to fetch and verify web objects from a peer. But in practice, JavaScript itself (according to ECMAScript 6 specification) does not provide API for traditional multithreaded programming. However, there is an API in the “living standard” for HTML called “web worker”, which allows developers to do multithreaded programming in browsers. Here is an introduction for web worker: 87 “Web Workers provide a simple means for web content to run scripts in background threads. The worker thread can perform tasks without interfering with the user interface. In addition, they can perform I/O using XMLHttpRequest (although the responseXML and channel attributes are always null). Once created, a worker can send messages to the JavaScript code that created it by posting messages to an event handler specified by that code (and vice versa.).”[13] In practice, when implementing the idea above using web worker, we need to create a web worker inside another web worker (called nested web worker). To our knowledge, until June 2016, chrome does not support nested web workers (https://groups.google.com/a/chromium.org/forum/#!topic/blink- dev/5R3B4RN4GHU). A different approach is required. We describe the practical approach we developed in the next section. 8.3 The “Promise”-based implementation of ConcurrentDFS Back to the JavaScript language itself, it is encouraged to use asynchronous events and this enables programs to have limited concurrency. For example, in our case, we are using the following JavaScript code snippet to initialize http request to the peer and handle the response: function get(url) { var oReq = new XMLHttpRequest(); oReq.onload = function() { /* handle the oReq.response here */ } oReq.open("GET", url); oReq.send(); } 88 get(peer_url); Figure 8-4 An example of using xmlhttprequest The above code snippet initializes an http connection to peer_url and attaches a function to handle the incoming response. But the code will not pause and wait for response. Instead, the code will proceed executing (while the http fetch continues in the background) until the end of the whole program. After that, the program will execute the response handler function (the function attached to oReq.onload). If there are many events dispatched during the program execution, those corresponding event handlers will be put into an event queue and these event handlers will start executing one after another (without pre-emption) after the program finish executing. More details about event queue model could be found here: https://developer.mozilla.org/en-US/docs/Web/JavaScript/EventLoop If we try to implement the ConcurrentDFS (depicted in Figure 8-2) by using asynchronous events, the intuitive implementation would be like this: function get(url) { var integrated_web_object; var xhr=new XMLHttpRequest(); xhr.open("GET", url); xhr.onload=function(){ var content=xhr.response; var urls=urlParser(xhr.response); var i=0; var linkedlist=[]; for(i=0;i 89 return integrated_web_object; } Figure 8-5 The intuitive implementation However, this intuitive rewrite of ConcurrentDFS in JavaScript actually does not work. Function “get(url)” is expected to return “integrated web object” identified by “url”. Nevertheless, as xhr is in asynchronous fetching mode, “get” will return immediately and what it returns is null. If we set xhr in synchronous mode, then the script degrades to a single-threaded DFS based solution, that is “verifiedIntegrate()” introduced in section 6.1. In ConcurrentDFS, every child threads construct their own “integrated web object” in parallel, while the parent thread will proceed only if all child nodes finish fetching (that’s what “ThreadJoin” does in Figure 8-2). Though we cannot implement ConcurrentDFS by utilizing asynchronous XMLHttpRequest alone, we can implement ConcurrentDFS by using the combination of XMLHttpRequst API and Promise API in JavaScript. Below we introduce our JavaScript implementation of ConcurrentDFS, based on the assumption that readers have knowledge about Promise API in JavaScript or they have read this introduction article (https://developers.google.com/web/fundamentals/getting- started/primers/promises) [26] about JavaScript Promise API. The Figure 8-6 below shows our core part implementation of ConcurrentDFS in JavaScript by using XMLHttpRequest API and Promise API. Line 1. var container={} 2. function get(url) { 3. return new Promise(function(resolve, reject) { 90 4. var req = new XMLHttpRequest(); 5. req.open('GET', url); 6. req.onload = function() { 7. if(integrityVerify(url,req.response)){ 8. resolve(req.response); 9. } 10. else{ 11. console.log(url+" cannot pass verification!"); 12. reject("cannot pass verification!"); 13. } 14. }; 15. req.send(); 16. }).then(function(rst){return rst;}); 17. } 18. 19. function ConcurrentDFS(curUrl, content) 20. { 21. var dummyPromise=Promise.resolve(); 22. return dummyPromise.then(function(){ 23. var urls=urlParser(content); 24. var i=0; 25. var toDo=[]; 26. for(i=0;i 91 50. var rst; 51. rst=linkReplacer(content, cldCmdContents); 52. container[curUrl]=rst; 53. return rst; 54. }); 55. } 56. 57. get(rootUrl) 58. .then(function(childContent){return ConcurrentDFS(childContent);}) 59. .then(function(integrated_web_content){writeIntoDOM(integrate_web_conte nt);}) 60. .catch(function(words){console.log(words+" some error happend!");}); Figure 8-6 ConcurrentDFS in JavaScript Function get() will return a promise whose task is to asynchronously fetch content of the input “url”, a URL of a certain web object. One input of function ConcurrentDFS, called “content”, is the content of a web object referred to by a URL named “curUrl”, the other input of function ConcurrentDFS. Function ConcurrentDFS returns the integrated web content identified by “curUrl”. In the function ConcurrentDFS, function urlParser in line 23 parses the web object “content”. It retrieves the URLs embedded in “content” and returns the URLs as a list. For each URL in that list, function ConcurrentDFS creates a new promise, which aims to return the integrated web object identified by that URL. We put these promises in a list called “toDo”, defined in line 25. This list of promises is used as the input of Promise.all() API in line 48. Promises.all() will not return until each of the input promise has fulfilled or rejected. Promise.all() return a list of values, each of which is the return value of an input promise in the list “toDo”. Specifically, Promise.all() returns a list of integrated web objects, each of which is identified by a URL parsed by the function urlParser() before. The list of integrated 92 web objects returned is named “cldCmdContents” in line 49. “cldCmdContents” along with “content” is the input of function linkReplacer() in line 51. This function replaces each embedded URL in “content” (the argument in line 19) with the corresponding integrated web object identified by the URL. The function returns the integrated web object identified by “curUrl”. The wrapper page invokes the function get() and function ConcurrentDFS in the last four lines. The input “rootURL” in get() is the URL of first content object. After the last four lines finishes executing, the integrated web object identified by first content included web content is written into the DOM, and browser will start to load this final integrated web object. The motivation to implement ConcurrentDFS in JavaScript is that parallel connections reduce greatly the time to fetch all web objects from the peer while maintaining the integrity of those objects. To avoid fetching the same object twice, ConcurrentDFS keeps a map (defined in line 1 in Figure 8-6) to store what it has already fetched. Specifically, the map stores any integrated web object the program has built during the execution to the corresponding URL that identify that integrated web object. Thus whenever the function ConcurrentDFS encounters a URL already seen before, it directly retrieves the integrated web object identified by this URL from the map. From Line 29 to line 36 in Figure 8-6 is this process. Due to the uncertainty of network latency to fetch a web object, the cache hit rate will also be uncertain as the following example illustrates. Our illustration below is based on Figure 8-1. 93 Assume we start run ConcurrentDFS start from node0. After node0’s content has been fetched, the program will run ConcurrentDFS on node 1, node 2, node 3, node 4 in parallel. The latency to fetch each of these four node’s content is different. And even for the same object, the latency of fetching it would vary from time to time. Assume node 9 and node1 has the same URL. Then we hope to build the integrated web object identified by either node 9 or node 1, but not both of them because these two integrated web objects are identical. However, this cache hit may or may not happen. Indeed, suppose node 1 finishes building the integrated web object earlier than node3. Then, when the program parses content of node3 and get the url of node9, it will not fetch any nodes in the sub-tree rooted at node9 but retrieve the integrated web object identified by node 9 directly from the map (“container” in line 1). On the other side, if one promise in the program is still building but not finish the integrated web object identified by node 1, while another promise is checking the map, the promise would start building the integrated web object identified by node 9. In this way, the program winds up fetching same content from the remote peer multiple times. 94 Chapter 9 Performance evaluation We use three experiments to evaluate the performance of our infrastructureless CDN system from different aspects. We adopt the “promise” based script introduced in chapter 8 as the user side software. 9.1 Performance of the user-side script 9.1.1 The experiment setup and preliminary results The goal of our first experiment is to measure the performance overhead caused by the user side wrapper page only. Specifically, we measure and compare the execution time of two scenarios. In the first scenario, assume all content web objects are cached in the user’s browser. We measure the time of retrieving all of these objects from the cache directly. This scenario reflects the Web download as it is executed currently, while removing influence of network performance. In the second scenario, the content web objects are still cached in the browser, but the user accesses them through the corresponding wrapper page introduced in our system. The wrapper page is also cached in the browser, so this scenario also removes any effect of network performance, concentrating on the overhead due to the wrapper page. We measure how long the user fetches all content objects through the corresponding wrapper page. We use the following setup. The browser machine’s CPU is Intel i5 6267u (base 2.9GHz), with 8GB RAM. The browser is Google Chrome Version 55.0.2883.95 (64-bit). The content page we use is the home page of 360.cn (called “qihu.htm”), 95 which is ranked No.24 in Alexa global top 500 web site. The home page itself is 271KB and it has 135 embedded web objects, whose total size is 2.5MB. The table below shows the comparison results of measurements repeated eight times. In other words, we measure 8 times how long it takes Chrome to retrieve “qihu.htm”(and embedded objects) directly from the cache and render them, with the results presented in the second row. We measure 8 times how long it takes Chrome to retrieve “qihu.htm”(and embedded objects) through wrapper page and render those web objects, with the results presented in the third row. Number 1 2 3 4 5 6 7 8 Without 529 621 531 526 569 528 518 525 wrapper page With 1710 1810 1820 1830 1840 1740 1790 1920 wrapper page Unit: millisecond Table 9-1 with wrapper page versus without wrapper page The average time of row 2 is 543.375 milliseconds. The average time of row 3 is 1807.5. In general, it takes Chrome around 3.3 times (1870.5/543.375) longer to fetch and render the web page “qihu.html” in the scenario of with the wrapper page compared to the scenario without wrapper page. 9.1.2 Explanation on the overhead of the wrapper page In the scenario 1, the browser goes through two stages to finally render the web page “qihu.html”. We refer the first stage as the “fetching stage” and the second 96 as the “rendering stage”. The fetching stage ends when all web objects have been fetched from the local cache. We use the rendering stage to refer to the remaining time that the browser spends on rendering the whole page. Through Chrome DevTools [24], we can record when the fetching stage ends. We calculate the duration of rendering stage as the difference between the corresponding total time and corresponding fetching duration. The total time refers to one of the 8 repeated measurements listed in the second row in Table 9-1. The table below shows details about how the two components contribute to the corresponding total time for each of the eight repeated measurements. Number 1 2 3 4 5 6 7 8 AVG 199.37 Fetching 185 223 195 192 226 198 211 165 5 34.97 35.91 36.72 36.50 39.72 37.50 40.73 31.43 Ratio % % % % % % % % 36.69% Renderin g 344 398 336 334 343 330 307 360 344 65.03 64.09 63.28 63.50 60.28 62.50 59.27 68.57 Ratio % % % % % % % % 63.31% Total 543.37 time 529 621 531 526 569 528 518 525 5 Unit: millisecond Table 9-2 the decomposition of without wrapper page results Now we explore what components comprise the time that browser spends in the second scenario, i.e., with wrapper page. The first component (Component 1) is the time to fetch and load the wrapper page. According to Table 9-3 below, it accounts for a very small ratio of the total time. We refer to the execution of our script in the wrapper page as the second component (Component 2). For each web object that user browser fetches through the wrapper page, the web object itself goes through three phases. First, the web object is fetched from the cache. Second, 97 the web object is checked against the corresponding SHA-256 hash value. Third, each of the inline URLs in the web object is replaced with the integrated web object identified by the URL. Therefore, the second component comprises these three parts: fetching, verifying, and assembling. Once the wrapper page finishes constructing the integrated web object identified by the first content object, the wrapper page instructs the browser to render this final integrated web object. The time of rendering the final integrated web object is the third component (Component 3). In summary, we use the equation to express components of the total time in the scenario 2. Total time = fetch and render wrapper page + script execution duration + browser rendering duration The total times of the 8 repeated measurements for the scenario 2 are those listed in the third row in Table 9-1. The table 9-3 below shows how much each component accounts for the total time. Number 1 2 3 4 5 6 7 8 AVG Wrapper page fetch & load (Component 1) 24 25 24 25 25 25 26 28 25.25 Ratio 1 1.31% 1.36% 1.38% 1.40% 1.30% 1.38% 1.42% 1.47% 1.38% Script execution (Component 2) 516 467 479 462 561 544 511 569 513.625 28.20 25.38 27.53 25.81 29.22 30.06 27.92 29.79 Ratio 2 % % % % % % % % 27.99% Browser render(Compo 1294.87 nent 3) 1290 1348 1237 1303 1334 1241 1293 1313 5 70.49 73.26 71.09 72.79 69.48 68.56 70.66 68.74 Ratio 3 % % % % % % % % 70.63% Total 1830 1840 1740 1790 1920 1810 1830 1910 1833.75 Unit: millisecond 98 Table 9-3 the decomposition of with wrapper page results We obtained the duration of component 1 through Chrome DevTools [24]. We calculate the component 2 (script execution duration) by taking the difference of two timestamps in the script of wrapper page. One timestamp is placed at the beginning of the script. The other timestamp is placed at the end of the script, i.e. the timestamps indicates the end of assembling phase of the first content object. Component 3 is calculated as the difference between the total time and the sum of component 1 and component 2. If we compare the AVG column in Table 9-2 and the AVG column in Table 9-3, we have the following findings. First, the overhead caused by fetching and loading the wrapper page is very small. As in the table 9-3, it accounts for only 1.38% of the average total time for the scenario 2. Even compared with the average total time for the scenario 1, it accounts for about 4.65% (25.25/543.375). Second, in both of the scenarios, the browser rendering duration takes the majority share of the total time. We think the reason is most content page web objects (129/136) are image objects, and images are time consuming to render. Third, the total performance penalty imposed by our wrapper page on average is 1290.375 milliseconds, and the browser rendering stage contributes most of the penalty. The penalty contribution ratio of each component in scenario 2 is defined like this: [(the component duration in the scenario 2) – (the corresponding component duration in the scenario 1)]/ total penalty time. The total penalty is 1290.375 milliseconds. For component 1 in the scenario 2, there is no 99 corresponding component in the scenario 1. Therefore, we set the penalty contribution ratio of it as the quotient between component 1 in the scenario 2 and total penalty time, that is 2% (25.25/1290.375). The component 2 in scenario 2 corresponds to the component 1 in the scenario 1 since the script in the scenario 2 fetches and inlines the embedded objects. Thus, the penalty contribution ratio of component 2 (script execution) is around 24% ([513.625-199.375]/1290.375). The penalty contribution ratio of component 3 (browser rendering) is around 74% ([1294.875-344]/1290.375). The penalty of the component 2 (script execution) comes from three aspects. The script uses “xmlhttpreqeust” API to fetch web objects, which is not as efficient as the browser internal fetching mechanism. The script needs to calculate the hash value for every web object. That is the second source of performance penalty. The script also needs to assemble multiple web objects into an integrated web object. That is the third source of performance penalty for script execution component. We think the reason why the component 3 of the scenario 2 has a high penalty contribution ration has something to do with the data URI scheme. That is, because we store every web objects of the content page using the data URI scheme, this data representation scheme would add overhead for Chrome to render them. Considering most of the content web objects are images, and the data URI scheme of an image web object is base64 encoded character, it would entail a large amount of decoding work for the browser. 9.2 End-to-end Performance evaluation 100 We now measure the end-to-end performance of our system in two extreme scenarios. First, we consider a situation where the user and peer are in close proximity to each other (terms of network latency) and they are connected with a high-bandwidth network path as compared to the origin server. This scenario is the most beneficial to our system. Second, we consider a situation where the user and the peer are still close to each other geographically but the connection has higher latency and lower bandwidth. In this situation we do not expect our system to have high performance for an individual access, and its main benefit in this case is scalability to a large volume of accesses. To emulate the first scenario, we set the peer and the user in the same residential local area network. For the second scenario, we leave the peer on the residential network but deploy the user on a well-connected campus network. The origin server is in the same remote location in both scenarios. In both scenarios, we measure how long it takes the user browser to fetch and render the web page “qihu.htm” through our wrapper page. “qihu.htm” is introduced in the experiment in section 9.1 and is mirrored on our origin server. The ADNS (authoritative DNS server) and origin server share the same physical machine. The domain name of content provider is www.junbo.ipl.eecs.case.edu. The goal for the two parts experiment is to explore what components comprise the overhead of the entire system, and how much each component contributes to the overhead. 9.2.1 Part 1 101 The three tables below show the hardware and network environment of our test bed. The network latency in the third table is measured in RTT (round trip time) in milliseconds between two points. Hardware environment CPU Memory Location User Intel i7 4700MQ 8GB Columbus (my 2.4GHz apartment) Peer Intel i5 8GB Columbus (my 6267u(base apartment) 2.9GHz) Origin server Intel Xeon (up to 1GB Oregon 3.3 GHz) Table 9-4 Hardware environments 1 Network bandwidth Upload Download User Up to 1 Mbps (up to Up to 15 Mbps (up to 50Mbps within LAN) 50Mbps within LAN) Peer Up to 1 Mbps (up to Up to 15 Mbps(up to 50Mbps within LAN) 50Mbps within LAN) Origin server Average 70Mbps Average 70Mbps Table 9-5 Network bandwidth 1 Network latency (unit: millisecond) 102 To User Peer Origin server From User 0 3 97 Peer 3 0 97 Origin Server 88 88 0 Table 9-6 Network latency 1 The user and the peer are in the same Local Area Network and behind the same home router. The user sends a request to the origin server to fetch the wrapper page. It will fetch content web objects through the wrapper page. For the very first time, the peer does not have content web objects cached locally. Then the peer has to fetch everything from the origin server. We call this case “cache miss”. To enact this option, we use Chrome’s “disable cache” option, which causes Chrome to send all requests with the “no-cache” cache control header. On the contrary, if the peer has those resources stored locally, we call it “cache hit”. In the “cache hit” scenario, before the browser sends out the request for the wrapper page, we clear the cache of the browser. Note that this is different from disable cache, which does not clear the cache. In the “cache hit” scenario, the cache function is enabled so that the browser would not add no-cache header. Also, the origin server has a cache- control: max-age=86400 header. This setup guarantees every content web object the browser fetches is from the peer’s cache. We present our results in two dimensions. One is “cache hit” versus “cache miss”. The other is “Component 1” versus “Component 2” versus “Component 3”. These three components have the same definitions as that in the scenario 2 in 103 section 9.1. Component 1 refers to the time of fetching and loading wrapper page. Component 2 refers to the time of script execution, and Component 3 refers to the time of browser rendering. Note that Component 2 in this experiment includes the time of communications between user and peer. In this part, that communication is with the same LAN (Local Area Network). In the next part, that communication is between two different networks. The table below shows the results of this part. For each of the two situations “cache miss” and “cache hit”, we measured it five times repeatedly. Each data in the table below is the average data of the 5 measurements. The component data is calculated as the same method as that in the scenario 2 in section 9.1. Component 1 Component 2 Component 3 C ache miss 237.57 5360.43 1224.86 Cache hit 225.2 1175.4 1271.4 Units: milliseconds Table 9-7 Part 1 results The two values of component 1 are very close. So are those of component 3. It is reasonable because fetching and loading wrapper page has nothing to do with the cache of the peer. The duration of browser rendering is only related to the web objects themselves and so is irrelevant to the peer cache. The difference of Component 2 of the two situations is 4185.03 (5360.43- 1175.4) milliseconds. We think the difference mainly comes from two parts. The first part is the time of interactions between the peer and the origin server in the “cache miss” case. The other part is the time of disk writing and reading time in the peer’s machine. Specifically, In the “cache miss” scenario, after the peer fetches a 104 web object from the origin, the peer will first store the web object in the local disk before sending it to the user. That disk writing and reading time is the second part of the difference. We refer to this difference as the “cache miss” penalty below. We use the following method to prove our hypothesis about “cache miss” penalty above. We measure 5 times how long it takes the browser in the peer machine to fetch “qihu.htm” from the local web server, which is also running on the peer machine. The web server does not have the web objects cached the locally and needs to fetch the web objects from the origin server. In average, it takes the peer browser 4257 milliseconds to fetch all the web objects of “qihu.htm”. This peer browser indirect fetching time represent sum of the interaction time between the origin server and the peer, and the time of peer’s local disk writing and reading time. This time (4257) is close to the difference (4185.03) we mentioned above. Therefore, this method proves our hypothesis. In this paragraph, we try to explore how much each of the two parts of the “cache miss” penalty accounts for. In the “cache miss” case, the apache server on the peer side has to fetch web objects from the remote origin server in Oregon on the user’s behalf. Part of the relevant configuration file of the apache server on the peer’s side is shown below. It configures the maximum concurrent http connections that a process can handle. # WinNT MPM # ThreadsPerChild: constant number of worker threads in the server process # MaxConnectionsPerChild: maximum number of connections a server process serves 105 ThreadsPerChild 150 MaxConnectionsPerChild 0 The limit of concurrent connections is determined by statement “ThreadsPerChild 150”, which means the server on the peer side allows 150 concurrent connections per process. However, the chrome browser on the user side limits the concurrent connections to 6. The 6 concurrent connections restriction applies to both the “cache hit” and “cache miss” cases. No matter how many concurrent connections the wrapper page script tries to create, the browser would create up to 6 concurrent connections during the execution. Therefore, in the “cache miss” scenario, the bottleneck that determines time spent for interactions between the peer side’s apache server and the remote origin server is actually imposed by the interactions between interactions of the peer and the user browser. Given that the browser and the peer are in the same home network, the time the peer spends fetching objects from the origin on a miss should be close to the time the browser spends fetching the same objects from the origin, despite the potentially high concurrency the peer is configured to allow. We measure 5 times repeatedly how long it takes the browser to fetch “qihu.htm” from the origin server directly on the peer machine. On average, it takes 3528 milliseconds. We use the result as an approximation of how long it takes the apache server on the peer side to fetch contents from the origin server totally in the cache miss scenario. Thus, we can deduce that the time of disk writing and reading 106 time in the peer machine in case of “cache miss” is about 729 milliseconds (4257- 3528). 9.2.2 Part 2 In this part of the experiment, we set the user in the Ohio State University campus network. The peer is in a residential network. The peer’s location is within 3 miles of the user. As in part one, we measure how long it takes the user browser to fetch and render the content web objects through our wrapper page in both peer “cache miss” and peer “cache hit” scenarios. We repeat the measurement in each scenario 5 times. Before we show our results in Table 9-11, we show the details of our experiment environments. Hardware environment CPU Memory Location User Intel i7 4700MQ 8GB Columbus (OSU 2.4GHz main campus) Peer Intel i5 8GB Columbus (my 6267u(base apartment) 2.9GHz) Origin server Intel Xeon (up to 1GB Oregon 3.3 GHz) Table 9-8 Hardware information 2 Network bandwidth Upload Download 107 User Average 29 Mbps Average 86Mbps Peer Up to 2.3 Mbps Up to 15 Mbps Origin server Average 70Mbps Average 70Mbps Table 9-9 Network bandwidth 2 Network latency (unit: millisecond) User Peer Origin server User 0 36ms 68ms Peer \ 0 90ms Origin Server \ 88ms 0 Table 9-10 Network latency 2 Due to network security policy in the user’s network, we cannot “ping” the user from the peer or the origin server. As in the part 1, we present our measurement results in the three components form. Each data in the table below is the average value of the five repeated measurements. Component 1 Component 2 Component 3 Cache miss 157.88 11413.25 1252.63 Cache hit 144.71 9928.43 1235.43 Unit: millisecond Table 9-11 Part 2 results The component 1 values in the two scenarios are close, so is the values of component 3. The reason for component 1 is that the wrapper page is from the origin server, so the fetching and loading time has nothing to do with peer cache. 108 The values of component 3 are only relevant to the content web objects themselves; so the peer cache does not affect Component 3 either. If we compare the component 2 of “cache hit” in Table 9-11 with the “cache hit” component 2 in Table 9-7, we can observe the difference of them is huge (9928.43-1175.4==8753.03). Similar thing happens for the comparison “cache miss” components 2 in part1 and part2 (11413.25-5360.43==6052.82). This kind of huge difference is caused by the limited upload bandwidth of peer’s network (2.3 Mbps). The following flow chart gives a better illustration. Origin server 1 6 7 2 10 peer’s router 9 8 3 5 user 4 peer Figure 9-1 The flow chart in “cache miss” This flow chart describes the process of a user fetching an object from the peer in case of “cache miss”. In the “cache hit” scenario, step 5,6,7 and 8 do not exist. For part1 of this experiment, step 3,4,9 and 10 happens within the same LAN. For part2 of this experiment, the user is in a different network from the peer. The transmission rate of step 10 in part 1 is up to 50Mbps while it is up to 1Mbps in 109 part2. The transmission rate of step 3 also drops from up to 50Mbps in part1 to up to 29Mbps (the user upload bandwidth) in part2. However, as user’s requests sizes are small and 29Mbps is still large, step 3 makes no difference in part1 and part2 and thus makes no contribution to the huge penalty. If we quantify the time that step 10 takes, we can better understand why results in part 2 take much more time than corresponding results in part 1. We have around 2.5MB data to transmit in step 10 and the bandwidth is up to 2.3 Mbps. Theoretically, the lower bound of transmission time of step 10 is (2.5MB*1024kB/MB)/(2355/8)kB/s==8.7s. That is, the component 2 of part 2 in both “cache miss” and “cache hit” situations should be larger than 8.7s. 9.3 How content pages structure affects the performance of the wrapper page 9.3.1 The experiment design In this section, we explore how the structure of a content page affects the performance of our wrapper page. Specifically, we measure how long it takes the browser to fetch and render two different content pages through the same wrapper page. The two content pages shares the same set of web objects. However, the two content pages organize them in two different ways. In one content page, the content web objects are organized as a three-level tree, in which the second level has the same number of nodes as that of third-level. In the other content page, the web objects are organized as a degenerate tree, each parent in which has only one associated child. The wrapper page used in both scenarios is the one that is 110 implemented based on JavaScript Promise API. All content web objects are cached in the local disk in both scenarios. We give an example for each of the two content page used in our experiment. Both kinds of the content page contain 2 images and 2 iframes files. We also show their corresponding structure graph. Figure 9-2 and Figure 9-3 show the instances of a degenerate web page. Figure 9-4 shows the structure of Figure 9-2 and Figure 9-3. In Figure 9-4, a circle stands for an image and a square represents an HTML file. 111 Figure 9-4 The degenerate tree structure Figure 9-5, Figure 9-6 and Figure 9-7 show the instances of a three-level web page. Figure 9-8 shows the structure of Figure 9-5, Figure 9-6 and Figure 9-7. In Figure 9-8, a circle stands for an image and a square represents an HTML file.
Figure 9-6 twoLevel1.html
112
Figure 9-7 twoLevel2.htmlFigure 9-8 The three-level tree structure
9.3.2 The theoretical time complexity analysis
We first show our theoretical complexity analysis of the time that a browser fetches and renders a content page in the two scenarios. Then we show our real measurements result and compare it with our theoretical complexity analysis. In the scenario of three-level tree, the n in the table is the number of all nodes of the tree except the root. In the scenario of degenerate tree, the n is the number of all nodes in the tree. As defined in the section 9.1, component 1, 2 and 3 together make up the total time that a browser fetch and render a content page through the wrapper page.
Component 1 is the time of the fetching and loading the wrapper page. Component 2 is the script execution time. Component 3 is time of browser loading the content page.
113 Three-level tree Degenerate tree Wrapper page fetch & load O (1) O (1) Script execution O (n) O (n) Browser rendering O (n) O (n2)
Table 9-12 The Theoretical complexity analysis
It is obvious that the complexity of Component 1 in both scenarios is O (1), because the time of fetching and loading the wrapper page is constant.
We have mentioned in section 9.1 that Component 2 comprises three parts: fetching, verifying and assembling. For Component 2 of Three-level tree, the complexity therefore is O(1+1+n)=O(n). Specifically, the complexity of both fetching and verifying is 1. Indeed, as the browser fetches content web objects from the local disk cache, there is no restriction that browser can only use 6 concurrent connections to fetch objects. Thus, in the three-level tree structure, the browser can concurrently fetch and verify all the web objects in the second-level and then concurrently fetch and verify all the web objects in the third-level. In this case, the browser’s fetching and verify time complexity is proportional to the height of the tree, which is a constant. Thus, the complexity of fetching and verifying is O (1). In our JavaScript Promise based implementation of wrapper page, we assemble the final content page by replacing each embedded URL in the first content object with the corresponding integrated web object. The replacing goes through URLs one after another, not in parallel. Therefore, the assembling process complexity is O (n).
For Component 2 of Degenerate tree, the complexity is O (n/2+n/2+n). The height of the degenerate tree of n nodes is n/2. The complexity of the time that the browser concurrently fetching and verifying content web objects is the height of the tree. Therefore, the complexity of both fetching and verifying part is n/2. The
114 assembling part complexity is n. The reason is same as that in the Three-level tree scenario.
From the Chrome DevTools [24], we observe that in both scenarios, the browser renders each integrated web object in a level order, after the browser finishes building the final integrated web object. Note that each integrated web object identified by a node contains all web objects (in data URI scheme) in the sub tree rooted at that node. In the three-level tree scenario, the time complexity of level order rendering is n (the first level)+n (the second level)+n/2(the third level).
Therefore, Component 3 complexity in three-level tree scenario is O (n). In the degenerate tree scenario, the complexity of level order rendering will be n+(n-2)+
(n-4) + … + 2 and the sum of this arithmetic series belongs to O (n2).
9.3.2 The experiment results and the conclusion
In each scenario, we measure the total time and each component time with a different n. For each scenario, we plot two scattered points graphs. One graph is
Component 2 time versus n, and the other graph is Component 3 time versus n. We calculate a trending line for each of the graph. Also, we calculate the R squared value to measure how close our trending line is to the scattered points.
We first show the results for the degenerate tree scenario.
115 N 4 8 12 16 20 24 28 32 36 40 44 Total time 143 264 431 653 880 1570 1810 2370 2940 3880 4440 Wrappe r page fetch & load 14 17 15 17 16 20 30 20 21 20 27 Script executi on 83 86 122 140 157 195 196 232 259 289 333 Browse r renderi ng 46 161 294 496 707 1355 1584 2118 2660 3571 4080 Unit: millisecond
Table 9-13 Degenerate tree results
Degenerate structure component 2 350 y = 6.1273x + 43.127 300 R² = 0.98162 250
200 component2 150 trendline
100
50
0 0 10 20 30 40 50 Figure 9-9 Component 2 Versus N in the Degenerate tree Structure
116
Degenerate structure component 3
4500 2 4000 y = 2.2928x - 7.7138x + 49.661 R² = 0.99582 3500 3000
2500 component3 2000 trendline 1500
1000 500 0 0 10 20 30 40 50
Figure 9-10 Component 3 Versus N in the Degenerate tree Structure
We now show the results for Three-level tree Structure scenario.
N 4 8 12 16 20 24 28 32 36 40 44 Total time 179 227 299 325 400 452 508 572 641 655 774 Wrap per page fetch & load 23 21 18 19 21 19 20 20 21 20 20 Script execu tion 78 89 92 103 133 143 143 151 161 163 188 Brow ser rende ring 78 117 189 203 246 290 345 401 459 472 566
Unit: millisecond
Table 9-14 Three-level tree results
117
Three level structure component 2
200 180 y = 2.6341x + 68.055 160 R² = 0.95836 140 120 100 component2 80 trendline 60 40 20 0 0 10 20 30 40 50
Figure 9-11 Component 2 Versus N in the Three-level tree Structure
Three level structure component 3
600 y = 11.739x + 24.273 500 R² = 0.99 400
300 component3 trendline 200 100 0 0 10 20 30 40 50
Figure 9-12 Component 3 Versus N in the Three-level tree Structure
In the Figure 9-9, we see that the time of Component 2 in degenerate tree scenario grows linearly with respect to n, and the Figure 9-10 shows Component 3 grows in a quadratic order. These agree with our theoretical analysis in section
9.3.2. The Figure 9-11 and Figure 9-12 show that both Component 2 and
118 Component 3 in Three-level tree scenario grow linearly, which also agree with the previous theoretical analysis.
Finally, let us look back at our original question in this section: How content pages structure affects the performance of the wrapper page. Our two content page structures actually represent two extreme cases. The three-level tree is perfectly balanced while the degenerate tree is extremely unbalanced and close to a linked list. According to the theoretical analysis and experimental results, the tree structure makes little difference to the time complexity of Component 2. However, it influences the time complexity of Component 3 a lot. If the content page structure is closer to be perfectly balanced, the time complexity of Component 3 is closer to O
(n). Otherwise, if the content page structure is more unbalanced, the time
Complexity of Component 3 is closer to O (n2).
119 Conclusion
In this thesis, we introduce and implement a prototype of an infrastructureless content delivery system, in which the content provider hires peers to serve contents to nearby end users. By leveraging peers’ computing and network resources, the content provider does not need to employ an expensive third-party CDN company.
In our system, the content provider rewards a peer based on the count of bytes the peer helps serve in a period. The incentive mechanism is expected to help the content provider attract more peers.
Our system ensures the integrity of the web objects that a user fetches from the peer. We also design a reliable accounting mechanism to prevent potential count of bytes inflation caused by a malicious peer and a malicious user. While we achieve these goals, our system is transparent to users, as our system implements user–side functionality through a JavaScript file and does not need a user to install or configure any software. We do require a peer to sign up and install our software, but the software automates the configuration of peer’s machine and peer’s router.
To improve performance of page downloads, we leverage the Promise API in
JavaScript to enable the user fetching web objects from the peer concurrently.
In our performance evaluation of the system, we measure the overhead caused by user side implementation, how the structure of the webpage obtained from the peer influences the overhead of the user side implementation, and we make an end-to-end performance evaluation of the entire system.
120 Our proof-of-concept implementation demonstrates that scalable infrastructureless content delivery is feasible and can be made secure and easy to deploy. Our performance study shows that the main factor limiting performance is peer upload bandwidth. Thus, in today’s prevalent Internet environment, our current prototype achieves scalable content delivery at the expense of significant overhead for individual web accesses. Future work involves allowing the user to download web objects comprising the Web page from multiple peers concurrently, thus bypassing the peer upload bandwidth bottleneck.
121 Bibliography
[1]Paarijaat Aditya, Mingchen Zhao, Yin Lin, Andreas Haeberlen, Peter Druschel, Bruce Maggs, and Bill Wishon. 2012. Reliable client accounting for P2P-infrastructure hybrids. In Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation (NSDI'12). USENIX Association, Berkeley, CA, USA, 8-8.
[2] Cheng Huang, Angela Wang, Jin Li, and Keith W. Ross. 2008. Understanding hybrid CDN-P2P: why limelight needs its own Red Swoosh. In Proceedings of the 18th International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV '08). ACM, New York, NY, USA, 75-80. DOI=http://dx.doi.org/10.1145/1496046.1496064
[3] Hao Yin, Xuening Liu, Tongyu Zhan, Vyas Sekar, Feng Qiu, Chuang Lin, Hui Zhang, and Bo Li. 2009. Design and deployment of a hybrid CDN-P2P system for live video streaming: experiences with LiveSky. In Proceedings of the 17th ACM international conference on Multimedia (MM '09). ACM, New York, NY, USA, 25-34. DOI: http://dx.doi.org/10.1145/1631272.1631279
[4] Hussein A. Alzoubi, Michael Rabinovich, and Oliver Spatscheck. 2007. MyXDNS: a resquest routing dns server with decoupled server selection. In Proceedings of the 16th international conference on World Wide Web (WWW '07). ACM, New York, NY, USA, 351-360. DOI=http://dx.doi.org/10.1145/1242572.1242620
[5] Mingchen Zhao, Paarijaat Aditya, Ang Chen, Yin Lin, Andreas Haeberlen, Peter Druschel, Bruce Maggs, Bill Wishon, and Miroslav Ponec. 2013. Peer-assisted content distribution in Akamai netsession. In Proceedings of the 2013 conference on Internet measurement conference (IMC '13). ACM, New York, NY, USA, 31-42. DOI=http://dx.doi.org/10.1145/2504730.2504752
[6] Liang Zhang, Fangfei Zhou, Alan Mislove, and Ravi Sundaram. 2013. Maygh: building a CDN from client web browsers. In Proceedings of the 8th ACM European Conference on Computer Systems (EuroSys '13). ACM, New York, NY, USA, 281-294. DOI=http://dx.doi.org/10.1145/2465351.2465379
[7] https://en.wikipedia.org/wiki/Internet_traffic
[8] https://en.wikipedia.org/wiki/HTML5_video
[9] https://xhr.spec.whatwg.org
[10] https://en.wikipedia.org/wiki/Network_address_translation
[11] https://en.wikipedia.org/wiki/Data_URI_scheme
122 [12] https://en.wikipedia.org/wiki/Same-origin_policy
[13]https://developer.mozilla.org/en- US/docs/Web/API/Web_Workers_API/Using_web_workers
[14] Michael J. Freedman, Eric Freudenthal, and David Mazières. 2004. Democratizing content publication with coral. In Proceedings of the 1st conference on Symposium on Networked Systems Design and Implementation - Volume 1 (NSDI'04), Vol. 1. USENIX Association, Berkeley, CA, USA, 18-18.
[15] Sitaram Iyer, Antony Rowstron, and Peter Druschel. 2002. Squirrel: a decentralized peer-to-peer web cache. In Proceedings of the twenty-first annual symposium on Principles of distributed computing (PODC '02). ACM, New York, NY, USA, 213-222. DOI=http://dx.doi.org/10.1145/571825.571861
[16] Manal El Dick, Esther Pacitti, and Bettina Kemme. 2009. Flower-CDN: a hybrid P2P overlay for efficient query processing in CDN. In Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology (EDBT '09), Martin Kersten, Boris Novikov, Jens Teubner, Vladimir Polutin, and Stefan Manegold (Eds.). ACM, New York, NY, USA, 427-438. DOI=http://dx.doi.org/10.1145/1516360.1516410
[17] Heverson Borba Ribeiro, Lau Cheuk Lung, Altair Olivo Santin, and Neander Larsen Brisola. 2007. Web2Peer: A Peer-to-Peer Infrastructure for Publishing/Locating/Replicating Web Pages on Internet. In Proceedings of the Eighth International Symposium on Autonomous Decentralized Systems (ISADS '07). IEEE Computer Society, Washington, DC, USA, 421-428. DOI=http://dx.doi.org/10.1109/ISADS.2007.79
[18] Antony I. T. Rowstron and Peter Druschel. 2001. Pastry: Scalable, Decentralized Object Location, and Routing for Large-Scale Peer-to-Peer Systems. In Proceedings of the IFIP/ACM International Conference on Distributed Systems Platforms Heidelberg (Middleware '01), Rachid Guerraoui (Ed.). Springer-Verlag, London, UK, UK, 329-350.
[19] Jeff Terrace, Harold Laidlaw, Hao Eric Liu, Sean Stern, and Michael J. Freedman. 2009. Bringing P2P to the web: security and privacy in the firecoral network. In Proceedings of the 8th international conference on Peer-to-peer systems (IPTPS'09). USENIX Association, Berkeley, CA, USA, 7-7.
[20] Amit Levy, Henry Corrigan-Gibbs, and Dan Boneh. 2016. Stickler: Defending against Malicious Content Distribution Networks in an Unmodified Browser. IEEE Security and Privacy 14, 2 (March 2016), 22-28. DOI=http://dx.doi.org/10.1109/MSP.2016.32
[21] Mark Allman, Christian Kreibich, Vern Paxson, Robin Sommer, and Nicholas Weaver. 2007. The strengths of weaker identities: opportunistic personas. In Proceedings of the
123 2nd USENIX workshop on Hot topics in security (HOTSEC'07), USENIX Association, Berkeley, CA, USA, , Article 9 , 6 pages.
[22] https://en.wikipedia.org/wiki/BitTorrent
[23] http://upnp.org/specs/arch/UPnP-arch-DeviceArchitecture-v1.1.pdf
[24] https://developer.chrome.com/devtools
[25] https://www.w3.org/DOM/DOMTR
[26]https://developers.google.com/web/fundamentals/getting- started/primers/promises
124