INFORMATION DISSEMINATION AND ROUTING IN COMMUNICATION NETWORKS
DISSERTATION
Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the
Graduate School of The Ohio State University
By
Yingjie Li, M.S., B.S.
*****
The Ohio State University
2005
Dissertation Committee: Approved by
Ming T. Liu, Adviser Ten H. Lai Adviser Dong Xuan Graduate Program in the Computer and Engineering °c Copyright by
Yingjie Li
2005 ABSTRACT
Communication networks can be either infrastructure-based (e.g., the Internet) or infrastructure-less (e.g., wireless ad hoc networks). Information dissemination and routing are two important research areas in networking, as the first is one of the ultimate goals for constructing a network and the second provides the underlying support for disseminating information among nodes within the network.
This dissertation presents three major research results, with the first falling into the category of information dissemination on the Internet, specifically, server replica placement, and the second and the third into the category of information dissemination and routing in wireless ad hoc networks, respectively.
The World Wide Web is a popular mechanism that creates a client-server in- formation sharing model to facilitate information dissemination across the Internet.
With the explosive growth of the WWW, popular Web sites experience increasingly heavy workloads and a large percentage of Internet traffic. To alleviate the work- load and the network traffic, these sites usually place a set of servers geographically distributed across the Internet and replicate their contents to the selected replicas.
The server replica placement scheme addresses the problem of properly placing a set of replicas among sets of candidate sites to optimize system performance gain.
ii More specifically, using an optimum replica placement algorithm, we study the re- lationship between the replica size and the optimized performance gain through a trace-driven simulation. We also perform a parametric study to find out the effect of client demand patterns on the optimized performance gain growth.
The field of wireless ad hoc networks has become a prosperous research field in re- cent years thanks to the rapid development and popularity of various mobile devices.
The lack of infrastructure allows a fast and inexpensive deployment of wireless ad hoc networks and, thus, makes them suitable for surveillance applications. However, the ad hoc networks’ flexibility and convenience do come at a price. The infamous characteristics of these networks, such as unstable wireless media, dynamic network topology due to link disruptions and node mobility, and power constrained wireless devices, require protocols designed for these networks to be fully distributed, energy efficient, fault (disruption) tolerant, and scalable.
Following these design rules, we propose a Scalable Data Storage and Retrieval
Service (SDSR), to enhance data dissemination for large-scale wireless ad hoc net- works, and a Disruption Tolerant Geographic Routing protocol (DTGR) to enhance the performance of geographic routing schemes in the presence of frequently occur- ring disruptions for wireless ad hoc networks.
iii This is dedicated to my parents Mr. Tao Li and Mrs. Yunmei Song
iv ACKNOWLEDGMENTS
I would like to express my sincere gratitude to several people who have helped me throughout my work with this dissertation. Without their guidance, support, and encouragement, this work would never have been finished.
Foremost, I would like to thank my adviser Professor Ming T. Mike Liu for guid- ing me and inspiring me to pursue a deeper understanding of the field of networking, for providing invaluable comments during my work with this dissertation, and for his warm hands during my educational and personal life. Without him, I would never have gone this far.
I would like to give special thanks to Professor Ten H. Lai, who has engaged me in numerous discussions, which stimulated and deepened many of my thoughts in this dissertation. I am also indebted to him for his continuous help and encouragement during my graduate study, especially during the difficult times.
I am very grateful to Dr. Dong Xuan for his inspiring discussions during my work with this dissertation and for serving as a member of my dissertation committee.
And it is my pleasure to offer my gratitude to Professor Mervin Muller for his helpful suggestions on this work. Thanks also to my fellows Dr. Min-Te Sun and Junmo
Yang for their collaboration, and to Dr. Chou-Yu Chiang and Dr. Chunlei Liu for
v their enjoyable discussions, helpful suggestions, and abundant support. Their co- work and comments have greatly honed and facilitated this dissertation. I also thank
Dr. Ming Liu and Dr. Quan Liu for their important discussions and comments.
Last but never least, my dear father Tao Li, mother Yunmei Song, and sisters
Weijie Li and Junjie Li, please accept my whole-hearted thanks for always being beside me, caring for me, and understanding me. It is your love and support that have led me through the peaks and valleys in my life and that have always motivated me to work harder and do better during my research journeys.
vi VITA
July 10, 1974 ...... Born - Yantai city, PR China
1995 ...... B.S. Computer Science and Telecom- munication Qingdao University Qingdao, Shandong PR China 1995-1998 ...... Software Engineer Yantai Stock Transaction LLC Yantai, Shandong PR China 1998-1999 ...... University Fellowship The Ohio State University Columbus, OH USA 2000 ...... M.S. Computer and Information Sci- ence The Ohio State University Columbus, OH USA June-Sep. 2003 ...... IBM T. J. Watson Research Center Hawthorne, NY USA 1999-2004 ...... Graduate Teaching Associate Department of Computer Science and Engineering The Ohio State University Columbus, OH USA Aug. 2005-present ...... Assistant Professor Department of Information System Sorrel College of Business Troy University Troy, AL USA
vii PUBLICATIONS
Research Publications
Ying J. Li, M. Sun and Junmo Yang. “DTGR: Disruption Tolerant Geographic Routing for Wireless Ad Hoc Networks,” Proc. of the International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS) 2005, pp. 734-743, Philadelphia, USA, July 2005.
Ying J. Li and Ming T. Liu. “SDSR: A Scalable Data Storage and Retrieval Service for Wireless Ad Hoc Networks,” Proc. of the International Conference on Informa- tion Networking (ICOIN) 2005, pp. 499-509, Jeju, Korea, January 2005.
Ying J. Li and Ming T. Liu. “Optimization of Performance Gain in Content Dis- tribution Networks with Server Replicas,” Proc. of the IEEE/IPSJ International Symposium on Applications and the Internet (SAINT) 2003, pp. 182-189, Orlando, Florida, January 2003.
Ying J. Li, C.-Y. Chiang, and Ming T. Liu.“Efficient Web Caching for GPRS net- works,” Proc. of the International Conference on Computer Networks and Mobile Computing (ICCNMC) 2001, pp. 85-90, Beijing, China, August 2001.
C.-Y. Chiang, Ying J. Li, Ming T. Liu, and Mervin E. Muller. “On Request For- warding for Dynamic Web Caching Hierarchies,” Proc. of the International Con- ference on Distributed Computing Systems (ICDCS) 2000, pp. 262-269, Taipei, Taiwan, April 2000.
viii FIELDS OF STUDY
Major Field: Computer Science and Engineering
Studies in: Computer Networking Prof. Ming T. Liu Computer Architecture Prof. Mario Lauria Computer Graphics Prof. Han-Wei Shen
ix TABLE OF CONTENTS
Page
Abstract ...... ii
Dedication ...... iv
Acknowledgments ...... v
Vita...... vii
List of Tables ...... xiv
List of Figures ...... xv
Chapters:
1. Introduction ...... 1
1.1 Overview ...... 1 1.2 Web Access ...... 3 1.2.1 Server Replica Placement for CDN ...... 4 1.3 Information Dissemination and Routing in Wireless Ad Hoc Networks 7 1.3.1 Data Storage/Retrieval Service for Wireless Ad Hoc Networks 8 1.3.2 Disruption Tolerant Wireless Ad Hoc Routing ...... 12 1.4 Organization of the Dissertation ...... 16
2. Background ...... 18
2.1 Information Dissemination Systems ...... 18 2.1.1 Client-Server-Based Web Access ...... 20
x 2.1.2 Peer-to-Peer-Based Systems ...... 25 2.2 Information Dissemination in Wireless Ad Hoc Networks ...... 26 2.3 Routing in Wireless Ad Hoc Networks ...... 29 2.3.1 Topology Based Routing Protocols ...... 31 2.3.2 Position Based Routing Protocols ...... 32 2.4 Greedy Perimeter Stateless Routing Protocol (GPSR) ...... 34 2.4.1 Routing under Network Disruptions ...... 37
3. Optimization of Performance Gain in Content Distribution Networks with Server Replicas ...... 42
3.1 Modeling of Server Replica Placement ...... 43 3.1.1 Minimum P-median Graph Problem ...... 44 3.1.2 Placement Algorithms ...... 44 3.2 Trace-Driven Simulation ...... 45 3.2.1 Network Model ...... 46 3.2.2 Aggregation of Clients’ Demands into AS Domains . . . . . 46 3.2.3 Construction of AS Level Topology ...... 51 3.2.4 Simulation Results ...... 58 3.3 Parametric Analysis ...... 61 3.4 Chapter Summary ...... 62
4. SDSR: A Scalable Data Storage and Retrieval Service for Wireless Ad Hoc Networks ...... 65
4.1 Preliminaries ...... 66 4.2 SDSR Service ...... 68 4.2.1 Hierarchical Partitioning ...... 69 4.2.2 Data Storage Server Selection ...... 70 4.2.3 Multicast Storage Tree Routing ...... 73 4.2.4 Data Query Algorithm ...... 79 4.2.5 Data Update Algorithm ...... 81 4.2.6 Fault Tolerance ...... 82 4.3 Analytical Results ...... 84 4.3.1 Communication Cost ...... 84 4.3.2 Resilience to Clustering Failures ...... 88 4.3.3 Data Placement Consistency Maintenance Cost ...... 88 4.4 Simulations ...... 89 4.4.1 Varying QSR ...... 90
xi 4.4.2 Varying Number of Nodes N ...... 91 4.4.3 Varying Replication Depth d ...... 92 4.5 Chapter Summary ...... 96
5. Disruption Tolerant Geographic Routing for Wireless Ad Hoc Networks . 97
5.1 Motivation and Problem Statement ...... 98 5.1.1 Motivation ...... 98 5.1.2 Problem Statement ...... 99 5.2 Reachability Value Computation Scheme ...... 101 5.3 Application: GPSR ...... 103 5.3.1 Adverse Effects of the Incorrect Removal of a Reachable Neighbor from the Neighbor Table ...... 103 5.3.2 Disruption Tolerant Geographic Routing (DTGR) ...... 106 5.3.3 DTGR Packet Forwarding Procedure ...... 106 5.3.4 Schemes for Forwarding a Packet to a Selected Unstable Neighbor ...... 112 5.4 Simulations ...... 113 5.5 Cost Effectiveness Discussions ...... 123 5.6 Chapter Summary ...... 126
6. Summary and Future Work ...... 128
6.1 Dissertation Summary ...... 128 6.2 Future Research Directions ...... 132
Appendices:
A. Abbreviations ...... 136
B. Simulation Models ...... 139
B.1 NS Package ...... 139 B.1.1 NS Basic Structure ...... 140 B.1.2 Agent ...... 142 B.1.3 Mobile Node ...... 142 B.1.4 Wireless Trace Format ...... 144 B.2 DTGR Design and Implementation ...... 144 B.2.1 DTGR Packet Format and Packet Header Data Structure . 146
xii B.2.2 DTGR Routing Agent ...... 148 B.2.3 DTGR Neighbor Table Entry Structure ...... 150 B.2.4 Timers ...... 152 B.3 DTGR Simulation ...... 153 B.3.1 NS Multistate Error Model and TCL Configuration Variables153 B.3.2 Simulation Run: An Example ...... 153 B.4 Summary ...... 163
Bibliography ...... 164
xiii LIST OF TABLES
Table Page
3.1 Performance gain versus |K| ...... 59
3.2 Threshold |K| versus α ...... 63
4.1 Comparison of the performance of different storage routing schemes in routing packet from s to its storage locations shown in Figure 4.2 . 77
4.2 Notations ...... 84
4.3 Performance metrics ...... 85
4.4 Total communication cost generated by each scheme ...... 87
4.5 Hotspot usage ...... 87
4.6 Total messages generated when N = 2500, M = 100, Q = 1000, and d =4...... 87
xiv LIST OF FIGURES
Figure Page
2.1 GPSR packet routing example, where S is the source node and D is the destination node. The solid line indicates that the packet is forwarded using greedy forwarding, and the dotted line indicates that the packet is forwarded using perimeter routing...... 34
2.2 GPSR protocol parameter definition ...... 40
2.3 GPSR packet forwarding procedure ...... 41
3.1 The cumulative distribution of reply bytes received by each AS . . . . 48
3.2 Aggregated content traffic (in bytes) ...... 49
3.3 Aggregated content delivery cost (in bytes AS hops) ...... 50
3.4 AS pair distribution by shortest path length ...... 52
3.5 The average path length from each AS to other nodes ...... 53
3.6 The number of direct connections from each AS to other nodes . . . . 54
3.7 The average shortest path length from the original server to other ASs within certain rank range (received reply bytes) ...... 56
3.8 The average shortest path length between the top 1% ASs and the ASs within certain rank range ...... 57
xv 3.9 Optimized total content delivery cost versus number of server replicas (|K|)...... 59
3.10 Server replicas’ AS ranking (in received reply bytes) versus chosen order ...... 60
3.11 Variation of cost with |K| and α ...... 62
4.1 Hierarchical partitioning of the network into four levels. Bold lines show each level-i grid that node s belongs to ...... 71
4.2 Example of SDSR hierarchical storage location selection, multicast routing tree construction, and query location selection. The solid circles are the storage locations selected by source node s, the dotted circles are the query locations selected by requester node r ...... 72
4.3 Example of the shortest path tree constructed from root s to the selected storage locations of the bottom left grid of the network shown in Figure 4.2 ...... 74
4.4 Example of the minimum spanning tree constructed from root s to the selected storage locations of the bottom left grid of the network shown in Figure 4.2 ...... 75
4.5 Total message overhead with different QSR values ...... 91
4.6 Total message overhead as the number of nodes increases, where QSR=10 ...... 92
4.7 Storage overhead for different hierarchical partitioning levels . . . . . 93
4.8 Query overhead for different hierarchical partitioning levels ...... 94
4.9 Total message overhead for different hierarchical partitioning levels . 95
5.1 Dropping of a packet destined for D due to the removal of a reachable neighbor j from i’s neighbor table ...... 104
xvi 5.2 Inefficient perimeter routing of a packet destined for D due to the removal of a reachable neighbor j from i’s neighbor table ...... 105
5.3 DTGR protocol parameter definition ...... 109
5.4 DTGR greedy forwarding procedure ...... 110
5.5 DTGR perimeter routing procedure ...... 111
5.6 Packet delivery success ratio under different node failure durations in a static network ...... 115
5.7 Packet delivery latency under different node failure durations in a static network ...... 116
5.8 Packet delivery success ratio under different node failure durations in a mobile network where the maximum node speed = 30 m/s . . . . . 117
5.9 Packet delivery latency under different node failure durations in a mobile network where the maximum node speed = 30 m/s ...... 118
5.10 Packet delivery success ratio in sparse, highly mobile networks . . . . 120
5.11 Packet delivery latency in sparse, highly mobile networks ...... 121
5.12 Packet delivery success ratio in stable networks ...... 123
5.13 Packet delivery latency in stable networks ...... 124
B.1 C++ and OTcl: the duality ...... 141
B.2 Simplified user view of NS ...... 141
B.3 Mobile node structure ...... 143
B.4 Wireless trace format ...... 145
B.5 DTGR packet format ...... 146
xvii B.6 DTGR temporary disruption procedure simulated by the NS multi- state error model ...... 154
B.7 DTGR TCL configuration variables used in the simulation script . . . 155
xviii CHAPTER 1
INTRODUCTION
1.1 Overview
Communication networks can be either infrastructure-based or infrastructure- less. Infrastructure-based networks are those that require certain existing network infrastructures, such as wire lines, base stations, etc., for their operation. The largest
Infrastructure-based communication network is the Internet, which is composed of billions of computers all over the world.
Infrastructure-less networks are those that require no infrastructure for their op- eration. Such networks are usually formed dynamically by a collection of network devices through wireless links, thus they are also referred to as wireless ad hoc net- works. The deployment of wireless ad hoc networks can be both fast and inexpensive due to the fact that a fixed infrastructure is not required. The fast and inexpen- sive deployment of wireless ad hoc networks has made infrastructure-less networks one of the best candidates for surveillance applications, such as rescue missions and battlefield communications.
1 Information (data) dissemination is an important issue in networking, since it is one of the ultimate goals of constructing networks. Here the information can be related to news, multimedia files, service advertisements, environmental conditions, and local events such as the presence of an object, etc.
Efficiently sharing information with other nodes in a network becomes a challenge as the network size/traffic increases. In wireless ad hoc networks, this is an even greater challenge because of the energy constraints of wireless devices as well as the frequent topology changes.
Besides information dissemination, routing is another important issue in net- working, as it determines the path along which information is transmitted from a source to a destination. In the Internet, a routing function is performed by desig- nated routers, while in wireless ad hoc networks, each node has to act as a router and forward data packets for other nodes that are out of each others transmission range. Thus the characteristics of wireless ad hoc networks necessitate and challenge the design of efficient and fault-tolerant routing protocols.
This dissertation addresses three major research problems, with the first falling into the category of client-server-based information dissemination, in particular,
Web access in wired Internet, the second into the category of peer-to-peer-based information dissemination protocol design in wireless ad hoc networks, and the third into the category of disruption tolerant routing protocol design in wireless ad hoc networks.
2 The purpose of this chapter is to set the foundation for presenting the above mentioned research topics. To begin with, the motivation and our contributions re- garding information dissemination in wired Internet are presented, followed by the characteristics of wireless ad hoc networks and the common requirements shared by all protocols designed in these networks. Based on these requirements, our contri- butions in designing information dissemination and routing protocols in wireless ad hoc networks are summarized. Lastly, the layout of this dissertation is presented.
1.2 Web Access
The Internet, also nicknamed the “information highway,” has grown and become the largest and most widely accepted medium for disseminating information among globally distributed network devices. Although not entirely, most of the credit for the popularity of the Internet should be accredited to the World Wide Web (WWW or Web) [96], which provides Internet users (clients) with easy access to all sorts of information available from the Internet.
The exponential growth rate of the Web has led to a significant increase in
Internet traffic, the workload of hot Web servers, as well as the dramatic degrada- tion in user perceived latency while accessing popular Web sites. There have been significant research efforts on enhancing web performance, e.g., reducing Internet bandwidth consumption and improving user-perceived latency.
Two important techniques for enhancing Web performance are caching and repli- cation. Caching is usually performed by caching proxies to reduce client-perceived
3 request latency as well as network communication costs. Replication is usually per- formed by Content Distribution Networks (CDNs) composed of server replicas to reduce the load on the original servers and the network.
Much research has been done on different aspects of caching and replication, such as client request forwarding [66] [32], cache replacement policy [97] [39] [11] [71] [76], inter-replica or inter-proxy communication [96] [77], object distribution [12] [52], etc.
However, less has been done on the proxy/replica placement. In this dissertation, we present our research results on the study of the server replica placement problem for enhancing CDN performance gain. The motivation and contributions of our research on server replica placement are presented in the following subsection.
1.2.1 Server Replica Placement for CDN
Motivation. CDNs aim to speed up the delivery of Web content and to reduce the load on the original servers and the network. A CDN, such as Akamai [3] or
Digital Island [40], deploys a set of servers throughout the Internet. A busy original server can sign up for the service and have some or all of its content replicated across the deployed servers. Clients requests for a certain content item are transparently directed to and served by the “good” replica. The notion of “good” usually means that the item can be returned quickly compared to the time it would take to retrieve it from the original server, with appropriate integrity and consistency [21].
To efficiently deliver content on behalf of the original server, a CDN needs to decide where the replicas should be placed and how many replicas are needed. Server replica placement of a CDN addresses the problem of properly placing a set of
4 replicas across the Internet to optimize the CDN performance gain. The CDN performance can be measured by client request latency, total network bandwidth consumption, or an overall cost function if each link is associated with a cost. How much performance improvement is achieved depends on the placement strategy, the size of the replicas and the traffic pattern.
Previous studies on server replica placement problems have focused on compar- ing the performance of different placement strategies on the optimization of certain performance metrics [12] [45] [72]. However, little has been done to study the per- formance gain growth as the number of replicas increases. Moreover, according to the best of our knowledge, no research has been done to study the effects of clients’ content demand and distribution patterns on the growth of performance gain.
Contributions. In [57], in addition to presenting several placement algorithms, we study the server replica placement on the optimized performance gain growth.
Specifically, we consider the following scenario: a CDN wants to place some replicas across the Internet beforehand on behalf of a busy Web site to minimize the overall content delivery bandwidth consumption in the network and to reduce the average delay to the clients. The problem is to find out how much the bandwidth consump- tion can be reduced as we increase replica size one by one under a fixed near optimal placement strategy.
Our study is important because, although a good placement strategy can further improve system performance with a larger number of replicas, increasing the number of replicas also results in higher replication cost (e.g., the total bandwidth consumed
5 to keep replicas consistent), which is proportional to the number of replicas. One would be willing to pay the cost associated with a large number of replicas only if it is outweighed by the performance gain.
We study the static server replica placement problem. Given a fixed replica size
|K|, we model the replica placement as a well-known NP complete Minimum P- median graph problem [28]. We then present several heuristic placement algorithms to solve the placement problem. We use a near optimal greedy algorithm to establish the relationship between the replica size and the optimized performance gain.
Many studies have been performed to analyze client access patterns through traces generated by proxy servers. These studies are useful in designing Web caching and pre-fetching algorithms to improve the performance of caching proxies. How- ever, the limitation of proxy traces in providing client access information seen from the server side makes them unsuitable in studying server replica placement prob- lems. We perform a trace driven simulation using traces generated by one specific original server and compute the threshold value of replica size required to achieve certain performance gain.
Previous work [42] [72] studied the replica placement problem on the basis of certain client demand and distribution patterns, but the effects of different pat- terns on the performance gain growth have not been studied. As these patterns are important in determining the threshold value of replica size for achieving a cer- tain performance gain, we perform a parametric study to establish the relationship between performance gain and client demand factor.
6 In addition, to improve the performance perceived by clients, replicas are usually chosen from a large number of locations where clients’ demands are highly aggre- gated. By observation we find that placing replicas in a small number of carefully chosen candidate sites (considering both client demand and the site’s outdegree) provides the same performance gain as placing replicas on all candidate sites while introducing less computation cost.
1.3 Information Dissemination and Routing in Wireless Ad Hoc Networks
The lack of fixed infrastructures allows a fast and inexpensive deployment of wireless ad hoc networks. However, the following characteristics of these networks complicate and challenge the deployment of such networks:
1. The lack of fixed infrastructures also indicates that there is no centralized
coordination or administration. Thus, in such networks, each node has to
operate in a distributed manner, acting as an independent router, generating
independent data, and taking part in network management;
2. In wireless ad hoc networks, nodes usually can move arbitrarily, which results
in frequent link disruptions, network topology changes, and possible packet
losses;
3. In wireless ad hoc networks, nodes communicate via wireless links, which are
unreliable and prone to disruptions and errors; and
7 4. Wireless devices usually have limited computing and battery power, which
limits services and applications that can be supported by each node, and
poses challenges in network scalability.
The above wireless ad hoc network characteristics require protocols for wireless ad hoc networks to be:
• fully distributed
• fault (or disruption) tolerant
• energy efficient
• scalable
Our research in wireless ad hoc networks covers the design of SDSR, a Scal- able Data Storage and Retrieval Service for enhancing information dissemination in large-scale wireless ad hoc networks, and the design of DTGR, a Disruption Toler- ant Geographic Routing Protocol for enhancing the performance of position-based routing protocols. The motivation and research contributions are summarized as follows.
1.3.1 Data Storage/Retrieval Service for Wireless Ad Hoc Networks
Motivation. Wireless ad hoc networks are constructed for sharing information
(data) among wireless hosts. The energy constrained wireless devices as well as the frequent network topology change make traditional information-sharing schemes
8 such as broadcasting and storing data centrally (CS) or locally (LS) undesirable, and thus necessitate and challenge the design of efficient and robust data dissemination schemes for large-scale wireless ad hoc networks.
Previous research on data dissemination schemes such as [34] [35] [92] [49] mainly focused on improving data accessibility/reliability for wireless ad hoc networks with high frequency of network partitions. However, they have done less for achieving data access scalability and efficiency for wireless ad hoc networks with reasonable node density and few occurrences of network partitions.
Contributions. In [58], we consider the problem of storing/retrieving data in a wireless ad hoc network in a scalable, efficient, and robust manner, where data can be related to service advertisements [13], files, locations of nodes, and local events like the presence of an object, etc. Such a network can be constructed either by personal devices such as notebooks or PDAs in a wide area civilian environment, or by military devices in a large battlefield. In our research, we assume a peer-to-peer system architecture where each node can act both as a server and a client and can store/retrieve data to/from the network.
We present a distributed and Scalable Data Storage and Retrieval service, SDSR, as a solution to the problem addressed above. SDSR is data-centric and location- based, it hashes each data key to normalized geographical coordinates (x,y), where x, y ∈ [0,1]. The normalized coordinates can be seen as a rendezvous point for storing/retrieving the data item in a unit grid.
9 SDSR achieves scalability, robustness, and load-balancing by partitioning the network into hierarchical grids of increasing size and replicating the data item into each hierarchical grid. The storage location in each grid is determined by using the normalized coordinates as an offset and scaling it to the corresponding grid size.
SDSR utilizes Greedy Perimeter Stateless Routing (GPSR) [48] to store the data item to nodes closest to the storage locations. These nodes serve as storage servers for the data item. SDSR retrieves a data item hierarchically, similar to storing the data item.
SDSR has several notable properties, which are listed below:
1. Scalability: (i) the number of storage servers for a data item increases as a
logarithm of the increase of network size; (ii) the per node data storage and
communication costs increase as a small fraction of the increase of the network
size; (iii) the task of storing and serving data items inserted into the network
is distributed among nodes in the network, thus avoiding bottlenecks.
2. Fault-tolerance: (i) it handles node joins, node fail-stops and node mobility
locally (by employing an indexing service); (ii) it is resilient to clustering
failures (nodes in a region fail simultaneously), since it replicates each data
item at multiple nodes which are distributed in different network regions.
3. Efficient utilization of network resources: (i) it constructs a multicast
routing tree to route the data item to the storage locations, which performs
better than rooted shortest path and minimum spanning trees in terms of the
combination of communication cost, network delay, and data delivery success
10 probability; (ii) it employs a lightweight indexing service for maintaining the
consistent view of the data storage and avoiding unnecessary broadcast in a
grid.
4. Query locality friendly: queries in networks usually exhibit locality [50]
[26], in the sense that nodes are more interested in data generated nearby
than data generated far away. Thus, to comply with query locality, a data item
should be stored near its source; moreover, to increase the data accessibility
and to avoid bottlenecks, multiple such storage servers should be selected for
storing the data item. SDSR stores each data item in such a way – it places
more storage servers near where most queries are generated and fewer far away.
5. Traffic localization: the hierarchical structure of storage servers for a par-
ticular data item constructed by SDSR guarantees that a query message for
a data item is propagated locally within a quadrant of the smallest grid con-
taining both the requester node and the source node.
We compare the performance of SDSR and other wireless data dissemination services regarding energy efficiency, query latency, hotspot usage, and resistance to clustering failures through numerical analysis and simulations. Our results show that in query- dominant, large-scale wireless ad hoc networks, SDSR performs better than existing schemes in terms of the above performance metrics. It complies well with query locality and scales well when the network size and the number of queries increase.
11 1.3.2 Disruption Tolerant Wireless Ad Hoc Routing
Motivation. Ad hoc networks consist of dynamic wireless links, i.e., new links are constructed and existing links are destroyed or disrupted. The disruptions of the wireless links can last for a long time, e.g., due to the destruction of wireless nodes under severe network conditions such as an earthquake, on a battlefield, etc.; or only for a short period of time, e.g., due to obstructions in between the communicating nodes. We define the latter as temporary disruptions. Temporary disruptions may occur frequently in ad hoc networks for the following reasons:
1. Obstructions present between a sender and a receiver - Obstructions can be
stationary or mobile. For instance, a network constructed by vehicles with
mounted communication devices may get temporarily disrupted due to the
presence of tall buildings.
2. Node mobility - Due to mobility, a node j may temporarily move out of the
transmission range of its neighboring node i and then move back within the
transmission range. This is especially true if j is at the edge of the coverage
area of node i.
3. Beacon collisions - In many wireless networks, nodes use beacons to validate
the availability of a link. Unfortunately, since beacons are commonly ex-
changed through a shared and uncoordinated channel, the repeated collisions
of a beacon from node i with the beacons of other nodes in its vicinity may
give its neighboring node i the wrong impression that the link is down even
12 when it is still up. In general, the increase in probability that a beacon will
collide is directly proportional to the increase in node density.
4. Noises introduced by other wireless devices - Many wireless technologies, such
as IEEE 802.11b and Bluetooth, share the same wireless spectrum. When
devices based on different wireless technologies are placed in close range, they
tend to interfere with each other. In addition, home/office appliances such as
microwave ovens and cordless phones can also generate interference when they
are in operation.
Temporary disruptions can affect a node’s perception of the status of the links between itself and its neighbors. At any given time, the state of a link between two neighboring nodes can be represented by a binary value of 1 or 0 , with 1 indicating that the link is up and, thus, the neighbor is reachable; and 0 indicating that the link is down and thus the neighbor is unreachable. 1
Due to temporary disruptions, the state of a link can toggle between 1 and
0, where the exact value may not be perceived correctly by the nodes. This is because, in most wireless networks, a node senses its link availability through beacon sampling. If the node receives a beacon from a neighbor, it considers that the state of the link to that neighbor is 1 and, thus, the neighbor can be used for packet forwarding. If a node does not receive a beacon from a neighbor within a certain
1With the state of a link as 1, we do not imply that the transmission error rate of the link is 0, but that the error rate is small enough for the nodes to directly communicate.
13 period of time, it concludes that the state of the link to that neighbor is 0 and does not consider the neighbor for packet forwarding.
Although inexpensive, beacon sampling can be misleading, because links deter- mined as down may actually be up (e.g., due to beacon collisions) or become up
(e.g., due to node mobility) before a node can perceive them.
Due to the incorrect perception of a node about its neighbor set, a neighbor considered by a node as unreachable might actually be reachable. We refer to such neighbors (and the corresponding links) as unstable neighbors (links). Unstable neighbors cannot be captured by the simple binary 0/1 neighbor set categoriza- tion used by the current position-based routing algorithms. In these algorithms, each node i constructs (and maintains) a neighbor table that contains only those neighbors that i perceives as reachable. In order to construct a neighbor table as accurately as possible, i usually treats unstable neighbors as being unreachable and, thus, does not store them in the neighbor table. As in position-based routing al- gorithms, nodes present in a node’s neighbor table are the only source where the node chooses its next hop for packet forwarding; excluding unstable neighbors may adversely affect the performance of these algorithms regarding the packet delivery success ratio and the packet delivery latency.
Contributions. In this research, we investigate the effect of a node’s wrong perception about its neighbor set (in particular, the effect of excluding unstable neighbors in packet forwarding) on the performance of position-based routing algo- rithms.
14 We first show, with the help of examples, that not involving unstable neighbors in the decision may either degrade the performance of the protocol or, even worse, render the network disconnected. We thus propose that node packet forwarding decisions should not completely rule out unstable neighbors. However, because such neighbors have a high probability of being unreachable, they should be considered as alternatives when depending on stable neighbors might only result in packet dropping or inefficient routing.
We propose associating each link (neighbor) with a reachability value to ac- commodate the possible incorrect perception of a node about the state of a link.
Associating a reachability value to each neighbor allows a forwarding node to choose from unstable neighbors for packet forwarding when a packet cannot be forwarded to any of the stable neighbors.
To illustrate the effectiveness of our proposed scheme, we apply it to Greedy
Perimeter Stateless Routing (GPSR) [48], a well-known position-based routing al- gorithm, and design two new routing algorithms, Disruption Tolerant Geographic
Routing-Simple Forwarding (DTGR-SF) and Disruption Tolerant Geographic Routing-
Waiting before Forwarding (DTGR-WF), in which nodes utilize reachability values to make forwarding decisions.
We compare the performance of the new routing algorithms with that of GPSR in terms of packet delivery success ratio and packet delivery latency. Through simulations, we show that our schemes achieve better performance than GPSR in cases where disruptions are involved; and in network conditions where GPSR has
15 shown to yield higher packet delivery success ratio and lower average packet delivery latency than other existing routing protocols, our schemes achieve the same high packet delivery success ratio as GPSR with reduced average packet delivery latency.
1.4 Organization of the Dissertation
The remainder of this dissertation is organized as follows. Chapter 2 presents background and related work for our research. In Chapter 3, we concentrate on the problem of server replica placement in CDNs. After presenting the graph model of the server replica placement problem and providing several algorithms, we study the effect of server replica size on the CDN performance. The study is performed through one trace-driven simulation on one specific server. After analyzing the server log file and finding its client demand and distribution patterns, we perform a parametric study to investigate the effect of the client content demand pattern on the performance gain growth. In the end, we provide a simple scheme to narrow down the size of the candidate sites for speeding up the computation.
Chapter 4 presents the design and evaluation of SDSR, the Scalable Data Storage and Retrieval service for wireless ad hoc networks. After presenting the system model and problem statement, we describe SDSR protocol suit in detail. In the end, we compare the performance of SDSR and other wireless data dissemination services regarding energy efficiency, query latency, hot spot usage and resistance to clustering failures through numerical analysis and simulations.
16 Chapter 5 is dedicated to showing our research results on the design and evalu- ation of DTGR - Disruption Tolerant Geographic Routing for wireless ad hoc net- work. First we show that temporary link disruptions in conjunction with neighbor table construction via beacon sampling can result in an incorrect perception by a node about its neighbor set, which, in turn, can adversely affect the performance of position-based routing protocols. We then propose a scheme that allows each node to associate each neighbor with a reachability value to accommodate the incorrect perception of the node about its neighbor set. After that, we present our schemes:
Disruption Tolerant Geographic Routing-Simple Forwarding (DTGR-SF) and Dis- ruption Tolerant Geographic Routing-Waiting before Forwarding (DTGR-WF), in which nodes utilize neighbors’ reachability values to make forwarding decisions. We then compare the performance of DTGR-SF and DTGR-WF with that of GPSR in various simulation settings. In the end, we discuss and justify the additional overhead introduced by DTGR.
Chapter 6 summarizes the main contributions of this dissertation and lists future research directions. For convenience, the abbreviations used in the dissertation are listed in Appendix A. In Appendix B, we first introduce Network Simulator (NS), which is the simulator we used in implementing and simulating DTGR, we then describe in detail the simulation models of DTGR.
17 CHAPTER 2
BACKGROUND
In this chapter, we first present related work on Web access and Web perfor- mance, we then introduce server replica placement schemes. After that, we briefly survey Distributed Hash Table (DHT), a data structure used by most peer-to-peer data dissemination systems in wired Internet, as it is similar to the Geographic Hash
Table (GHT) data structure used in SDSR, our Scalable Data Storage and Retrieval service for large-scale wireless ad hoc networks. We then survey related work on data dissemination and routing protocols in wireless ad hoc networks. In the end, we ex- plain in detail GPSR - the widely used position-based routing algorithm in wireless ad hoc and sensor networks.
2.1 Information Dissemination Systems
Information dissemination systems can either be client-server-based or peer-to- peer-based. Client-server-based systems are used by most Internet applications, such as Web, FTP etc., where a data item is stored in a small number (usually one) of designated servers. These nodes serve as data providers, and a large number of
18 other nodes (clients) act as the data consumers. To download the data item, the client locates the appropriate server and sends a request to the server and receives a response.
In peer-to-peer-based systems, each node acts both as a server and a client and stores/queries data items to/from a network. Nodes can join and leave the network at any time, and the operation of the system depends on the participation of each node in the network, not the designated servers.
Each of these two systems have their own advantages and challenges. The major advantages of client-server-based systems are that: (i) the performances of these sys- tems are usually independent of the storage and computation capacity of the client side; and (ii) the availability and delivery of the data items are relatively reliable.
This is because in these systems, data items are usually provided by designated high performance servers via high bandwidth connections. The major challenge in deploying these systems is that the server may become a bottleneck and, thus, may not be able to handle the requests efficiently as the number of clients increases. To make the systems scalable to a large number of clients, expensive bandwidth and resources are required.
Compared with client-server-based systems, the main advantage of peer-to-peer- based systems is that they scale automatically as the number of nodes increases.
This is because each node in the network can serve as the data source. This sug- gests that peer-to-peer systems have the potential of supporting fast and low cost deployment of large-scale applications. The downside is that the performance of a
19 peer-to-peer system largely depends on the participation of its users, as well as the variety of the available content.
Our research regarding information dissemination is based on both client-server- based (server replica placement in CDNs) and peer-to-peer-based information dis- semination systems (SDSR). Next, we introduce existing work in these two systems.
2.1.1 Client-Server-Based Web Access
Web Components. The Web basically consists of at least two components - servers and clients. Servers are typically the originators of the information items and, thus, are also called original servers, while clients are the consumers of the information items. In this dissertation, we use “Web server,” “Web site,” and
“original server” interchangeably. Each information item hosted by the server is identified by its Universal Resource Locator (URL) [96], which is composed of the hosting server’s domain name and the path to the item on the server. A Web page
– a special kind of information item, contains URLs to other items and creates hyperlinks linking the Web page to these items.
Web browsers such as Netscape [65] and IE [23] are typical client softwares that enable users to download information items from the Web by clicking on the hyper- links, which triggers requests for information items to be sent to the Web servers specified in the URLs. Responses are generated by the Web servers and sent back to the clients. The application layer protocol supporting the communication between servers and clients is Hyper Text Transfer Protocol (HTTP) [96]. The HyperText
20 Transport Protocol (secure)(HTTPs) provides standard encrypted communication mechanisms on the Web.
The explosive growth of Web has led to a dramatic increase in Internet traffic, the workload of popular Web servers, and the significant degradation in client perceived latency while accessing these servers. To improve the Web performance, another component, Web proxies, has been introduced. A Web proxy stores copies of the data items for one (several) original servers either by caching or replicating, and acts as a “front end” of this server (these servers) to the clients.
Web proxies can be divided into three kinds - caching proxies, Surrogates, and
CDN replica proxies. Caching proxies are usually operated by large corporations or ISPs and are located near a set of clients, e.g., all people within a company.
They mediate the communication between these clients and a subset or all of the servers by caching popular Web documents hosted by these servers and replying to clients’ requests for the requested documents. To achieve scalability, multilevel caching proxies can be placed along the path between clients and original servers
[95].
Surrogates (server accelerators) are installed right in front of an original server to enhance the server’s performance through resource sharing and load balancing.
CDN replica proxies (replicas for short) are deployed by commercial CDN compa- nies, such as Akamai, through the Internet. Popular Web sites can select some of these replicas and replicate the heavily requested documents to these replicas to re- duce client request latency, balance workload, and reduce total network bandwidth
21 consumption. In this dissertation, we are interested in the problem of CDN replica proxy placement to improve system performance gain.
Related Work on Improving Web Performance Using Caching and
Replication. There has been a considerable amount of research on Web perfor- mance, ranging from Web workload characterization [5] [8] to developing techniques to enhance Web performance using caching and replication. Significant research has been conducted on different aspects of caching and replication, such as client request forwarding/redirection [14] [30] [24] [46] [85], cache replacement policy [97]
[39] [11] [71] [76], inter-replica or inter-proxy communication [96] [77], cache/replica performance studies [20] [6] [4] [15] [52] [64], multimedia service [19] [93] [75], Web caching/replication protocols for wireless networks [56] [25], and dynamic caching protocol design [16], etc.
Replica Placement Problem for Improving Web Performance. Web
Proxy placement addresses the problem of where proxies should be placed in the network and which data objects should be replicated at those proxies to improve Web performance. As the goal of placing a cache proxy is to reduce the access latency for a set of clients, the proxy is usually placed at the edge of the network and close to these clients, which limits the performance of the cache proxy in improving total system performance, such as reducing total network bandwidth usage. Similarly, surrogates benefit a Web server by alleviating the server’s workload, yet they provide only limited support in improving total system performance.
22 Replicas, on the other hand, are placed within the network. With appropriate placement they can (i) benefit the Web clients by reducing their request latency, (ii) benefit the original servers by alleviating their workload, and (iii) benefit Internet
Server Providers (ISPs) by reducing network bandwidth consumption.
We are interested in the replica placement problem, which can be formalized as follows: Given a network consisting of a set of clients C, nodes N, objects K, and links L (a link is added between two nodes if there is a physical communication channel between them), each client i ∈ C is assigned to a node j ∈ N for each object k ∈ K, which results in a cost according to certain cost function. The proxy placement problem is to optimize the cost function under certain constraints. For example, the cost function might reflect the average client access latency or the total bandwidth cost under the constraint that all the requests from a client can only be served by one specific replica.
Related work such as [54] [53] [72] [42] [12] [52] all model the problem as an optimization problem and compare the performance of different placement strategies in optimizing certain performance metrics, such as total bandwidth cost [53][72], client request latency [54] [53] [42] [52], or replica size [12].
Refs [42] [72] focus on server replica placement, where each replica is a complete replica of the original server, others [12] [52] study the object replica placement, where the placement decisions are made on a per object granularity. Some [42] [72] consider the problem of replica placement for one original server, while others [12]
[52] for several servers. Some [42] [72] study the static replica placement problem by
23 assuming the clients’ distribution and access patterns are known beforehand, while others [12] focus on the dynamic object replica placement problem under client QoS and server capacity constraints. In [52], Krishnamurthy et al. considered both situations.
In [54], assuming the underline network topologies are trees, Li et al. modeled the proxy placement problem as a problem of minimizing the overall request latency for accessing one single original server. They solved the problem using a dynamic programming algorithm with the high computation complexity of O(N 3K2) for se- lecting K replicas among N candidate sites. Krishnan et al. in [53] considered the placement of proxies inside the network to minimize the overall traffic in the network and to reduce the average delay to the clients. They presented optimal solutions for simple network topologies such as line, ring, and tree. The complexity of their dynamic programming-based algorithm for tree topology is O(NHK), where H is the height of the tree.
The above schemes provide optimal solutions to the proxy placement problem only when the topology constraints are satisfied, e.g., the network topology is indeed a tree, and hence do not perform well in graphs, e.g., Internet topology. Using real network trace and Internet topology, Qiu et al. [72] studied the replica proxies placement problem and proposed several placement algorithms. They showed that a greedy algorithm performs better than others in reducing total traffic cost and that its performance is relatively insensitive to imperfect inputs such as client locations and network topology.
24 Assuming that the replicas can only be placed in certain fixed locations, Jamin et al. studied the placement problem for Internet instrumentation in [42]. They also studied the effect of replica size on the optimized system gain (regarding client response time and replica load distribution). Their work on the relationship between replica size showed the diminishing benefits of placing additional replicas using an elaborate strategy.
Our work is based on the study of static server replica placement for one orig- inal server. Without restricting the locations of the replicas, we analyze the client demand and distribution patterns and study the effects of these patterns on the size of replicas when a certain optimization gain should be achieved. In addition, we give a simple scheme to narrow down the candidate sites.
2.1.2 Peer-to-Peer-Based Systems
A number of peer-to-peer data storage/query systems, such as CAN [73], Chord
[87], Tapestry [100], and Pastry [79], have been proposed to simplify the construction of Internet-scale applications. In these systems, each data item is associated with a key, which is a sequence of digits of certain length, and each node is associated with an ID that is drawn from the same space as the keys. Each node is responsible for storing a certain range of keys.
The major similarity shared by these systems is that they all use DHT data structure to hash each data key to the logical ID of a node. This node serves as a rendezvous point for data storage and lookup. Each node maintains a routing table of a small set of nodes in the network. Nodes close to each other in their logical ID
25 space are neighbors and are interconnected by overlay routing tables so that data lookup and storage packets can be routed closer and closer to the responsible node.
The notion of “closeness” varies among different DHT systems but, in general, is defined in terms of the distance between the ID of the current node and the data key.
Different DHT systems use different routing algorithms to route the lookup through the overlay network to the node responsible for the data item. The direct adoption of DHT systems in wireless ad hoc networks might not be cost effective because neighboring nodes close to each other in their logical ID space might be far apart in the physical location space, thus some packets might traverse the wireless network multiple times before reaching their respective destinations.
2.2 Information Dissemination in Wireless Ad Hoc Net- works
With regard to wireless mobile ad hoc networks, a list of data replication schemes have been proposed to improve data accessibility/reliability either upon network partitions [34] [35] [92] [49], or in highly dynamic environments with few occurrences of network partitions [59].
In [34], assuming that each mobile node can hold a limited number of data repli- cas and that the access frequency for each data item is fixed and known beforehand,
Hara proposed several replica allocation methods that periodically reallocate repli- cas of each data item in the network based on its access frequency. In [35], these schemes were extended to handle data updates. How a query for a data item is
26 handled was not discussed in both papers. The problem of replication allocation was also studied by Wang et al. in [92], where the allocation process is triggered dynamically by the detection of possible future partitions, based on certain criteria such as connection stability.
In [49], assuming that a set of information replica servers are constructed a priori and known by each mobile client beforehand, Karumanchi et al. proposed several traditional strict quorum systems for mobile clients to update and query information.
To mitigate the effect of query failures resulting from network partitions, several heuristic algorithms were used in selecting the update server set and the query server set from the list of servers with high probability of being unreachable. In PAN [59],
Luo et al. presented a Probabilistic quorum system for Ad hoc Networks (PAN) to provide reliable data storage service in highly dynamic network environments.
PAN applies an asymmetric quorum construction scheme for efficiency and relies on a gossip-based multicast protocol for quorum accesses in order to improve the robustness and to combat the high probability of channel failures resulting from node mobility [59].
Different from the above, SDSR focuses on achieving data access scalability and efficiency for wireless ad hoc networks with reasonable node density and few occurrences of network partitions.
Geographic Hash Table (GHT). GHT [74] proposed by Ratnasamy et al. for information dissemination in wireless sensor networks is similar to DHT. In GHT, the rendezvous point for a data item results from a mapping between data keys and
27 geographic locations in the network. More specifically, GHT hashes all data objects of the same kind to one fixed location in the current network area, and the node closest to this location serves as the storage server for these hashed data objects.
SDSR is similar to GHT, since both schemes provide location-based data-centric storage/retrieval service to large scale wireless networks. Some important differences between SDSR and GHT are:
1. GHT is designed for the asymmetric sensor network model and, thus, fits for
an environment where the amount of data generated is much larger than the
amount of data queried. SDSR is designed for the symmetric model, where
each node can insert/retrieve data in the network and where queries are more
intensive than those in GHT;
2. In GHT, each hashed (x,y) coordinates represent one fixed geographical loca-
tion in the current network whose boundary is known a priori. In SDSR, each
hashed (x,y) coordinates represent a fixed location in a unit square and can
be scaled to get a geographical location for this data item in a network of any
size; and
3. In GHT, each data item is not replicated across the network. In SDSR, each
data item is replicated to multiple locations in the network to reduce query
communication cost and bottleneck effects, and to increase the resilience to
clustering failures.
28 In GHT, since there is no relationship between the location of a data item’s source node and the hashed storage location, the data item might have to be stored farther away from where it is created, which introduces high storage cost. To reduce the storage cost of GHT, the authors in [74] proposed Structured Replication GHT (SR-
GHT). SR-GHT hashes each data key to a root location in the network, divides the whole network uniformly into 4d subregions (d is the replication hierarchy depth) and then computes 4d − 1 mirrors of the root location, with one mirror in one subregion.
The source node of a data item stores the data item to the closest mirror. Note, the data itself is not replicated in each subregion. The tradeoff is that SR-GHT increases the query cost since starting from the root location, each query has to go through all the mirrors to get the required data objects.
2.3 Routing in Wireless Ad Hoc Networks
In wireless ad hoc networks, each node can communicate directly with other nodes that are within its radio transmission range. To communicate with nodes that are outside this range, the node needs to use intermediate nodes as routers to relay the messages hop by hop.
The dynamic feature of wireless ad hoc networks due to the unstable wireless media as well as nodes join, leave and fail stops results in frequent and unpredictable network topology changes, and thus challenges the design of an efficient and reliable routing scheme.
29 Routing has been the most active research area in wireless ad hoc networks.
Based on the cast type, routing protocols proposed so far can be categorized into four groups - broadcast, multicast, geocast, and unicast. Broadcast is used when a node wants to send a message to all the other nodes in the network. Wireless media implicitly supports 1-hop broadcast, which means a message transmitted by a node is usually received by all the other nodes within its transmission range. Several schemes have been proposed to alleviate the broadcast storm problem introduced by broadcasting using simple flooding. A survey of related work can be found in
[88].
Multicast is used to support multipoint communications (e.g., video conferenc- ing). Ad hoc multicast routing can be divided into three types: tree based, such as
MAODV [80] and ITAMAR [83], mesh based [29] [27], and stateless [43]. Stateless multicast routing schemes are proposed to reduce the overhead introduced by the other two in maintaining and delivering the tree or mesh in time. Cordeiro et al. surveyed existing multicast routing research in [18]. Geocast is used to forward messages to a group of nodes residing within a certain geographic region. A survey of related work for both infrastructure-based and infrastructure-less networks can be found in [61].
The purpose of unicast routing is to establish and maintain a communication path between a pair of nodes such that packets can be delivered reliably and ef-
ficiently. A large percentage of ad hoc routing protocols falls into this category.
Based on whether an end-to-end path needs to be constructed before packets can
30 be transmitted, these protocols can be categorized into two groups - topology-based and position-based.
2.3.1 Topology Based Routing Protocols
In topology-based routing, an end-to-end path from the source to the destination has to be established either beforehand (proactively) or on demand (reactively) before the communication starts. The establishment of the path is based on the information about the current network topology, which is defined by the connectivity between pairs of nodes.
Topology-based routing protocols can be divided into three parts - proactive, reactive, and hybrid. In proactive routing, each node maintains routing information about the available paths in the network. Even if these paths are not currently used, the routing information is updated periodically or whenever there is a change in the network topology. Examples of proactive routing protocols are Destination
Sequenced Distance Vector (DSDV) [69], Fisheye State Routing (FSR) [68], Distance
Routing Effect Algorithm for Mobility (DREAM) [7], and Optimized Link State
Routing (OLSR) [41], etc.
As routing information is periodically updated in position-based routing proto- cols, routes between pairs of nodes are always available and up to date, thus they can optimize end-to-end delay. The major disadvantage of proactive routing protocols is that the maintenance of unused paths consumes a significant amount of network bandwidth if the network topology changes frequently.
31 In response to the above observation, On-demand (reactive) routing protocols such as Ad hoc On Demand Distance Vector (AODV) [70], Dynamic Source Routing
(DSR) [44], Temporally Ordered Routing Algorithm (TORA) [67], and Associativity-
Based Routing (ABR) [90], etc., have been proposed. To reduce the network band- width consumption for maintaining unused routes, these protocols maintain infor- mation for active routes only. Some limitations of reactive protocols are: (i) long route discovery delay, because a route discovery is typically required before any packets can be exchanged; and (ii) the cost of maintaining active routes may still be high if the network topology changes frequently.
Hybrid topology routing protocols such as Zone Routing Protocol (ZRP) [33] and Scalable Location Update Routing Protocol (SLURP) [99] try to combine local proactive routing and global reactive routing for better efficiency and scalability.
However, as these protocols still need to maintain at least routes that are currently in use, the degree of the topology changes that they can support is still limited. A survey of topology-based routing protocols can be found in [81] and [2].
Although topology-based routing protocols can usually find the shortest path between any pair of nodes, they are not scalable (regarding the topology information stored in each node and the network update message overhead) as the network size increases or when the network topology is very dynamic.
2.3.2 Position Based Routing Protocols
Position-based routing protocols exploit each node’s geographic location infor- mation to eliminate some of the limitations of topology-based routing, e.g., the
32 overhead of constructing and maintaining an endto-end path from the source to the destination. Each node obtains its geographic location through GPS [91] or some other location services as proposed in [94]. Neighboring nodes exchange with each other their location information periodically through beaconing.
Position-based routing protocols route a packet to its destination based on the destination’s geographic location instead of its IP address. More specifically, when a source node wants to send a packet to a destination node, it inserts the destination location (obtained through some location service such as GLS [55]) and forwards the packet to one of its neighbors. When an intermediate node receives the packet, it forwards the packet solely based on its own location, the packet’s destination loca- tion, and the locations of its neighbors. To which neighbor the packet is forwarded is based on different forwarding criteria, e.g., to the neighbor that is closest to the destination (MFR) [89], or to the nearest neighbor of the sender that is closer to the destination (NFP) [36].
In position-based protocols, there is no end-to-end route establishment before sending a message and there is no route maintenance during packet transmis- sion. Each packet is routed independently from the source to the destination.
Thus, position-based protocols are more robust to network topology changes than topology-based routing protocols. In addition, position-based routing protocols are more scalable than topology-based routing protocols as the network size increases, because each node only needs to maintain its local topology. The problem is that
33 they may not be able to find the optimal route due to the lack of global topology information. A survey of position-based routing algorithms can be found in [62].
2.4 Greedy Perimeter Stateless Routing Protocol (GPSR)
D l
k
j i
S
Figure 2.1: GPSR packet routing example, where S is the source node and D is the destination node. The solid line indicates that the packet is forwarded using greedy forwarding, and the dotted line indicates that the packet is forwarded using perimeter routing.
34 GPSR is a geographic routing protocol for wireless networks. It combines two forwarding methods – greedy forwarding and perimeter routing. Greedy forwarding allows a packet to be forwarded to the neighbor geographically closest to the desti- nation, while perimeter routing allows a packet to circumvent a void when there is no neighbor closer to the destination than the current forwarding node.
In GPSR, each node actively maintains the following information:
1. Its own geographic position obtained through either GPS or other techniques;
2. A neighbor table containing the addresses and geographic locations of its 1-hop
neighbors; and
3. A subset of 1-hop neighbors that form a planar subgraph. Some planarized
graphs, such as Relative Neighborhood Graph (RNG) and Gabriel Graph (GG)
can be constructed by each node locally using the geographical locations of its
neighbors. The planar subgraph is required by the perimeter routing mode to
route the packet out of a void.
A packet in GPSR can be in either greedy mode or perimeter mode. When a node has a packet to send in greedy mode, it inserts inside the packet the location of the destination, sets the packet into greedy mode, and forwards it using greedy forwarding. If the packet cannot be forwarded in greedy mode due to the presence of a void, the node employs perimeter routing and forwards the packet in perimeter mode.
35 In perimeter routing, each forwarding node applies right-hand rule to select the next hop so the packet can traverse along planar faces that converge toward the destination. The packet remains in perimeter mode until it reaches a node i closer to the destination than the node where the perimeter forwarding starts. In this case, i sets the packet back to greedy mode and starts greedy forwarding procedure.
The routing of the packet is terminated when the packet reaches the destination or when there is no neighbor to which the packet can be forwarded using either of the above two methods, in which case, the forwarding node drops the packet.
The greedy forwarding and perimeter routing procedures briefly described above are shown with the help of an example in Figure 2.1. Source node S first forwards the packet in greedy mode to i, which has no neighbor closer to D than itself, and, hence, i sets the packet in perimeter mode and sends it to k. When the packet reaches k, which is closer to D than i, k sets the packet back to greedy mode and forwards it to l using greedy forwarding, which in turn forwards the packet to the destination D in greedy mode.
Since in position-based routing protocols, each node selects the next hop for packet forwarding from nodes stored in its neighbor table, the maintenance of the neighbor table is important for the performance of these protocols. GPSR, in par- ticular, uses a simple neighbor table maintenance scheme: each node periodically broadcasts its address and location to its neighbors. Any neighboring node i on receiving the beacon, stores the received information in its neighbor table. If i has not received a beacon from its neighbor j for a time-out interval, i considers j as
36 unreachable and removes j from the neighbor table. The time-out interval is set as a multiple of a beacon interval. Figure 2.2 and Figure 2.3 show GPSR parameter definition and GPSR packet forwarding procedure, respectively.
2.4.1 Routing under Network Disruptions
Routing in wireless networks in the presence of disruptions can be categorized into two types. The first type is routing under severe network failures, where the network disconnection or disruption is not temporary but can last for a significant period of time. An example of such networks can be sparse networks composed of mobile robots participating in rescue tasks after an earthquake. Schemes such as
Disruption Tolerant Networking (DTN) [22], message ferrying [101], and MV routing
[10] fall into this category.
To enable communication between disconnected peers under severe network dis- ruptions, approaches proposed so far require additional capacity and functionality of the peers and/or new relay devices to be deployed in these networks. For exam- ple, in [22], mobile nodes are required to have large storage and power capacity so that messages can be transferred using store and forwarding. In message ferrying schemes proposed by Zhao et al., special mobile devices called ferries are required to travel along predictable routes within the network so that disconnected peers can adjust their movements to meet with the ferries for message uploads and downloads, while in [10], mobile ferries adjust their movements to meet the routing demand of disconnected peers.
37 The second type is routing under normal conditions, where the network dis- connection or disruption is frequent yet temporary. An example of such networks can be ad hoc networks that are temporarily disconnected due to node movements.
Topology-based and position-based routing schemes, introduced before, fall into this category. In topology-based algorithms, an end-to-end path must be constructed be- fore the packet is sent from the source. In case of link disruptions, a new path needs to be discovered and updated information needs to be flooded in the network. The resultant overhead makes these algorithms unscalable as the network size increases and less adaptive as the network becomes more dynamic.
On the other hand, using position-based routing protocols such as GPSR, each node makes routing decisions strictly based on its own location, the location of its neighbors, and the location of the destination. Since these protocols do not need to. maintain routing tables and only require local re-routing in case of link disruptions, they are inherently more robust to network topology changes and scale better as the network size increases than topology-based routing algorithms
For most of the position-based routing protocols, each node maintains the in- formation about its neighbors in a neighbor table. The neighbor table maintenance usually is performed through beacon sampling, which is inexpensive yet misleading.
Since the set of nodes stored in the neighbor table is the only source from which the node chooses its next hop for packet forwarding, existing protocols usually try to construct the neighbor table as accurately as possible. For example, assuming that each sensor node knows the packet reception rate (PRR) and the locations of
38 its neighbors, Seada et al. [84] proposed “blacklisting,” which rules out unreliable neighbors (neighbors whose PRR is below a certain value) before making greedy for- warding decisions. Since ruling out of neighbors should not disconnect the network,
“blacklisting” is only applicable in networks with high node density.
Different from the above, we show that unreliable neighbors should not be totally ruled out in the routing process, as considering reliable neighbors only might result in packet dropping or inefficient routing.
39 Figure 2.2: GPSR protocol parameter definition
40 Figure 2.3: GPSR packet forwarding procedure
41 CHAPTER 3
OPTIMIZATION OF PERFORMANCE GAIN IN CONTENT DISTRIBUTION NETWORKS WITH SERVER REPLICAS
Server replica placement of CDNs addresses the problem of properly placing a set of replicas among a set of candidate sites to optimize system performance gain.
The placement decision can be made either on a per content item granularity or on a per server granularity. In the latter case, a replica holds copies of all the content items of the original server. The placement can also be made either dynamically, according to requests from clients, or statically, based on “know” client demands and client distribution patterns.
In this research, we study the static server replica placement problem. Given a fixed replica size |K|, we model the replica placement as a well-known minimum p-median graph problem [28] and propose several heuristic solutions.
With a near optimal greedy algorithm, we establish the relationship between replica size and the optimized performance gain by adding one replica one at a time and computing the performance gain. To compute the threshold value of replica size required to achieve certain performance gain, we perform one trace-driven simulation
42 on one specific server. In the end, we perform a parametric study to study the effect of content demand pattern on the performance gain growth and propose a simple scheme of narrowing down the candidate sites to speed up the computation.
The rest of the chapter is organized as follows: Section 3.1 formulates the server replica placement problem and introduces several algorithms to solve the problem;
Section 3.2 shows the results from the trace-driven simulation; the parametric study is given in Section 3.3; and Section 3.4 summarizes this chapter.
3.1 Modeling of Server Replica Placement
The server replica placement is modeled as follows. The network is modeled as a graph G=(V, E). The content received by each client node is associated with a content delivery cost that is computed as the product of the number of bytes received by the client and the distance between the client and its assigned source node. The source node can be either the server node or a replica node, depending on which one is closer to the client.
Given a value |K|, we want to place |K| replica nodes within the network so that the total content delivery cost is minimized. Here, the distance metric can reflect either the link delay or the cost of sending content from the source to the destination. Note that the replica set is determined by both the set size |K| and the location of each replica. Assuming that (i) there is no capacity constraints on each replica and the original server, (ii) the cost of placing replicas is 0, and (iii) each replica is a full copy of the original server, we model the the server replica placement
43 problem as a well-known uncapacited minimum p-median graph problem, which is described in the following subsection.
3.1.1 Minimum P-median Graph Problem
Given N nodes, suppose we want to select P nodes as centers. Each node j is assigned to i, the center node closest to it. The cost between j and i is computed c(j) × d(i, j), where c(j) is the demand traffic generated by j, and d(i, j) is the distance between i and j. There is no cost associated with selecting the center nodes. The goal is to select P centers so as to minimize the sum of the cost. For the uncapacited minimum p-median problem, there is no constrain on the amount of service that can be provided by any center.
3.1.2 Placement Algorithms
Since the minimum p-median problem is known to be NP complete, finding an optimal solution is not feasible. In a network with a fixed number of clients and replicas, the total content delivery cost generated depends on both the client demand traffic and the distance between each client and its assigned replica. To solve the minimum p-median problem, the following heuristic algorithms can be used:
1. Random: ignoring the above two factors, the random algorithm randomly
picks |K| replicas with uniform probability.
2. Hot spot: by considering client demand traffic, the hot spot algorithm places
replicas on the top |K| sites receiving the highest demands.
44 3. Highest outdegree: by assuming that a node with a higher outdegree can reach
more nodes with shorter distance, this algorithm places replicas on the top |K|
sites with the highest outdegrees.
4. Greedy-Single: by considering both factors defined above, the greedy-single
algorithm selects |K| replicas iteratively on the network in a greedy fashion.
In our research, we use the greedy-single algorithm to compute the optimized content delivery cost. The algorithm is described as follows. For one original server
(OS), suppose we want to choose |K| replicas among N sites. In the first iteration, we exhaustively evaluate each node in N and select the node, which, combined with the OS, yields the lowest total delivery cost. In the second iteration, we search for a second replica site, which, in conjunction with the site already selected and the
OS, yields the lowest cost. In general, in computing the cost, we assume that each client gets all its content from the source node closest to it. The iteration continues until all the |K| replicas have been chosen. In our study, the OS is a fixed source node and when |K| = 0, all the clients get their content directly from the OS.
3.2 Trace-Driven Simulation
Using the above greedy-single algorithm, we perform a trace-driven simulation to study the relationship between replica size and the optimized performance gain.
Our simulation is performed on a network model, constructed based on a real Web server workload and a simplified Internet AS-level topology. In this section, we
first describe our network model (including both Web server workload analysis and
45 AS-level topology construction), we then present our simulation results. In the end, we present a simple scheme to narrow down the set of candidate sites based on the client demands and outdegree distribution patterns observed through the workload analysis.
3.2.1 Network Model
In our network model, each node represents one AS, and each edge represents a connection between two ASs. Without going into detail, at the highest level, the
Internet can be seen as a routing network in which the ASs are the nodes. Each AS is weighted by the aggregated content demand from all clients within its domain, and each edge is weighted by the shortest path length between the two ASs measured in AS hop counts. Since the quality of communication within an AS is often much better than between different ASs, we place at most one replica within each AS domain and neglect the content delivery cost generated by clients within the same domain as the replica.
3.2.2 Aggregation of Clients’ Demands into AS Domains
We use the the log file of the Department of Computer and Information Science at
The Ohio State University collected in February 2002 for our trace-driven simulation.
The log file contains 24 hours of requests that are globally diverse. After filtering out all the other requests except the HTTP GET requests with successful results, the log file contains 126,886 requests making up 2.47 gigabytes. These requests were issued by 11,123 unique IP addresses. To aggregate together the reply traffic of the
46 same AS, we need to group these IP addresses into their AS domains. We achieve this by the following steps. First, we use the approach proposed by Krishnamurthy et al. in [52] to cluster IP addresses that are topologically close together. After this step, we group 11,123 unique IP addresses into 16,816 clusters. Each entry in the cluster is represented in the form x1.x2.x3.x4/l, where x1.x2.x3.x4 is the network prefix and l is the network mask length. Since several clusters can belong to the same AS, we then group these clusters into their AS domains by using tool ipas
[38]. Ipas takes each cluster as an input and outputs the AS to which this cluster belongs. After the above steps, 16,816 small clusters are grouped into 1,950 ASs.
Client content demand analysis. Figure 3.1 shows the cumulative distri- bution of traffic load received by each AS from the OS. We observe that a small percentage of ASs requires a large percentage of traffic. For example, the top 10%
(20) ASs account for 76.6% of the total traffic, and the top 20% ASs account for
80.7% of the total traffic. This distribution shows that our log file fits for the well- known 20/80 rule; similar results are also found in [5]. Since the second half ASs receive only 2% of the traffic, it is safe to neglect them. Therefore, in our graph model, V only contains the top half ASs, which receive the largest amount of reply traffic.
Figure 3.2 shows the aggregated content delivered to each AS in bytes. Figure
3.3 shows the aggregated content delivery cost in bytes AS hops. In both figures, the ASs are ranked in the order of received reply bytes; e.g., the first AS receives the most reply bytes. Here, both x-axis and y-axis are in log scale.
47 Figure 3.1: The cumulative distribution of reply bytes received by each AS
48 Figure 3.2: Aggregated content traffic (in bytes)
49 Figure 3.3: Aggregated content delivery cost (in bytes AS hops)
50 Figure 3.3 shows that the content delivered to the top 950 ASs receiving the most reply traffic follows a Zipf-like distribution of α = 1.1. Similar results are also found in [51]. The Zipf-like distributions are common in a variety of Web measurements
[8]. Using a Matlab curving fitting tool, we get α = 1.1. The similarity between both figures shows that, in general, the order of content delivery cost, from the original server to each AS, is mainly determined by the amount of traffic delivered.
The distance between the OS and the AS also has a noticeable effect on the order regarding the content delivery cost.
3.2.3 Construction of AS Level Topology
In this AS-level topology, each node represents one AS in V and each edge represents the shortest path length (in AS hop counts) between two nodes. To obtain the shortest path between each AS pair, we use the data provided by NLANR [63], which contain the summary of Internet routing data collected in the route views project [78] from 1997 to early 2002. The summary provides information on all the direct connections between all ASs. Using this information, we run all pair shortest path first algorithm to get the shortest paths between all AS pairs, from which we extract those related to the ASs in our node set V.
Figure 3.4 shows the shortest path distribution of AS pairs in V, where the x- axis is the shortest path and the y-axis is the percentage of AS pairs having the same shortest path length. We observe that the longest path length between these
AS pairs is six and that most AS pairs have a path length of two or three. The distribution follows the Gaussian approximation with the mean value of 2.63. The
51 Figure 3.4: AS pair distribution by shortest path length
distribution of the shortest path lengths between the original server to all its AS clients ((not shown here)) follows the same trend. It is shown that the shortest path lengths between all ASs in the Internet follow a Gaussian distribution with the mean value of 3.65 [60].
Figure 3.5 shows the average path length and Figure 3.6 shows the outdegree of each AS node in V. Here, ASs are ranked in the order of received reply bytes. It is observed that the higher the outdegree of an AS, the shorter the average shortest path length from this AS to all the other nodes. In fact, in Figure 3.6, the top eleven ASs with the highest outdegrees contain the top 10 ASs in Figure 3.5, which have the shortest average shortest path lengths. Furthermore, the top seven ASs in both figures are exactly the same. The explanation for this relationship is that, in
52 Figure 3.5: The average path length from each AS to other nodes
53 Figure 3.6: The number of direct connections from each AS to other nodes
54 general, the AS nodes with higher outdegrees should be able to reach more nodes with shorter paths. It is also observed that the top ASs receiving the most reply bytes tend to have more nodes with shorter average shortest path lengths and higher outdegrees. For example, for the top ten ASs with the shortest average path lengths, four of them are also among the top ten ASs receiving the most reply bytes, and all of them, except two ASs, are among the top 69 nodes receiving the most reply bytes. The two exceptions are: (i) the AS with the highest outdegree (616) and the shortest average path length (1.37) is ranked as the 211th in received reply bytes, and (ii) the AS with the 6th highest outdegree (118) and the shortest average path length (1.8) is ranked as the 266th in received reply bytes.
Figure 3.7 shows the average shortest path length distribution from the original server’s point of view. The graph is plotted as follows: first we rank ASs in the order of received reply bytes, and then we divide these ASs into several chunks (1%, 1%-
5%, 5%-10%, 10%-20%, 20%-50%, 50%-100%). Each chunk consists of ASs within a certain rank range. For example, 1% includes the top 1% ASs receiving the most reply bytes, and 1%-5% includes the top 1% to the top 5% ASs receiving the most reply bytes. We then average the shortest path lengths from the OS to all ASs within each rank range.
We observe that the average distance from the OS to those ASs receiving more reply bytes tends to be smaller than the average distance from the OS to those ASs receiving fewer reply bytes.
55 Figure 3.7: The average shortest path length from the original server to other ASs within certain rank range (received reply bytes)
56 Figure 3.8: The average shortest path length between the top 1% ASs and the ASs within certain rank range
57 Figure 3.8 shows the average shortest path length between the top 1% ASs receiving the most reply bytes and the other ASs within each range. We observe that the ASs receiving more bytes also tend to be closer to each other.
3.2.4 Simulation Results
To study the effect of replica size |K| on the optimized performance gain, we run the greedy-single algorithm to choose up to 200 replicas. We start from |K|=0 (no replicas are placed) and increase |K| by 1 each time until |K|=200. For each |K|,
|K| replicas are chosen and the associated total content delivery cost is computed.
The greedy-single algorithm is fed by three input parameters: |K| (200, in our simulation), the content aggregated on each node, and the distance metric.
Variation of the optimized performance gain with the replica size |K|.
Figure 3.9 shows the optimized total content delivery cost. We observe that the
first few replicas can greatly reduce the delivery cost, however, after |K| reaches a threshold value, there is a diminishing performance gain when we place additional replicas one by one in the network. For example, Table 3.1 shows that, together with the original server, the first replica can decrease the delivery cost by 65%, the
first 5 by 82%, and after that, adding 45 more replicas can only further decrease the total delivery cost by 8%.
Scheme to narrow down the replica candidate sites. How well the total content delivery cost can be minimized also depends on the size and topological placement of candidate sites. Usually, candidate sites are chosen to be those receiv- ing the most content [72]. To avoid keeping any good candidate from being chosen
58 Figure 3.9: Optimized total content delivery cost versus number of server replicas (|K|)
|K| Performance gain 1 65% 5 82% 50 90% 150 95%
Table 3.1: Performance gain versus |K|
59 as a replica, the size of candidate sites tends to be set to a large value, e.g., all the clients.
Figure 3.10: Server replicas’ AS ranking (in received reply bytes) versus chosen order
Figure 3.10 shows the relationship between the order in which a candidate is chosen as a replica and its ranking as the number of bytes received. The x-axis shows the chosen order of each replica, e.g., x=1 refers to the first chosen replica; the y-axis shows the replica rank as the number of bytes received, e.g., y=1 means that the replica is the site receiving the most reply bytes. The figure shows that
60 the replica’s chosen order generally follows the replica’s rank order in received bytes with few exceptions. For example, the first ten chosen replicas are AS1, AS211, AS2,
AS3, AS7, AS5, AS6, AS9, AS8, AS266, where the number represents the receiving byte rank. For example, the second chosen replica byte rank is 211. According to Figure 3.10, AS211 has the highest outdegree, and AS266 has the sixth highest outdegree. The ASs with higher outdegrees can be seen as transit nodes that can reach more nodes with shorter distances.
From the observations, we conclude that placing replicas on a small number of carefully chosen candidate sites allows us to achieve the same performance gain as placing replicas on all candidate sites, but with reduced computation cost. The carefully chosen candidate sites should contain a few sites who receive the most reply traffic and a few sites which have the highest outdegrees.
3.3 Parametric Analysis
The trace-driven simulation studies the effect of replica size on the optimized performance gain based on the traffic pattern of one specific server. In order to study the effects of various traffic patterns on the performance-gain growth, we perform a parametric study. We generate three different Zipf-like client demand distributions with α = 0.8, 1.2 and 1.6, respectively. For simplicity purpose, we adopt the distance information retrieved from the trace-driven simulation. Figure 3.11 shows the effect of α on the threshold value of |K|. It is observed that as α increases, the number of replicas required for achieving certain optimal performance gain decreases. For
61 example, as shown in Table 3.2, in order to achieve 70% performance gain, 10 replicas are required for α = 0.8, 2 for α = 1.2, and only 1 for α = 1.6.
Figure 3.11: Variation of cost with |K| and α
3.4 Chapter Summary
In this chapter, we presented our research results on the study of the effects of server replica size and client traffic patterns on the growth of CDN performance
62 Performance gain/α 0.8 1.2 1.6 60% 1 1 * 70% 10 2 1 80% 62 7 2 90% > 200 45 7 95% > 200 155 19
Table 3.2: Threshold |K| versus α
gain. We performed one trace-driven simulation to study the relationship between the optimized performance gain and the replica size. The simulation was based on the log file of one specific Web server. From the simulation we found that there is a diminishing return to placing more replicas to minimize total content delivery cost, and that a surprisingly small number of replicas (less than 5) is enough to significantly improve performance gain (82%). By analyzing the server log file, we found that its client demand follows a Zipf-like distribution with α = 1.1, and its client distance (in AS hop counts) distribution follows Gaussian-like distribution with a mean of 2.63.
We performed a parametric study with different values of α and established the relationship between α and the threshold value of replica size. The results showed that as α increases, the number of replicas required to achieve a certain performance gain decreases.
In addition, we observed that placing replicas among a small number of carefully chosen candidates (e.g., those clients that have requested the largest amount of
63 content, or those clients that have the highest outdegrees) allows us to achieve the same performance gain as placing replicas among all clients but with reduced computation cost.
In the future, we would like to study the replica placement problem under the constraint that each replica has limited storage space and, thus, might not be able to store all the content of the original server. When storage constraint is considered, more detailed Web server trace analysis might need to be conducted. For example, instead of only obtaining the total size of all the documents requested by each candidate site, we might need to take additional information such as each document’s size and its popularity into consideration when making replica placement decisions.
64 CHAPTER 4
SDSR: A SCALABLE DATA STORAGE AND RETRIEVAL SERVICE FOR WIRELESS AD HOC NETWORKS
The energy constrained wireless devices, as well as the frequent network topology changes, make the traditional information dissemination schemes, such as broadcast- ing, storing data centrally (CS) or locally (LS), undesirable and, thus, necessitate and challenge the design of energy efficient and robust data dissemination schemes.
In this chapter, we study the problem of storing/retrieving data in a wireless ad hoc network in a scalable, efficient, and robust manner, where data can be any resources of interests, such as service advertisements [13], files, locations of nodes, local events like the presence of objects, etc. Such a network can be constructed either by personal devices such as notebooks or PDAs in a wide area civilian en- vironment, or by military devices in a large battlefield. Our study is based on a peer-to-peer system architecture, where each node acts both as a server and a client and can store/retrieve data to/from the network.
We propose Scalable Data Storage and Retrieval service (SDSR), an efficient, robust, and load-balancing data storage/retrieval service to disseminate information
65 in large-scale wireless ad hoc networks. SDSR hashes each data item to normalized geographical coordinates (x,y), where x, y ∈ [0,1]. The normalized coordinates can be seen as a rendezvous point for storing/retrieving the data item in a unit grid. SDSR achieves scalability, robustness, and load-balancing by partitioning the network into hierarchical grids of increasing size and replicating the data item into each hierarchical grid. The storage location in each grid is determined by using the normalized coordinates as an offset and scaling them to the corresponding grid size.
SDSR retrieves a data item in the same way as it hierarchically stores the data item.
Our analysis shows that in a query-dominant, large-scale wireless ad hoc network,
SDSR performs better than existing schemes in terms of energy efficiency, data query latency, hotspot usage, and resilience to clustering failures. SDSR scales well when the network size and the number of queries increase.
The folllowings of this chapter are organized as follows: Section 4.1 presents our system model and SDSR problem statement; in Section 4.2, we describe SDSR in detail; in Section 4.3, we present the mathematical analysis of SDSR and compare its performance with those of existing data dissemination schemes according to various performance metrics; We compare SDSR with other schemes through simulation in
Section 4.4 and then summarize the chapter in Section 4.5.
4.1 Preliminaries
In this section, we first present SDSR system model and assumptions, we then present some definitions and SDSR problem statement.
66 System Model. We consider a large-scale wireless ad hoc network in a 2-D coordinate plane. Nodes are connected if they are within unit distance of each other. Nodes and edges are represented by sets V and E, respectively, and the resultant undirected graph by G, where G = (V , E).
Assumptions. We make the following model assumptions: 1) we assume a connected network where, with high probability, there are multiple nodes in each unit square area; 2) each node knows its geographic location via some location determination technologies such as GPS service; 3) each data item has a unique key;
4) a hash function uniformly hashes each key to normalized (x,y) coordinates; 5) energy is a scarce resource for nodes in the network (since communication consumes most energy of wireless networks [82], minimizing communication cost should be an important criterion in designing energy efficient data dissemination schemes); and
6) the longer the Euclidean distance between two nodes, the more the network-level hops is required for thse two nodes to communicate. Next, we present the definition of the terms used in this chapter.
Definitions. We use j and k to denote nodes or locations in the network. Let dist(j, k) denote the Euclidean distance between j and k in G, and e(j, k) denote the edge between j and k. Data delivery success probability, Pj,k, between any two nodes j and k is defined as the probability of successfully delivering a data item from j to k. It is computed as plen(j,k), where p is the success probability of delivering the data item between two 1-hop neighbors, and len(j, k) is the length of the path traversed by the data item from j to k (note that len(j, k) ≥ dist(j, k)). Communication cost,
67 Cj,k, between any two nodes j and k is defined as O(m ∗ dist(j, k)) where m is the number of messages transmitted between j and k. Network delay, tj,k, between any two nodes j and k is defined as O(dist(j, k)). Next we give the problem statement.
Problem statement. Given a wireless ad hoc network, where each node is capable of generating/retrieving data items, the data storage and retrieval problem is to design a distributed, scalable, and energy efficient scheme that stores each data item at a set of storage locations such that:
1. the number of storage locations for each data item and per node data storage
and communication cost increase as a fraction of the increase in network size;
2. each data item is routed from its source node to its storage locations with low
total communication cost and high data delivery success probability;
3. the request latency experienced by a data requester node is proportional to
the distance between the requester node and the source node; and
4. more storage locations are placed in the network area where more queries for
the data item are generated, while fewer storage locations are places in the
area where fewer queries are generated.
4.2 SDSR Service
In this section, we first present the hierarchical network partition scheme used by SDSR, followed by the data storage server selection scheme. We then introduce the multicast tree routing algorithm used by SDSR for efficiently routing each data
68 item to their selected storage locations. We conclude this section by presenting the data query and data update schemes, followed by the description of the indexing service employed by SDSR to achieve fault tolerance.
4.2.1 Hierarchical Partitioning
We construct hierarchical partitioning of the network into squares of increasing size similar to the grid partitioning described in GLS [55]. As shown in Figure 4.1, the smallest square is a level-1 grid and four level-1 grids form a level-2 grid, and so on. The largest grid covering the whole network area is a level-N grid (e.g., level-4 in Figure 4.1). Grids do not overlap with each other in the sense that each level-i grid belongs to exactly one level-(i+1) grid. Such a partitioning guarantees that each node belongs to exactly one grid in each hierarchy level. The four level-i grids of the same level-(i+1) grid are neighboring level-i grids of each other. Since each node knows its location, it knows which grid it belongs to.
Furthermore, for ease of presentation, we number the squares in each grid as 0,
1, 2, and 3, with 0 representing the bottom left square of the grid, 1 representing the bottom right square, 2 representing the top left square, and 3 representing the top right square of the grid. Each level-i (0 < i < N) grid can be addressed by its grid location, which is defined as the concatenation of numbers aN−1aN−2...ai+1ai, where ai is the square number of the grid in its level-(i+1) grid.
For example, in Figure 4.1, the grid location of the level-1 grid where s resides is 333. If s was located in its level-1 neighboring grid numbered as 2, then the grid location of the level-1 grid where s resides would have been 332. In this chapter,
69 when we refer to a node’s location, we use its (x,y) coordinates and grid location interchangeably.
After partitioning the network into hierarchical grids of different levels, we com- pute and select storage locations for each data item based on its key value and the location of its source node. Next we show how the storage servers are selected in the grids at each level.
4.2.2 Data Storage Server Selection
The source node s of a data item f selects one storage server for f in each of its three neighboring level-i (0 < i < N) grids as follows. First, s hashes key(f) to get a normalized geographic location (x,y) such that x ∈ [0, 1] and y ∈ [0, 1]. Then, s computes the storage location for f relative to a neighboring level-i grid by scaling the normalized geographic location with 2(i−1) ∗ l, where l is the length of the level-i grid. Finally, s stores f in each of its neighboring level-i grids at the node nearest to the geographic location (x ∗ 2(i−1) ∗ l, y ∗ 2(i−1) ∗ l) relative to the lower left corner of the corresponding level-i grid.
To guarantee that each computed storage location belongs to exactly one grid at each level, we specify that the storage locations along the border of two horizontally adjacent grids belong to the left grid, and those along the border of two vertically adjacent grids belong to the bottom grid.
As shown in Figure 4.2, s with location of (7.75,7.75) hashes key(f) to get
(x, y) ≡ (0.5,0.5) and then stores f in its neighboring level-1 grids at locations
(6.5,7.5), (6.5,6.5), and (7.5,6.5), which are at the offset of (0.5,0.5) from the (x,y)
70 s Level−1
Level−2
Level−3
Level−4
Figure 4.1: Hierarchical partitioning of the network into four levels. Bold lines show each level-i grid that node s belongs to
71 (0,16) (16,16)
(4,12) (12,12) L 4,2 L4,3
(6,5,7,5) s (7.75,7.75) (5,7) L1,2 (6.5,6.5) L 2,2 (7.5,6.5) L1,0 L1,1 (2,6) L (5,5) (7,5) 3,2 L2,1 L2,0 r(7,2,4.2) (12,4) L4,1 (2,2) (6,2) L3,0 L3,1
(0,0) (8,0) (16,0)
Figure 4.2: Example of SDSR hierarchical storage location selection, multicast rout- ing tree construction, and query location selection. The solid circles are the storage locations selected by source node s, the dotted circles are the query locations se- lected by requester node r
72 coordinates of the lower left corners of the corresponding grids - (6,7), (6,6), and
(7,6), respectively. Node s repeats the process at each of the neighboring level-i grids by scaling the offset by 2(i−1) ∗ l, i.e., for neighboring level-2 grids the offset is
(1.0,1.0), for neighboring level-3 grid the offset is (2.0,2.0), and so on. Node s stores f at nodes nearest to locations (5,7), (5,5), and (7,5) in each neighboring level-2 grid; at nodes nearest to locations (2,6), (2,2), and (6,2) in neighboring level-3 grids; and (4,12), (12,12), and (12,4) in neighboring level-4 grids; respectively. To efficiently distribute the data item from the source node to its storage servers, we construct a multicast routing tree, which is presented in the following section.
4.2.3 Multicast Storage Tree Routing
The goal of multicast tree routing is to construct a tree T , with source node s, and the corresponding data storage locations, L1,2, L1,0, L1,1, etc. as the nodes in the tree so that a data item can be routed from root s to the storage locations along the edges of T with optimal total communication cost and optimal network delay/data delivery success probability.
Traditional tree construction schemes, such as rooted hortest path tree shown in
Figure 4.3 or rooted minimum spanning tree shown in Figure 4.4, optimize either the network delay/data delivery success probability or the total communication cost but not both.
Specifically, since the shorter the distance between a source node and a storage server, the fewer the network-level hops required to deliver packets from the source node to the storage server and the higher the data delivery success probability, the
73 L1,2 S
L2,2
L1,0 L1,1 L3,2
L2,0 L2,1
L3,0 L3,1
Figure 4.3: Example of the shortest path tree constructed from root s to the selected storage locations of the bottom left grid of the network shown in Figure 4.2
74 S L1,2 L2,2
L1,0 L1,1 L3,2
L2,0 L2,1
L3,0 L3,1
Figure 4.4: Example of the minimum spanning tree constructed from root s to the selected storage locations of the bottom left grid of the network shown in Figure 4.2
75 rooted shortest path tree yields optimized network delay/data delivery success prob- ability; however it suffers from increased total communication cost. This is because the source has to send m copies of the data item where m is close to the total number of destinations.
On the contrary, the minimum spanning tree minimizes the total communication cost in storing the data item at the storage locations, because storage locations along the same path can share the same data copy and, thus, the communication cost.
However, it increases the network delay/data loss probability by introducing longer paths from the source to the storage locations.
For example, in Figure 4.4, one possible path from s to L3,1 is s −→ L1,2, L1,0,
L1,1, L2,1, L2,0, L3,1. The length of the path is much longer than the shortest path s −→ L3,1, and, thus, introduces longer network delay and lower data delivery success probability for data items destined to L3,1 from s.
We propose a new multicast tree routing algorithm that aims to strike a compro- mise between minimizing the total communication cost and the data delivery path length. The algorithm is as follows: initially, the tree T only contains the source node s, the tree then grows iteratively, such that in the ith-iteration all level-i stor- age locations join the tree by forming an edge with a node in the tree, which is at minimum Euclidean distance from them, under the constraint that no two nodes at the same level can have an edge between them. This constraint gives us the property that the depth d of a level-i node in the tree is always ≤ i.
76 Performance metrics total communication cost total storage path length Multicast tree 36.37 55.96 Shortest path tree 50.35 50.35 Minimum spanning tree 34.96 80.3
Table 4.1: Comparison of the performance of different storage routing schemes in routing packet from s to its storage locations shown in Figure 4.2
As shown in Figure 4.2, in the first iteration, all level-1 storage locations, L1,2,
L1,0, and L1,1, join the tree such that the edges formed are e(L1,2, s), e(L1,0, s), and e(L1,1, s). In the second iteration, all level-2 storage locations, L2,2, L2,0, L2,1, join the tree such that the edges formed are e(L2,2,L1,2), e(L2,0,L1,0), and e(L2,1,L1,1).
Similarly, other storage locations in the network join the tree. The source node sends out one copy of the data item along each separate path. As shown in Figure
4.2, s sends one copy of the data item to L1,2,L1,0, and L1,1 separately.
Table 4.1 compares the performance of the above three storage routing schemes with regard to the total communication cost incurred and the it total path length traversed while storing a unit packet from root s to the selected storage locations in Figure 4.2. It shows that our proposed multicast tree gives near optimal results in optimizing both total communication cost and total path length.
77 (1) sn.j ∧ f.j ∧ ¬ hf .j −→ (x.j, y.j), hf .j := hash(key(f.j)), true []
(2) sn.j ∧ f.j ∧ hf .j ∧ ¬s.j −→ storage.j; s.j := true []
(3) sn.j ∧ f.j ∧ hf .j ∧ s.j ∧ ¬m.j −→ multicast.j; m.j := true []
(4) (sn.j ∧ hf.j ∧ s.j ∧ m.j ∨ ¬sn.j) ∧ f.j ∧ ¬r.j −→ route.j; r.j := true
Multicast storage tree algorithm. In the above program, each node j main- tains the following variables:
1. sn.j: a boolean variable. A true value of sn.j signifies that j is the source
node of the data;
2. f.j: data present at node j;
3. hf .j: a boolean variable. A true value of hf .j signifies that data f has been
hashed. A false value signifies that f has not been hashed;
4. x.j: x-coordinate of the offset value;
5. y.j: y-coordinate of the offset value;
6. s.j: a boolean variable. A true value of s.j signifies that the storage locations
of data f have been computed;
7. m.j: a boolean variable. A true value of m.j signifies that the multicast tree
of data f has been computed; and
78 8. r.j: a boolean variable. A true value of r.j signifies that the data have been
successfully routed to the higher level neighbors of j in the multicast tree.
A node j uses the following modules (functions) for (i) computing the locations of the storage servers (module storage.j); (ii) constructing the multicast tree on top of the storage servers (module multicast.j); and (iii) routing data items to the storage locations using the multicast tree (module route.j).
1. storage.j: module for calculating the storage locations for f. It takes f,
current location of j, and hf .j as input, and outputs storage locations.
2. multicast.j: module for constructing the multicast tree. It takes the output of
storage.j module as input, and outputs a multicast tree used by source node
for routing messages.
3. route.j: module for routing messages. It uses the underlying GPSR to route
data item f along the edges of the multicast tree.
4.2.4 Data Query Algorithm
To locate a particular data item f, which was stored by using the above multicast tree program, a requester node r first hashes key(f) to get the normalized (x, y) coordinates. It then computes the possible storage locations for f in the network using the normalized (x, y) coordinates as an offset.
Query location selection. In SDSR, a requester node r computes the query locations in a similar way as the source node computes the storage locations for f.
79 Starting from the level-1 hierarchy, r selects one query location from each of its 3 neighboring level-i (0 For example, in Figure 4.2, the set of query locations chosen by r located at (7.2,4.2) are (6.5,4.5), (6.5,5.5), (7.5,4.5), and (7.5,5.5) for level-1 hierarchy; (5,5), (7,5), (7,7), and (5,7) for level-2 hierarchy; (2,6), (2,2), (6,2), and (6,6) for level-3 hierarchy; and (4,4), (12,4), (12,12), and (4,12) for level-4 hierarchy. SDSR query algorithm. Requester r queries the set of computed query locations in a hierarchical way, starting from its level-1 grids. In each level, instead of querying all the query locations, r only queries the storage location that is closest to itself. The query terminates either when r gets f from any one of the query locations or if it does not get f from the highest level query locations. For example, in Figure 4.2, the query path for f from r is: (7.5,4.5) −→ (7,5) −→ (12,4). The storage server of f that is closest to location (7,5) answers the query during the second query step. Querying only the closest query location in each level does not work when the source node s, the requester node r, and the closest level-1 query location of f are in the same level-1 grid. To handle this, if r cannot locate f in the query location of its own level-1 grid, r broadcasts within the grid before querying the closest query location in the level-2 hierarchy. SDSR query scheme is scalable since it not only limits the number of steps needed to satisfy a query but also bounds the geographic region in which the query 80 will propagate. This is because the query stays inside a grid, which is one of the four low level squares of the smallest grid containing both the source and the requester. It is important to mention that the above query scheme works in a static network with no faults. In Section 4.2.6, we provide additional solutions to help SDSR handle faults introduced by node mobility, node failure, and clustering failure. 4.2.5 Data Update Algorithm In our model, each data item can only be updated by its source node. After a node updates a data item, it stores the updated item at its storage locations. The set of storage locations selected may change from time to time as the node moves within the network. But this change is local and is bounded by O(D) where D is the diameter of the smallest grid, which contains the node’s current location and its previous location where the last update occurs. Take s in Figure 4.1 as an example. Initially, the level-1 grid location of s is 0333, and it selects storage locations for f using the scheme described in Section 4.2.2. When next time s updates f, if s is still in 0333, then the set of selected storage locations for f remains unchanged. If s moves to 0332, then s needs to remove the previous storage location in 0332 and select a new storage location in 0333. If s moves to 0322, then s needs to remove the three previous storage locations in 033 and recruit three new storage locations in 032, and one storage location in 033. So in order to maintain a concurrent view of f, s needs to remember its previous location where f is updated last time. The maintenance of storage locations is local and is 81 bounded by O(D), where D is the diameter of the smallest grid containing current and previous locations of s. 4.2.6 Fault Tolerance Wireless ad hoc networks have frequent topology changes (modeled as faults) due to node joins, node fail-stops, and node mobility. We achieve fault-tolerance by employing an indexing service, which is explained below. Motivation. The storage service of SDSR stores a data item f at the node j nearest to the corresponding storage location. But, when a new node k joins the network, it may be closer to the storage location of f than the current storage server j. Transferring f from the current storage server to the node nearest to the storage location is undesirable in the presence of frequent node joins (and moves), as it results in high communication cost, especially when the size of f is large. On the other hand, not storing f at the node nearest to the storage location (in the presence of node joins) results in broadcasting the query for f. In oberving the above conerns, we propose an indexing service for avoiding the high communication cost due to either frequent data transfers or frequent query broadcasts. Indexing service. The new node k, instead of getting f from the storage server j, serves as an index for f and only stores the information in its memory that there exists a storage server of f in the grid. k also does not store the location of j, as it requires j to update k whenever its location changes. Thus, we avoid the data transfer cost by only storing the indexing information. 82 Since a query for a data item is routed to the node nearest to the storage location, on receiving the query for f, k broadcasts the query in the grid, as it has the information that f exists in the grid. When storage server j receives the query, j replies back to k with f, which replies f back to the query node r. With f stored, k is able to serve the subsequent queries for f directly. Another advantage of the indexing approach is that if f is not in the grid, then the query is not broadcasted by k, thus unnecessary communication cost is avoided. The index list for a storage location is copied to each node along the perimeter of the storage location. Thus, when the node closest to the storage location fails, the second closest node can serve the subsequent queries correctly. Node mobility. Node movement has minimal effects on the data storage struc- ture as long as storage servers move in their original level-1 grids. If storage server j of data f moves outside its level-1 grid, j first transfers f to the node nearest to the storage location in the original grid, then deletes f and seizes to serve as a storage server for f. Node fail-stop. The source node of a data item periodically sends soft state hello messages with low frequency to the storage servers to detect storage server fail-stops and recruit new servers accordingly. In order to handle clustering failures, a requester node can query multiple storage servers in each hierarchy level and retrieve the data item from the first of those that reply with the data. 83 N the number of nodes in the network. Similar to [74], we use O(N) to represent the cost of flooding√ a data item to the whole network, and use O( N) to represent he cost of sending a message between any two nodes in the network d the replication depth used by SDSR M the number of data items stored in the network Q the number of queries generated in the network Rsdsr the number of replicas for each data item in SDSR (Rsdsr = 3d) Table 4.2: Notations 4.3 Analytical Results In this section, we first compare SDSR with existing data dissemination schemes with regard to the total communication cost and hotspot usage. Then we compare SDSR with GHT with regard to resilience to clustering failures. Table 4.2 lists the notations we use to compute the communication cost and Table 4.3 lists the performance metrics used in our comparison. 4.3.1 Communication Cost In this section, we compare the total communication cost and the hotspot usage of the following data dissemination schemes: Local Storage, Centralized Storage, GHT, and SDSR. 84 Total storage message complexity Cs the total number of storage messages generated Total query message complexity Cq the total number of query messages generated Total reply message complexity Cr the total number of reply messages generated Total message complexity Ctotal the total number of messages generated Hotspot message complexity H the maximum number of messages processed by each node in the network Table 4.3: Performance metrics 1. LS. Storing a data item locally incurs no cost. The cost of querying a data item is O(N) since each query message has to be flooded to all the nodes and the cost of √ sending a reply is O( N). √ Cs = 0 Cq = QN Cr = Q NH = Q √ 2. CS. The cost of storing a data item to a centralized server is O( N), the √ √ cost of querying a data item is O( N), and the cost of sending a reply is O( N). √ √ √ Cs = M ∗ NCq = Q NCr = Q NH = M + Q √ 3. GHT. The cost of storing a data item to a fixed location is O( N), the cost √ √ of querying a data item is (O( N), and the cost of sending a reply is O( N). √ √ √ Cs = M NCq = Q NCr = Q NH = max(M/N, 1) + Q/min(M,N) √ 4. SDSR. The storage cost for a data item is 6 N(1 − 1/2d), since we store three copies of the data item in each hierarchy level and the multicast tree routing 85 enables the sharing of storage cost among storage servers along each path. The √ query cost for a data item is N(2d + 2d−1 − 2d + 1)/((d − 1)2d−1). Reply is sent √ back at the cost of N/d(1 − 1/2d−1). √ √ d d d−1 d−1 Cs = 6M N(1 − 1/2 ) Cq = Q N(2 + 2 − 2d + 1)/((d − 1)2 ) √ d−1 Cr = Q N/d(1 − 1/2 ) H = max(3dM/N, 1) + Q/(min(M,N)(3d + 1)) Table 4.4 shows the total communication cost generated by each scheme, and Table 4.5 shows the maximum number of storage and query messages processed by one node in each scheme. From Table 4.4 and Table 4.5, we make the following observations (i) LS incurs the largest total communication cost with increased N; (ii) CS yields the highest hotspot message counts; (iii) SDSR has the lowest hotspot usage, since, firstly, the task of storing data is distributed among all the nodes; and secondly, each data item has Rsdsr replicas, which are distributed across the network; and (iv) SDSR introduces the highest storage cost, since it replicates data in a multilevel hierarchy for achieving better fault tolerance and reduced query latency, yet it yields the lowest query cost and reply cost. From the above we conclude that SDSR is preferable to other schemes with regard to the total communication cost when (i) N is large enough; (ii) Q > M; and (iii) network query exhibits locality. For example, let N = 2500, M = 100, Q = 1000, and d = 4, then the total communication cost of each scheme is shown in Table 4.6. 86 Schemes total messages√ (Cs + Cq + Cr ) LS 0 +√QN + √Q N √ CS M√N + Q√N + Q√N GHT M √N + Q N + Q N d SDSR 6M√ N(1 − 1/2 )+ d d−1 d−1 Q√N(2 + 2 − 2d + 1)/((d − 1)2 )+ Q N/d(1 − 1/2d−1) Table 4.4: Total communication cost generated by each scheme Schemes Hotspot usage of each scheme LS Q CS M + Q GHT max(M/N, 1) + Q/min(M,N) SDSR max(3dM/N, 1) + Q/(min(M,N)(3d + 1)) Table 4.5: Hotspot usage Schemes Total messages LS 255000 CS 105000 GHT 105000 SDSR 77827 Table 4.6: Total messages generated when N = 2500, M = 100, Q = 1000, and d = 4 87 4.3.2 Resilience to Clustering Failures Clustering failures occur when all nodes within the same grid fail. GHT is vul- nerable to clustering failures, since it replicates each data item only locally around its hashed location but not across the network. SDSR performs better in handling clustering failures, as it replicates each data item at nodes that are distributed in different network regions. If each level-1 grid has the same clustering fault proba- bility q, then the probability that all the grids containing data item f fail is q for GHT, while only q3d+1 for SDSR. 4.3.3 Data Placement Consistency Maintenance Cost In this subsection, we show the cost associated with maintaining data consistency across all storage servers for a particular data item for GHT and SDSR respectively. GHT utilizes GPSR perimeter routing to identify a data item f’s home node h (the node closest to the hashed geographic location for f). After f reaches its home node h, since there is no node closer to the hashed location than h, GPRS enters perimeter mode and routes f around the perimeter encircling the storage location (home perimeter) before coming back to h. To ensure a consistent placement of each data item at its home node while network topology changes, GHT replicates f to all the nodes along its home perimeter, and uses perimeter refresh protocol to periodically refresh the consistency between f and all these nodes along the home perimeter. 88 GHT’s perimeter refresh protocol suffers from (i) high storage space cost, because besides the home node, each node along the perimeter of each storage location needs to store all the data items hashed to that location; and (ii) high communication cost, because the home node needs to periodically send the refreshment around its home perimeter for each data item. SDSR employs a lightweight indexing service to reduce the storage cost and the communication cost incurred by consistency requirements. SDSR indexing service allows the current home node j for a storage location l to have a consistent view of all the data items that have been hashed to this storage location. j shares its index list for location l with each node along the perimeter of l, thus when j fails, the node that is second closest to l can handle the subsequent queries correctly. It is evident that since SDSR does not replicate each data item to all the nodes along the perimeter of a location server, it consumes much less storage space than GHT. Also, since only the index list but not the data item, is copied along the perimeter of a location server, SDSR introduces lesss communication cost than GHT when data size or the number of data items hashed to a storage location is large. 4.4 Simulations In this section, we compare the performance of SDSR with other schemes through simulation, and present our simulation results. In [74], Ratnasamy et al. verified the correctness and robustness of GHT using detailed simulation in NS, which included a full MAC-layer behavior, packet loss, 89 and node dynamics, etc. Since SDSR can be seen as GHT with hierarchical level extension, we believe that SDSR will be viable in the same simulation environment as that of GHT. As the detailed NS simulation can not scale to more than 200 nodes and we would like to compare the performance of SDSR with other schemes in large-scale wireless networks, we use higher level simulations (without radio details) in our work. In our simulations, we assume that nodes are stationary and that the packets are transmitted to neighbors instantaneously and error-free. We are interested in finding out how SDSR performs as Query Storage Ratio (QSR), replication depth (d), and number of nodes (N) vary individually. We compare SDSR to GHT, LS, and CS. In each simulation, nodes are randomly placed in the network and data items are inserted/queried to/from the network according to the scheme simulated. Unless specified otherwise, the density of the network is set to be 1/256m2, and the communication radius is 40m, the same as those used in [74]. We compute each simulation result from the average value of 10 runs with 10 randomly generated topologies. 4.4.1 Varying QSR In this simulation, we set N = 100, d = 2, M = 10, and compute the total com- munication cost with varied QSR. Figure 4.5 shows the simulation results. We see that as QSR increases, the total communication cost for all four schemes increases, and the cost generated by SDSR increases slower than the other three schemes, and 90 Figure 4.5: Total message overhead with different QSR values the one generated by LS increases the fastest. LS generates the highest total com- munication cost under all cases because each query needs to be flooded to the whole network. When QSR is low, SDSR generates higher communication cost than GHT; while when QSR is high, SDSR generates the least total communication cost. The above results indicate that SDSR performs better than the other schemes in query dominant wireless ad hoc networks. 4.4.2 Varying Number of Nodes N In this section, we show the results of varying N, number of nodes in the network. We set M = 100 and QSR = 10 in all cases. The replication level of SDSR is adjusted such that the average number of nodes in each level-1 grid remains the 91 Figure 4.6: Total message overhead as the number of nodes increases, where QSR=10 same under all simulation runs. From Figure 4.6 we see that SDSR incurs the lowest communication cost among all schemes, and that LS has the highest total communication cost. 4.4.3 Varying Replication Depth d The replication depth d is an important design parameter, as it determines the number of storage servers for each data item, the average number of nodes in each grid, and thus can affect the data storage/query cost. In this simulation, we let N = 10000, QSR = 10, and compare the performance of SDSR with GHT with 92 Figure 4.7: Storage overhead for different hierarchical partitioning levels varied replication depth d. Figure 4.7 shows the total storage cost, Figure 4.8 shows the total query cost, and Figure 4.9 shows the total communication cost. From Figure 4.9 we can see that the total storage cost and query cost of GHT remain constant due to the reason that GHT does not partition the network into hierarchical levels. Figure 4.7 shows that SDSR has higher storage cost than GHT and that its storage cost increases as d increases; however, the increase is slow since the multicast routing protocol allows storage servers along the same path to share a single copy of the data item. Figure 4.8 shows that the total query cost of SDSR is 93 Figure 4.8: Query overhead for different hierarchical partitioning levels 94 Figure 4.9: Total message overhead for different hierarchical partitioning levels 95 much less than that of GHT. This is because, as opposed to GHT, where each query is bounded by the entire network, in SDSR each query is bounded by the smallest grid, which contains both the query node and a storage server for the data item. 4.5 Chapter Summary In this chapter, we presented the design and evaluation of SDSR, a data-centric, location-based data storage and retrieval service for enhancing data dissemination in large-scale wireless ad hoc networks. SDSR provides data storage and retrieval service on which various applications can be built. Our analytic and simulation results show that in a query dominant, large-scale wireless ad hoc network, SDSR performs better than existing schemes in terms of total communication cost, data query latency, hotspot usage, and resilience to clustering failures. It complies well with query locality and scales well when the network size and the number of queries increase. We are interested in extending SDSR so that it can handle node storage capacity limit. We are also interested in finding out how SDSR can be efficiently employed in hierarchically clustered wireless ad hoc networks. 96 CHAPTER 5 DISRUPTION TOLERANT GEOGRAPHIC ROUTING FOR WIRELESS AD HOC NETWORKS In wireless ad hoc networks, temporary link disruptions can frequently occur due to the presence of obstacles, node mobility, etc. These disruptions can result in the incorrect perception by a node about its neighbor set when the neighbor set is constructed via beacon sampling, e.g., beacon collisions may result in the removal of a node j from the neighbor set of a node i, although j is within the transmission range of i. Such a behavior can adversely affect the performance of position-based routing algorithms as it may lead to inefficient routing or packet dropping. To address this, we propose a scheme that allows each node to associate each of its neighbors with a reachability value that is a measure of the stability of the link between the node and the neighbor. We then design two new routing algo- rithms, Disruption Tolerant Geographic Routing-Simple Forwarding (DTGR-SF) and Disruption Tolerant Geographic Routing-Waiting before Forwarding (DTGR- WF), in which nodes utilize reachability values to make forwarding decisions. We compare the performance of DTGR-SF and DTGR-WF with that of Greedy Perime- ter Stateless Routing (GPSR) in various simulation settings. Our simulation results 97 show that our proposed algorithms perform better in settings where disruptions are present. In networks with few occurrences of disruptions, our schemes achieve the same high performance as that of GPSR. The remainder of this chapter is organized as follows: Section 5.1 presents our re- search motivation and problem statement; Section 5.2 presents our proposed scheme for link reachability value computation; Section 5.3 applies reachability value to GPSR and presents routing algorithms DTGR-SF and DTGR-WF; simulation re- sults are shown in Section 5.4; Section 5.5 discusses the overhead introduced by DTGR; and finally, Section 5.6 makes concluding remarks. 5.1 Motivation and Problem Statement In this section, we present the motivation and problem statement of our research. 5.1.1 Motivation As in position-based algorithms, each node selects the next hop for packet for- warding solely from nodes stored in its neighbor table, the correctness of the neighbor table is crucial in determining the performance of these algorithms. In general, the construction and maintenance of the neighbor table are through a simple beacon sampling mechanism: each node periodically broadcasts beacons to its neighbors to announce its ID (e.g., IP address) and location. When a node i receives a beacon from a node j, i stores j’s ID and location in its neighbor table. If i does not receive a beacon from its neighbor j within a time-out interval T , where T is a multiple 98 of beacon interval B, i concludes that j is unreachable and, thus, immediately re- moves j from its neighbor table. The problem is that node i’s own perception that neighbor j is unreachable may not be correct. This is because neighbors usually contend with each other to send beacons over a shared channel [98], and hence, the beacon from neighbor j can be lost due to the collision with other beacons, even if j is indeed reachable. The results presented by Huang et al. in [37] further illustrate the above problem. Through analysis and simulation, the authors showed that in IEEE 802.11 ad hoc network, where all nodes are neighbors of each other, the probability that a specific node successfully sends a beacon in one beacon interval is low even in a small network, e.g., 19.7% for a network of five nodes; and the probability decreases fast as the network size increases, e.g., 4% for a network of twenty five nodes. Given a network with twenty five nodes, even if we set the neighbor time-out interval to be as long as twenty beacon intervals, the probability that a reachable neighbor is incorrectly considered as unreachable is still as high as (1 − 4%)20 = 44.2%. The beacon loss ratio is expected to be much higher if we take the transmission of data packets into consideration. The link disruptions resulting from background noises, multi-path fading, and obstructions such as moving objects, etc. further worsen the situation. 5.1.2 Problem Statement We consider a large-scale mobile ad hoc network in a 2-D coordinate plane. All nodes are assumed to have the same transmission range R. A link e(i, j) exists 99 between two nodes i and j iff the Euclidean distance, dist(i, j), between them is less than or equal to R. Nodes i and j are neighbors if e(i, j) exists. The set of nodes and links are represented by sets V and E, respectively, and the resultant undirected graph is represented by G, where G = (V,E). We use si,j to denote the state of e(i, j), where si,j ∈ {0, 1}. The value 1 indicates that nodes i and j can directly communicate with each other, and 0 indicates that i and j can not directly communicate with each other. We use pi,j to denote the state of e(i, j) perceived by the node i through beacon sampling, where pi,j ∈ {0, 1}. pi,j = 1 indicates that i perceives the state of e(i, j) as 1 and, thus, j is reachable, 3 and pi,j = 0 indicates otherwise. A node i has an incorrect perception of the state of the link e(i, j) iff pi,j 6= si,j, e.g., si,j = 0 and pi,j = 1, or si,j = 1 and pi,j = 0, with the latter being the focus of the study in this chapter. Temporary disruptions in conjunction with beacon sampling may result in the incorrect perception by a node about its neighbor set. This may result in the removal of a reachable neighbor from the neighbor set of the node, which in turn may adversely affect the performance of position-based routing algorithms in terms of packet delivery success ratio and packet delivery latency. To address the above problem, our goal is to propose a scheme that (i) improves the performance of position-based routing algorithms in the presence of temporary disruptions; (ii) does not compromise the overall packet delivery success ratio and packet delivery 3 Note that (i) both si,j and pi,j may change as time varies and (ii) pi,j can be different from pj,i. 100 latency in networks with no or few occurrences of temporary disruptions; and (iii) introduces minimal overhead in terms of space and time requirements. 5.2 Reachability Value Computation Scheme To achieve the above goal, we propose assigning a reachability value for each link e(i, j). In this section, we first present our scheme for computing the reachability value for a link. We then present the construction of the neighbor table using the computed reachability values. Each node i associates a link between itself and its neighbor j with a value ri,j ∈ [0, 1], which represents the stability metric of the link as perceived by node 4 i. For each node i, a link e(i, j) is stable if e(i, j) ∈ E and ri,j = 1, or unstable if e(i, j) ∈ E and rthreshold ≤ ri,j < 1, where rthreshold is a constant between (0, 1). The link e(i, j) is considered as unreachable if 0 ≤ ri,j < rthreshold. For each node i, a neighbor j is stable if e(i, j) is stable, unstable if e(i, j) is unstable, and unreachable if e(i, j) is unreachable. Since unstable neighbors have higher probability of being unreachable than stable neighbors, they should be considered with lower priority than stable neigh- bors when a node makes forwarding decisions. These unstable neighbors should be utilized only as alternatives when selecting next hop from only stable neighbors might force the routing algorithm either to drop the packet or to route the packet inefficiently. Associating each neighbor with a reachability value allows a node to 4 Note that (i) ri,j may change as time varies and (ii) ri,j can be different from rj,i. 101 efficiently utilize both stable neighbors as well as unstable neighbors in packet for- warding. Each node i, in addition to storing the address and location of each of its neigh- bors j, also maintains j0s reachability value in the neighbor table. Node i uses beacons to approximate ri,j as follows ( 1, if l ≤ T rˆ = i,j max(1 − 2(l−T ) × 0.1, 0), otherwise. In the above formula, T is a given constant representing the time-out interval, l is defined as the number of consecutive beacon intervals during which node i has not received a beacon from j. If node i has received at least one beacon from j in the last T beacon intervals, it assigns j a reachability value of 1. Otherwise, starting from the (T + 1)th beacon interval, i decreasesr ˆi,j exponentially. We achieve this by exponentially increasing the value by whichr ˆi,j is decreased. For example, if l = 4 and T = 3, thenr ˆi,j = 0.8. When the reachability value of neighbor j is below rthreshold, i removes j from its neighbor table. Notice that if i receives at least one beacon from an unstable node j or from a new node, i assigns the node a reachability value of 1. The introduction of a reachability value to differentiate between stable and unstable neighbors allows a forwarding node to consider stable neighbors as first priority in making packet forwarding decision, while utilizing unstable neighbors as alternatives when a packet cannot be forwarded to any of the stable neighbors in the current routing mode. 102 5.3 Application: GPSR The removal of a reachable neighbor may prevent the position-based routing al- gorithms from choosing an efficient neighbor for packet forwarding. In this section, using GPSR as an example, we first identify the possible scenarios where the perfor- mance of position-based routing algorithms is compromised due to the removal of a reachable neighbor from the neighbor table. We then apply the link reachability scheme on GPSR and present the resultant Disruption Tolerant Geographic Routing algorithms. 5.3.1 Adverse Effects of the Incorrect Removal of a Reachable Neighbor from the Neighbor Table In both Figure 5.1 and Figure 5.2, i is the forwarding node for a packet destined for node D. si,j=1 and pi,j=0 and, thus, node j is a reachable neighbor of i, but i has incorrectly removed j from its neighbor table. The removal of j from the neighbor table causes i to either drop the packet (as shown in Figure 5.1) or to route the packet in perimeter mode (as shown in Figure 5.2). Dropping the packet requires the packet to be retransmitted, and forwarding the packet in perimeter mode usually introduces a much longer route, higher packet delivery latency, and a higher packet loss ratio than forwarding the packet to j in greedy mode. 103 D D k v j i j u i (a) (b) Figure 5.1: Dropping of a packet destined for D due to the removal of a reachable neighbor j from i’s neighbor table 104 D o n j m i l k Figure 5.2: Inefficient perimeter routing of a packet destined for D due to the removal of a reachable neighbor j from i’s neighbor table 105 5.3.2 Disruption Tolerant Geographic Routing (DTGR) We apply the reachability attribute to GPSR and design Disruption Tolerant Geographic Routing (DTGR) protocol. In DTGR, each node constructs and main- tains its neighbor table as presented in Section 5.2. In the rest of this section, we explain the DTGR forwarding procedure, and schemes to forward a packet to a selected unstable neighbor in detail. 5.3.3 DTGR Packet Forwarding Procedure A forwarding node i always attempts to select a stable neighbor to whom it will forward the packet. It selects unstable neighbors only under the following critical conditions: 1. Before setting a greedy mode packet into perimeter mode. In GPSR, when a greedy mode packet gets stuck at a node i that does not have a closer neighbor to the destination, i sets the packet into perimeter mode and routes it along a planar subgraph using right-hand rule. Although perimeter routing allows a packet to recover from local maxima, it usually introduces longer routes, higher packet delivery latency, and higher packet loss ratio, since nodes that can be reached via a path with few hops might become far apart after planarization. In DTGR, we use the reachability value to assist node i in continuing to forward the packet in greedy mode. Thus when node i receives a packet m in greedy mode, i first tries to forward m in greedy mode using neighbors 106 with reachability value of 1, as shown in step 1 of DTGR Greedy.i.module in Figure 5.4. If all i’s neighbors with reachability value of 1 are further to the destination than i, then, instead of setting the packet into perimeter routing mode immediately, i tries to select a neighbor from those with a reachability value of less than 1 to continue forwarding the packet in greed mode, as shown in step 2 of DTGR Greedy.i.module in Figure 5.4. 2. Before dropping a packet. In GPSR, a forwarding node i drops a packet when there is no neighbor j to which it can forward the packet in either greedy mode or perimeter mode. In DTGR, instead of dropping the packet, forwarding node i tries to utilize the unstable neighbors to forward the packet. In greedy mode, unstable neighbors will be tried as described above. If no such unstable neighbors are available, i will try to forward the packet in perimeter mode on the local planar subgraph constructed from its stable neighbors. If the packet cannot be forwarded to any of the stable neighbors in perimeter mode either, then instead of dropping the packet, i first constructs a planar subgraph consisting of both stable and unstable neighbors, and then forwards the packet in perimeter mode to an unstable neighbor, if available. 3. DTGR graph planarization. The routing of a packet out of the local maxima using perimeter routing requires the removal of cross links from the underlying connectivity graph. In GPSR, each node has only one level local graph, i.e, the node maintains local graph formed by links between the node 107 and its stable neighbors only. Assuming that the underlying network is al- ways connected, each node runs planarization algorithms such as RNG or GG in a distributed manner to remove local cross links. However, in a network where temporary link disruptions occur frequently, the underlying network constructed only by stable nodes and links might not be connected all the time and, thus, the resultant planar subgraph also might not be connected, which may result in packet dropping (see Figure 5.1(b)). In DTGR, each node has at least two levels of local graphs: the first level formed by links between the node and its stable neighbors (neighbors with reachability value 1) and the second level formed by links between the node and all its neighbors with rthreshold ≤ r < 1. After determining that a packet (either in greedy mode or perimeter mode) has to be forwarded in perimeter mode, the node first checks if there exists a next hop neighbor by applying right-hand rule on the first level planar subgraph as shown in step 2 of DTGR Greedy.i.Module and step 2 of DTGR Perimeter.i Module in Figure 5.5. If no such neighbor exists, the node then tries to select the next hop neighbor by applying the right-hand rule on the second level planar subgraph as shown in step 4 of DTGR Greedy.i.Module and step 3 of DTGR Perimeter.i Module in Figure 5.5. Since (i) long links have high probability of being unstable, and (ii) both RNG and GG prefer shorter links to longer ones, the probability that a stable cross link is removed due to short unstable links is low. 108 Figure 5.3: DTGR protocol parameter definition 109 Figure 5.4: DTGR greedy forwarding procedure 110 Figure 5.5: DTGR perimeter routing procedure 111 5.3.4 Schemes for Forwarding a Packet to a Selected Unstable Neighbor Once node i has selected an unstable neighbor j to which it will forward the packet, it can utilize the following forwarding schemes: • Simple Forwarding (DTGR-SF): In this scheme, i forwards the packet to neighboring unstable node j immediately after i selects j. • Wait before Forwarding (DTGR-WF): In this scheme, i waits for a cer- tain time twait before forwarding the packet to j. After twait, i checks if there is a better candidate k than j (e.g., a stable neighbor to which the packet can be forwarded in greedy mode) and forwards the packet to k accordingly. Otherwise, i forwards the packet to j. In the above two schemes, if node i could not forward the packet to j successfully, i selects another node to whom it can forward the packet. If i fails to forward the packet to this node too, i drops the packet. DTGR-SF is simpler to implement and introduces no extra delay, yet it might result in a lower packet forwarding success ratio due to its best effort property. DTGR-WF should yield a higher packet delivery success ratio while introducing longer packet delivery latency, since it waits for a certain time for a stable neighbor to be available before forwarding the packet to an unstable neighbor. 112 5.4 Simulations Both DTGR-SF and DTGR-WF are implemented in NS [86], and their perfor- mance are compared with that of GPSR using the wireless extensions provided by [31]. In all of the following simulations, the time-out interval T is set to 4.5B, where B is the beacon interval. As in [48], each beacon’s transmission is jittered by 50% of the beacon interval B in order to avoid possible neighbor beacon synchronization. The random waypoint model [9] is used to simulate node mobility. In all simulations, rthreshold used by DTGR-SF and DTGR-WF is 0.6. To simulate temporary link disruptions resulting from obstructions, noises, and beacon collisions, each mobile node is configured with the NS multistate error model, where each node’s state is either on (0 packet loss ratio) or off (100% packet loss ratio) at various periods. The on and off time durations follow the normal distribu- tion. The link-based error model is not used, since, as far as we know, NS does not have a link object to support this function in wireless network settings. For each simulation setup, three different randomly generated motion patterns and topologies are run, and the average of their performance is calculated as our final result. The performance of DTGR-SF, DTGR-WF, and GPSR are compared by varying disruption durations under various models of traffic, mobility, network density, and topology. The performance metrics used are (i) packet delivery success ratio, which is defined as the ratio of the total number of packets received by all destinations in the network over the total number of packets sent by all sources in the network; and 113 (ii) average packet delivery latency, which is defined as the average delivery time of all packets that are successfully received by all destinations in the network. Networks with temporary disruptions resulting from obstructions, noises or beacon collisions. The simulations are performed in both static and mobile networks with 50 nodes randomly placed in a 1500 × 600 rectangular area. In the mobile networks, the maximum speed of each node is 30 m/s. For both static and mobile network settings, the beacon interval is 0.8s, and the twait of DTGR-WF is 1.6s. Each simulation is run for 500s, during which 15 CBR sources transmit 64B packets at 1KB/s. In all simulations, each node’s state is periodically switched between on and off at different times, the average on time is fixed as 20s with a standard deviation of 5s, and the average off time varies from 2.0s to 3.5s, with a fixed standard deviation of 0.5s. Figure 5.6 and Figure 5.7 show the performance of both DTGR-WF and GPSR in a static network when the average node off duration varies. Figure 5.6 shows that as node off duration increases, the packet delivery success ratios of both schemes de- crease. However, compared with GPSR, DTGR-WF achieves a much higher packet delivery success ratio under all cases. For example, when the average node failure time is 2s, the delivery success ratio of DTGR-WF is 86.8%, which is 34.2% higher than that of GPSR. Also, the performance of DTGR-WF degrades more gracefully than that of DTGR-SF, thus, DTGR-WF is more resilient to temporary network disruptions. Figure 5.7 shows that DTGR-WF has a longer average delivery latency due to its wait-before-forwarding property. 114 Figure 5.8 shows the performance of the above schemes in a dynamic network, in which each node moves at a maximum speed of 30 m/s. Here the packet delivery success ratio has the same trend as that in the static network, with DTGR-WF showing a significant gain over GPSR. As shown in Figure 5.9, the delivery latency obtained from both schemes is higher than that found in the static network, and the difference in latency between DTGR-WF and GPSR is less than that found in static networks. 100 80 60 40 Delivery Success Ratio (%) 20 GPSR DTGR-WF 0 0 2.0 2.5 3.0 3.5 Node Failure Time (s) Figure 5.6: Packet delivery success ratio under different node failure durations in a static network 115 5 GPSR DTGR-WF 4 3 2 Delivery Latency (s) 1 0 0 2.0 2.5 3.0 3.5 Node Failure Time (s) Figure 5.7: Packet delivery latency under different node failure durations in a static network 116 100 80 60 40 Delivery Success Ratio (%) 20 GPSR DTGR-WF 0 0 2.0 2.5 3.0 3.5 Node Failure Time (s) Figure 5.8: Packet delivery success ratio under different node failure durations in a mobile network where the maximum node speed = 30 m/s 117 5 GPSR DTGR-WF 4 3 2 Delivery Latency (s) 1 0 0 2.0 2.5 3.0 3.5 Node Failure Time (s) Figure 5.9: Packet delivery latency under different node failure durations in a mobile network where the maximum node speed = 30 m/s 118 Networks with temporary disruptions resulting from node mobility. We are also interested in finding out how these protocols perform in a sparse network as the degree of node mobility increases. The disruptions in such networks can be seen as frequent link insertions and deletions resulting from the frequent joining and leaving of neighbors. In a sparse network, designing a routing protocol to be resilient to high random mobility is more challenging as each node has a small neighbor set. The simulations are set such that there is a pause time of 0s, 50 nodes move randomly in a 1340 × 1340 grid, and there are 5 neighbors/node on average. Each simulation lasts 900s, during which 30 CBR sources transmit 64B packets at 2KB/s. The beacon interval is 1s and the twait in DTGR-WF is 2s. We do not count the packets dropped by the sender at the beginning of the simulation when all nodes are stationary, because the low density of the network can easily result in a situation where the sender is truly an isolated node. Figure 5.10 and Figure 5.11 show the performance of the various schemes as the maximum speed of the node varies. Again, DTGR-WF exhibits a higher packet delivery success ratio compared to GPSR and DTGR-SF, with an average gain of 37% over GPSR. The packet delivery success ratio of DTGR-SF lies between those of DTGR-WF and GPSR, with an average gain of 14.5% over that of GPSR. For example, at 40 m/s, the packet delivery success ratio is 69.1%, 55.5%, and 49.1% for DTGR-WF, DTGR-SF, and GPSR, respectively. The performance gain of DTGR- WF over GPSR is 40.7% and the performance gain of DTGR-WF over DTGR-SF is 24.5%. As for the data delivery latency, it is interesting that, despite the extra 119 2.0s that DTGR-WF might spend before forwarding a packet, it achieves the lowest average latency in almost all cases. GPSR, on the other hand, has the highest average latency. DTGR-SF’s performance falls between those of DTGR-WF and GPSR. DTGR-WF has better performance than the other two schemes because new nodes may become stable neighbors of a DTGR-WF forwarding node during the wait period, thus allowing the packet to be routed in greedy mode. In other words, DTGR-WF allows the packet to be routed using a smaller number of hops, thus results in lower latency. 100 80 60 40 Delivery Success Ratio (%) 20 GPSR DTGR-SF DTGR-WF 0 10 20 30 40 Speed (m/s) Figure 5.10: Packet delivery success ratio in sparse, highly mobile networks 120 5 GPSR DTGR-SF DTGR-WF 4 3 2 Delivery Latency (s) 1 0 10 20 30 40 Speed (m/s) Figure 5.11: Packet delivery latency in sparse, highly mobile networks 121 Stable networks. The above results indicate that DTGR-SF and DTGR-WF perform better than GPSR when disruptions are caused either by temporary node unreachability or by frequent link insertions and deletions. Next we investigate how these protocols perform in the same settings as those used in by GPSR in [48], where GPSR has shown to provide a higher packet delivery success ratio and a lower latency than other ad hoc routing protocols. As was the case in [48], we randomly place 50 nodes in a grid of size 1500 × 300 (20 neighbors/node on average), where each node moves with a maximum speed of 20 m/s, the beacon interval is set as 1.5s, and the twait in DTGR-WF is set as 3.0s. Each simulation has 30 CBR flows originating from 22 source nodes. In addition to the 0-, 60-, and 120s pause times used in [48], we also include 300-, 600-, and 900s pause times. Each simulation lasts for 900s. Figure 5.12 and Figure 5.13 present the packet delivery success ratio and delivery latency, respectively. Figure 5.12 reveals that all three schemes show similar packet delivery success ratios. In terms of data delivery latency, DTGR-SF yields the lowest latency, followed by DTGR-WF and GPSR. This is because (i) in a dense network where each node has an average of 20 neighbors and temporary disruptions are not considered, the possibility that a packet gets stuck in a local maxima is rare, thus allowing a packet to be forwarded in greedy mode most of the time; and (ii) the high node density increases the possibility that beacons from a reachable neighbor will collide with other beacons. In this case, GPSR removes the neighbor from the neighbor table, while DTGR-SF and DTGR-WF keep the neighbor in the neighbor 122 table with a low reachability value so it can still be used to forward packets in greedy mode. 100 80 60 40 Delivery Success Ratio (%) 20 GPSR DTGR-SF DTGR-WF 0 0 60 120 300 600 900 Pause Time (s) Figure 5.12: Packet delivery success ratio in stable networks 5.5 Cost Effectiveness Discussions In comparison with GPSR, the higher packet delivery success ratio of DTGR incurs the following overhead: (i) higher storage cost because of the larger number of entries stored in the neighbor table and the additional reliability attribute contained in each entry; and (ii) possible longer delivery latency when DTGR-WF is used. The above overhead is justifiable because: 123 0.5 GPSR DTGR-SF DTGR-WF 0.4 0.3 0.2 Delivery Latency (s) 0.1 0 0 60 120 300 600 900 Pause Time (s) Figure 5.13: Packet delivery latency in stable networks 124 1. Similar to GPSR, DTGR is nearly stateless, in the sense that with DTGR, each node only stores in its neighbor table the location, address, and reliability value of each of its neighbors. If we use 16 bytes to store each neighbor (8 bytes floating-point value for position coordinates, 4 bytes for address, and 4 bytes for reachability value), then the storage space required for a 100-entry neighbor table is only 1.6 KB. Considering that a network with twenty neighbors per node is already very dense [48], we believe that the extra space required for DTGR routing table is negligible. 2. The extra wait time of DTGR-WF before forwarding the packet to an unstable neighbor might result in longer average packet delivery latency. As shown in Figure 5.7 and Figure 5.9, when the disruptions are due to obstructions, and beacon collisions, etc., the average packet delivery latency of DTGR-WF is higher than that of GPSR. However, under some cases, for example, when the network is sparse and the nodes are mobile, the average packet delivery latency of DTGR-WF is the lowest, as shown in Figure 5.11. This might be due to the fact that the extra wait time of DTGR-WF allows a new stable neighbor to be used for forwarding the packet in greedy mode. Also, it is worth mentioning again that in DTGR-WF, unstable neighbors are considered only under critical conditions when considering reliable neighbors only might result in packet dropping or inefficient routing. 125 5.6 Chapter Summary In this chapter, we showed that temporary link disruptions in conjunction with neighbor table construction via beacon sampling can result in the incorrect percep- tion by a node about its neighbor set, which, in turn, can adversely affect the perfor- mance of position-based routing protocols. We then proposed a scheme that allows the node to associate each of its neighbors with a reachability value to accommodate the incorrect perception. We designed two new routing algorithms, Disruption Tol- erant Geographic Routing-Simple Forwarding (DTGR-SF) and Disruption Tolerant Geographic Routing-Waiting before Forwarding (DTGR-WF), in which nodes uti- lize reachability values to make forwarding decisions. We compared the performance of DTGR-SF and DTGR-WF with that of GPSR in various simulation settings. Some key results are that: (i) when temporary disruptions due to obstructions, beacon collisions, etc., are present, DTGR-WF performs significantly better than GPSR in terms of packet delivery success ratio with the tradeoff of increased packet delivery latency. For example, in network settings where nodes are static, DTGR- WF achieves an average 46.8% higher packet delivery success ratio than GPSR; (ii) when temporary disruptions due to node mobility are present, DTGR-WF performs the best in terms of packet success ratio with an average gain of 37% over GPSR and 20% over DTGR-SF. It also achieves the lowest average latency in almost all the cases. GPSR, on the other hand, achieves the lowest packet delivery success ratio and the highest packet delivery latency; and (iii) in network conditions where GPSR has shown to yield a higher packet delivery success ratio and lower average packet 126 delivery latency than existing routing protocols, our schemes achieve the same high packet delivery success ratio as GPSR with reduced average packet delivery latency. Some directions we are currently exploring are: (i) investigating the relationship between various types of link disruptions (e.g., non-deterministic fading models) and the performance of ad hoc and sensor network routing protocols; (ii) designing an adaptive algorithm that constructs and maintains a neighbor set based on the current network conditions, e.g., node density, mobility, etc.; and (iii) deriving the reachability function using the feedback from link level information such as the packet loss ratio, etc. 127 CHAPTER 6 SUMMARY AND FUTURE WORK In this chapter we summarize the main contributions of this dissertation and then present our future research directions. 6.1 Dissertation Summary This dissertation addresses three major research problems, with the first falling into the category of server replica placement problem for improving Web access, the second into the category of data storage/retrieval service for improving information dissemination in wireless ad hoc networks, and the third into the category of dis- ruption tolerant routing in wireless ad hoc networks. Our main contributions are summarized as follows. Server replica placement in CDNs. In Chapter 3, we studied the static server replica placement problem in CDNs, with the main focus on the effects of server replica size and its clients’ traffic pattern on the CDN performance gain growth. We modeled the replica placement as a well-known NP complete minimum p-median graph problem and presented several heuristic algorithms to solve the 128 problem. Using a near optimal greedy algorithm, we performed one trace-driven simulation to establish the relationship between optimized performance gain and replica size. The simulation was performed based on one specific Web server log file. From the simulations we found that there are diminishing returns to placing more replicas for minimizing total content delivery cost, and that a surprisingly small number of replicas (less than five) are enough to significantly improve optimized performance gain (82%). After aggregating together content received by clients within the same Au- tonomous System (AS), we found that, for this specific Web server, its clients’ demand follows a Zipf-like distribution with α = 1.1; the distances (in AS hop counts) among these ASs follow a Gaussian distribution with mean of 2.63, which is highly skewed to the top ASs receiving most content. We performed a parametric study with different values of α and established the relationship between the threshold value of replica size and α. The results show that as α increases, the number of replicas required to achieve a certain performance gain decreases. In addition, we observed that placing replicas among a small number of carefully chosen candidates (top clients receiving most content and few clients with the highest outdegrees) allows us to achieve the same performance gain as placing replicas among all clients, while with reduced computation cost. This simple scheme for narrowing down the candidate sites was used to speed up the computation. Data storage/retrieval service in wireless ad hoc networks. In Chapter 4, we presented the design and evaluation of SDSR, a data centric, rendezvous−based 129 data storage and retrieval service for enhancing data dissemination in large-scale wireless ad hoc networks. SDSR achieves scalability, robustness, and load-balancing by partitioning the network into hierarchical grids of increasing size and replicating the data item at each hierarchical grid. The number of storage servers for a data item increases as a logarithm of the increase of the network size. SDSR supports query locality by placing more storage servers near the locations where most queries are generated, and fewer far away. SDSR is resilient to clustering failures, as it replicates each data item at multiple nodes that are distributed in different network regions. We proposed a rooted multicast storage tree routing protocol to route each data item from its source to all its storage locations. The proposed multicast storage tree routing achieves better performance than traditional rooted shortest path tree and rooted minimum spanning tree when both energy efficiency and network delay/data delivery success probability are considered in combination. SDSR query protocol retrieves a data item in the same way as the storage pro- tocol hierarchically stores the data item. SDSR query protocol is scalable, since it not only bounds the number of steps needed to satisfy a query to the logarithm of the size of the network but also guarantees that a query message for a data item is propagated locally within a quadrant of the smallest grid containing both the query node and the source node. The data update protocol employed by SDSR is scalable as the update messages are propagated within the smallest grid containing the source node’s current location 130 and its previous location where the last update occurred. In addition, we proposed a lightweight indexing service to handle faults resulting from node joins and node mobility locally. We compared the performance of SDSR and other wireless data dissemination services regarding the energy efficiency, hotspot usage, and resilience to clustering failures through numerical analysis and simulations. Our results showed that in a query-dominant, large-scale wireless ad hoc network, SDSR performs better than existing schemes in terms of total communication cost, data query latency, hotspot usage, and resilience to clustering failures. It complies well with query locality and scales well when the network size and the number of queries increase. Disruption tolerant wireless ad hoc routing. In Chapter 5, we first il- lustrated that temporary link disruptions can frequently occur in wireless ad hoc networks due to the presence of obstacles, node mobility, etc. These disruptions can result in a node’s incorrect perception of its neighbor set when the neighbor set is constructed via beacon sampling, e.g., beacon collisions may result in the removal of a node j from the neighbor set of a node i, although j is within the transmission range of i. We then proposed a simple scheme that allows each node to associate its neighbors with a reachability value to accommodate the incorrect perception of the node about its neighbor set. To illustrate the effectiveness of our proposed scheme, we applied it to GPSR and designed two new routing algorithms, Disruption Tolerant Geographic Routing- Simple Forwarding (DTGR-SF) and Disruption Tolerant Geographic Routing-Waiting 131 before Forwarding (DTGR-WF), in which nodes utilized reachability values to make forwarding decisions. We finally compared the performance of DTGR-SF and DTGR- WF with that of GPSR in various simulation settings. Some key results were that: 1. when temporary disruptions due to obstructions, beacon collisions, etc., are present, DTGR-WF performs significantly better than GPSR in terms of packet delivery success ratio with the tradeoff of increased packet delivery latency. For example, in network settings where nodes are static, DTGR-WF achieves an average 46.8% higher packet delivery success ratio than GPSR; 2. when temporary disruptions due to node mobility are present, DTGR-WF performs the best in terms of packet success ratio with an average gain of 37% over GPSR and 20% over DTGR-SW. It also achieves the lowest average latency in almost all the cases. GPSR achieves the lowest packet delivery success ratio, as well as the highest packet delivery latency; and 3. in network conditions where GPSR has shown to yield higher packet delivery success ratio and lower average packet delivery latency than existing routing protocols, our schemes achieve the same high packet delivery success ratio as GPSR with reduced average packet delivery latency. 6.2 Future Research Directions As we discussed throughout this dissertation, there are still many interesting and challenging problems remaining in the research areas we have covered. 132 Server replica placement in CDN. Currently, assuming that each replica does not have capacity constraints, we study the complete replica placement problem for one original server. In general, each replica in a CDN may only have space to hold partial content for one original server. We would like to consider the replica storage constraints in the placement decision. The uncapacited minimum p-median can be considered as the problem model. When storage constraints are considered, more detailed Web server trace analysis might need to be conducted. For example, instead of only obtaining the total size of all the documents requested by each candidate site, we would like to obtain each document’s size and its popularity in each candidate site from the log file. We would also like to study adaptive replica placement algorithms that support various QoS requirements. Data storage/retrieval service in wireless ad hoc networks. In our cur- rent work, we assume that each node has unlimited storage space, thus data items can always be stored in the correct node. This is not always the case. We are con- sidering the following possible schemes to extend SDSR to work with node storage limits: 1. Construction of virtual node. A storage server can cluster nodes within its vicinity to form a virtual node to extend its storage space. When a data item arrives and the storage server does not have enough storage space, it sends the data item to a node in its cluster. The node can be selected randomly or based on such criteria as selecting the node with the largest available space or selecting the node second closest to the storage location. The storage server 133 can have a pointer pointing to the selected node (indexing service can be extended to provide this service), so the request for the data item can be routed correctly; 2. Storage of data blocks. For a large file, instead of storing the whole file to each of the three storage servers in each level of the hierarchy, we can divide the file into three pieces and store each piece in each of the storage servers. Using Figure 4.1 as an example, s can store the first 1/3 block of the data item in number 1 grid of each of the hierarchy, the second 1/3 block in number 2 and the last 1/3 in number 3. By storing only blocks of files instead of the whole files at each server, we can alleviate the problem of unbalanced use of storage space. Additionally, accessing different blocks in parallel can be used to reduce the query latency; and 3. Currently, we assume that nodes are uniformly distributed in the network. In a non-uniform wireless network with skewed node distribution, the distribu- tion of data might be skewed. We are currently investigating this problem. Clustering is widely used to construct scalable, energy-efficient wireless net- work topology. Studying clustering-based data dissemination schemes is also one of our future research directions. Disruption tolerant wireless ad hoc routing. As to the design of disruption tolerant wireless ad hoc routing protocol, we plan to extend our research as follows: 134 1. Investigate the relationship between various types of link disruptions (non- deterministic fading models) and the performance of ad hoc and sensor network routing protocols; 2. Design an adaptive algorithm which constructs and maintains the neighbor set based on the current network conditions, e.g., node density, mobility, etc.; and 3. Derive the reachability function using the feedback from link level information such as packet loss ratio, etc. 135 APPENDIX A ABBREVIATIONS ABR: Associativity-Based Routing AODV: Ad hoc On-Demand Distance Vector AS: Autonomous System CAN: Content Addressable Network CDN: Content Distribution Network DHT: Distributed Hash Table DREAM: Distance Routing Effect Algorithm for Mobility DSDV: Destination-Sequenced Distance-Vector Routing DTGR: Disruption Tolerant Geographic Routing DTGR-SF: Disruption Tolerant Geographic Routing - Simple Forwarding DTGR-WF: Disruption Tolerant Geographic Routing - Wait Before Forwarding DTN: Disruption Tolerant Networking FSR: Fisheye State Routing GG: Gabriel Graph GHT: Geographic Hash Table 136 GLS: Geographic Location Service GPS: Global Position System GPSR: Geographic Perimeter Stateless Routing HTTP: Hyper Text Transfer Protocol HTTPs: Hyper Text Transfer Protocol(Secure) IP: Internet Protocol ITAMAR: Independent-Tree Ad hoc Multicast Routing MAODV: Multicast Extensions of AODV NLANR: National Laboratory for Applied Network Research OLSR: Optimized Link State Routing PAN: Probabilistic quorum system for Ad hoc Networks PRR: Packet Reception Rate PDA: Personal Digital Assistant QoS: Quality of Service OS: Original Server RNG: Relative Neighborhood Graph GHT: Geographic Hash Table SDSR: Scalable Data Storage Retrieval service SLURP: Scalable Location Update Routing Protocol SR-GHT: Structured Replication GHT TORA: Temporally Ordered Routing Algorithm URL: Universal Resource Locator 137 WWW or Web: World Wide Web ZRP: Zone Routing Protocol NS: Network Simulator 138 APPENDIX B SIMULATION MODELS In this chapter, we first introduce Network Simulator (NS), the simulator that we used to implement and simulate Disruption Tolerant Geographic Routing-Simple Forwarding(DTGR-SF) and Disruption Tolerant Geographic Routing-Wait before Forwarding (DTGR-wF) routing protocols. We then describe the implementation and simulation models for DTGR in detail. B.1 NS Package NS is a collaborative project developed by UCB, LLNL, Xerox PARC, and USC/ICI and is maintained by USC/ISI. NS is primarily useful for simulating net- work protocols of various layers, such as routing, transport, and application layers, and for studying the impact of transmission media (wired and wireless LAN, satel- lite, etc.) on Internet protocols. NS also contains tools that analyze and display the simulation results. NS is written in both C++ and OTcl to separate the data and control imple- mentations and thus to increase the simulation efficiency. More specifically, in order 139 to speed up the packet and event processing time, the core of NS, such as simula- tion scheduler and network component objects, is generally written and compiled in C++, because C++ allows efficient run-time manipulation of packets headers and bytes. OTcl, on the other hand, is used to control simulation configuration and examine simulation scenarios, as OTcl allows easy manipulation of simulation parameters and simulation environments. As shown in Figure B.1, NS supports a class hierarchy in C++ (compiled hierar- chy) and a corresponding class hierarchy in OTcl Interpreter (interpreted hierarchy). For each compiled object in the compiled hierarchy, there is a matching OTcl object in the interpreted hierarchy, so that the compiled C++ objects can be controlled by the OTcl interpreter. The following subsections focus on the introduction of NS basic structure, its network components, the NS objects used in our simulation, such as Agent, Mobile Node, and Timer, and the format of NS wireless simulation traces. The related figures are from the NS Simulator Tutorial/Workshop slides [1], the NS by Example [17], and the NS Manual [86]. B.1.1 NS Basic Structure Figure B.2 shows that NS is an event-based OTcl script simulator that is com- posed of a simulation event scheduler, network component object libraries, and network plumbing module libraries. A user writes an OTcl script to set up and run the simulation, which includes initializing an event scheduler, setting up the net- work topology using network objects and plumbing functions in the libraries, and 140 Figure B.1: C++ and OTcl: the duality Figure B.2: Simplified user view of NS 141 telling the traffic sources when to start and stop sending packets through the event scheduler. The results of each simulation are written into one or several output files and can be processed using various analyze tools and viewed using the graphical tool NAM. A packet in NS is an event, which is issued by each network component that needs to process the packet. At the beginning of each simulation, all the events are issued and stored in an event queue. During the simulation, the event scheduler keeps track of simulation time and fires all the events in the event queue in the order of their scheduled time, by invoking the corresponding object to process the associated packet. B.1.2 Agent An Agent in NS represents an end point object that actually produces and consumes packets and is used to implement protocols at various layers. For example, our DTGR-SF and DTGR-WF routing protocols are implemented as Agent in NS. B.1.3 Mobile Node In NS, class MobileNode is used to create an object that simulates the function- ality of a wireless mobile node. Besides the functionalities of a basic Node object, a MobileNode object has additional functionalities such as movement, periodical po- sition updates, the ability to transmit and receive on a channel, etc. Figure B.3 shows the structure of a wireless mobile node. 142 Figure B.3: Mobile node structure 143 B.1.4 Wireless Trace Format Figure B.4 shows the wireless trace format. Each entry in the wireless trace begins with one of the four characters: ‘s,’ ‘r,’ ‘d,’ and ‘f,’ with ‘s’ meaning that this entry represents an event of sending a packet, ‘r’ meaning an event of receiving a packet, ‘d’ meaning an event of dropping a packet, and ‘f’ meaning an event of forwarding a packet. The character is followed by flag/value pairs similar to NAM traces. For flags with two letters, the first letter indicates the flag type, e.g., ‘N’ for Node Property, ‘I’ for IP Level packet information, ‘H’ for next hop information, ‘M’ for MAC level packet information, and ‘P’ for packet-specific information. The value for the ‘-Hd’ flag can be -1 or -2, with -1 indicating that the packet is a broadcast packet and -2 indicating that the destination node has not been set. B.2 DTGR Design and Implementation This section gives the detailed explanation of the design and implementation of DTGR, which are based on the design and implementation of GPSR provided in [47]. It starts with the illustration of DTGR packet format, followed by DTGR routing agents implementation and the neighbor table entry structure, and it ends with the illustration of several important Timers used by DTGR. 144 Figure B.4: Wireless trace format 145 Figure B.5: DTGR packet format B.2.1 DTGR Packet Format and Packet Header Data Struc- ture As illustrated in Chapter 5, the DTGR packet format is shown in Figure B.5. The data structure for the new DTGR packet header that carries the relevant data is defined in class hdr dtgr and is coded as follows: # define MAX PERI HOPS STATIC 128 class hdr dtgr { public: struct PeriEnt hops [MAX PERI HOPS STATIC]; struct PeriEnt peript ; //structure to define a perimeter header 146 nsaddr t periptip [3]; int nhops ; //number of hops the packet has traversed int currhop ; //current hop int mode ; // packet is in greedy mode (0) or perimeter node (1) //header size int size() { return (nhops * 12 + 3 * sizeof(int)); } void add hop(nsaddr t addip, double addx, double addy, double addz) { //maximum number of hops a perimeter mode packet can traverse, //used to avoid loop if (nhops == MAX PERI HOPS STATIC) { fprintf( stderr, “out of slots.”); abort(); } hops [nhops ].x = addx; hops [nhops ].y = addy; hops [nhops ].z = addz; hops [nhops ].ip = addip; nhops ++; } static int offset ; inline static int& offset() { return offset ; } // added for gpsr inline static hdr dtgr *access(const Packet* p) {// added inline keyword for dtgr return (hdr dtgr *) p − > access(offset ); }}; 147 B.2.2 DTGR Routing Agent In DTGR simulation, when a node receives a packet, it passes the packet to the DTGR routing agent. The routing agent handles the packet based on the packet’s type, forwarding mode and the current routing algorithm type (passed from the simulation TCL file). For example, if the packet is a broadcast packet, then the routing agent forwards the packet to all its neighbors in the neighbor table; if it is a unicast packet and the node is not the packet’s destination, then the routing agent selects a neighbor as the next hop for packet forwarding based on the packet’s current forwarding mode, the forwarding node’s own location, the packet destination location, as well as the location and reachability of each of the neighbors of the current forwarding node. Class DTGR Agent implements the main functionalities of DTGR routing pro- tocol including DTGR greedy forwarding procedure, DTGR perimeter routing pro- cedure, the beacon handling procedure, etc. It also defines the bindings between the OCT variables and the corresponding C++ variables, the Timers used by the routing agent, and the node’s neighbor table, etc. class DTGR Agent : public Tap, public Agent { public: DTGR Agent(void); void beacon callback(void); // generate a beacon (timer-triggered) void deadneighb callback(DT NeighbEnt *ne); // neighbor gone void periprobe callback(DT NeighbEnt *ne); // gen perimeter probe (timer-triggered) void lastperi callback(void); // turn off peri probes when unused for timeout 148 void planar callback(void); // planarization callback .... // consider unstable node in forwarding void dt forwardPacket( Packet *, int = 0); void addNextHopTimer(Packet *p , int rtxflag ,int index ); void removeNextHopTimer(int index); protected: DT NeighbTable *ntab ; DTGR BeaconTimer *beacon timer ; // Alive beacon timer DTGR LastPeriTimer *lastperi timer ; // Last perimeter used timer DTGR PlanarTimer *planar timer ; // inter-planarization timer DTGR NextHopTimer *nexthopTimerList ; //nexthop Trace *tracetarget; // Trace Target virtual void recv(Packet *, Handler *); //handle received DTGR packet Packet *makeAlive(void); void beaconIn(Packet *, hdr dtgr *); void trace(char *fmt, ...); ..... void init(); int use mac ; // whether or not to simulate full MAC level int use peri ; // whether or not to use perimeters int verbose ; // verbosity (binary) .. 149 int peri proact ; // whether or not to pro-actively send pprobes int use implicit beacon ; // whether or not all data packets are beacons int use planar ; // whether or not to planarize graph int use loop detect ; // whether or not to fix loops in peridata pkts int use timed plnrz ; // whether or not to replanarize with timer double bint ; // beacon interval double bdesync ; // beacon desync random component range double bexp ; // beacon expiration interval double pint ; // perimeter probe interval double pdesync ; // perimeter probe desync random cpt. range double lpexp ; // perimeter probe generation timeout LocDbase *ldb ; // location database MobileNode *mn ; // my MobileNode PriQueue *ifq ; // my ifq ... int algo type ; //DTGR algorithm type int nexthoptimercounter ;//used by DTGR-WF double nexthop timeout ; //the wait time before selecting next hop }; B.2.3 DTGR Neighbor Table Entry Structure Class DT NeighbEnt defines the fields and functionalities of a neighbor entry stored in a DTGR node’s neighbor table. The major fields of a neighbor entry 150 include the neighbor’s IP address, its x,y,z coordinates, and its reachability. The reachability defines the stability metric of the neighbor. The major functions include the local planarization, etc. class DT NeighbEnt { public: DT NeighbEnt(DTGR Agent *ina): ... Graph planarization using stable neighbors only void planarize(DT NeighbTable *, int, double, double, double); Graph planarization using both stable neighbors and unstable neighbors void dt planarize(DT NeighbTable *nt, int algo, double x, double y, double z); .. nsaddr t dst; // IP of neighbor double x, y, z; // location of neighbor last heard double reachability; //reachability of the neighbor struct PeriEnt *peri; // perimeter via this neighbor int perilen; // length of perimeter int maxlen; // allocated slots in peri DTGR DeadNeighbTimer dnt; // Timer for expiring the neighbor // Timer for generating perimeter probe to the neighbor DTGR PeriProbeTimer ppt; int live; // when planarizing, whether edge should be used } 151 B.2.4 Timers Timers in NS are used to schedule events. Refer to Chapter 11 of [86] for a more detailed explanation of the usage of Timers. Several important Timers implemented in DTGR are: (i) Beacon Timer, which is used by a node to schedule the next beacon event; (ii) Neighbor time-out Timer, which is used to fire the neighbor removal Event; and (iii) DTGR-WF Timer, which is used by DTGR-WF to fire the next hop neighbor selection event if an unstable neighbor is selected. The following code shows the implementation of the Beacon Timer implemented for the DTGR Routing Agent. Beacon Timer implementation: class DTGR AgentTimer: public TimerHandler { public: DTGR AgentTimer(DTGR Agent *a ) { a = a ; } virtual void expire(Event *e) = 0; protected: DTGR Agent *a; }; class DTGR BeaconTimer : public DTGR AgentTimer { public: DTGR BeaconTimer(DTGR Agent *a ) : DTGR AgentTimer(a ) {}; // invoked after beacon expires, fires an beacon event virtual void expire(Event *) { a − > beacon callback(); }}; 152 B.3 DTGR Simulation In this section, we explain the DTGR simulation model, the simulation com- mand, and the format of the simulation input and output files in detail through examples. B.3.1 NS Multistate Error Model and TCL Configuration Variables In DTGR, each mobile node is configured with the NS multistate (0 and 1) error model for simulating temporary link disruption, where the state on representing 0 packet loss ratio and off representing 100% packet loss ratio. The on and off time durations follow the normal distribution. The temporary disruption procedure is show in Figure B.6. The TCL configuration variables used in the DTGR simulation script are shown in Figure B.7. B.3.2 Simulation Run: An Example The simulation was performed on Sun Solaris workstations of the Computer Sci- ence and Engineering Department at The Ohio State University and the Computer Science and Software Engineering Department at Auburn University, respectively. An example of running simulation for DTGR protocol is shown as follows. The simulation is run using NS command: ns error.tcl dtgr out scen-n10-p300-M5-t200- x500-y500 1.5 1 3.0. The second argument error.tcl in the command line is the TCL simulation script, which simulates DTGR-WF in a network environment where temporary link disruptions exist, the third argument out is the simulation output file, 153 Figure B.6: DTGR temporary disruption procedure simulated by the NS multistate error model 154 Figure B.7: DTGR TCL configuration variables used in the simulation script 155 the fourth scen-n10-p300-M5-t200-x500-y500 is the name of the waypoint random motion topology file used in the simulation, the fifth scen-n10-p300-M5-t200-x500- y500 specifies the randomly generated network topology and each mobile node’s movement trace, the sixth 1.5 is the beacon interval, the seventh 1 specifies DTGR- WF algorithm, and the last argument 3.0 is the wait time used by DTGR-WF. The traffic file (cbr-n10-s1-mc5-r4) used in the simulation is imported in error.tcl. Next we introduce each input file and the final output file with examples. Network topology and mobile node motion file. The topology file scen- n10-p300-M5-t200-x500-y500 specifies the randomly generated network topology and each mobile node’s movement trace. The file contains 10 nodes, the speed of each node is in [0-5] m/s, the simulation time is 200s, and the grid size is 500 × 500. Refer to Figure B.4 for the meaning of each field. scen-n10-p300-M5-t200-x500-y500: # nodes: 10, pause: 300.00, max speed: 5.00, max x: 500.00, # max y: 500.00 # $node (0) set X 329.015508744685 $node (0) set Y 11.236929180399 $node (0) set Z 0.000000000000 $node (1) set X 404.252702040729 $node (1) set Y 394.735721546465 $node (1) set Z 0.000000000000 $node (2) set X 186.417535942711 $node (2) set Y 461.827968377671 156 $node (2) set Z 0.000000000000 ..... $node (9) set X 85.520610978894 $node (9) set Y 420.922150847457 $node (9) set Z 0.000000000000 $god set-dist 0 1 2 $god set-dist 0 2 3 $god set-dist 0 3 2 $god set-dist 0 4 1 .... $god set-dist 6 9 1 $god set-dist 7 8 3 $god set-dist 7 9 3 $god set-dist 8 9 1 # # Destination Unreachables: 0 # Route Changes: 0 # Link Changes: 0 # # Node — Route Changes — Link Changes # 0 — 0 — 0 # 1 — 0 — 0 157 # 2 — 0 — 0 # 3 — 0 — 0 .... # 6 — 0 — 0 # 7 — 0 — 0 # 8 — 0 — 0 # 9 — 0 — 0 Traffic file. The traffic file (cbr-n10-s1-mc5-r4) used in the simulation is im- ported in error.tcl. This traffic file specifies 10 nodes with 5 cbr connections, and 2KB/s data rate. cbr-n10-s1-mc5-r4: # nodes: 10, max conn: 5, send rate: 0.25, seed: # 1 connecting to 2 at time 2.5568388786897245 # set udp (0) [new Agent/UDP] $ns attach-agent $node (1) $udp (0) set null (0) [new Agent/Null] $ns attach-agent $node (2) $null (0) set cbr (0) [new Application/Traffic/CBR] $cbr (0) set packetSize 512 $cbr (0) set interval 0.25 $cbr (0) set random 1 $cbr (0) set maxpkts 10000 158 $cbr (0) attach-agent $udp (0) $ns connect $udp (0) $null (0) $ns at 2.5568388786897245 ”$cbr (0) start” # 4 connecting to 5 at time 56.333118917575632 # set udp (1) [new Agent/UDP] $ns attach-agent $node (4) $udp (1) set null (1) [new Agent/Null] $ns attach-agent $node (5) $null (1) set cbr (1) [new Application/Traffic/CBR] $cbr (1) set packetSize 512 $cbr (1) set interval 0.25 $cbr (1) set random 1 $cbr (1) set maxpkts 10000 $cbr (1) attach-agent $udp (1) $ns connect $udp (1) $null (1) $ns at 56.333118917575632“$cbr (1) start” # # 4 connecting to 6 at time 146.96568928983328 # set udp (2) [new Agent/UDP] $ns attach-agent $node (4) $udp (2) set null (2) [new Agent/Null] 159 $ns attach-agent $node (6) $null (2) set cbr (2) [new Application/Traffic/CBR] $cbr (2) set packetSize 512 $cbr (2) set interval 0.25 $cbr (2) set random 1 $cbr (2) set maxpkts 10000 $cbr (2) attach-agent $udp (2) $ns connect $udp (2) $null (2) $ns at 146.96568928983328 “$cbr (2) star” # # 6 connecting to 7 at time 55.634230382570173 ... # 7 connecting to 8 at time 29.546173154165118 ... # #Total sources/connections: 4/5 # Sample result. The content file out, which is processed to get the packet success ratio of the simulated scenario, is partially listed below. The verbose version of the output (not shown here) gives the detailed information about the number of hops traversed by each packet and is used to compute the average packet delivery latency. Refer to Figure B.4 for the meaning of each field. 160 out: M 0.0 nn:10 x:1500 y:1500 rp:dtgr M 0.0 sc:scen-n10-p300-M5-t200-x500-y500 cp:/export/3/yingjie/ns-allinone-2.27/ns-2.27/gpsr/cbr-n10-s1-mc5-r4 seed:0.0 M 0.0 prop:Propagation/TwoRayGround ant:Antenna/OmniAntenna s 2.556838879 1 AGT — 16 cbr 512 [0 0 0 0] ——- [1:0 2:0 32 0] [0] 0 1 r 2.605571816 2 AGT — 16 cbr 552 [13a 2 1 800] ——- [1:0 2:0 128 2] [0] 1 1 s 2.702696351 1 AGT — 19 cbr 512 [0 0 0 0] ——- [1:0 2:0 32 0] [1] 0 1 s 2.841533007 1 AGT — 21 cbr 512 [0 0 0 0] ——- [1:0 2:0 32 0] [2] 0 1 D 2.841533007 1 IFQ ARP 19 cbr 552 [13a 6 1 800] ——- [1:0 2:0 128 3] [1] 0 1 r 2.916940496 2 AGT — 21 cbr 552 [13a 2 3 800] ——- [1:0 2:0 127 2] [2] 2 1 r 2.951137163 2 AGT — 19 cbr 552 [13a 2 6 800] ——- [1:0 2:0 127 2] [1] 2 1 s 3.017869085 1 AGT — 22 cbr 512 [0 0 0 0] ——- [1:0 2:0 32 0] [3] 0 1 r 3.030431369 2 AGT — 22 cbr 552 [13a 2 6 800] ——- [1:0 2:0 127 2] [3] 2 1 D 3.149847244 6 IFQ ARP 22 cbr 552 [13a 2 6 800] ——- [1:0 2:0 127 9] [3] 1 1 r 3.241291906 2 AGT — 22 cbr 596 [13a 2 6 800] ——- [1:0 2:0 127 2] [3] 2 1 s 3.349889287 1 AGT — 23 cbr 512 [0 0 0 0] ——- [1:0 2:0 32 0] [4] 0 1 r 3.376073093 2 AGT — 23 cbr 596 [13a 2 6 800] ——- [1:0 2:0 127 2] [4] 2 1 r 3.386307270 2 AGT — 22 cbr 552 [13a 2 9 800] ——- [1:0 2:0 126 2] [3] 3 1 s 3.675863228 1 AGT — 26 cbr 512 [0 0 0 0] ——- [1:0 2:0 32 0] [5] 0 1 161 r 3.763000492 2 AGT — 26 cbr 552 [13a 2 9 800] ——- [1:0 2:0 126 2] [5] 3 1 s 3.835911777 1 AGT — 27 cbr 512 [0 0 0 0] ——- [1:0 2:0 32 0] [6] 0 1 r 3.853788083 2 AGT — 27 cbr 552 [13a 2 9 800] ——- [1:0 2:0 126 2] [6] 3 1 r 3.898977890 2 AGT — 27 cbr 596 [13a 2 9 800] ——- [1:0 2:0 126 2] [6] 3 1 s 4.104310413 1 AGT — 31 cbr 512 [0 0 0 0] ——- [1:0 2:0 32 0] [7] 0 1 r 4.151525630 2 AGT — 31 cbr 596 [13a 2 9 800] ——- [1:0 2:0 126 2] [7] 3 1 s 4.399117377 1 AGT — 33 cbr 512 [0 0 0 0] ——- [1:0 2:0 32 0] [8] 0 1 r 4.488027706 2 AGT — 33 cbr 596 [13a 2 9 800] ——- [1:0 2:0 126 2] [8] 3 1 r 4.525082086 2 AGT — 33 cbr 596 [13a 2 6 800] ——- [1:0 2:0 127 2] [8] 2 1 s 4.708470110 1 AGT — 34 cbr 512 [0 0 0 0] ——- [1:0 2:0 32 0] [9] 0 1 r 4.750271438 2 AGT — 34 cbr 596 [13a 2 6 800] ——- [1:0 2:0 127 2] [9] 2 1 s 5.077877321 1 AGT — 38 cbr 512 [0 0 0 0] ——- [1:0 2:0 32 0] [10] 0 1 ... D 200.000000000 7 IFQ END 9643 cbr 552 [13a 5 7 800] ——- [4:0 5:0 126 5] [1756] 2 2 D 200.000000000 8 IFQ END 9648 cbr 552 [13a 3 8 800] ——- [4:1 6:0 127 3] [1396] 1 2 162 B.4 Summary In this appendix, we first gave an overview of NS, the widely adopted simula- tor that we used to implement and simulate DTGR-SF and DTGR-WF. We then explained the design, implementation, and simulation of DTGR in detail. 163 BIBLIOGRAPHY [1] “The 5th UCB/LBNL Network Simulator (NS): June 1999 Tutorial”. http://www.isi.edu/nsnam/ns/ns-tutorial/ucb-tutorial.html, 1999. [2] M. Abolhasan, T. A. Wysocki, and E. Dutkiewicz. “A Review of Routing Protocols for Mobile Ad Hoc Networks”. Elsevier Journal of Ad hoc Networks, 2:1–22, 2003. [3] Akamai. http://www.akamai.com. [4] J. Almeida and P. Cao. “Measuring proxy performance with the Wisconsin Proxy Benchmark”. Computer Networks and ISDN Systems, 30(22–23):2179– 2192, 1998. [5] M. F. Arlitt and C. L. Williamson. “Web Server Workload Characterization: The Search for Invariants”. In Proc. of the 1996 International Conference on Measurement and Modeling of Computer Systems Conference (SIGMET- RICS’96), pages 126–137, 1996. [6] P. Barford and M. Crovella. “Measuring Web Performance in the Wide Area”. ACM SIGMETRICS Performance Evaluation Review, 27(2):37–48, 1999. [7] S. Basagni, I. Chlamtac, V. R. Syrotiuk, and B. A. Woodward. “A Dis- tance Routing Effect Algorithm for Mobility (DREAM)”. In Proc. of the 4th ACM International Conference on Mobile Computing and Networking (Mobi- Com’98), pages 76–84, 1998. [8] L. Breslau, P. Cao, L. Fan, G. Philips, and S. Shenker. “Web Caching and Zipf-like Distributions: Evidence and Implications”. In Proc. of the 18th An- nual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM’99), pages 126–134, 1999. 164 [9] J. Broch, D. Maltz, D. Johnson, Y. Hu, and J. Jetcheva. “A Performance Com- parison of Multi-Hop Wireless Ad Hoc Network Routing Protocols”. In Proc. of the 4th Annual ACM/IEEE International Conference on Mobile Computing and Networking (MobiCom’98), pages 85–97, 1998. [10] B. Burns, O. Brock, and B. N. Levine. “MV Routing and Capacity Building in Disruption Tolerant Networks”. In Proc. of the 24th Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM’05), pages 398–408, 2005. [11] P. Cao and S. Irani. “Cost-Aware WWW Proxy Caching Algorithms”. In Proc. of the 1997 Usenix Symposium on Internet Technologies and Systems (USITS’97), pages 193–206, 1997. [12] Y. Chen, R. H. Katz, and J. Kubiatowicz. “Dynamic Replica Placement for Scalable Content Delivery”. In Proc. of the International Workshop on Peer- to-Peer Systems, pages 306–318, 2002. [13] L. Cheng. “Service Advertisement and Discovery in Mobile Ad Hoc Networks”. In Proc. of the Workshop on Ad hoc Communications and Collaboration in Ubiquitous Computing Environments, 2002. [14] C.-Y. Chiang, Y. J. Li, M. T. Liu, and M. E. Muller. “On Request Forward- ing for Dynamic Web Caching Hierarchies”. In Proc. of the 20th IEEE In- ternational Conference on Distributed Computing Systems (ICDCS’00), pages 262–269, 2000. [15] C.-Y. Chiang, M. T. Liu, and M. E. Muller. “Modeling Web Caching Schemes for Performance Studies”. In Proc. of the 1999 International Conference on Parallel Processing (ICPP’99), pages 516–523, 1999. [16] C.-Y. Chiang, M. Ueno, M. T. Liu, and M. E. Muller. “Caching Neighborhood Protocol: A Foundation for Building Dynamic Web Caching Hierarchies”. In Proc. of the 2000 International Conference on Parallel Processing (ICPP’00), pages 243–250, 2000. [17] J. Chung and M. Claypool. “NS by Example”. http://nile.wpi.edu/NS/. [18] C. D. M. Cordeiro, H. Gossain, and D. P. Agrawal. “Multicast over Wireless Mobile Ad Hoc Networks: Present and Future Directions”. IEEE Network, 17(1):52–59, 2003. 165 [19] D. Cranor, M. Green, C. Kalmanek, D. Shur, S. Sibal, J. E. Van der Merwe, and C. J. Sreenan. “Enhanced Streaming Services in a Content Distribution Network”. IEEE Internet Computing, 5(4):66–75, 2001. [20] B. Davison. “Simultaneous Proxy Evaluation”. In Proc. of the 4th Interna- tional Web Caching Workshop (WCW’99), pages 170–178, 1999. [21] M. Day, B. Cain, G. Tomlinson, and P. Rzewski. “A Model for Content Internetworking (CDI)”. IETF Internet-Drafts, draft-ietf-cdi-model-02.txt. [22] Defense Advanced Research Projects Agency (DARPA). “Disruption Tolerant Networking (DTN)”. http://www.darpa.mil/ato/solicit/DTN/, 2004. [23] Internet Explorer. http://www.microsoft.com/windows/ie/default.mspx. [24] Z. Fei, F. Bhattacharjee, E. W. Zegura, and M. H. Ammar. “A Novel Server Selection Technique for Improving the Response Time of a Replicated Service”. In Proc. of the 17th Annual Joint Conference of the IEEE Computer and Communications Societies (INFCOM’98), pages 783–791, 1998. [25] T. B. Fleming. “Improving the Performance of the World Wide Web over Wireless Networks”. In Proc. of the 1997 IEEE Global Telecommunications Conference (GLOBECOM’97), 1997. [26] J. Gao, L. J. Guibas, J. Hershberge, and L. Zhang. “Fractionally Cascaded Information in a Sensor Network”. In Proc. of the 3rd International Sympo- sium on Information Processing in Sensor Networks (IPSN’04), pages 311– 319, 2004. [27] J. J. Garcia-Luna-Aceves and E. L. Madrug. “The Core-Assisted Mesh Proto- col”. IEEE Journal on Selected Areas in Communications, pages 1380–1394, 1999. [28] M. R. Garey and D. S. Johnson. “Computer and Intractability: A guide to the Theory of NP-completeness”. San Francisco, CA: Freeman, 1979. [29] M. Gerla, S.-J. Lee, and W. Su. “On-Demand Multicast Routing Protocol (ODMRP) for Ad Hoc Networks”. IETF Internet-Drafts, draft-ietf-manet- odmrp-02.txt, 2000. [30] C. Grimm, J.-S. Vckler, and H. Pralle. “Request Routing in Cache Meshes”. Computer Networks And ISDN Systems, 30(22-23):2269–2278, 1998. 166 [31] The CMU MONARCH Group. “Wireless and Mobility Extensions to ns-2”. http://www.monarch.cs.cmu.edu/cmu-ns.html. [32] J. D. Guyton and M. F. Schwartz. “Locating Nearby Copies of Replicated Internet Servers”. In Proc. of the 1995 ACM Special Interest Group on Data Communications Conference (SIGCOMM’95), pages 288–298, 1995. [33] Z. J. Haas, M. R. Pearlman, and P. Samar. “The Zone Routing Protocol (ZRP) for Ad Hoc Networks”. IETF Internet-Drafts, draft-ietf-manet-zone- zrp-04.txt, 2002. [34] T. Hara. “Effective Replication Allocation in Ad Hoc Networks for Improving Data Accessibility”. In Proc. of the 20th Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM’01), pages 1568–1576, 2001. [35] T. Hara. “Replica Allocation with Data Update”. Mobile Methods in Ad Hoc Networks and Applications, 8(4):343–354, 2003. [36] T.-C. Hou and V. O. K. Li. “Transmission Range Control in Multihop Packet Radio Networks”. IEEE Transactions on Communication, 34(1):38–44, 1986. [37] L. F. Huang and T. H. Lai. “On the Scalability of IEEE 802.11 Ad Hoc Networks”. In Proc. of the 2002 ACM International Symposium on Mobile Ad Hoc Networking and Computing (MobiHoc’02), pages 173–182, 2002. [38] IPAS. http://www.iepm.slac.stanford.edu/tools/ipas/. [39] S. Irani. “Page Replacement with Multi-Size Pages and Applications to Web Caching”. In Proc. of the 1997 Annual ACM Symposium on Theory of Com- puting (STOC’97), pages 701–710, 1997. [40] Digital Island. http://www.digitalisland.com/. [41] P. Jacquet, P. Muhlethaler, and A. Qayyum. “Optimized Link State Routing Protocol”. IETF Internet-Drafts, draft-ietf-manetolsr-00.txt, 1998. [42] S. Jamin, C. Jin, A. Kurc, D. Raz, and Y. Shavitt. “Constrained Mirror Placement on the Internet”. In Proc. of the 20th Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM’01), pages 31–40, 2001. 167 [43] L. Ji and M. S. Corson. “Differential Destination Multicast - A MANET Multicast Routing Protocol for Small Groups”. In Proc. of the 20th Annual Joint Conference of the IEEE Computer and Communications Societies (IN- FOCOM’01), pages 1192–2002, 2001. [44] D. B. Johnson and D. A. Maltz. “Dynamic Source Routing in Ad Hoc Wireless Networks”. In Mobile Computing, volume 353. Kluwer Academic Publishers, 1996. [45] J. Kangasharju, J. Roberts, and K. W. Ross. “Object Replication Strategies in Content Distribution Networks”. Computer Communications, 25(4):367–383, 2002. [46] J. Kangasharju, K. W. Ross, and J. Roberts. “Performance Evaluation of Redirection Schemes in Content Distribution Networks”. In Proc. of the 5th Web Caching and Content Distribution Workshop, pages 207–214, 2000. [47] B. Karp. “Geographic Routing”. http://www.icir.org/bkarp/gpsr/gpsr.html. [48] B. Karp and H. T. Kung. “Greedy Perimeter State Routing for Wireless Networks”. In Proc. of the 6th Annual ACM/IEEE International Conference on Mobile Computing and Networking (MobiCom’00), pages 243–254, 2000. [49] G. Karumanchi, S. Muralidharan, and R. Prakash. “Information Dissemina- tion in Partitionable Mobile Ad Hoc Networks ”. In Proc. of the 18th IEEE Symposium on Reliable Distributed Systems (SRDS’99), pages 4–13, 1999. [50] D. Kempe, J. M. Kleinberg, and A. J. Demers. “Spatial Gossip and Resource Location Protocols”. In Proc. of the 33rd ACM Symposium on Theory of Computing (STOC’01), pages 163–172, 2001. [51] B. Krishnamurthy and J. Wang. “On Network-Aware Clustering of Web Clients”. In Proc. of the 2000 ACM Special Interest Group on Data Com- munications Conference (SIGCOMM’00), pages 97–110, 2000. [52] B. Krishnamurthy, C. Wills, and Y. Zhang. “On the Use and Performance of Content Distribution Networks”. In Proc. of the ACM Special Interest Group on Data Communications Conference Internet Measurement Workshop (SIGCOMM IMW’01), pages 169–182, 2001. [53] P. Krishnan, D. Raz, and Y. Shavitt. “The Cache Location Problem”. IEEE/ ACM Transactions on Networking, 8(5):568–582, 2000. 168 [54] B. Li, M. J. Golin, G. F. Italiano, X. Deng, and K. Sohraby. “On the Opti- mal Placement of Web Proxies in the Internet”. In Proc. of the 18th Annual Joint Conference of the IEEE Computer and Communications Societies (IN- FOCOM’99), pages 1282–1290, 1999. [55] J. Li, J. Jannotti, D. DeCouto, D. Karger, and R. Morris. “A Scalable Lo- cation Service for Geographic Ad Hoc Routing”. In Proc. of the 6th Annual ACM/IEEE International Conference on Mobile Computing and Networking (MobiCom’00), pages 120–130, 2000. [56] Y. J. Li, C.-Y. Chiang, and M. T. Liu. “Efficent Web Caching for GPRS Net- works”. In Proc. of the 2001 International Conference on Computer Networks and Mobile Computing (ICCNMC’01), pages 85–90, 2001. [57] Y. J. Li and M. T. Liu. “Optimization of Performance Gain in Content Dis- tribution Networks with Server Replicas”. In Proc. of the 2003 IEEE/IPSJ International Symposium on Applications and the Internet (SAINT’03), pages 182–189, 2003. [58] Y. J. Li and M. T. Liu. “SDSR: A Scalable Data Storage and Retrieval Service for Wireless Ad Hoc Network”. In Proc. of the 2005 International Conference on Information Networking (ICOIN’05), pages 499–509, 2005. [59] J. Luo, J. P. Hubaux, and P. Th. Eugster. “PAN: Providing Reliable Storage in Mobile Ad Hoc Networks with Probabilistic Quorum Systems”. In Proc. of the 2003 ACM International Symposium on Mobile Ad Hoc Networking and Computing (MobiHoc’03), pages 1–12, 2003. [60] D. Magoni and J. Pansiot. “Analysis of the Autonomous System Network Topology”. ACM SIGCOMM Computer Communication Review, 31(3):26– 37, 2001. [61] C. Maihfer. “A Survey of Geocast Routing Protocols”. IEEE Communications Surveys and Tutorials, 6(2):32–42, 2004. [62] M. Mauve, J. Widmer, and H. Hartenstein. “A Survey on Position-Based Routing in Mobile Ad Hoc Networks”. IEEE Network Magazine, 15(6):30–39, 2001. [63] NLANR Measurement and Operations Analysis Team. “NLANR Measure- ment and Operations Analysis Team”. http://moat.nlanr.net. 169 [64] A. Myers, P. Dinda, and H. Zhang. “Performance Characteristics of Mirror Servers on the Internet”. In Proc. of the 18th Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM’99), pages 304–312, 1999. [65] Netscape. http://www.netscape.com. [66] V. Pai, M. Aron, G. Banga, M. Svendsen, P. Drushel, W. Zwaenepoel, and E. Nahum. “Locality-Aware Request Distribution in Cluster-Based Network Services”. In Proc. of the 8th International Conference on Architectural Sup- port for Programming Languages and Operating Systems (ASPLOS-VIII), pages 182–189, 1998. [67] V. D. Park and M. S. Corson. “A Highly Adaptive Distributed Routing Al- gorithm for Mobile Wireless Networks”. In Proc. of the 16th Annual Joint Conference of the IEEE Computer and Communications Societies (INFO- COM’97), pages 1405–1413, 1997. [68] G. Pei, M. Gerla, and T.-W. Chen. “Fisheye State Routing in Mobile Ad Hoc Networks”. In Proc. of the 20th IEEE International Conference on Distributed Computing Systems (ICDCS’00) Workshop on Wireless Networks and Mobile Computing, pages D71–D78, 2000. [69] C. E. Perkins and P. Bhagwat. “Highly Dynamic Destination-Sequenced Distance-Vector Routing (DSDV) for Mobile Computers”. In Proc. of the 1994 ACM Special Interest Group on Data Communications Conference (SIG- COMM’94) on Communications Architectures, Protocols and Applications, pages 234–244, 1994. [70] C. E. Perkins, E. M. Royer, and S. R. Das. “Ad Hoc On-Demand Distance Vec- tor (AODV) Routing”. IETF Internet-Drafts, draft-ietf-manet-aodv-13.txt, 2003. [71] S. Podlipnig and L. Bszrmenyi. “A Survey of Web Cache Replacement Strate- gies”. ACM Computing Surveys, 25(4):374–398, 2003. [72] L. Qiu, V. N. Padmanabhan, and G. M. Voelker. “On the Placement of Web Server Replica”. In Proc. of the 20th Annual Joint Conference of the IEEE Computer and Communications Societies (INFCOM’01), pages 1587–1596, 2001. 170 [73] S. Rathasamy, P. Fransis, M. Handley, R. Karp, and S.Shenker. “A Scalable Content Addressable Network”. In Proc. of the 2001 ACM Special Interest Group on Data Communications Conference (SIGCOMM’01), pages 161–172, 2001. [74] S. Rathnasamy, B. Karp, S. Shenker, D. Estrin, R. Govindan, Y. Li, and F. Yu. “Data Centric Storage in Sensornets”. Mobile Networks and Applica- tions (MONET), Special Issues on Wireless Sensor Networks, 8(4):427–442, 2003. [75] R. Rejaie, M. Handley, H. Yu, and D. Estrin. “Proxy Caching Mechanisms for Multimedia Playback Streams in the Internet”. In Proc. of the 4th Inter- national Web Caching Workshop, 1999. [76] L. Rizzo and L. Vicisano. “Replacement Policies for a Proxy Cache”. Technical Report RN/98/13, UCL-CS, 1998. [77] A. Rousskov and D. Wessels. “Cache Digests”. In Proc. of the 3rd International WWW Caching Workshop, 1998. [78] Route Views. http://www.antc.uoregon.edu/route-views/. [79] A. Rowstron and P. Druschel. “Pastry: Scalable, Distributed Object Loca- tion and Routing for Large-Scale Peer-to-Peer Systems”. In Proc. of the 18th IFIP/ACM International Conference on Distributed Systems Platforms (Mid- dleware’01), pages 233–241, 2000. [80] E. M. Royer and C. E. Perkins. “Multicast Operation of the Ad Hoc On- Demand Distance Vector Routing Protocol”. In Proc. of the 5th Annual ACM/IEEE International Conference on Mobile Computing and Networking (MobiCom’99), pages 207–218, 1999. [81] E. M. Royer and C.-K. Toh. “A Review of Current Routing Protocols for Ad-Hoc Mobile Wireless Networks”. IEEE Personal Communications, pages 46–55, 1999. [82] F. Sailhan and V. Issarny. “Cooperative Caching in Ad Hoc Networks”. In Proc. of the 4th International Conference on Mobile Data Management (MDM’03), pages 13–28, 2003. [83] Sajama and Zygmunt J. Haas. “Independent-Tree Ad Hoc Multicast Routing (ITAMAR)”. Mobile Networks and Applications, 8(5):551–566, 2003. 171 [84] K. Seada, M. Zuniga, B. Krishnamachari, and A. Helmy. “Energy Efficient Forwarding Strategies for Geographic Routing in Lossy Wireless Sefnsor Net- works”. In Proc. of the Second ACM Conference on Embedded Networked Sensor Systems (SenSys’04), pages 108–121, 2004. [85] A. Shaikh, R. Tewari, and M. Agrawal. “On the Effectiveness of DNS-based Server Selection”. In Proc. of the 20th Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM’01), pages 1801–1810, 2001. [86] Network Simulator. “The UCB/LBNL/VINT Network Simulator NS (version 2)”. http://mash.cs.berkeley.edu/ns. [87] I. Stoica, R. Morris, D. Liben-Nowell, D. R. Karger, M. F. Kaashoek, F. Dabek, and H. Balakrishnan. “Chord: A Scalable Peer-to-Peer Lookup Protocol for Internet Applications”. IEEE/ACM Transactions on Network- ing, pages 17–32, 2003. [88] I. Stojmenovic and J. Wu. “Broadcasting and Activity-Scheduling in Ad Hoc Networks”. In Mobile Ad Hoc Networking. IEEE press Wiley, 2003. [89] H. Takagi and L. Kleinrock. “Optimal Transmission Ranges for Mobile Com- munication in Linear Multihop Packet Radio Networks”. IEEE Transactions on Networking, 32(3):246–257, 1984. [90] C.-K. Toh. “Associativity-Based Routing for Ad Hoc Mobile Networks”. Wire- less Personal Communications, 4(2):103–139, 1997. [91] US Naval Observatory (USNO). “US Naval Observatory (USNO) GPS Oper- ations”. http://tycho.usno.navy.mil/gps.html, 2001. [92] K. H. Wang and B. Li. “Efficient and Guaranteed Service Coverage in Parti- tionable Mobile Ad-hoc Networks”. In Proc. of the 20th Annual Joint Confer- ence of the IEEE Computer and Communications Societies (INFOCOM’01), pages 1089–1098, 2001. [93] Y. Wang, Z.-L. Zhan, D. Du, and D. Su. “A Network Concious Approach to End-to-End Video Delivery over Wide Area Networks Using Proxy Servers”. In Proc. of the 17th Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM’98), pages 660–667, 1998. 172 [94] A. Ward, A. Jones, and A. Hopper. “A New Location Technique for the Active Office”. IEEE Personal Communications, 4(5):42–47, 1997. [95] D. Wessels. “Squid Internet Object Cache”. http://squid.nlanr.net/Squid/, 1998. [96] D. Wessels and K. Claffy. “Internet Cache Protocol (ICP), Version 2”. RFC2186, 1998. [97] S. Williams, M. Abrams, C. R. Standridge, G. Abdulla, and E. A. Fox. “Re- moval Policies in Network Caches for World Wide Web Documents”. In Proc. of the 1996 ACM Special Interest Group on Data Communications Conference (SIGCOMM’96), 1996. [98] Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specification. IEEE Standard for Information Technology, 1999. [99] S.-C. M. Woo and S. Singh. “Scalable Routing Protocol for Ad Hoc Networks”. Wireless Networks, 7:513–529, 2001. [100] B. Y. Zhao, L. Huang, S. C. Rhea, J. Stribling, A. D. Joseph, and J. D. Kubi- atowicz. “Tapestry: A Global-Scale Overlay for Rapid Service Deployment”. IEEE Journal on Selected Areas in Communications, 22(1):41–53, 2004. [101] W. Zhao, M. Ammar, and E. Zegura. “A Message Ferrying Approach for Data Delivery in Sparse Mobile Ad Hoc Networks”. In Proc. of the 2004 ACM International Symposium on Mobile Ad Hoc Networking and Computing (MobiHoc’04), pages 187–198, 2004. 173