A Delay Tolerant Networking and System Architecture for Developing Regions Michael Demmer Electrical Engineering and Computer Sciences University of California at Berkeley Technical Report No. UCB/EECS-2008-124 http://www.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-124.html September 25, 2008 Copyright 2008, by the author(s). All rights reserved. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission. A Delay Tolerant Networking and System Architecture for Developing Regions by Michael Joshua Demmer B.S. (Brown University) 1998 A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Computer Science in the GRADUATE DIVISION of the UNIVERSITY OF CALIFORNIA, BERKELEY Committee in charge: Professor Eric Brewer, Chair Professor Scott Shenker Professor AnnaLee Saxenian Fall 2008 A Delay Tolerant Networking and System Architecture for Developing Regions Copyright 2008 by Michael Joshua Demmer 1 Abstract A Delay Tolerant Networking and System Architecture for Developing Regions by Michael Joshua Demmer Doctor of Philosophy in Computer Science University of California, Berkeley Professor Eric Brewer, Chair Technology has shown significant potential in developing countries, as appropriate de- signs matched with real world need can effectively bridge information gaps, provide greater trans- parency, and improve communication efficiency. Unfortunately, many developing regions environ- ments have a lack of affordable network connectivity. Even where there is connectivity, networks are often characterized by frequent, lengthy, and unpredictable link outages, along with limited bandwidth and congested usage. These challenged network conditions, along with the desire to ex- tend the reach of the network, motivate a different approach to developing applications that is more tolerant of intermittent network characteristics. To address these issues, we have developed an overall system framework aimed at easing the development and deployment of applications in challenged network environments. Our ap- proach is built on a robust implementation of the Delay Tolerant Networking architecture, a generic store-and-forward overlay network that uses medium-term storage within the network to buffer 2 messages during link outages. We present an approach to data routing in these environments that achieves effective results by leveraging the fact that many intermittent network topologies still have an underlying topological stability. We extend the DTN architecture to include a publish/subscribe session layer, providing a more natural fit for many applications and a more robust and efficient framework for communication. Finally, we leverage this framework in TierStore, a distributed shared storage system that eases the adaptation of existing storage-oriented applications and the development of new ones. In this dissertation, we present the design rationale for these contributions, describe and evaluate our implementation efforts, and discuss ways in which our system framework can ease the burden of application development and make deployments more robust. Professor Eric Brewer Dissertation Committee Chair i For my loving wife Rachel ii Contents List of Figures v List of Tables vii 1 Introduction 1 1.1 Potential Benefits of Technology in Developing Regions .............. 3 1.2 Network Connectivity Challenges .......................... 6 1.3 Our Approach ..................................... 9 1.4 Dissertation Outline .................................. 12 2 Background 13 2.1 Store and Forward Networking ............................ 13 2.2 Caching and Replication ............................... 16 2.3 Consistency vs. Availability ............................. 18 2.4 Offline Applications .................................. 20 2.5 Design Themes .................................... 22 3 Implementing the Delay Tolerant Networking Architecture 25 3.1 Delay Tolerant Networking Overview ........................ 28 3.2 Implementation Structure ............................... 32 3.3 Daemon / Router Interface .............................. 37 3.4 Bundle State Management .............................. 39 3.5 Endpoint Identifiers .................................. 42 3.5.1 Links and Adjacencies ............................ 43 3.6 Convergence Layer Interface ............................. 45 3.7 Application Interface ................................. 47 3.8 Simulator Framework ................................. 50 3.9 Evaluation ....................................... 52 3.9.1 Overhead Comparison ............................ 54 3.9.2 Intermittency Tolerance ........................... 58 3.10 Conclusions ...................................... 64 iii 4 Delay Tolerant Link State Routing 65 4.1 Routing Protocol Design Space ............................ 68 4.1.1 Standard Approaches ............................. 68 4.1.2 MANET Routing ............................... 70 4.1.3 DTN Routing ................................. 71 4.2 DTLSR Design .................................... 73 4.2.1 Features of Link State ............................ 73 4.2.2 Modifying Standard Link State ........................ 76 4.3 The DTLSR Protocol ................................. 77 4.3.1 Messages and Flooding ............................ 78 4.3.2 Update Frequency / Expiration ........................ 80 4.3.3 Calculating Best Paths ............................ 81 4.3.4 Administrative Areas ............................. 83 4.3.5 Local Advertisements ............................ 84 4.4 Evaluation ....................................... 85 4.4.1 Protocols Compared ............................. 87 4.4.2 Simulation Scenario ............................. 88 4.4.3 Delivery Results ............................... 89 4.4.4 Delay Results ................................. 91 4.5 Conclusions ...................................... 93 5 A Publish / Subscribe Session Layer for Delay Tolerant Networks 94 5.1 Motivations ...................................... 98 5.2 Design Considerations ................................ 101 5.2.1 Service Model and Session Names ...................... 101 5.2.2 Application Roles .............................. 102 5.2.3 Sequence Identifiers and Obsolete Messages ................ 105 5.2.4 Group Membership and Bundle State .................... 107 5.3 Implementation Details ................................ 110 5.3.1 Session Service Interface ........................... 110 5.3.2 Sequence Identifiers and Vector Clocks ................... 112 5.3.3 Session Membership Protocol ........................ 114 5.4 Related Work ..................................... 116 5.5 Conclusions ...................................... 117 6 TierStore: A Distributed Filesystem for Challenged Networks in Developing Regions 119 6.1 TierStore Design ................................... 122 6.2 Related Work ..................................... 124 6.3 TierStore in Detail .................................. 127 6.3.1 System Components ............................. 129 6.3.2 Objects, Mappings, and Guids ........................ 129 6.3.3 Versions .................................... 130 6.3.4 Persistent Repositories ............................ 132 6.3.5 Updates .................................... 133 6.3.6 Immutable Objects and Deletion ....................... 135 iv 6.3.7 Publications and Subscriptions ........................ 138 6.3.8 Update Distribution .............................. 139 6.3.9 Views and Conflicts ............................. 140 6.3.10 Manual Conflict Resolution ......................... 143 6.3.11 Automatic Conflict Resolution ........................ 145 6.3.12 Object Extensions .............................. 146 6.3.13 Security .................................... 147 6.3.14 Metadata ................................... 148 6.4 TierStore Applications ................................ 149 6.4.1 E-mail Access ................................ 149 6.4.2 Content Distribution ............................. 151 6.4.3 Offline Web Access .............................. 152 6.4.4 Data Collection ................................ 153 6.4.5 Wiki Collaboration .............................. 154 6.5 Evaluation ....................................... 155 6.5.1 Microbenchmarks .............................. 156 6.5.2 Multi-node Distribution ........................... 157 6.5.3 Ongoing Deployments ............................ 160 6.6 Conclusions ...................................... 160 7 Conclusions and Future Work 162 7.1 Dissertation Review .................................. 162 7.2 Design Themes .................................... 164 7.3 Application Examples ................................. 165 7.3.1 Voice Message Phone ............................ 166 7.3.2 Educational Content Distribution ...................... 167 7.3.3 Microfinance Transaction Log Synchronization ............... 168 7.3.4 Remote Medical Consultation ........................ 169 7.4 Future Research Opportunities ............................ 170 7.4.1 Rethinking the Networking API ....................... 171 7.4.2 Exposing Network Reachability ....................... 173 7.4.3 Link Predictions and Erasure Coded Reliability ............... 174 7.4.4