Resilient Overlay Networks by David G. Andersen B.S., Computer Science, University of Utah (1998) B.S., Biology, University of Utah (1998) Submitted to the Department of Electrical Engineering and Computer Science in partial fulfillment of the requirements for the degree of Master of Science in Computer Science and Engineering at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY May 2001 ­c Massachusetts Institute of Technology 2001. All rights reserved. Author........................................................................... Department of Electrical Engineering and Computer Science May 16, 2001 Certifiedby....................................................................... Hari Balakrishnan Assistant Professor Thesis Supervisor Accepted by ...................................................................... Arthur C. Smith Chairman, Department Committee on Graduate Students Resilient Overlay Networks by David G. Andersen Submitted to the Department of Electrical Engineering and Computer Science on May 16, 2001, in partial fulfillment of the requirements for the degree of Master of Science in Computer Science and Engineering Abstract This thesis describes the design, implementation, and evaluation of a Resilient Overlay Network (RON), an architecture that allows end-to-end communication across the wide-area Internet to detect and recover from path outages and periods of degraded performance within several seconds. A RON is an application-layer overlay on top of the existing Internet routing substrate. The overlay nodes monitor the liveness and quality of the Internet paths among themselves, and they use this information to decide whether to route packets directly over the Internet or by way of other RON nodes, optimizing application-specific routing metrics. We demonstrate the potential benefits of RON by deploying and measuring a working RON with nodes at thirteen sites scattered widely over the Internet. Over a 71-hour sampling period in March 2001, there were 32 significant outages lasting over thirty minutes each, between the 156 communicating pairs of RON nodes. RON’s routing mechanism was able to detect and recover around all of them, showing that there is, in fact, physical path redundancy in the underlying Internet in many cases. RONs are also able to improve the loss rate, latency, or throughput perceived by data transfers; for example, about 1% of the transfers doubled their TCP throughput and 5% of our transfers saw their loss rate reduced by 5% in absolute terms. These improvements, particularly in the area of fault detection and recovery, demonstrate the benefits of moving some of the control over routing into the hands of end-systems. Thesis Supervisor: Hari Balakrishnan Title: Assistant Professor 2 To my mother, whose unparalleled chicken stock expertise was invaluable. 3 Contents 1 Introduction 12 1.1TheInternetandOverlayNetworks.......................... 12 1.2 Contributions . .................................. 15 2 Background and Related Work 17 2.1InternetRouting.................................... 17 2.1.1 TheOrganizationoftheInternet....................... 17 2.1.2 Routing.................................... 19 2.2OverlayNetworks................................... 22 2.2.1 MBone.................................... 22 2.2.2 6-bone . .................................. 22 2.2.3 TheX-Bone.................................. 22 2.2.4 Yoid / Yallcast . ............................. 23 2.2.5 End System Multicast ............................ 23 2.2.6 ALMI..................................... 23 2.2.7 Overcast . .................................. 24 2.2.8 ContentDeliveryNetworks......................... 24 2.2.9 Peer-to-peerNetworks............................ 24 2.3 Performance Measuring Systems and Tools . .................. 24 2.3.1 NIMI..................................... 25 2.3.2 IDMaps.................................... 25 2.3.3 SPAND.................................... 25 2.3.4 Detour.................................... 26 2.3.5 Commercial Measurement Projects . .................. 26 4 2.3.6 LatencyandLossProbeTools........................ 27 2.3.7 AvailableBandwidthProbeTools...................... 27 2.3.8 LinkBandwidthProbeTools......................... 27 2.3.9 StandardsforTools.............................. 27 3 Problems with Internet Reliability 28 3.1Slowlinkfailurerecovery............................... 29 3.2 Inability to detect performance failures . ....................... 29 3.3 Inability to effectively multi-home . ....................... 30 3.4 Blunt policy expression . ............................. 31 3.5DefiningOutages................................... 31 3.6Summary....................................... 33 4 RON Design and Implementation 34 4.1DesignOverview................................... 34 4.2SystemModel..................................... 36 4.2.1 Implementation................................ 37 4.3BootstrapandMembershipManagement....................... 38 4.3.1 Implementation................................ 39 4.4DataForwarding................................... 40 4.4.1 Implementation................................ 42 4.5 Monitoring Virtual Links . ............................. 44 4.5.1 Implementation................................ 44 4.6Routing........................................ 45 4.7 Link-State Dissemination . ............................. 45 4.8PathEvaluationandSelection............................ 46 4.8.1 Implementation................................ 48 4.9PolicyRouting.................................... 49 4.10PerformanceDatabase................................ 50 4.10.1Implementation................................ 51 4.11Conclusion...................................... 52 5 5 Evaluation 53 5.1 RON Deployment Measurements ........................... 54 5.1.1 Methodology ................................. 54 5.1.2 OvercomingPathoutages.......................... 58 5.1.3 OutageDetectionTime............................ 62 5.1.4 OvercomingPerformanceFailures...................... 63 5.2 Handling Packet Floods . ............................. 72 5.3OverheadAnalysis.................................. 73 5.4RON-inducedFailureModes............................. 76 5.5SimulationsofRONRoutingBehavior........................ 77 5.5.1 Single-hop v. Multi-hop RONs . ....................... 77 5.5.2 RON Route Instability ............................ 77 5.6Summary....................................... 77 6 Conclusion and Future Work 79 6 List of Figures 1-1TheRONoverlayapproach.............................. 14 2-1AnexampleofInternetinterconnections....................... 18 2-2Tier1and2providers................................. 18 3-1LimitationsofBGPpolicy.............................. 31 3-2 TCP throughput vs. loss (Padhye ’98) . ....................... 32 4-1TheRONoverlayapproach.............................. 35 4-2TheRONsystemarchitecture............................. 35 4-3TheRONpacketheader................................ 40 4-4RONforwardingcontrolanddatapath........................ 41 4-5ImplementationoftheIPencapsulatingRONclient................. 43 4-6Theactiveprober................................... 44 4-7 The RON performance database. ........................... 51 5-1 The 13-node RON deployment, geographically . .................. 55 5-212RONnodes,bynetworklocation......................... 56 5-3Loss&RTTsamplingmethod............................ 57 5-4Packetlossratescatterplotcomparison........................ 59 5-5 Packet loss rate scatterplot comparison, dataset 2 .................. 60 5-6Detailedexaminationofanoutage.......................... 61 5-7CDFof30-minutelossrateimprovementwithRON................. 62 5-8 CDF of 30-minute loss rate improvement with RON, dataset 2 ........... 63 5-9Thelimitationsofbidirectionallossprobes...................... 65 5-10 5 minute average latency scatterplot, dataset 1 . .................. 66 7 5-11 5 minute average latency scatterplot, dataset 2 . .................. 67 5-12IndividuallinklatencyCDFs............................. 68 5-13Individualpathlatenciesovertime.......................... 69 5-14Individuallinklossratesovertime.......................... 70 5-15 RON throughput comparison, dataset 1 . ....................... 72 5-16 RON throughput comparison, dataset 2 . ....................... 73 5-17 TCP trace of a flooding attack mitigated by RON .................. 74 5-18Testbedconfiguration.................................. 74 5-19RONencapsulationlatencyoverhead......................... 75 8 List of Tables 3.1Internetlinkfailurerecoverytimes.......................... 30 4.1 Performance database summarization methods . .................. 52 5.1ListofdeployedRONhosts.............................. 54 5.2Ronoutageimprovements.............................. 61 5.3RONlossratechanges................................ 64 5.4 RON loss rate changes in dataset 2 . ....................... 64 5.5 Cases in which the latency and loss optimized paths differ. ............ 71 5.6 TCP throughput overhead with RON . ....................... 76 5.7RONrouteduration.................................. 78 9 These are just a few of the treasures for the stockpot — that magic source from which comes the telling character of the cuisine. - Marion Rombauer Becker, The Joy of Cooking Art is I, science is we. - Claude Bernard Acknowledgments Emerging from my first two years at MIT with my sanity, and a completed thesis, is not something I could have completed alone. I am deeply indebted to many people for their support and patience while I paced, grumbled, bounced,