Internet Routing Instability Craig Lab ovitz, G. Rob ert Malan, and Farnam Jahanian UniversityofMichigan Department of Electrical Engineering and Computer Science 1301 Beal Ave. Ann Arb or, Michigan 48109-2122 flab ovit, rmalan, [email protected] network reachability and top ology information, has a num- Abstract b er of origins including router con guration errors, transient This pap er examines the network inter-domain routing in- physical and data link problems, and software bugs. Insta- formation exchanged b etween backb one service providers at bility, also referred to as \route aps", signi cantl y con- the ma jor U.S. public Internet exchange p oints. Internet tributes to p o or end-to-end network p erformance and de- routing instabili ty, or the rapid uctuation of network reach- grades the overall eciency of the Internet infrastructure. ability information, is an imp ortant problem currently fac- All of these sources of network instabili ty result in a large ing the Internet engineering community. High levels of net- numb er of routing up dates that are passed to the core Inter- work instabili ty can lead to packet loss, increased network net exchange p oint routers. Network instabili ty can spread latency and time to convergence. At the extreme, high lev- from router to router and propagate throughout the net- els of routing instabilityhave lead to the loss of internal work. At the extreme, route aps have led to the transient connectivity in wide-area, national networks. In this pap er, loss of connectivity for large p ortions of the Internet. Over- we describ e several unexp ected trends in routing instabili ty, all, instability has three primary e ects: increased packet and examine a numb er of anomalies and pathologies ob- loss, delays in the time for network convergence, and addi- served in the exchange of inter-domain routing information. tional resource overheard (memory, CPU, etc.) within the The analysis in this pap er is based on data collected from Internet infrastructure. BGP routing messages generated by b order routers at ve The Internet is comprised of a large number of intercon- of the Internet core's public exchange p oints during a nine nected regional and national backb ones. The large public month p erio d. We show that the volume of these routing up- exchange p oints are often considered the \core" of the In- dates is several orders of magnitude more than exp ected and ternet, where backb one service providers peer, or exchange that the ma jority of this routing information is redundant, trac and routing information with one another. Backb one or pathological . Furthermore, our analysis reveals several service providers participatin g in the Internet core must unexp ected trends and ill-b ehaved systematic prop erties in maintain a complete map, or default-free routing table, of all Internet routing. We nally p osit a numb er of explanations globally visible network-layer addresses reachable through- for these anomalies and evaluate their p otential impact on out the Internet. the Internet infrastructure. The Internet is divided into a large numb er of di er- ent regions of administrative control commonly called au- tonomous systems. These autonomous systems (AS) usually 1 Intro duction have distinct routing p olicies and connect to one or more remote autonomous systems at private or public exchange Since the end of the NSFNet backb one in April of 1995, the points. Autonomous systems are traditionall y comp osed of Internet has seen explosive growth in b oth size and top olog- network service providers or large organizational units like ical complexity. This growth has placed severe strain on the college campuses and corp orate networks. At the b oundary commercial Internet infrastructure. Regular network p er- of each autonomous system, p eer b order routers exchange formance degradations stemming from bandwidth shortages reachability information to destination IP address blo cks [2], and a lack of router switching capacity,have lead the p op- or pre xes, for b oth transit networks, and networks origi- ular press to decry the imminent death of the Internet [13]. nating in that routing domain. Most autonomous systems Routing instability, informally de ned as the rapid change of exchange routing information through the Border Gateway Supp orted by National Science Foundation Grant NCR-9321060 Proto col (BGP ) [12 ]. and a generous gift from the Intel Corp oration. Unlikeinterior gateway proto cols, such as IGRP and This pap er app ears in the Pro ceedings of the ACM SIGCOMM '97. OSPF, that p erio dical ly o o d an intra-domain network with c Copyright 1997 by the Asso ciation for Computing Machinery, Inc. all known routing table entries, BGP is an incremental pro- Permission to make digital or hard copies of part or all of this work for p ersonal or classro om use is granted without fee provided that to col that sends up date information only up on changes in copies are not made or distributed for pro t or direct commercial ad- network top ology or routing p olicy. Moreover, BGP uses vantage and that copies b ear this notice and the full citation on the TCP as its underlying transp ort mechanism in contrast to rst page. Copyrights for comp onents of this work owned by others manyinterior proto cols that build their own reliabili tyon than ACM must b e honored. Abstracting with credit is p ermitted. To copy otherwise, to republish, to p ost on servers, or to redistribute top of a datagram service. As a path vector routing pro- to lists, requires prior sp eci c p ermission and/or a fee. Request p er- to col, BGP limits the distribution of a router's reachability missions from Publications Dept., ACM Inc., fax +1 (212) 869-0481, information to its peer, or neighb or routers. A path is a se- or (p [email protected]). 1 quence of intermediate autonomous systems b etween source Routing information is dominated by pathological , or and destination routers that form a directed route for pack- redundant up dates, whichmay not re ect changes in ets to travel. Router con guration les allow the stipulation routing p olicy or top ology. of routing policies that may sp ecify the ltering of sp eci c Instability and redundant up dates exhibit a sp eci c routes, or the mo di cation of path attributes sent to neigh- p erio dici ty of 30 and 60 seconds. b or routers. Routers may b e con gured to make p olicy deci- sions based on b oth the announcement of routes from p eers Instability and redundant up dates show a surprising and their accompanying attributes. These attributes, such correlation to network usage and exhibit corresp onding as Multi Exit Descriptor (MED), may serveashints to help daily and weekly cyclic trends. routers chose b etween alternate paths to a given destination. Backb one b order routers at public exchange p oints com- Instability is not dominated by a small set of autono- monly have thirty or more external, or inter-domain, p eers, mous systems or routes. as well as a large number of intra-domain p eering sessions Instability and redundant up dates exhibit b oth strong with internal backb one routers. After each router makes a high and low frequency comp onents. Much of the high new lo cal decision on the b est route to a destination, it will frequency instabili ty is pathological. send that route, or path information along with accompa- nying distance metrics and path attributes, to each of its Discounting p olicy uctuation and pathological b ehav- p eers. As this reachability information travels through the ior, there remains a signi cant level of Internet for- network, each router along the path app ends its unique AS warding instability. numb er to a list in the BGP message. This list is the route's ASPATH. An ASPATH in conjunction with a pre x provide This work has led to sp eci c architectural and pro- a sp eci c handle for a one-way transit route through the to col implementation changes in commercial Internet network. routers through our collab orati on with vendors. Routing information shared b etween p eers in BGP has two forms: announcements and withdrawals. A route an- The remainder of this pap er is organized as follows: Sec- nouncement indicates a router has either learned of a new tion 2 describ es the infrastructure used to collect the rout- network attachment or has made a p olicy decision to prefer ing stability data analyzed in this pap er. Section 3 provides another route to a network destination. Route withdrawals further background on Internet routing and related work. are sent when a router makes a new lo cal decision that a net- Section 4 describ es a numb er of anomalies and pathologies work is no longer reachable. We distinguish b etween explicit observed in BGP routing information. It de nes a taxon- and implicit withdrawls. Explicit withdrawls are those asso- omy for discussing the di erent categories of BGP up date ciated with a withdrawl message; whereas an implicit with- information, and p osits a numb er of plausible explanations drawl o ccurs when an existing route is replaced by the an- for the anomalous routing b ehavior. Section 5 describ es key nouncement of a new route to the destination pre x without trends and characteristics of forwarding instability. Finally, an intervening withdrawl message. A BGP update may con- the pap er concludes with a discussion on the p ossible im- tain multiple route announcements and withdrawals.