ASSEMBLER ABGP-COMPATIBLE MULTIPATH INTER-DOMAIN ROUTING PROTOCOL

Universidad Carlos III de Madrid/University of Twente June 2011

José Manuel Camacho Camacho

Supervisor: Francisco Valera Pintor (UC3M) Co-Supervisor: Geert Heijenk (UT)

Contents

1 Introduction 9

2 The Border Gateway Protocol 12

3 Protocol Requirements 17

3.1 Flexible Multipath Routing ...... 17

3.2 BGP-Compatible Advertising Scheme ...... 18

3.3 Controlled Routing Table Growth ...... 18

3.4 Stable under Common Configurations ...... 19

4 Path ASSEMBLER 20

4.1 Decision Process: The K-BESTRO Algorithm ...... 21

4.2 Route Dissemination: Path Assembling ...... 23

4.3 Example: An ASSEMBLER-Capable Autonomous System ...... 25

4.3.1 Downstream Advertisement ...... 26

4.3.2 Upstream Advertisement ...... 26

5 Deployment Considerations 27

5.1 Deployments with Legacy Routers ...... 27

5.2 Multipath Routing Policies ...... 28

5.3 Enhanced Traffic Engineering ...... 28

6 Stability Analysis 30 5 Contents

6.0.1 On Dispute Wheels in Unipath and Multipath Scenarios ...... 30

6.0.2 Synchronous Model of Path ASSEMBLER ...... 33

6.0.3 Path ASSEMBLER Convergence ...... 36

6.0.4 Asynchronous Convergence ...... 38

6.0.5 Stable Multipath Policy Guidelines ...... 40

7 Implementation of an ASSEMBLER-Capable Router 43

7.1 The Evaluation Testbed ...... 44

7.2 The Control Plane ...... 44

7.2.1 The Standard BGP Daemon ...... 45

7.2.2 Path ASSEMBLER Extensions in XORP ...... 46

7.2.3 Modifying the RIB and FEA Processes ...... 48

7.3 The Data-Forwarding Plane ...... 49

7.4 Disclosed Path-Diversity ...... 50

8 Related Work and Conclusions 53

8.1 Conclusions ...... 54

8.2 Future Work ...... 54

Bibliography 58

Abstract

Multipath routing offers several potential advantages compared to unipath in terms of re- sources usage, reliability and security. The idea of using several paths concurrently to send traffic towards a destination has already been explored and deployed for cost-based routing solutions, like those typically found in intra-domain routing. Nevertheless, in policy-based routing scenarios, like inter-domain routing, existing multipath solutions have not been em- braced yet, mainly because of the backwards compatibility requirements with BGP and the impossibility of performing a global coordinated upgrade of the whole Internet.

This work presents the design and implementation of a multipath inter-domain routing protocol that is backwards compatible with BGP and does not require any kind of inter-AS coordinated deployment. The protocol supports the current policies of ASes and defines a more flexible set of path selection rules to fully exploit the multipath infrastructure of an AS.

The protocol is shown to advertise multipath information consistently in regular unipath BGP updates. In addition, the protocol stability analysis is provided to characterize its be- havior and which policies are supported without creating oscillations.

The second part of the work presents an implementation of the protocol in a real software router using XORP. The implementation of the protocol is combined with a multipath FIB designed using CLICK in a testbed to carry out performance measurements of the protocol. Chapter 1

Introduction

The provision of multiple paths between two nodes has been envisioned for many years as a natural way to enhance communication networks. Once multiple paths are in place, nodes can divert traffic from failed links or split load among them, achieving fast recovery [37] and load balancing [19, 15] respectively. Those techniques should improve the reliability and the performance of the network.

Recent contributions [36, 25, 27] point out that the usage of multipath routing can be advantageous in inter-domain scenarios. Since most of the ASes through the Internet already have redundant connections with their neighbors [24], by embracing multipath inter-domain routing they could benefit from a more flexible use of their resources [35]. The reachability information advertised through these redundant connections should provide ASes with mul- tiple alternatives to route the traffic towards a destination. Those alternative paths could be used simultaneously and enable the aforementioned recovery and balance techniques. Un- fortunately, in most cases the unipath nature of the Border Gateway Protocol [28] impedes making use of those multiple paths concurrently.

Most ASes have no choice but to rely on techniques such as prefix deaggregation [28] or load sharing [8] to relax the constrains of BGP. Nevertheless, those techniques present their own limitations. By deaggregating prefixes, and autonomous system can handle the traffic corresponding to each sub-prefix differently and forward each split traffic flow through different ASes. In the case of load balancing, the balancing is widely used in intra-domain among equal-cost paths. Each packet, or a flow of packets (e.g. packets sharing the same origin and destination transport addresses) are routed through the available paths. However, with the current load sharing approach [7, 13, 15] used in BGP, the egress point for a certain prefix can be changed periodically in terms of minutes, but it cannot be changed for each packet or flow. The control plane of the network cannot keep up with the necessary changes since every time a packet follows an alternative path, the control plane of BGP generates a new BGP advertisement to avoid routing inconsistencies and loops. The generated churn in the network makes load balancing unfeasible. In practice, only stub ASes exploit their multi- homing connections to perform load balancing among different egress ASes, given that they do not have to re-advertise BGP information.

The previous example shows that ASes are keen on more flexible routing configurations. However, in spite of the potential benefits that using multipath routing can bring about in inter-domain scenarios, so far, the lack of economic incentives to replace BGP has hindered 10 Chapter 1 Introduction

Internet-wide multipath deployments. Moreover, the latter imposes that any approach to deploy multipath inter-domain must be BGP-compatible.

Aimed at hastening large-scale deployments, some backwards compatible solutions have appeared in the literature in the latest years. BGP extensions such as [18, 6] provide multipath capabilities by taking advantage of the multiple interconnections between two ASes. Those paths have the same BGP attributes, such that every selected path can be advertised with the same BGP update. Whereas the latter ensures backwards compatibility with BGP, the multipath set yielded by these solutions is rather limited, e.g. traffic cannot be forwarded across different egress ASes simultaneously even though available paths exist.

An alternative to use richer sets of multiple paths (i.e. multipath sets) is forwarding packets among all available paths and advertise only one. That would require additional mechanisms to detect traffic loops [37, 25] or advertise paths that may be less attractive to legacy routers (e.g. routers advertise the longest received path [31]). Other solutions rely on a separate protocol to incrementally request or advertise additional paths [36, 32] and they can provide more flexible multipath configurations. Yet, they require that at least two neighbor ASes must coordinate to deploy that type of solutions, which represents a main drawback for those approaches.

In this work, a novel protocol for multipath inter-domain routing, ASSEMBLER, is pre- sented. ASSEMBLER stands for AS-SEt-based Multipath BLending Routing since the pro- tocol operation resembles a mixing of paths. It is the first inter-domain routing protocol that features both, flexible multipath routing and backwards compatibility with BGP, without any kind of coordination between ASes or additional protocols.

Furthermore, not only is ASSEMBLER backwards compatible with BGP, but also it ad- heres to its philosophy. It is able to support and map to routing policies the existing business relationships among ASes. Current routing policies, path import and export rules, and traffic engineering techniques are supported and in some cases extended. ASSEMBLER advertise- ments do not incur in any penalization when compared to BGP thanks to its path assembling technique and the selection process (so-called K-Best Routing Optimizer) can be locally tuned to cover a myriad of multipath configurations ranging from unequal AS path length multi- path through different egress ASes to a fallback configuration that mimics exactly the BGP behaviour.

This work is an original unpublished contribution that began with the early idea pointed out by Dr. Alberto Garcia-Martinez suggested in [27] of exploiting prefix aggregation to de- ploy multipath solutions compatible with BGP. The contribution of this work is the result of a series of discussions among the main author, the supervisor and Dr. Garcia-Martinez. The enumeration of requirements for a backwards compatible multipath inter-domain routing pro- tocol, the evolution of the original idea to the current definition of the assembling technique, the analysis of the implications of adding the assembling technique to BGP, analysis of in- teroperability in mixed environments, the definition of a proof-of-concept multipath decision process, implementation of the protocol in a state-of-the-art software router and the stability analysis and resulting stability guidelines can be fully attributed to the main author.

The structure of the work is as follows, after reviewing briefly BGP in Chapter 2, Chapter 3 introduces the requirements that are aimed for the protocol design. The protocol itself in presented in Chapter 4 along with an example to show the flexibility supported by ASSEM- BLER in its configurations. A group of important deployment considerations are detailed in Chapter 5. The stability of the protocol is proven and configuration guidelines to guarantee stability are given in Chapter 6. Chapter 7 presents the implementation of the protocol and 11 Introduction

the validation using a virtual testbed. The work is completed with a comparison between AS- SEMBLER and the existing multipath inter-domain proposals in the related work in Chapter 8 along with the conclusions and future work. Chapter 2

The Border Gateway Protocol

This chapter is aimed at introducing the basics of the Border Gateway Protocol used in inter- domain routing. The terminology and the concepts presented in this chapter are used through- out the work to describe the multipath extensions for BGP. The Border Gateway Protocol (hereafter BGP [28]) is the de-facto standard for advertising reachability information in the Internet, where several independent organizations interconnect to create a large scale net- work and profit from the exchanged traffic between end-hosts. Those organizations are the so-called Internet Service Providers (i.e. ISPs). Each ISP runs one or more Autonomous Sys- tems (i.e. AS) or domains, which are networks that hauls traffic according to an economic- driven policy. The AS networks interconnect among them and exchange traffic. When the exchanged traffic between two ASes is uneven or one of them has a better location in the network (e.g. a main provider or tier-1), it is said that they keep a transit relation, one AS plays the role of the provider, offering hauling service towards a destination to the other AS, its customer. The provider charges a per-bit rate to the customer for the coursed traffic from and to the customer network. On the other hand, when the exchanged traffic is roughly the same or both ASes are of similar importance, the two ASes have a peering relation, they both act as peers without charging each other.

Hence, the fact that some paths may provide larger profit than others makes that cost- based protocols such as OSPF cannot be used in this context, since the path with lowest sum of weights is not necessarily the most profitable. In inter-domain scenarios ISPs must define routing policies according to their business model, such that routers select the most profitable path for the ISP. Moreover, the advertisement of some paths may cause the ISP to incur in extra losses for carrying undesired traffic, therefore in addition to the import policy, ISPs must also define an export policy that states to which ASes a path must not be announced. The BGP standard provides the necessary mechanisms to disseminate the reachability infor- mation, techniques to implement routing policies and path attributes to enforce them. To that extent, BGP defines for each path which attributes may be used to describe the path charac- teristics. The attributes are used in a decision process to select the bests path according to the policy.

The dissemination of reachability information happens in three different steps. Firstly, one or more neighbor ASes advertise reachability information to different border routers in the AS through external BGP (i.e. eBGP) sessions. Secondly, after the eBGP dissemination happens between ASes, the internal BGP (i.e. iBGP) redistribution takes place, such that every BGP router inside the AS is aware of the available paths learnt at different border 13 The Border Gateway Protocol

RIB Process Decision Egress Filtering Ingress Filtering

Adj-RIB-In Adj-RIB-Out FIB

Figure 2.1: BGP Process Architecture routers and the path selection becomes a distributed process, with every BGP router taking a consistent decision according to all the received announcements. The decision process carried out at each router selects the best path for that router. In some cases, a router will play the role of egress traffic point (i.e. it has learnt its most suitable paths through eBGP) and in others the routers will play the role of an intermediate node or an ingress point (i.e. their best path comes from an iBGP session). The third step consists in advertising further the decision made by each router to neighbor ASes.

The regular operation of each BGP router in the AS consists in establishing and maintain- ing a session with other BGP routers and exchanging BGP updates with them for different prefixes. Upon the reception of an advertisement for a prefix, the BGP router receives the path to that prefix along with a set of values for the different BGP attributes such as preference val- ues, the neighbor advertising the path and the ASes that the traffic will cross. BGP attributes are rewritten by the router depending on their scope, e.g. an attribute can be meaningful to the border router, to the entire AS, to the neighbor ASes or end-to-end.

The received paths are passed from left to right through the blocks depicted in Fig.2.1 starting at the import filter, which checks that the paths are compliant to the routing policy and tags some of their attributes, such as the local preference. Afterwards, if multiple paths are available for the same prefix, the decision process is carried out to select which is the most suitable given the routing policy. Every path received from any BGP session towards the same prefix is compared with other paths for that prefix. The selection of one path or another depends on the attribute values assigned to each path. The paths are compared on different attributes following the rules defined in Table 2.1, which represents the BGP decision process [28]. The process executes rules sequentially until only one path is left within the candidate set, i.e. the winner. Then, the winner is passed onto the Routing Information Base RIB in order to be deployed in the FIB.

The first decision rule (the WEIGHT attribute is typically not used) is based on the LO- CAL_PREF attribute value. The latter reflects the preference of the network administrator for a certain path or set of paths and overrides the values of other attributes since it is the first decision rule. According to the typical business relations between ASes mentioned above, the paths coming from customer ASes are preferred over paths coming from peers, since the AS relaying the traffic earns money using them. Paths from peer ASes are preferred over paths from providers since no money is either paid or received per coursed traffic. Finally, paths coming from providers are economically less attractive since that implies that the AS using them pays for the coursed traffic. These preferences are mapped to numeric integer values of 14 Chapter 2 The Border Gateway Protocol

Table 2.1: BGP Decision Process 1.- Keep paths with highest LOCAL_PREF value

2.- Keep paths with shortest AS path

3.- Keep paths with lowest ORIGIN value

4.- For each advertising AS, select the path with lowest MED value

5.- If there is a remaining path with session TYPE eASSM, delete paths with TYPE iASSM 6.- Keep paths with lowest IGP cost

7.- Keep paths with lowest BGP_ID

8.- Select the path advertised from the lowest network address

the LOCAL_PREF (e.g. 60 to a provider paths and 100 to customer paths).

Since two paths may have the same LOCAL_PREF value, e.g. two paths coming from two different customer ASes or two path coming from the same AS but received at two differ- ent border routers, additional rules are required to decide on one path. Before analyzing the next rule, the AS_PATH concept must be introduced. The necessity of hiding connectivity information consistently to avoid the economic losses as pointed out above and the scale of the network conditioned the design of BGP to be an extension of distance vector protocols called path vector protocols. In particular for BGP, the path vector is an AS-level represen- tation. Each AS is assigned with a unique identifier called AS Number such that each time a border router of an AS advertises a path outside its own AS, it appends the local AS Number to the path. The collection of AS Numbers of ASes crossed along the path is the value of the AS_PATH attribute. The AS_PATH is in turn formed by a collection of segments. Each segment can be either an ordered sequence of AS Numbers, called AS_SEQUENCE or an unordered set of AS Numbers delimited by braces, called AS_SETs. The AS_PATH length is computed by counting the length of each AS_SEQUENCE as the number of ASes within the sequence and the length of AS_SETs as length one.

The third rule is related to the way the reachability information is generated by the first AS. If the advertisement was dynamically generated by redistributing intra-domain routing information into BGP, the ORIGIN attribute gets a lower value. Otherwise (e.g. statically configured) the ORIGIN is higher. The reason for this comes from the idea that in case something fails, a dynamic configuration will advertised that the reachability information formerly propagated is not valid anymore, whereas static configurations are not responsive to network failures.

If at this point of the decision process there is more than one path available, either they come from the same AS but from different border routers or from different ASes but having the same AS_PATH length. In the former case, if the AS receives the same path through different routers (exit points), it may have to satisfy the preferences of the neighbor AS. Think for instance in the case of a customer AS advertising one path through two different BGP sessions with the same provider, the provider should respect the preferences of its customer. 15 The Border Gateway Protocol

To that extent, some BGP attributes are used to influence the treatment received by a path in a neighbor AS, like the multi-exit discriminator, i.e. MED. The paths coming from the same AS and not removed until here have the same LOCAL_PREF and the same AS_PATH length. Then, the advertising AS can suggest that it prefers to receive traffic over one path or another by properly setting the MED value. Rule 4 removes paths coming from the same AS that are not of minimum MED value.

Another way of influencing the decision of a neighbor AS is the use of BGP Communities, which are designed to give an homogeneous treatment to the paths containing them (e.g. assign a certain LOCAL_PREF value). BGP Communities are optional attributes and not every AS support them. Typically, the actions taken over a path with a certain community are posted publicly by provider ASes such that its customer can use them. Communities used between peers are typically negotiated privately between the peers. BGP Communities are processed before the decision process is executed and they do not have an specific rule in the process, although they can influence the outcome of a certain rule by modifying the BGP attributes of a paths.

After the preferences of the customers are processed taking into account the MED values, the BGP router by means of Rule 5 gives preferences to paths learned from a BGP session with a router in an external AS to paths received from an iBGP session. Otherwise, all the routers may end up selecting an internal advertisement and loops and oscillations may occur.

Rule 6 perform what is known as hot-potato routing, this is if there are several alternatives compliant with the routing policy of the AS, try to course the traffic towards the closest egress point of the AS, since the traffic will stay as less time as possible and the operational costs per bit will be the lowest. Therefore, among the multiple possibilities those with the lowest internal cost are chosen.

The remaining decision rules do not enforce a given preference or an optimization over the selected paths but they perform a tie-break, such that only one path is left after the decision process. Rule 7 selects the routes coming from the BGP neighbor router with the lowest BGP_ID exchanged in the BGP session establishment. If there is more than one path coming from the same neighbor router (e.g. two routers have two connections in parallel) then the path advertised from the network interface with the lowest network address is chosen, as stated in Rule 8.

After the decision process, exporting rules are configured per BGP session. When the router selects a certain path, there are some BGP session through which the path must not be advertised. Regarding external connections with other BGP border routers, the router config- uration usually follows export rules as defined in [10]. For instance, if a router selects and eBGP path coming from a provider AS, it must advertise that path only through customers, since the AS will pay the provider for the traffic towards that destination and the customers are paying to the AS. Otherwise, if the AS announces to the rest of its providers it pays for relaying the traffic to the advertising provider and it is charged by the rest of the providers for sending traffic to it. The case in which the AS announces the path to its peers is similar except for the fact that the AS is not charged by the peers. Paths coming from peers can be only advertised to customers for the same reason. Only paths coming from customers can be advertised to providers and peers.

For iBGP, the export rules are typically simpler and the most relevant is the one that states that paths learned by means of an iBGP session must not be re-advertised through another iBGP session. Notice that paths advertised through iBGP does not add any kind of information about the hops that would be performed inside the AS, therefore there is no way 16 Chapter 2 The Border Gateway Protocol

to detect internal loops among iBGP speakers.

If the selected rule does not match any of the discarding egress filters, it can be advertised through that BGP session in a BGP update message. BGP updates contains typically multiple entries, each of them regarding a different prefix. Since BGP is designed as a unipath routing protocol, each router is expected to select and use only one path per prefix. Therefore, two advertisements regarding the same prefix are included in the same update (this should not happen in practice since the routers update the information of a prefix if it has not been advertised yet) or an update is received after another, the last information received overwrites the previous information announced by that peer.

Finally, to conclude with this brief description of BGP, although internal scalability tech- niques such as route reflectors, route servers and confederations are not covered in this work and no multipath extensions are proposed to them, the multipath protocol proposed in this work is able to co-exists (under certain conditions) with legacy BGP routers within the same AS and interoperate with already existing multipath intra-AS solutions like BGP-AddPaths [32]. Chapter 3

Protocol Requirements

The main goal of ASSEMBLER is to provide ASes with a backwards compatible solution for inter-domain routing that enables multipath routing. In this chapter, the defining requirements for ASSEMBLER are introduced prior to describing the relevant parts of the protocol in depth. Those requirements motivate the design choices that are presented in the next chapter and provide a clear view of the main features of the protocol. The discussion addresses the issues of target multipath configurations, backwards compatible updates, stability and data plane growth.

3.1 Flexible Multipath Routing

ASSEMBLER must flexibly let administrators choose the characteristics and the amount of paths used in the routing system. The multipath protocol must feature enough flexibility to concurrently select paths that, (1) have different next AS, (2) have different AS path length and (3) have different internal cost. Moreover, the protocol must be able to select a subset of the paths matching the previous conditions using a deterministic tie-break. ASSEMBLER must empower administrators with the tools to implement such a broad range of routing policies. In some cases, the administrator would like to provide a router with the whole set of paths and in other cases, administrators may prefer keeping only those with certain attributes, (e.g. shortest AS path length).

The first requirement that we impose on ASSEMBLER is that regardless how the selec- tion process of multiple paths is tuned, the BGP winner path is always included in the multi- path set. In addition to the BGP winner, according to criterium (1), a router can either include paths through different egress ASes or limit the multipath set to go through the same next AS. The criterium (2) defines if equal AS length multipath (i.e. ELMP) is enabled or additional paths, longer than the shortest one, can be selected. Criterium (3) implies that internal equal cost multipath routing (i.e. ECMP) is supported and additionally, the administrator can tune how much the internal paths deviates from the hot-potato routing behaviour [30]. A strong requirement is that ASSEMBLER must allow every AS to select the type of multipath that they need independently from other ASes, i.e. without any type of coordination. 18 Chapter 3 Protocol Requirements

150.0/16 150.0/16 AS10,AS9 AS9 AS6 AS7 AS5 AS1 R6 R7 7{6,10,9} 7,9 R5 7,9 R8 7,9 R9

1,7{6,10,9} R2 R4 1,7,9 AS2 1,7,9 AS4 1,7{6,10,9} AS3 160.1/16 AS30 160.1/16 AS30

Figure 3.1: Model of a transit AS with Path ASSEMBLER Routers

3.2 BGP-Compatible Advertising Scheme

ASes advertise each other reachability information by means of BGP updates. Whereas pro- cessing regular BGP updates should not present any shortcoming for multipath routers, ad- vertising multiple paths per network prefix in a single BGP update is not a trifle. Multipath routers should respect the structure and the semantic of the attributes included in the updates, such that legacy routers can keep on processing them. Concatenating multiple paths to the BGP message is not enough, provided that the update of a path has implicit the withdrawn of previous updates.

Therefore, ASSEMBLER must carry out some additional processing to merge informa- tion from multiple paths and accommodate them into regular BGP updates. To that extent, path assembling (Section 4.2), a particular case of prefix aggregation [28] seems an outstand- ing candidate. It is a crucial requirement that generated advertisements must be representative of the aggregated paths, such that a router (legacy or not) can perform any regular BGP pro- cessing over the advertisement, as if the paths were announced separately. For instance, when a router receives an announcement containing an aggregate of paths, it must be able to derive the local preference for the aggregate or apply MED values comparison consistently. The protocol must identify those cases in which the advertisements are not representative and do not perform aggregation. Even for BGP, RFC4271 [28] identifies different situations where it is not consistent to aggregate multiple prefixes due to conflicting attributes. Thus, the proto- col must avoid those situations in which inconsistent network advertisements may be created as a consequence of the aggregation process.

3.3 Controlled Routing Table Growth

The design must address the well known problem of the inter-domain routing table growth. Whereas routers feature more processing and memory capacity at the control plane, the situa- tion at the data-forwarding plane is completely different. The hardware that forwards packets at wire-speed is expensive and its storage space constrained. The adoption of multipath does 19 3.4. Stable under Common Configurations

nothing but worsening the problem as multiple next-hops are stored per prefix. A growth in the amount of paths selected can potentially rise issues with the limitations of the data plane.

Therefore, the protocol must be aware of those constrains and must limit the amount of paths relayed to the data plane. The requirement for the protocol is to be able to select a subset of k-best paths per prefix, such that the size of each routing table entry is limited. Every path in the subset must be compliant with the routing policy and the k-best tie-break must be deterministic.

3.4 Stable under Common Configurations

BGP has been proven to be unstable under conflicting routing policies. The existing relation among those routing policies that cannot be fulfilled simultaneously is calleddispute-wheel [12]. In presence of dispute-wheels BGP is not guaranteed to converge and the network may end up in a permanent oscillation. Permanent oscillations do not happen often in practice since routing policies are typically overruled by the business relationships among ASes. It can be proven that when routing policies align with those relationships dispute-wheels cannot be created.

The work in [5] presents a more abstract framework for the analysis of the stability in policy-based routing protocols. The framework extends the concept of dispute-wheels to reflexive policy relations. The concept of reflexive relations is more powerful in the sense that it covers the BGP dispute-wheels and allows to extent stability results to multipath policy- based routing protocols. ASSEMBLER must be able to converge in absence of reflexive relations among policies. Relaying on the abstract framework in [5], the protocol must be proven stable in those situations in which conflicting policies do not exists, specially in those that align with business relations among ASes. Chapter 4

Path ASSEMBLER

ASSEMBLER is a novel multipath inter-domain routing protocol inspired in the BGP prefix aggregation to compact the multipath information. ASSEMBLER stands for AS-Set-based Multipath BLEnding Routing, since the protocol blends the additional AS_PATHs and stores the result in AS_SETs. ASSEMBLER keeps backwards compatibility and allows for a pro- gressive deployment of multipath-capable routers. The specification of ASSEMBLER relies on two main cornerstones: a multipath selection process and a BGP-compatible multipath advertising scheme.

Fig.4.1 shows the block diagram of an ASSEMBLER process running in the control plane of a router. There are some differences between Fig.4.1 and a BGP process diagram (see Fig.2.1). The import policy (ingress filteringin Fig.4.1) is applied first, like in BGP. The BGP decision process has been replaced by the multipath selection algorithm K-BESTRO (pronounced cabestro) that stands for K-Best Routes Optimizer. K-BESTRO is presented in detail in Section 4.1. The output of the K-BESTRO block is a set of K paths instead of a single winner path. K-BESTRO features three parameters to tune the characteristics of the multipath set. The parameter ELMP defines the maximum difference in AS path length be- tween the shortest and the longest AS path. ECMP defines the difference in internal cost among selected paths. Finally, the KBEST parameter limits the maximum size of the multi- path set, which should be set depending on the capacity of the routing table to store prefixes with multiple next-hops.

The paths in the multipath set are passed to the RIB in order to be installed in the data plane (through the FIB). Afterwards, they undergo the export policy. The export pol- icy (egress filtering) generates the same advertisement for all the peering sessions that the router maintains. Therefore, a neighbor is either advertised or the export policy discards the whole multipath set as soon as one path matches a filter. Otherwise, different paths could be discarded for each peering session, generating different advertisements. Adding neighbor- specific announcements [34] is out of the scope of this paper.

Next to the egress filtering block, there is the new block called Assembling, which is responsible for generating the advertisements. The assembling algorithm ensures backwards compatibility, creating special BGP announcements that can be processed by legacy routers, do not incur in penalization when competing with regular unipath BGP announcements in the selection process and allow multipath capable ASes to use several paths concurrently. The algorithm takes its name from the way of constructing those announcements that resembles an 21 4.1. Decision Process: The K-BESTRO Algorithm

RIB Assembling K-BESTRO Egress Filtering Ingress Filtering

Adj-RIB-In Adj-RIB-Out FIB

Figure 4.1: Path ASSEMBLER Process Architecture assembling of pieces (e.g. AS_NUMBERs in this case). The announcement is an aggregated version of the multipath set that cannot be distinguished from the outcome of a regular prefix aggregation. See Section 4.2 for details about the assembling procedure for external (i.e. eASSM) and internal (i.e. iASSM) ASSEMBLER peering sessions (eASSM/iASSM are also used to refer BGP peering sessions, unless stated otherwise). Finally, the advertisement containing the assembled path is propagated to the neighbor routers.

4.1 Decision Process: The K-BESTRO Algorithm

The decision process of ASSEMBLER is carried out over the set of advertisements for a given prefix that are not discarded by the import policy. The decision rules resemble those for BGP. Meanwhile the BGP decision process clearly has a tie-breaking character and paths are trimmed from the set of candidates on the look out for the most suitable path. The re- quirements regarding flexibility of K-BESTRO completely redefine its philosophy and it is inspired in the decision process of Morpheus [33]. In the design of Morpheus, the decision process creates a ranking of the candidate paths according to some configurable criteria rather than discarding them. Afterwards, the set of best paths is selected, possibly according to dif- ferent criteria, this time applied over the paths already sorted (e.g. select the first k paths in the ranking with MED value equal to 10). Therefore, K-BESTRO can be seen as a particular instance of a Morpheus ranking with the criteria presented in the next paragraphs. Each of them is mapped into a phase of the algorithm depicted in Table 4.1,

The ranking criteria of K-BESTRO rank the BGP winner in first position and the rules respect the semantic of the BGP attributes Rules 1 to 5 discard paths like a regular BGP decision process. The BGP winner is never discarded by those rules and rules 6.a-d give higher ranks following the order used by BGP to tie-break the paths. Therefore the BGP winner is always ranked first. Generally speaking, the algorithm must always advertise the winner and propagate other aggregatable paths whenever possible.

In addition, in order to keep the semantic of BGP attributes and make the paths sortable, some of them must be discarded before the remaining paths are sorted in a rank. Otherwise, inconsistent multipath decisions can be made with respect to the semantic of the attributes. For instance, it is not consistent that two paths with different LOCAL_PREF appear in the final ranking or in the selected multipath set. It is not sounded either that two paths coming 22 Chapter 4 Path ASSEMBLER

from the same AS and with different MED values are simultaneously used, since the customer is explicitly stating that it prefers one path to the other to receive the traffic.

The first phase starts with rule 1, which keeps just the paths with highest local preference. The next step in BGP is keeping paths with shortest AS path length. Instead, K-BESTRO considers paths that satisfy the relation AS_PATH_LENGTH <= shortest-l+ELMP, being shortest-l the shortest AS path length value found in the candidate set and ELMP is the parameter introduced earlier. The latter is implemented in rule 2.

The ranking criteria of K-BESTRO keep the order of the BGP rules For example, the latter implies that the BGP AS path length rule must not be overridden by the MED rule, i.e. a path with lower MED but longer AS path must not cause any path with shorter AS path length to be overlooked by the algorithm. K-BESTRO ensures that every path taken into account in the ranking honours the highest LOCAL_PREF, highest ORIGIN, lowest MED per AS and session TYPE (i.e. eASSM or iASSM) criteria exactly as BGP does.

The second phase applies these criteria (rules 4-5). For each AS advertising a path for the prefix, the algorithm looks for the paths of shortest length through that AS and the lowest MED value of that path. Every path from that AS and different MED value is removed at rule 4.c. The phase is completed by leaving paths only from one session TYPE. If there is a path from an external session, i.e. eASSM, paths with session TYPE iASSM are removed (rule 5). This is needed to avoid that two border routers try to course traffic through the external path of the other (see [28] for further details).

The final ranking of paths must be performed over monotonically increasing and bounded attribute values Applying rules 1-5 leads to consistent results regarding the selected mul- tipath set. Once the considered paths are compliant with those rules, the ranking can be performed upon the remaining attributes without violating the specified routing policy and the order of BGP rules. For example, two paths with equal preference, origin and coming from different ASes, can be ranked according to their AS path length without creating any inconsistency.

The algorithm executes the third phase (rule 6) and ranks the paths according to the cri- terium of shortest AS_PATH_LENGTH first. If a several paths in a subset draw in AS path length, it sorts the subset from lower internal cost to higher. Within the subset, if the first ranked path has a cost of lowest-c it removes the paths with internal cost higher than lowest- c+ECMP, where ECMP is a parameter that can be tuned by the administrator. While either tunnelling or IGP equal cost multipath are used inside the AS, ECMP should be equal to 0. If at this point some paths have the same AS path length and interior cost towards the next- hop, the paths with lower BGP_ID are ranked first. If several paths have the same BGP_ID attribute, then the ones advertised from the interface with the lowest address are ranked first.

The K-BESTRO algorithm selects paths in order of appearance in the ranking and selected paths are aggregatable As described at the beginning of the chapter, the selected multipath set ends up aggregated into a single BGP advertisement. Therefore, the selected paths must be aggregatable, otherwise the generated announcement is not representative of the multipath set and that may lead to routing inconsistencies. RFC4271 [28] defines that two path with different MED values should not be aggregated. This restriction only applies to paths advertised through iBGP sessions. Similarly, only paths with the same NO_EXPORT:X Community (i.e. do not export this path to a specific peer X) can be aggregated, since as 23 4.2. Route Dissemination: Path Assembling

mentioned above, neighbor-specific configurations are not supported by ASSEMBLER. If the first ranked path does not include the NO_EXPORT:X community for any peer, the algorithm should overlook other paths in the ranking including any NO_EXPORT:X community when selecting the multipath set.

This last criteria is implemented in the fourth phase (rules 7-10) is executed. The param- eter KBEST is defined by the administrator and limits the maximum size of the multipath set. The fourth phase takes care of selecting a maximum of KBEST paths that can be ag- gregated and advertised together. According to the previous paragraph, if the first ranked path is tagged with the BGP Community NO_EXPORT:X, then the multipath set contains only the first KBEST ranked paths with the same community. Otherwise, paths with any NO_EXPORT:X community are deleted from the rank to avoid aggregation conflicts. There- after, if the ranked paths come from an iASSM session, then select the first KBEST paths in the ranking. Else if they come from eASSM sessions, select the first ranked KBEST paths coming from the same AS and with the same MED value as the first ranked path, as stated in RFC4271.

The algorithm finishes relaying the set of KBEST paths to the egress filtering block that implements the export policy.

4.2 Route Dissemination: Path Assembling

The decision process constructs a multipath set compliant with the preferences of the admin- istrator. The multipath set must be advertised to every neighboring AS with an established peering session. Advertising an array of paths for each network prefix is not supported by BGP. The algorithm presented in this section is applied to the set of paths to embed the mul- tipath information into a single BGP advertisement.

The algorithm follows the philosophy used in prefix aggregation to compact the multipath information. Prefix aggregation defined in [28] defines how the attributes of two advertise- ments can be combined under some conditions, such that two contiguous prefixes propagated within each advertisement can be combined into a larger prefix and advertised in a single BGP update message. The attributes of the new message are the result of aggregating those in the two advertisements. Our path assembling can be understood as the aggregation of several advertisements carrying the same prefix.

Besides other attributes like the NEXT-HOP or the ORIGIN, of special interest is the AS_PATH attribute aggregation. When two contiguous prefixes are aggregated, it is neces- sary to keep the AS_PATH information to maintain path loop-freeness. In [28] the minimum requirements for the path aggregation algorithm are specified. Any algorithm compliant with those minimum specifications can safely combine the AS_PATH information from several announcements. The algorithm used by ASSEMBLER meets the minimum requirements of [28] and creates an AS_PATH following the most commonly found format in current routing tables. Data sets collected at some Internet vantage points [23, 24] brings out that recorded aggregations construct always an AS_PATH with an AS_SEQUENCE followed by an AS_SET. For example, the aggregate of paths P = 1, 2, 3, 5 and Q = 2, 3, 4, 5 should look like A = 2, 3, {1, 4, 5}.

In addition, the algorithm tries to be consistent in the assembling and keep meaningful information. For example, it creates AS_PATHs whose length is equal to the path length 24 Chapter 4 Path ASSEMBLER

Table 4.1: K-BESTRO Algorithm 1.- Keep paths with highest LOCAL_PREF value

2.- Look for the shortest AS path, store the length in shortest-l and keep paths with AS_PATH_LENGTH<=shortest-l+ELMP. 3.- Keep paths with lowest ORIGIN value

4.- For each advertising AS,

4.a.-Look for the subset of paths with lowest AS path length

4.b.-Select the lowest MED value in that subset

4.c.-Delete the paths from that AS with different MED value

5.- If there is a remaining path with session TYPE eASSM, delete paths with TYPE iASSM 6.- Rank the paths according to,

6.a.-Paths with shortest AS_PATH_LENGTH go first

6.b.-If a subset paths have the same length, paths with lowest internal cost goes first Discard paths within the subset with internal cost>lowest-cost+ECMP (Default ECMP=0) 6.c.-If equal cost, lowest BGP_ID goes first

6.d.-If same BGP_ID, lowest peer address goes first

7.- If the first ranked path has the NO_EXPORT:X Community,

7.a.-Then select only the first KBEST ranked path with the same NO_EXPORT:X Community 7.b.-Else, delete paths with any NO_EXPORT:X Community

8.- If the ranked paths have session TYPE iASSM, select the first KBEST paths 9.- Else if the ranked paths have session TYPE eASSM, select the first KBEST paths from the same AS and MED value as the first ranked path 10.- Return the selection. K-BESTRO ENDs

BGP would advertise. Thus, using assembling neither represents a penalty nor an advantage to multipath nodes, what we believe is fair. Moreover, the algorithm also preserves the last AS added to the AS_PATH as it is consider meaningful in some policies (e.g. neighboring AS filtering). The algorithm does not preserve the position of the origin AS, given that typically ASes rely on RIRs to check the origin and ASes included in an advertisement [22]. 25 4.2. Route Dissemination: Path Assembling

Table 4.2: Assembling Algorithm 1.- Create an empty AS_SEQUENCE and AS_SET.

2.- Pick up the shortest path from the multipath set and initialize shortest to its AS_PATH_LENGTH. 3.- Copy the most to the left AS_NUMBER of the shortest path into the AS_SEQUENCE. 4.- Keep parsing the AS_NUMBERs in the shortest path (if repeated, process it only once): Check its presence in other paths in the set. 4.a.-If present in all paths and after AS_NUMBERs already in the AS_SEQUENCE, concatenate it to the AS_SEQUENCE. 4.b.-Else add it to the AS_SET

5.- Append the AS_SET at the end of the AS_SEQUENCE to create the AS_PATH. 6.- For each remaining path, parse every AS_NUMBER. If the number has not been previously added to the AS_PATH, add it to the AS_SET. 7.- Compare the length of the resulting path, if longer than shortest run 7.a, otherwise run 7.b, 7.a.-Starting from the most to the right AS_NUMBER in the AS_SEQUENCE, move as many AS_NUMBERs into the AS_SET until the path length is equal to shortest. 7.b.-Append at the beginning of the AS_SEQUENCE the most to the left AS_NUMBER as many times as needed until the path length is equal to shortest. 8.- If the assembled path is advertised through an iASSM sessions run 8.a, run 8.b otherwise, 8.a.-Return the resulting AS_PATH

8.b.-Append the local AS_NUMBER at the beginning of the path and return the resulting AS_PATH

The algorithm is displayed in Table 4.2. The AS_SEQUENCE is constructed with the AS_NUMBERs common to all the paths in the multipath set as suggested in RFC4271. The order between AS_NUMBERs is kept. If two AS_NUMBERs X and Y appear always one af- ter the other in every path (even though some AS_NUMBERs may appear in the middle), they are said to be in order (rule 4.a). If two ASes, common to all paths, are not in order, then the second in appearance within the shortest path is put in the AS_SET (rule 4.b). The assembled AS_PATH is the result of concatenating the AS_SET at the end of the AS_SEQUENCE (rule 5). The remaining AS_NUMBERs not common to every path are added to the AS_SET (rule 6). Afterwards, rules 7.a-b check that the AS_PATH_LENGTH of the resulting AS_PATH is exactly the same as the shortest path in the multipath set. Rules 8.a-b deal with the fact that the local AS_NUMBER is not added to the advertisements until it is propagated to a peering router outside the AS. 26 Chapter 4 Path ASSEMBLER

4.3 Example: An ASSEMBLER-Capable Autonomous Sys- tem

This example refers always to the AS depicted in Fig.3.1. The figure represents a transit AS (AS1) with three customers (AS2,AS3,AS4), one peer (AS5) and two providers (AS6 and AS7) connected to AS1 by means of ASSEMBLER-capable routers. Router (R5,R7,R9) set (ELMP=0,ECMP=0,KBEST=1) and (R2,R4,R6,R8) aggregate the maximum number. Routers establish a full-mesh of intra-AS peering sessions to redistribute routing informa- tion. The example present two cases, a prefix propagated downstream from providers to customers and another prefix propagated upstream from the customers.

4.3.1 Downstream Advertisement

In the first case, two paths are advertised to AS6 and AS7 towards 150.0/16. The paths are propagated further and four paths reach AS1 and all of them are assigned with the same local preference. The egress routers for the paths in AS1 are R6 and R7. The router R6 can aggregate up to three paths depending on the ELMP parameter. If ELMP=0, then only the path from AS7 is selected according to rule 3 in Table 4.1. Otherwise, if ELMP=1 or higher the three paths can be aggregated as depicted in the figure. The aggregated path from R6 is constructed using the algorithm in Table 4.2. The first AS_NUMBER of the shortest path, AS7 leads the AS_SEQUENCE. Only AS9 is common to all paths and it is aggregated to the AS_SEQUENCE, as well. The remaining ASes are added to the AS_SET, AS6 and AS10. The length of the aggregated path is checked and it is 3 where it should be 2, therefore AS9 is moved into the AS_SET to compensate the length. Finally, the update is re-advertised through iASSM. Router R7 is unipath and selects only the path through AS7. Routers R2 and R8 receives both announcements from R6 and R7. The announcements have equal AS path length and equal IGP cost, therefore they are aggregated by the routers, although in this case the resulting aggregated paths are the same as for R6. Routers R4 and R9 are advertised as well. Both paths from R6 and R7 have the same AS_PATH_LENGTH therefore the ranking is done according to the IGP cost. Routers R4 and R9 has its ECMP parameter equal to 0 and consider only paths with lowest internal cost. Both, R4 and R9 select in this case the path through R7. Routers R2,R4 and R9 propagate the paths towards AS1 clients adding the AS_NUMBER 1 to the advertised AS_PATH.

4.3.2 Upstream Advertisement

In this second case, a couple of customers of AS1, AS3 and AS4 advertise a path towards their customer AS30. The effect of the MED values on the ranking function is shown in this second case. AS3 advertises two paths towards AS30, one to R2 with MED=20 and another one to R4 with MED=10. AS4 does not use MED values. Every path is assigned with the same local preference. Hence, three routers end up with a path from eBGP sessions and redistribute them through iASSM. The router R4 discards the internal path through R2-AS3 with highest MED and selects the eASSM path from AS3 and from AS4. Nevertheless, the two paths cannot be aggregated since the path from AS3 has MED value, therefore only the path through AS3, which has a lower BGP_ID, is ranked first and selected. R2 discards its own eASSM path an selects the internal path through R4 with the lowest MED for AS3. Assuming there is no restriction for R2 on the IGP cost to R9, it can aggregate the internal path through R9 as well, since the aggregated path is advertised only through eASSM sessions and the MED values are not taken into account. Chapter 5

Deployment Considerations

When it comes to the deployment of ASSEMBLER in a real AS there are some consider- ations to be taken into account beforehand. This chapter outlines them and points out how the main shortcomings that may appear during the deployment can be solved using the ap- propriated settings. As mentioned during the introduction and shown in the previous chapter, ASSEMBLER does not required of any kind of coordination between ASes to take advantage of multiple paths with high flexibility. Therefore, the deployment issues may rise while de- ploying it inside an AS. The issues are collected in three categories, problems related to mixed configurations, inconsistent multipath routing policies and traffic engineering techniques.

5.1 Deployments with Legacy Routers

The incremental deployment when legacy routers are present depends on the intra-AS tech- nique used to forward the traffic. Two different types of intra-AS techniques to forward the traffic are considered: internal redistribution and tunnelling. Internal redistribution implies that every router inside the AS understands reachability information and fills in the routing tables accordingly. An internal protocol such as iBGP/iASSM is required to perform the re- distribution. Tunnelling techniques rely on the encapsulation of packets. Only the ingress and the egress routers need to be aware of the paths advertised to the AS. In this discussion we consider IP over IP tunnelling and MPLS tunnelling [29] as representative techniques.

If the AS performs a full deployment, such that every legacy router is replaced inside the AS, both internal redistribution and tunnelling can be used without any kind of limitation. The only potential advantage of tunnelling is the addition of eiBGP configurations [6] but that topic is out of the scope of this work. On the other hand, if the AS is not planning a full upgrade of the network, ASSEMBLER can still be deployed progressively. The difference in this case is that any router in the network can be randomly replaced if tunnelling is used, whereas special attention must be paid for internal redistribution. Using redistribution and legacy routers, ASSEMBLER routers must not aggregate paths received from internal peering sessions. The reason is that that internal aggregation may lead to routing inconsistencies. For instance in Fig.3.1, assuming that R2, R6 and R7 are unipath routers, if R2 receives the paths from R6 and R7 through the iASSM sessions and chooses the one from R6. R8 is multipath and aggregates the paths through R6 and R7, however it does not advertise the aggregate through iASSM to R2 to avoid internal loops RFC4271. If the IGP path between R2 and R6 28 Chapter 5 Deployment Considerations

passes through R8. The router R2 announces to the AS2 and AS3 its choice through AS6, however when packets get to R8, the multipath router can forward some of them towards R3 which is inconsistent with the network view that R2 is advertising.

5.2 Multipath Routing Policies

In addition to the considerations regarding the deployment of ASSEMBLER along with legacy routers, some other issues may arise related to the policy configurations of the AS- SEMBLER routers. Defining simultaneously import and export policies for several paths that match a given criteria is supported nowadays in regular BGP routers. For instance, Cisco IOS uses the route-maps to define policies for several paths at once. ASSEMBLER-capable routers should support the same definition of policies. A BGP router can be transparently replaced and provide the same functionality configuring the same route-maps and setting K-BESTRO with the combination (ELMP=0,ECMP=0,KBEST=1).

However, the combination of policies for multiple paths with more lax K-BESTRO con- figurations may lead to inconsistent states regarding the export of paths. In contrast to BGP, the ASSEMBLER decision process yields a set of KBEST paths instead of a winner path. If several paths are assigned with the same LOCAL_PREF, and the import policy is not de- signed appropriately, a path coming from a provider may end up in the selected multipath set with a path coming from a customer, which cannot be exported together. No ASSEM- BLER advertisement is generated towards the providers, although BGP would advertise the path from the customer. Therefore, LOCAL_PREF assigned by the import policy of a router should be the same only for paths coming from the same type of neighbor AS.

5.3 Enhanced Traffic Engineering

Once several paths are selected and advertised by an ASSEMBLER router they can be used simultaneously to forward packets. Outgoing traffic engineering (hereafter TE) is usually based on the local preference defined in the import policy. Nevertheless, in BGP only one path at a time is selected and the most flexible outgoing TE techniques are load-sharing [8] and tuning of IGP costs. Multipath BGP defined in [6, 18] allows for load-balancing among multiple parallel connections between two ASes but they only support load sharing to split the traffic among multiple ASes. ASSEMBLER allows to perform load-balancing across multiple ASes and in contrast to the proposals in [6, 18], the traffic can be switched from one egress AS to another without re-advertising additional routing information. In addition, how much traffic balance over each path becomes a new TE parameter.

Regarding TE using IGP costs, currently ASes send the traffic to the closest egress point in the network, which can be used as a form of performing TE. ASSEMBLER extends this TE technique and administrators can modify the ECMP parameter of K-BESTRO to define how close to this hot-potato routing they want to stick to.

On the other hand, the most widespread incoming TE techniques acccording to [4, 26] are prefix deaggregation, path prepending and TE with BGP Communities. Prefix deaggregation and BGP Communities for TE are supported. Yet, in order to respect the TE performed by neighbor ASes using path prepending, a maximum value must be setup for the ELMP parameter. That maximum value does not have to be public since in practice downstream 29 5.3. Enhanced Traffic Engineering

ASes tune the amount of AS numbers to prepend on a trial and error basis [26]. Chapter 6

17 17 Stability Analysis 3.4. Stability3.4. Analysis Stability Analysis 17 17 3.4. Stability3.4. Analysis Stability Analysis some unipathsome configuration unipath configuration guidelines guidelines must be slightly must be modified slightly becausemodified they because no longer they no can longer can be directlybe used directly in multipath used in multipath scenarios. scenarios. 17 17 3.4. Stability3.4. Analysis Stability Analysis someThe unipath stabilitysomeThe configuration unipath proof stabilityis configuration presented proof guidelines is presented after guidelines must the be guidelines after slightly must the be guidelines modified and slightly it isstructuredmodified because and it is they becausestructured as follows:no longer they as after nofollows: can longer after can thebe directly analyticalbethe used directly analytical model in multipath used of modelthe in protocol multipath scenarios. of the is protocolscenarios. presented, is presented, the synchronous the synchronousconvergenceconvergence of the proto- of the proto- 17 17 col for anycol of for its any flavours, of its in flavours, absence in of absence conflicting of conflicting routing policies routing between policies ASes between is3.4. shown. ASes Stability is3.4. shown. Analysis Stability Analysis Even though several equally preferred paths are available,some unipathsome BGP configuration unipath routers configuration guidelines selects guidelines must be just slightly must one be modified slightly path.17 modifiedbecause17 they because no longer they no can longer can If convergenceThe stabilityIf convergenceThe is proofstability not achieved, is is presented proof not achieved, then is presented after it isthe3.4. shownthen Stabilityguidelinesafter it isthat3.4. the shown Analysis Stability the guidelines and conflicting that Analysis it is the structured and conflicting relation it is structured as among relation follows: policies as among afterfollows: policies after According to the results in [5], the fact that BGPactuallythebe converges directly analytical exists.theactuallybe used directly analytical model Afterwards, in exists. tomultipath usedof model thea Afterwards, instable protocol it multipath scenarios.is of proven the is protocol itsolution presented,scenarios. is that proven if is nodes presented, thatthe followdoessynchronous if nodes the thenot followsynchronous guidelinesconvergence the guidelines presentedconvergence of the presented in proto- Sec- of the in17 proto- Sec- 17 3.4. Stability3.4. Analysis Stability Analysis tioncolsome for XXx, unipath anycoltionsome it of foris XXx,configurationitsunipath not any flavours, possible it of is itsconfiguration not inflavours, guidelines toabsence possible have in conflicting guidelinesof toabsence must conflicting have be conflicting of slightly policiesmust conflicting routing be modified slightlyand policies policies17 routing themodifiedbecause protocol and17 between policies the they should because protocol ASes between no longerbe is they should shown.able ASes no can to longer be is shown. able can to The stabilityThe proof stability is presented proof is presented after3.4. the Stabilityguidelines after3.4. the Analysis Stability guidelines and Analysis it is structured and it is structured as follows: as after follows: after guarantee that the network is stable ifsome routers unipathsome configuration unipath select configuration guidelinesfindIfbe several convergence directly a guidelines muststableIffindbe be used convergence directlysolution.slightly must a paths isstable in benot multipathmodified slightly used achieved,In solution.instead. orderis in becausemodified not multipath scenarios. to achieved, In then theycomplete because order itno scenarios.The is longer to they then shown completethe no can itrelaxation stability longer is that shown thecan the proof, stability conflicting that another the of proof, conflicting relation demonstration, another among relation demonstration, policies which among policies17 which 17 be directlybe used directly in multipath used in multipath scenarios.the analytical scenarios.the analytical model of model the protocol of the is protocol presented, is presented, the synchronous the synchronousconvergenceconvergence3.4. of Stability the3.4. proto-Analysis Stability of the Analysis proto- reliesactually on exists. theactuallyreliessomeGeneral onAfterwards, unipath exists. thesome AsynchronousGeneral configuration unipath Afterwards, it isconfiguration Asynchronous proven guidelines Convergence it is that guidelines provenmust if beConvergence nodes slightly mustthat Theorem follow be if modified slightlynodesXXx, Theoremthe17 becausemodified follow guidelines is17 theypresented becauseXXx,the no guidelines longerpresentedis they presented such no can longer presentedthat in canSec- suchthe that in Sec- the the selection process may trigger oscillations that didcol for not anycol happen of for its any flavours, of beforeits in flavours, absence in in of absence the conflicting3.4. Stability unipath of3.4. conflicting Analysisrouting Stability Analysis case.policies routing between policies ASes between is shown. ASes is shown. someThe unipath stabilitysomeThe configuration unipath proof stabilityis configuration presented proof guidelinesconvergencetion is presented afterThe XXx,guidelines must the stabilitytionconvergence beguidelines itafterbe slightlyof isThe must XXx, directly the not proof bestability guidelines modifiedrealbe and possible slightly it used directly ofis itis (asynchronous) isin notthe presentedstructuredmodifiedmultipathproof because and used toreal possible it have inis is (asynchronous) they multipathbecausestructured scenarios. aspresented after conflictingfollows: no toprotocol longer theyhave the scenarios. as after nofollows:guidelines canafter conflicting longer policiesis protocol guaranteed,the after can guidelines and and policies is guaranteed,it the is as structured protocoland well. and it the is asstructuredshould protocol well. as follows: be should asable after follows: to be able after to If convergence is not achieved, then it is shown that the conflicting relation among policies 17 17 thebe directly analyticalbethe used directly analytical model in multipath used of modelthe in protocol multipath scenarios. offind the is aprotocolscenarios. presented, stablefindIf is convergence solution. presented, a the stablesynchronous thesolution.Insynchronous order is notconvergence to Inachieved, complete orderconvergence of tothe then complete proto-the of itstability the is proto- shown the proof, stability that17 another the proof, conflicting17 demonstration, another3.4. relation demonstration, Stability3.4. which amongAnalysis Stability Analysis policies which Those results are the main motivation of this section,the analytical sincethesome analytical model ASSEMBLER unipathsome of configuration model theunipath protocol configuration of the guidelines is protocol presented, guidelines selects must is be presented, slightly must the synchronous bemultiple modified slightly the modifiedbecausesynchronousconvergence they because no longer theyconvergence ofno can longer the proto- can of the proto- col for anycol of for its any flavours, of its in flavours, absenceactually in of absence conflicting exists.actually of conflictingThe Afterwards,routing exists. stability policiesThe routing proofAfterwards, stability it between policiesis is presented proofproven ASes between is it presented afteris that is3.4. provenshown. ASes the if Stability guidelinesnodes after is3.4. thatshown. the Analysis Stability follow guidelinesif and nodes it Analysis is the structured and follow guidelines it is structured the as follows: guidelines presented as after follows: presented in after Sec- in Sec- reliescol for on anyrelies thecol ofbeGeneral for itsdirectly on any flavours, thebe used of directly AsynchronousGeneral its inin multipathflavours, used absence Asynchronous in multipath scenarios. in Convergence of absence conflicting scenarios. Convergence of conflicting Theorem routing policies TheoremXXx, routing is between presentedpoliciesXXx, is ASes between presented such is that shown. ASes such the is that shown.17 the 17 Ifsome convergenceThe unipath stabilityIfsome convergenceThe configuration isunipath proofstability not achieved, is configuration is presented proof not guidelines achieved, then is presented after it guidelines mustisthe shownthen be guidelinesafterthe it slightly mustisthat analytical the shown the beguidelines modifiedtheand conflictingslightly that analytical modelit is the structured modifiedbecauseand conflicting of relation modeltheit is protocol they structured because as of among relation follows: no thelonger is protocol they policiespresented, as among afterfollows: no can longeris presented, policies the after cansynchronous the synchronousconvergenceconvergence of the proto- of the proto- paths and additional information is advertised betweenconvergencetion XXx,convergencetion routers. it of is XXx, the not real possible it of is (asynchronous) the notThe real to possible have (asynchronous) goal conflicting to protocol have of conflicting our ispolicies protocol guaranteed, stability and is policies guaranteed, the as protocolwell. and the as should protocolwell.3.4. Stability be3.4. should able Analysis Stability to be Analysis able to actuallythebe directly analytical exists.theactuallybe used directly analytical model Afterwards, in exists. multipath usedof model the Afterwards, in protocol it multipath scenarios.is ofIf proven the convergence3.4.1 is protocol it presented,scenarios. is that provenIf Onif is convergencecolnodes3.4.1 presented, thatthe forDispute is followsynchronous any ifnotcol nodesOn of theachieved, for theitsDispute followsynchronous anyis flavours,Wheels guidelines notconvergence of itsthe achieved, then in flavours, guidelines presentedabsence Wheelsconvergence in it of Unipath is in the of thenshownabsence presented in conflicting proto- Sec- in of it the ofUnipath isthat inand conflictingproto- shown routing Sec- the Multipath conflicting policies that routing and the between policiesMultipath conflicting relation Scenarios ASes between is among shown. relation ASes Scenarios is policies shown. among policies find a stablefindIfsome solution. convergence aThe stable unipath stabilityIfsome convergenceThe solution.In configuration isunipath orderproofstability not achieved, is configuration to is presentedIn proof notcomplete guidelines order achieved, then is presented after toit guidelines mustis complete thethe shownthen be guidelinesafter stability it slightly mustisthat the shown the beguidelines modifiedandproof,conflicting slightly stability that it is the structured modifiedbecauseandanother conflicting relation proof, it is they structured because as demonstration, amonganother relation follows: no longer they policies as among afterfollows: nodemonstration, can longer policies which after can which tioncolsome for XXx, unipath anycoltionsome it of foris XXx,configurationitsunipath not any flavours, possible it of is itsconfiguration not inflavours, guidelines toabsence possibleactually have in conflicting guidelinesof toabsence must conflicting have exists.actually be conflictingbe of slightly policiesmust conflictingdirectly Afterwards, routing beexists. modifiedbe slightlyandused policies directly policies routing the in Afterwards,modifiedbecause protocolmultipath and usedit between policies is the in proven they shouldmultipath because protocol scenarios. ASes between no itislonger be that is they should scenarios.proven shown.able ASes if no can to nodes longer be isthat shown. able can follow toif nodes the follow guidelines the guidelines presented presented in Sec- in Sec- analysis is to show that the deploymentThe stability ofThe ASSEMBLER proof stability is presented proofrelies is presented after on the therelies guidelines afteractuallytheGeneral in analytical the on a guidelinesexists. thetheactually and network analyticalAsynchronousGeneral modelit Afterwards, is exists. structured and of model the it Afterwards,Asynchronous is protocol it structured asis of does provenConvergencefollows: the is protocol it presented, asis that provenafter follows:not ifConvergence is nodes presented, thatthe Theorem affectafter followsynchronous if nodes the the TheoremfollowXXx,synchronous guidelines theconvergence the is guidelines presented presentedXXx,convergence of the is presented in proto-presented suchSec- of the thatin proto- Sec- such the that the findIfbe convergence directly a stableIffindbe used convergence directlysolution. a isstable in not multipath used achieved,In solution. orderis in not multipath scenarios. to achieved, In then complete orderThe it scenarios. is to thengoal shown completethe it stabilityThe of is that shown this the the goal proof, stability sectionconflicting that of another the this proof, conflicting is relation section to demonstration, another introduce among relation is demonstration, to policies introduce whicha among discussion policies which a discussion about the about impact the of impact multipath of multipath solu- solu- the analyticalthe analytical model of model the protocol oftion the is protocolXXx, presented,tion it istioncolsome is presented, XXx, the for XXx, not unipathsynchronous anycoltionsome possible it of theforis is XXx,configurationitsunipath not anysynchronous notflavours, possible itconvergence of to possible is itsconfiguration nothave inflavours, guidelines toabsence possibleconvergence have conflicting toof in theconflicting guidelinesofhave toabsence must conflicting proto- have of be conflicting conflictingthe of policiesslightly policiesmust conflicting proto- routing be modified slightlyand policies and policies routing the themodifiedbecause protocol and between policies protocoland the they should because protocol the ASes between no should protocollonger be is they should shown.able ASes no can to be longer be is should shown. able able can to to be able to reliesactually on exists. theactuallyreliesGeneral onAfterwards, exists. the AsynchronousGeneral Afterwards, it isconvergence Asynchronous proven Convergence it is that provenconvergence ifConvergence nodes of thatThe Theorem the follow if stability nodesrealTheofXXx,Theorem the (asynchronous) follow theproof guidelinesstability is real presented isXXx,the presented proof guidelines(asynchronous) presented is presented issuch presented protocolafter presentedthat in the Sec- suchthe guidelines after is protocolthat in guaranteed,the Sec- the guidelines and is it isguaranteed, structured and as well.it is structured as follows:as well. as after follows: after stability. col for anycol of for its any flavours, of its in flavours, absencetionsfind3.4.1 in a ofin absence stable conflicting thetionsfind On stabilityfindIfbe ofsolution.3.4.1 convergencea directly conflictingin Dispute a stable routing stable theIffindbe On of used stabilityconvergence directlysolution.solution.In policies policy-baseda routing isstable Dispute inorder Wheels not multipath used achieved,betweenIn solution. of policies to orderis In in policy-based not complete multipath orderscenarios. Wheels into ASesachieved,routing Inbetween then complete order Unipathto itis scenarios. is shown. completetothe ASes thenprotocols. shown incomplete the routing stability it stabilityUnipathis is that shown.and shown the the protocols. Beforeproof, stabilityMultipathproof, conflicting stability that and another the discussingproof, another conflicting Before relationMultipath proof, demonstration, another Scenarios demonstration, among discussinganother relation the demonstration, variety policies which amongScenarios demonstration, the policiesof whichwhich variety solu- of which solu- convergencetionThe XXx, stabilitytionconvergence it of isThe XXx,the not proof stability real possible it ofisis (asynchronous) notthe presented proof toreal possible have is (asynchronous) presented after conflicting to protocol have the guidelines after conflictingthe policiesis protocol analytical guaranteed,the guidelinesthe and and policies is analyticalguaranteed,it model the is as structured protocoland well. and of modelit the the is as protocolstructuredshould protocol well. as of follows: the be is protocol should presented, asable after follows: to be is presented, able the aftersynchronous to the synchronousconvergenceconvergence of the proto- of the proto- If convergenceIf convergence is not achieved, is not achieved, then it is thenshownreliesactually it isthat shown on the exists. theactuallyrelies conflicting thatGeneral onAfterwards, the exists. the conflicting Asynchronous relationGeneral Afterwards, it is amongAsynchronous relationproven Convergence it policies is that among proven ifConvergence nodes policies that Theorem follow if nodesXXx, Theoremthe follow guidelines is presentedXXx, the guidelines presented is presented such presentedthat in Sec- suchthe that in Sec- the findthe analytical a stablefindthe solution. analytical a model stable of solution.In model the order protocol of totionsrelies In thecomplete order is protocol and presented, on to completetions instabilitiestherelies iscol stabilityGeneral presented, the and for on thesynchronous any theinstabilities proof,col stability of in theAsynchronous forGeneral its unipath anothersynchronous any flavours, proof,convergence of itsdemonstration, in Asynchronousanother and in flavours, unipath absenceconvergence multipath Convergence ofdemonstration, in the of and absence whichproto-conflicting of multipath Convergenceand the of whichconflictingtheTheorem proto- routing relationships and policies routing TheoremXXx,the relationships between policies is between presentedXXx, ASes between is them, between shown.presented ASes such some is that shown. them, no-such the some that no- the actually exists.actually Afterwards, exists. Afterwards, it is proven it is that proven ifconvergencetion nodes thatThe XXx, follow if stabilitytionconvergence nodes it of isThe XXx, thethe not follow proof stabilityguidelines real possible it ofisis (asynchronous) the notthe presented proof guidelines toreal possiblepresented have is (asynchronous) presented after conflicting to presented protocolin have the Sec- guidelines after conflicting policiesis protocol in guaranteed,the Sec- guidelines and and policies is guaranteed,it the is as structured protocoland well. and it the is asstructuredshould protocol well. as follows: be should asable after follows: to be able after to reliescol for on anyrelies thecol ofGeneral for its on any flavours, the of AsynchronousGeneral its in flavours, absencetationconvergence AsynchronousThe in Convergence of absence is conflicting goalintroducedtationconvergence ConvergenceIf ofThe of convergence conflicting Theorem isthe thisrouting goal introducedrealIf hereby. section convergence policiesof TheoremXXx, routing(asynchronous) is thethis not Theis is realbetween achieved,hereby. presentedsectionpoliciesXXx, to is notation(asynchronous)introduce not is ASes betweenachieved, thenpresented The issuch protocoltois itcomes notation that isshown.introduce aASes thenshown suchdiscussion the from itis is protocolthat isthat shown. comes guaranteed, shown athe the discussion conflictingabout frameworkthat from isguaranteed, the the theas conflicting relation aboutwell.impact framework defined among the relation as of well. impact inmultipath policies defined among XXxChikin. of policies inmultipath solu- XXxChikin. solu- tion XXx,tion it is XXx, not possible it is not to possible have conflicting to havefind conflictingthe policies analytical a stablefindthe and policies solution. analytical a model thestable protocol and of solution.In model theorder the protocol should protocol of to In thecomplete order beis protocol should presented, able to complete the to is be stability presented, able the thesynchronous to proof, stability the anothersynchronous proof,convergence demonstration, anotherconvergence ofdemonstration, the whichproto- of the which proto- convergenceIf convergence3.4.1convergenceIf On convergence of3.4.1 Dispute isthe not real On achieved, of (asynchronous) Dispute theis Wheels not realtions achieved, then (asynchronous) Wheels in3.4.1 it in Unipath is protocol the thenshowntions On in stability itactually3.4.1 Unipath isthat protocol inand Disputeguaranteed, shown the the exists. MultipathactuallyOn of conflicting stabilityis that and guaranteed,policy-based Afterwards,Dispute theasWheels exists. Multipath well. conflicting relationof Scenarios Afterwards, policy-based as it well.Wheels isin among routing relation provenScenarios Unipath it policies is that among protocols. provenin routing if nodes Unipath policies andthat follow protocols. if Before nodes Multipath the and follow guidelines discussing Before Multipath the guidelinesScenarios presented discussing the variety presented in Scenarios Sec- the of in variety solu-Sec- of solu- The section starts with a discussionfind a to stablefind motivate solution. a stable solution.In order the to In complete order stability to complete thereliescol stability for on the anyrelies theproof,colanalysis. stability ofGeneral for its on another any flavours, the proof, of AsynchronousGeneral its demonstration, another in flavours, absence Afterwards, Asynchronous demonstration, in Convergence of absence conflictingwhich Convergence of conflicting whichTheorem routing the policies TheoremXXx, routing sta- is between presentedpoliciesXXx, is ASes between presented such is that shown. ASes such the is that shown. the actually exists.actually Afterwards, exists. Afterwards, it is proven it is that proven if nodestion that XXx, follow iftion nodes it is the XXx, not follow guidelines possible it is the not guidelines topresented possible have conflicting to presented in have Sec- conflicting policies in Sec- and policies the protocol and the should protocol be should able to be able to relies on thereliesGeneral on the AsynchronousGeneraltions AsynchronousIn Convergence and policy-basedtions instabilitiesconvergence ConvergenceIfIn convergence and3.4.1 Theorem policy-based instabilitiesconvergenceIf scenarios On convergence in of3.4.1 TheoremXXx, Dispute unipathisthe not real On is achieved, scenariosof wherepresented (asynchronous)XXx, inDispute theis Wheels and not unipath real is achieved,multipatha thenpresented (asynchronous)such path Wheelswhere in it and Unipaththat is protocol vector thenshown such multipaththe a and in path it Unipaththatis protocol protocol andthe guaranteed, shown vector the relationships Multipath and conflicting is that and guaranteed, is protocol the theas running, Multipath well. conflicting relationships relation Scenarios betweenas is well. paths running, among relation Scenarios propagatethem, betweenpolicies among paths some policies them, propagate from no- some from no- tionThe XXx, goaltion itTheof is XXx, notthis goal possiblesection it ofis notthis is to possible section to have introduce conflicting is to to have introduce a discussion conflictingfind policies a astable discussionfind about and policies solution. a the stable the protocol aboutimpact and solution.In theorder the of should protocol impact multipath to In complete order be of should able tomultipath solu- complete the to be stability able solu- the to proof, stability another proof, demonstration, another demonstration, which which convergence of the real (asynchronous)The protocol goalactuallyThe of is guaranteed, this goal exists.actually section of Afterwards, asthis exists. well. is section to Afterwards, introduce it is proven is to it isintroduce that a proven discussion if nodes that followa if discussion nodes about the follow guidelines the theabout impact guidelines presented the of impact multipathpresented in Sec- of in multipath Sec- solu- solu- bility of ASSEMBLER is studied. Intions order3.4.1 in thetionsconvergence On stability3.4.1 to inDispute the prove On of stabilityof policy-based Dispute theWheels real ofonetation the policy-based (asynchronous) Wheels in routingnode stabilityUnipathis introducedtationone to protocols. in routingrelies another Unipath node protocol and is on protocols. introduced Before Multipathto therelies hereby. of as is anotherandGeneral guaranteed, they ondiscussingASSEMBLER, BeforeMultipath the The are hereby.AsynchronousGeneral Scenariosas discussing notation chosen as thethey well. variety Asynchronous The Scenarios are in theConvergence comes notation of chosen the variety solu- ranking Convergencefrom of in comesweTheorem solu- the the procedure rankingextend framework from TheoremXXx, the is procedure presentedXXx, as framework definedmost is presented such preferred as in that defined most XXxChikin. such the preferred and that in the XXxChikin.an- and an- find a stablefind solution. a stable solution.In order to In complete order3.4.1 to complete the Ontion stability3.4.1The Dispute XXx, the goal proof,tion Onstability itTheof is XXx, notthis anotherDispute Wheelsgoal proof, possiblesection it ofis demonstration, notthis another is to possible sectionWheels toin have introduce demonstration, Unipath conflicting is to to which have introducein a discussion conflicting Unipath policies and which a discussion about Multipath and policies theand the protocol aboutimpact and Multipath the the ofScenariosshould protocol impact multipath be of should able multipath Scenariossolu- to be able solu- to tions in thetions stabilityconvergence in theconvergence of stability of policy-based the real of of (asynchronous) the policy-based real routing (asynchronous) protocol protocols. routing is protocol guaranteed, protocols. Before is guaranteed, as discussing well. Before as well. discussing the variety the of variety solu- of solu- tionsrelies and ontions instabilitiesthereliesGeneral and on theinstabilities in AsynchronousGeneral unipathnounced in Asynchronous and unipath multipath Convergence and tonouncedthe multipathtions Convergenceandfind3.4.1routers theTheorem a in stable therelationships totionsfind and On stabilitythe solution.3.4.1withTheoremXXx, thea inDispute stable routers the relationships On ofpeeringis stability between solution.In policy-basedpresentedXXx, Dispute order Wheels with ofis them,sessions. to between In policy-basedpresented peering suchcomplete order Wheels in routing some thatUnipath tothem, completeno-suchsessions. theWhen protocols. in routing somestability Unipaththat and a theno-the protocols. path Before Multipathproof,When stability andP discussing another= Beforea Multipathproof, pathvi Scenarios,v demonstration, discussinganother thePi 1= variety, Scenarios . demonstration,v .i . the,v , of viwhich variety0 solu-1is, .preferred . of . which , solu- v0 is preferred the network model in [5] to include tationassembledThe is goalintroducedtationThe of is this goal introduced hereby. section ofpaths. this The is hereby. section totions notation introduce The is andAfterwards, to comes notation introducetionsa instabilities discussion from and comes a the discussion instabilities about framework from in unipath the the the aboutimpact framework defined in the andstabilityof unipath impact inmultipath multipath defined XXxChikin. of and inmultipath solu- XXxChikin. multipath and condition the solu- relationships and the relationshipsof between− them, between− some them, no- some no- convergenceconvergence of the real of (asynchronous) the realover (asynchronous)InThe a policy-based pathprotocol goaloverQtionsreliesInThe of is protocola= guaranteed,policy-based andthispath onv goalitions instabilitiestherelies,v scenarios section isQjGeneralof,v guaranteed, and= onj as this theinstabilitiesv well.1i is in,,v scenariosAsynchronousGeneral where.section unipathto .j .,v as ,introduce vj well.0 in Asynchronousa and1accordingis unipath, path where . multipathto .Convergence . ,introduce vectora v and0 discussion aaccording multipath pathConvergence andto protocol the theTheorem vector a relationships preferences discussion and about to TheoremXXx,the is protocol the relationships running, the ispreferences between presentedXXx, about impactof isv isipaths running,them,, between thepresented that such of someimpact of propagatemultipath thatrelationship them,v pathsi no-such the, that some of that propagate multipath relationshipsolu- from no- the of fromsolu- of tions3.4.1 in thetions On stability3.4.1 in Dispute theOn of stability policy-based Dispute Wheels of policy-based Wheels in routing Unipath protocols.in routing Unipath and protocols. Before Multipath and discussing Before− Multipath Scenarios discussing the− variety Scenarios the of variety solu- of solu- preferenceonetation node is introducedonepreferencetation tocantationconvergence another node beThe is is denoted introduced goalintroduced totationconvergence hereby.can as anotherThe of they isthe bethis goal introduced like,real hereby. denotedsection The are hereby.of as (asynchronous) thethis notationchosen they The is real hereby. section to like, notation The (asynchronous)introduce are in The is comes chosennotation protocolto the comes notation introduce a rankingdiscussion from from isin protocol comes comes guaranteed, the a the the discussionprocedure about frameworkranking from is framework from guaranteed, the theas aboutwell. theimpact framework procedure defined as framework the as definedmostof well. impact inmultipath defined XXxChikin. preferred as of inmostdefined inmultipath solu- XXxChikin. XXxChikin. preferred and solu-in an-XXxChikin. and an- ASSEMBLER is derived. Once thattions conditionIn and policy-basedtions instabilitiesIn and policy-based instabilities scenarios in unipathis scenariosstated, where intions and unipath multipatha inpath wherethe andtions vector the multipath astabilitytions and path protocolin3.4.1 the in guidelinesvector the therelationshipstions and Onof stability is3.4.1 protocol the policy-based in running,Dispute the relationshipsOn of stability between is policy-basedof pathsDispute running, Wheels policy-based offor propagatethem, between routing policy-based paths Wheels in routing some routing Unipath them, propagate from protocols. no- routing protocols.in routing some Unipath and from no- protocols. protocols. Beforepolicies Before Multipath and discussing Before discussingMultipath Before Scenarios discussing the variety discussing Scenarios the the varietyof variety solu- the of of solu-variety solu- of solu- tationThe is introducedgoaltationThe of is this introducedgoal hereby. section of this The is hereby. section tonounced notation introduce The is to comes notation tonounced introduce a discussion the from comesrouters a the to discussion about framework from the with routers the about frameworkpeeringimpact defined with the of impact insessions. multipath defined peering XXxChikin. of in multipath solu-! XXxChikin.sessions. When solu- a path! WhenP a= pathvi,vPi 1=, .v .i .,v , vi0 1is, . preferred . . , v0 is preferred one3.4.1 nodeone to On another3.4.1 node Dispute to On as another they Dispute Wheels are astions chosen they Wheels in andare Unipath in chosen thetions instabilities intions ranking Unipath inIn and andthe policy-based procedure instabilitiestions Multipathinstabilities ranking inIn and and unipathpolicy-based procedure instabilitiesMultipath scenariosas in most Scenarios unipath in and unipath preferred scenarios aswhere in multipath and most Scenarios unipath multipatha pathpreferred andP where and! an-vector multipath and multipathQ a and path and protocol the theP vectoran- relationships! relationships and andQ is protocol the running, the relationships relationships between is paths running, between− propagatethem, between paths some them, between them, propagate− from no- some them, from(3.1) no- no- some(3.1) no- proposed in [10] are reformulated fortionsmultipath in thetions stability in the of stability policy-based scenarios. of policy-basedIn routing policy-based protocols. routingQIn=The The policy-basedv protocols. goalBefore,v scenariosQThe of,v condition= discussingthis goal Beforev section,,v scenariosof .where .thisdiscussing .,v ,the is v section to variety a introduce, path .where .is theis . of , tovectorvariety v solu- used introduce a a discussion path of protocol solu- avector to discussion about prove is protocol the running, about impact the is ofvimpact paths running,multipath of propagate multipathv solu- paths propagatesolu- from from nounced tonounced the routers to the with routers peeringovertation with sessions. peeringa is path introducedovertation sessions.Whenonetation a3.4.1 node is path a ispath introduced introducediWhentationoneto hereby. On anotherPj3.4.1 nodeDispute= aisj path introducedvi to hereby.On,v1 Thei asanother hereby.Pi they Dispute1j= Wheels, notation . Thev . are hereby.ij .,v as0, v notationi1chosen The0according they1is Wheels, in .preferred The comes .are notationUnipath . in, comes vnotation chosen0 theisaccording intofromranking preferred from Unipathin comestheand comes the the the preferencesprocedure Multipath ranking framework from frameworkto from and the the procedure Multipath as the framework preferences definedmost Scenarios offramework defined preferred asi in, defined mostScenariosthat XXxChikin. of inpreferred and relationship definedin XXxChikin.i XXxChikin.an-, that and in relationship an- XXxChikin. of of tionsIn and policy-basedtions instabilitiesIn and policy-based instabilities scenarios in unipath scenarios where inone and unipath amultipath node path where and vector to a multipathtions andanother path protocol the in vector therelationshipstions andas stability is protocol the in running,they the relationships− of stability betweenare is policy-based− paths running, chosen of− propagate them, between policy-based− paths routing in some propagatethe them, from no- protocols. ranking routing some from no- protocols. Before procedure discussing Before as discussing the most variety preferred the of variety solu- ofand solu- an- overThe a path goaloverQThe of a= thispathv goali,v sectionQj of,v=j thisv1i is,,v .sectionto .preferencej .,v ,introduce vj0 1according is, . to . . ,introducepreferenceone a v0can discussionnouncedaccording to node be the a denoted preferences discussion tonounced to aboutcan to the another the routers be the preferences tolike, denotedabout impactof the withv asi routers, the that theypeering of impactof like,multipath relationship withv arei, sessions. that peering of chosen multipath relationshipsolu- of sessions.When in solu- a thepath of When rankingP = a pathvi,vPi procedure1=, .v .i .,v , vi0 1is, .aspreferred . . , most v0 is preferred preferred and an- their stability. onetation node is introducedonetation to another node is introduced to hereby. as another they− The are hereby. as notationchosen they− The are in comes chosennotation thetions ranking from in comes and the the proceduretions instabilitiesranking framework from and the procedure instabilities as framework in definedmost unipath preferred as in in mostanddefined XXxChikin. unipath multipath preferred and in and an-XXxChikin. multipath and and the an- relationships and the relationships between− them, between− some them, no- some no- preferencetions in thepreferencetionscan stability bein the denoted ofcan stability policy-based be like, denoted ofnounced policy-basedAfterwards, like, routing tonounced protocols. routing theoverAfterwards,InThe routers aP policy-based path protocols. goaltoBeforeisoverQ the announcedInThe ofwith a= policy-based discussingthispath routersv goalP Beforei,v scenarios sectionpeeringQisj of,v=j announced thisdiscussing withv the1 andiis,,v scenarios where.sectionto . varietysessions.j .,vpeering, introduce it vj0 a theis1according is, path of where . to andchosen. variety solu-. ,introduce! vectorasessions. Whenv0 discussion a itaccording pathto of is protocolthe as solu-a vector chosena preferences pathmostdiscussion! When about to is protocol theP running, preferred the as preferences= a about impactofpath mostv isvii paths,v running,, the thatPi of preferred by impactof propagate1multipath= relationship,v . anpathsiv, .i that.,v of , arbitrary v propagatei multipath relationship0solu- from by1is of, . preferred an . . , fromsolu- num-arbitrary v0 ofis preferred num- In policy-basedtation is introducedtation scenarios is introduced hereby. where− The hereby. a notation path− TheP vector comes notationQ from protocol comesP the framework fromQ is the running, framework defined− paths in defined XXxChikin. propagate in− XXxChikin.(3.1) from (3.1) nouncedtions and tonouncedtions instabilities the and routers to instabilities the in with unipath routers peering inwith and unipath sessions. multipath peering andP ! sessions. When multipathpreferenceonetions andQIn nodethe policy-basedP in a path therelationships! Whenonepreference andtoQcan stability anotherP node the be a= path relationshipsdenotedv toi,v ofcan as betweenanotherP scenariosi policy-based they be1=, like, . denotedv . arei .,v as , them, between vi chosen0 they1is where, like, routing . preferredsome .are . ,in them, v(3.1) chosen0!the no- aprotocols.is path preferredranking some in(3.1) the no-vector! Before procedure ranking protocol discussing procedure as most the is preferred as running, variety most preferred andof solu- paths an- and propagate an- from In policy-basedIn policy-based scenarios scenarios whereberover of a a path where nodespathberover vector! aQ pathofv ai= protocol+ nodespathn vectorv!,vitions,vi+Qj is protocoln,vv in running,i=+j the1n,v .1,v stabilityi .,,v is− .i+ ,.pathsj running, v.n,v ,i+1 vj of10 propagate, policy-based1− .accordingin, . paths . the., .v ,i propagate+1 v from network.0 according routingin to the the from protocols. network. preferences The to assigned the Before preferences The of discussing pathassignedvi, that to the of relationshipv varietyi pathv+in, thatis toofP solu- relationshipv"i+= ofn is P " = of overtation a is path introducedovertationQ a= is pathv introducedi,v hereby.Qj,v=j v1 Thei,,v hereby. . .onej .,v notation , vj0 node1 Theaccording, . . comes . notation ,one vto0 nouncedtionsaccording another tofrom node comesthe and the preferences tonouncedtionsto instabilities frameworkto fromas the another the andthey routers− the preferences to− instabilities offramework the inare definedwithv as unipath routersi,chosen that they peering− of in− in relationshipwith anddefinedv XXxChikin. unipath arei, sessions. multipaththatin peering chosenthe in relationship andP XXxChikin. of! rankingsessions. When multipath andQ in theP a the ofpath relationships! When andprocedureQ rankingP the a= path relationshipsvi,v betweenP procedurei as1=, .v most .i .,v , them, between vi0 1is preferred, as. preferredsome . . , mostthem, v(3.1)0 no-is preferred some preferred and(3.1) no- an- and an- one nodeone to another node to as another they− are as chosen they− are in chosen the ranking inIn the policy-based procedure ranking procedure scenariosas most preferred aswhere most a preferredpath and an-vector! and protocol an-! is running,− paths propagate− from preferencepreferencecan be denotedcan be like, denotedvpreferencei+n like,,vi+nvpreferenceican+1overn, .,v .be a .i+ ,path denoted vnoveri+1canQ1In,,v a= . policy-basedpathbe .iv .,i like, ,,v . denoted v .Qji .,v+1 ,=j v,vv01i,,v=( scenariosi ., .jlike, .,v , .v vj .0i ,+1according v,n .where0,v . .=( ,i v+0 n avaccording topathi+1 then, .,v vector . preferences .i+ , to vn protocolthei+11, preferences) .P .of .and is ,vi v running,,i that+1 its of) relationship relationshipPvi paths,and that propagateits relationship of relationship of fromcom- of of com- nounced tonounced− thetation routers isto introducedtation− the with is routers introduced hereby. peering− with The hereby. sessions. notation peering− The comes notation sessions.When− from comes a the path When framework from−P the= a frameworkpathv definedi,vPi 1 in=, defined XXxChikin..v .i .,v , vi in0 XXxChikin.1is, . preferred . . , v0 is preferred nouncedAfterwards, tonounced theAfterwards, routersP tois the announced with routersP peeringis announced with andAfterwards, sessions. peering it is andchosen! sessions.WhenpreferenceoneAfterwards, it isnode asP a chosen pathmost! Whenpreferenceisone tocan announcedanotherP preferrednode as be= apath mostP denotedv toican,v asis anotherPi preferredthey bybe announced1=, like, denoted. anv and . arei .,v as , arbitrary vi chosen0 they byit1is like,, .is preferred an .are . and ,inchosen num-arbitrary v chosen0! theis it preferredranking is in num-as thechosen most! procedure ranking preferred as procedure as most most− preferred byas most an preferred arbitraryand by an- an and arbitrary num- an- num- In policy-basedIn policy-based scenarios scenarios whereposition a path wherewithPposition vector! aQ pathP protocolcanP vectorwith! beQ is protocolP then running,can expressed beis− paths running, then propagate expressed− like, pathsP propagate(3.1) from! Q like,(3.1) fromP ! Q − (3.1) (3.1) berover of a nodespathberoverQ ofv ai=+ nodespathnv,vi,vi+Qjn,vvi=+j1n,v .1,vi .,,v .i+over ,.j v.n,v ,i+1 vj10,1 .aaccordingin, . . path the., .v ,overi+1 v network.0Qnouncedaccordingin to a=Afterwards, the the pathv network. preferencesi The tonounced,v to theQjAfterwards, assigned the,v routers=Pj preferences Thetoisv1 ofi the, announced,vwith . pathassignedv .routersij,P .,v that , peeringis tovj of0 announced relationshipv with1accordingi pathv and+,in, . sessions. that. peeringis it . to ,P isv relationshipv"i0 andchosen+= of!naccording sessions.When tois itP theis as" a chosen= pathmostof! preferencesWhen toP preferred as= a thepath mostvi,v preferencesPi preferred by of1=, . anv .vi .,vi , arbitrary, vi0 that by1is, . preferred anof . relationship. , num-arbitrary v0i,is that preferred num- relationship of of one nodeone to another node to as another they− − are asber chosen they− − of are nodes in chosenber the ranking ofv iniIn+ nodes the policy-basedn,v procedure rankingi+Invni policy-based+1−n procedure, scenariosas,v . . mosti .+ ,n vi preferred+1 scenarios aswhere1−,most .in . . a , the preferredpathv and wherei+1P network. an-vector!in aQ path and the protocolP vectoran-! network. TheQ is protocol running, assigned The is− paths running, assigned path propagate− paths to v propagatei path(3.1) from+n is toP(3.1) fromv"i+=n is P " = vpreferencei+n,vi+nvpreferenceican+1n, .,v .be .i+ , denoted vni+1can1,,v . be.i ., like, , . denoted v .i .+1 , v,v0 =(i, like, . .v .i ,+ vn0,v=(i+berovernvi+ of1n, a .,v nodespath . .iber+over , vniQ+1 ofv1 ai,=)+− .P nodespathn .v .,vandi ,,v vi+Qij+1n its,vvi=+)j1 relationshipPn,v− .1,viand .,,v .i+ ,.j v.n,v its ,i+1 vj10 relationship,1 .accordingin, of. . the.,com- .v ,i+1 v network.0 accordingin toof the thecom- network. preferences The to assigned the preferences The of pathassignedvi, that to of relationshipvi pathv+in, thatis toP relationshipv"i+= ofn is P " = of nounced to the routers with peeringpreference sessions.preferencecan Whenone be node a pathdenotedone tocan anotherP node= bev like, to denoted,v as another they− ,− . . are . as , like, v chosen they−is− preferred are in chosen" the ranking in the" procedure ranking procedure as most preferred as most preferred and an- and an- nounced− to− the routersv with,v peeringv ! sessions.,−,v . . . , v When−,,v . a . path ., ,i . v .Pi . ,1= v,vvi=(,v, .i0 .v1 ., , . v . . ,,v v=(0 isv preferred,,v . . . , v ,) .P . .and , v its)P relationshipand its relationship of com- of com- positionAfterwards,withpositionAfterwards,P canPwithis be announcedP thencanP expressedis be announcedi then+ andn expressed like,iti is+Pn and choseni+v1preferenceni itQlike,+n is,v asiP chosen+i+ mostn!nvpreferenceii+1Qcan+1n, preferred .,v as .be .i+i ,most denoted vni+1can1i,+1,v− preferred . by be.i ., like, ,0 .denoted an v .i .+1 , arbitrary v,v0− by=(ii, like,+ . an .vPn .0i ,+ arbitrary num- vn(3.1)!0,vi=(+Pi+nni"v+i+11 num-nPn, .,v .! .ii+ ,+ vnPin+11i"+1,)1 .P . .and , vi+1 its) relationshipPi+1and its relationship of com- of com-(3.2) (3.2) Q = v ,v ,v , . . . , v −!nounced! tonounced−− the routers to− the withv routers peering with sessions. peering! sessions.When−− (3.1) a path When! −P−= a pathvi,vPi 1=, .v . .,v , v0 is, . preferred . . , v is preferred over a pathover a pathi Qj =j v1i,vj,vj0Afterwards,1according, . . . , v0 accordingAfterwards, to theP preferencesis to announced the preferencesP ofisi announced, that and of relationshipv iti, that isP and chosen relationship of!Q it is asP chosen of! mostQ preferred as mosti preferred byi an1 arbitrary by0 an arbitrarynum-(3.1) num-(3.1) ber of nodesber ofvi+ nodesn,vi+vni+1−n,,v . .i .+position ,n vi+11−, .in . . , the vwithpositioni+1 network.positioninPAfterwards, thecanwith network.with Theposition beAfterwards,P assignedP thencanPcanwith Theis be expressed announced beP then assigned pathcanP then expressedis to be announcedv thenexpressedi path and+ like,n expressedislike,it toP isv"Pi! and chosen+=n! like,is itQlike,P is asP" chosen= most! Q preferred as most− preferred by an arbitrary− by an arbitrary num-(3.1) num-(3.1) 6.0.1 On Dispute Wheels inpreference Unipathpreferencecan be denotedcan− be like, denoted and− like, Multipathover a pathoverQ a= pathvi,vQj,v Scenarios=j v1i,,v . .j .,v , vj0 1according, . . . , v0 according to the preferences to the preferences of vi, that of relationshipvi, that relationship of of vi+n,vi+vni+1n,,v . .i .+ ,n vi+11,,v . .i ., , . v .i .+1 , v,v0 i=(, . .v .i ,+ vn0,v=("i+bervnvi+ of1n,,v . nodes ."i,v .ber+ ,n vi+1 of1v,i)+ . nodesP .n .,vand ,, vi .+i+1 .vn its .i+) ,1P relationship−n v,,v .and .i .+ ,n v itsi+11− relationship, .in .of . , thecom- vi+1 network.in of thecom- network. The assigned The assigned path to vi path+n is tovPv"i+=n is PP" == ber of nodesPber!! P of" i+P nodesn! Pi"+nvi+1n,vi−+ni+11, .−in . . ,the vi(3.2)+1 network.in the(3.2) network. The assigned The assignedpath to i path+n is to v"i+n is P " = Afterwards,− P is− announcedP and it isP chosenpreferenceQ− asP most!preferenceQcan− preferred be− denotedcan bybe like, denoted an− arbitrary like, num-(3.1)" (3.1)" position withpositionAfterwards,P canwith beP thencan expressedis be announced then expressed like, and!vi like, it+n is,v choseni+!vni+1n,,v . as .i .+ , mostn vi+11,,v . preferred .i ., , . v .i .+1 , v,v0 byi=(, . an.v .i ,+ varbitraryPn0,v=(!i+Pvni"+1 num-nP,,v . .!i .+ ,Pn vi"+11,) .P . .and , vi+1 its)P relationshipand its relationship of com-(3.2) of com-(3.2) vi+n,vi+nvi+1n,,v . . .i+ , vni−+11,,v . .Pi ., ,− . v .i .+1 , v,v0 =(i, . .v .i ,+ vn0,v=("Pi+!nvQi−+1n,,v ." .! .i+ ,− vni+11,) .P . .and , vi+1 its) relationshipP and its(3.1) relationship of com- of com- ber of nodesber ofvi+ nodesn,vi+nvi+1n,,v . .i .+ , vni+11To, .in .indicate . ,the vi+1− network.positioninToAfterwards, thethat indicateP network.withposition TheP "Afterwards,P assignediscan that awith Theis propagated be announcedPP then assignedpath"canPis expressedis to be a announcedv propagatedtheni path and+ versionn expressedislike, itP toP isv!"i and chosen+=!nP of versionislike," it−PP is as.P" chosen In!= mostQ orderP of" preferred asP most. to In assign preferred order by an arbitrary toP by" assignto anv arbitrarynum-i+nP,"Pto num-(3.1)must(3.2)vi+n, P (3.2)must − − Afterwards,Afterwards,is− announcedP is announced and it is and chosen it is as chosen most− preferred as most preferred by an arbitrary by an arbitrarynum- num- position withposition" berP ofcan nodes"withber be ofvP theni+ nodesncan,v expressedi+ benvi+1n then,,v . .i .+ , vn expressedi like,+11, .in . . ,the vi+1 network. like,in the network. The assigned The assignedpath to vi path+n is toPv"i+=n is P " = vi+Ton,v indicatei+nvi+1nTo,,v .that . indicate .i+ , vnPi+11,is,v . that. ai ., , propagated . v .Pi .+1be , vis,v0 assigned=( ai, propagated. .v version .i ,+ vPn0be,v=(i to+ assignedP ofnv version"ivP+1iPn,.,v . In Therefore, . .i+ ,orderP of vn to"i+1P1,.v) to.Pi In .. . assignand , Therefore,order vPi+1 its" )is toP relationshipP assign feasibleandto v itsPP relationship", isP ofto asmust feasiblecom-v long, P of asmustcom- asP longis. as P is. Afterwards,− Afterwards,P is−" announcedP isber" announced and of it nodes isber and chosen! it ofv−i is+ as nodes chosenn! most,vi+− preferrednv asi+ most1n,,v . .i .preferred by−+ , vn an"i+11 arbitrary, .iin− by+ .n . ,the an" vi arbitrarynum-+1(3.2) network.i+nin the num-(3.2) network. The assigned The assignedpath to vi path+n is toPv"i+=n is P " = The goal of this section is to introduceposition with P acan discussion beP then expressed like, aboutvi+n,vi the+nvi+1n,,v . .impact .i+ , vnPi+11,,v . .i ., , . v .Pi .+1 ,of v,v0 =(i, .multipath .v .i ,+ vn0,v=("i+nviP+1n,,v ." . .i+ , vni+1P in1,) .P . .theand , vi+1 its)P relationshipP andv itsP relationshipP of com-v P of com- be assignedbeposition to assignedvi. Therefore,with to vican. Therefore,P be" is then feasible expressedP " is as feasible long like,ToAfterwards, as as indicateP longis.− ToAfterwards, as that− indicateP is.is−" announcedis that aP propagated−is" announcedis and a propagated it version isP and chosen! P of it version"− isP as. chosen In! most orderP of"− preferred as. to In most assign order preferred by to an" assignto arbitraryi by+n, an" to arbitrarynum-must(3.2)i+n, num-(3.2)must ber of nodesber ofvi+ nodesn,vi+nvi+1n,,v . .i .+v , vni+11,,v .in . . ,the vi+1 network.in, . the . . , network. v TheP assigned,v , The . . . assignedpath , v to=(vi path+vn is to,vPv"i+=n is P," .= . . , v )P and its relationship of com- − i+−n i+nvi+be1positionn,v assignedi+nwithbepositioni+1 to1 assigned,v .ican. i Therefore,.with , be v toiP+1 thenvican.,v0 Therefore, expressedPi be," is . then . feasible .i ,+ expressed vP like,n0" is=(" asi feasible+ longnv like,i+1 asn as,vP" longis.i+n asi+11P, .is. . . , vi+1)P and its relationship of com- −ber of nodesber− ofvi+ nodesn,vi+nvi+1n,,v . .i .+ , vni+11, .in .P . ,the vi+1 network.Pin−" theP network. TheP assigned−" The assignedpath to vi path+n is toPv"i+=n is P "(3.2)= (3.2) vi+n,vi+nvi+1n,,v . . .i+ , vni+11,,v . .i ., , . v .i .+1 , v,v0 =(iUsingTo, . .v .i indicate ,+ vn0,v this=("i+ToUsingnvi two+1 thatn indicate,,v . ." .isimple+ ,thisP vni"+11is,) two .thatP . a .conceptsand , propagated v simpleiP+1 its−") relationshipPisand a differentconcepts propagated its− version relationship of com-! relationsdifferent of version ofPcom-. In! betweenrelations orderof P . to In the between assign order policies toP the" assignto of policiesv thei+Pn, different" Pto ofmustvi the+n, differentP must stability of policy-based routing protocols.To indicate− SomeTo that indicateP− " is that notationa propagatedPposition" is a propagated versionwithPposition is! ofPintroducedP version−"PPcan. Inwith! orderofP be−"PP then. to Incan assign order expressed bebefore toP then" assignto v expressedi like,+Pn," Pto themust(3.2)vi+n like,, discussion.P must(3.2) positionUsingwithposition thisUsingP twocanwith simple this beP then twocan concepts expressed simple be then differentconcepts expressed like, relationsdifferentvi like,+n,vi+ betweenrelationsnvi+1n,,v . . .i+ ,thebetween vni+11 policies,,v . .i ., , .the v .i .+1of ,policies v,v0 the=(i, . different .v .i ,+of vn0,v the=("i+ differentnvi+1n,,v . ." .i+ , vni+11,) .P . .and , vi+1 its) relationshipP and its relationship of com- of com- be assignedbe toassignedvToi. indicate Therefore,− toTov that indicatei. Therefore,P− "Pis" that ais propagated feasibleP " isP a" propagatedis version as feasibleP long! ofP version−" asPP. asP In! long orderofPis.−"P . as to In assignP orderis. toP " assignto vi+Pn," Ptomust(3.2)vi+n, P must(3.2) nodesbe assigned andbenodes the toassignedv pathsi and. Therefore, the announcedto v pathsi. Therefore,P announcednodes" is by feasible themP and" is can by asnodes feasible the longthem beposition expressed.Using paths as andcan asP longwithpositionis. be this the announcedUsing expressed.P as two A pathscanPwith particularis. simple this beP then announcedtwo Acan conceptsby expressedparticularsimple type be them then of differentconcepts relationsexpressed like, cantype by them of be relationsdifferent relations like, expressed. can betweenrelations be expressed. the betweenA policiesparticular the of A policies the particular type different of of the relations different type of relations be assignedbe toassignedvi. Therefore, to vi. Therefore,P " is feasibleP " is as feasible long as asP longis. as P is. The notation comes from the frameworkbetween pathsdefinedbetween caused paths by caused inroutingbetween [5]. by policies routing arepathsbetween policies" thenodesnon-anti-reflexive caused are and" pathsnodes the thenon-anti-reflexive by paths causedand routing the announcedrelations. paths by policies routing announcedrelations. by For them aare formal policies can by" For the them be anon-anti-reflexive formal expressed. canare" be the expressed.non-anti-reflexive A particular Arelations. particular type of relations typerelations. For of a relations formal For a formal P ! P " P P " P (3.2)P " P(3.2) P " (3.2) (3.2) To indicateTo that indicateP " is that a propagatedP " is a propagated versionbetween of versionP . In! pathsbetween order of P caused. to In paths assign order by caused routing toP " assignto byv policiesi+ routingPn," Ptomust! arev policies" the, Pnon-anti-reflexivemust are"! the non-anti-reflexiverelations.relations. For a formal For a formal andUsing rigorousand thisUsing definition rigorous two simplethis definition of twoanti concepts simpleand ofnon-anti-reflexiveantiTo conceptsdifferent indicateand non-anti-reflexive differentrelationsTo that indicaterelations relationsbetweenP " is seethatrelations athebetween propagated XXx, policiesP " seeis herebythe a ofXXx, policies propagated the we version hereby different just ofP thein-i+! wen ofP different version just"PP. in- InP order of" P . to In assign order toP " assignto v(3.2)i+Pn," Pto(3.2)mustvi+n, P must andUsing rigorousand thisUsing definition rigorous twoTo indicate simplethisTo definition of twothat indicateanti conceptsP simple" isand that a of propagatedPnon-anti-reflexiveanti conceptsdifferent" is aand propagated versionnon-anti-reflexive differentrelations of versionP . Inrelations! relationsbetween order of P . to In assign seeorderrelations thebetween XXx, toPpolicies" assignto v see herebyithe+Pn, of"XXx,P policiestomust thev we hereby different, P just ofmust thein- we different just in- be assignedbe to assignedvi. Therefore, to v . Therefore,P " is feasibleP is as feasible longandUsing rigorous as asP longandis. thisUsing definition rigorous as twoP simpleis.this definition of twoanti concepts simpleand ofnon-anti-reflexiveanti conceptsdifferentand non-anti-reflexive differentrelationsrelations relationsbetween seerelations thebetween XXx, policies see herebythe ofXXx, policies the we hereby different just of thein-i+ wen different just in- troducenodes and anodestroduce fair the definition paths and a fair the announced definition based pathsi onannouncedbe by anbased assigned them example on" canby an themfor example beto the expressed.v can. shake Therefore, for be theexpressed. ofv A clarity. shake particularP of If" A clarity. thereis particulartype feasible is IfP ofan thererelations alternat- type as is long of an relations alternat- as P is. P nodes andnodesbe thebe assigned assigned paths andi be the announced to assignedv pathsi.i Therefore,. Therefore, to announcedvi. by Therefore,P " themis feasible" Pis canby" feasibleis as them feasible be long expressed. ascan asP long longis. be as expressed. asP Ais. particularis. A particular type of relations type of relations ! troduce! atroduce fairtroducenodes definition" and a anodes fairtroduce fair the" definition definition paths based and a fair the announced definition on based paths anbased onannounced example by anbased onthem example on an canby an for example themfor example be the the expressed. can shake forbe for theexpressed. of the of A clarity. shake particular clarity. shake of If A clarity. there ofparticular typeIf clarity. is there If ofan thererelations alternat- type is isIf an of an there relationsalternat- alternat- is an alternat- ingbetween sequenceTo indicate pathsbetweeningTo sequence of caused that preferenceindicate pathsP " by ofis caused that apreferencerouting (propagatedP) andbyis policies routinga composition propagated( version) and are policies composition the of version (non-anti-reflexiveP are). relations In the order of (non-anti-reflexiveP). to created relations In assign orderrelations. among toP created" assigntorelations.v a Fori set+ amongPn, a ofPto formal pathsmustv a For set, a ofP formal pathsmust Using this two simple concepts! " different! relationsbetween! betweenpaths! caused pathsthe!policies by caused routing! of by! policiesthe routing different"! are policiesi+n thenon-anti-reflexive"non-anti-reflexive" are the non-anti-reflexive"non-anti-reflexive" relations.relations. For a formal For a formal and rigorousandUsing definition rigorous this definition of twoanti simpleingbetweenand of non-anti-reflexive sequenceantiTo concepts indicateand pathsbetweeningnon-anti-reflexive differentingTo sequence of sequenceTocaused that preferenceindicate indicaterelations paths relationsingP "To sequence ofby ofis thatcaused preferenceindicatethatrelations see apreferencerouting between (Ppropagated XXx,"P ofis) that"andby apreference seeis (herebypropagated thepolicies!P routinga) XXx, composition" andpolicies propagated(is awe version) composition propagated(hereby! and justare version) policiesof and in-compositionthe the we compositionof ofdifferent version ( just version (P!P are). in- relations relationsIn Inthe order of order of (!P ().P to created relations In). assign tocreated orderrelations In assign order among toP createdrelations." assignto among toPv acreatedi set+ among" assignPnto," ofPtorelations.v a paths Formustiv a set+ amongi+ setPnn, a," of ofPto formal pathsmust pathsmustv a For set, a ofP formal pathsmust Pbe1,P assigned2,...,PPbe1,P to assignednv2,Q,...,Pi.1 Therefore,,Q to2n,...,Qv,Qi.1 Therefore,,QPn"2,is,...,Q we feasible sayPn" that,is we as feasible the longand say relationUsing thatrigorous as asP the longandis. this is relationUsing definition non-anti-reflexiverigorous as twoP is. simple this is! definition non-anti-reflexive of twoanti concepts simpleand relation of!non-anti-reflexiveanti conceptsdifferentand if relation thenon-anti-reflexive relationsdifferent! if therelations betweenrelations!relations see thebetween XXx, policies see hereby the XXx,of policies the we hereby different just of in-the we different just in-i+n troducenodes and atroducenodes fair the definition paths and a fair the announced definition based paths on announcedand by based anUsingrigorousthem example on canbyand an this forthem examplePbe be1Using definition rigorous,P assignedthe expressed.two2 can,...,P shake forPbesimple bethis1,P toassigned the expressed.ofn definitionv2,Q of,...,PiAtwo clarity. shake.1 Therefore,anti particular,Q concepts tosimple2 ofn,...,Qv If,Qandi Aclarity.. there of1 Therefore, particular,QP typenon-anti-reflexivenanti"2 conceptsdifferent,is ,...,Q we If feasibleof an thereand say relationsalternat-P typen" that,is non-anti-reflexive we asrelationsdifferent feasible anof the long say alternat-relations relation that as asPrelations the longis. betweenrelations is relation non-anti-reflexive as P is.relations see is thebetween non-anti-reflexive XXx, policies relation see hereby the XXx,of policies if relationthe the we hereby different just ifof the in-the we different just in- sequencesequence is cyclic, is cyclic, Pbe1,P assigned2,...,PPbe1,Ptroduce tonodes assignednv2,Q,...,Pi. and1 Therefore, a,Qtroducenodes fair the to2 definitionn,...,Q pathsv,Q andi a. fair1 Therefore, the,Q announcedP definitionn based" paths2,is,...,Q we feasible on announced say by based anPnthem example" that,is onwe as feasible canby anthe long say forthem example be relation the that expressed.as can as shakeP for the long beis. theis expressed.of relation non-anti-reflexiveA clarity. asshake particularP ofis. If is Aclarity. there particularnon-anti-reflexive type is If ofan there relationsalternat- type relation is anof alternat-relations if relation the if the between pathsbetween caused paths by caused routing! nodes by policies routing! and arenodes policies thesequence the "non-anti-reflexive paths and aresequence isthe the cyclic, announced"non-anti-reflexive paths is cyclic,relations. announced by themrelations. For a canby formal For them be aexpressed. formal can be expressed. A particular A particular type of relations type of relations ingUsing sequenceing this sequenceUsing of two preference simple this of two preference concepts (! simple)troduce and compositionconceptsdifferent (!) andatroduce fair compositionrelationsdifferentbetween definition (!) a relationsbetweenrelations fairpathsbetween (! definition) caused based createdrelations thebetween paths policies by on caused among routing created basedthe an! ofpolicies example by athe policies amongset routing ondifferent! of an of paths a arethepolicies for setexample thedifferent of the"non-anti-reflexive paths are shake forthe "non-anti-reflexive the of clarity. shakerelations. of If clarity.relations. there For a formal is If Foran there alternat-a formal is an alternat- and rigorousand definition rigorous definition of antisequenceandv of2vnon-anti-reflexiveanti7...v0andsequence isvnon-anti-reflexive cyclic,5...ingv0Using sequencerelations ising this cyclic, sequenceUsing of two preference seerelations simple this XXx, of two preference conceptssee (hereby! simple) XXx, and we compositionconceptsdifferent ( hereby! just) and in- we compositionrelationsdifferent just (!) in-relations betweenrelations (!) createdrelations thebetween policies among created the of policies a the amongsetdifferent of of paths a the setdifferent of paths v2v7...v0 v5...Pv0,P ,...,PP ,P,Q,...,P,Q ",...,Q,Q ,Q! ",...,Q" ! ! " " ! ! " " ! ! " ! v2v7...v0 ! nodes1 !2 andnodes1 then2 paths and1 P the1 announced2!n pathsQ1n vP!1n announced:between2P,! by wenQ! themn say...!n that,P! canby wepathsnbetween...! the them sayand be...! relation expressed. thatcausedQ rigorous! can2 ...v! thepaths2and bevP is!3... relation2 expressed. definitionnon-anti-reflexivev byrigorousQ!0 A2 causedQ! routing particular1 P! is!2"definition non-anti-reflexiveP of!1 A byQanti particularpolicies1 type! routing!and" relationP of! of1"non-anti-reflexiveanti relations type! areandpolicies if(3.3)! relation the ofv thenon-anti-reflexive"5... relations"v"non-anti-reflexive0 if(3.3)! are therelations! the" ""non-anti-reflexive seerelations! ! XXx," seerelations. hereby! XXx, we hereby justrelations. For in- we a just formal in- For a formal ∆1 v1: ! troduce! v2 avtroduce3... fairv0 definition a fair definition based oningv basedan2vUsing sequence8... examplev on0 ing an thisv for6...P examplenodes sequenceUsing1v of,P the0two2 preference,...,P and shakeP fornodessimple this1 the,P then of of2,Q paths,...,Ptwo and clarity. shake preference1,Q conceptsP ( the1! announcedsimple2!n,...,Q of),Q IfpathsQ and clarity. there1n,QvP!1n announced: composition2conceptsdifferentP,! (,...,Q is by! wen IfQ an!) themtheren say and... alternat-!n that,P! canisby wen compositionrelationsdifferent... an! the them say be... alternat-! ( relation! expressed. thatQ! can)2 ... relationsv! the2 betweenrelationsbevP is!3... relation2 expressed. non-anti-reflexivevQ! (!0 A2Q!) particular1 createdrelationsP! is the2between non-anti-reflexiveP!1 AQ policies particular1 type among! createdrelationPthe of1 relations of typepolicies a if(3.3) relationthe amongset the ofdifferentrelations of ifof(3.3) paths thea the setdifferent of paths v1 v1 v2v8...v0 v6...sequencebetweenv0 pathssequencebetween is cyclic, caused paths is cyclic, by caused routing by policies routing are policies thetroducenon-anti-reflexive are a thetroduce fairnon-anti-reflexive definition a fairrelations. definition basedv2v on7...relations.v For basedan2vv08... example av formal on0 For anv for example av formal the shake for the of clarity. shake of If clarity. there is If an there alternat- is an alternat- v2 v2 v1 ! and rigorous! andsequencebetween definition rigorous" pathssequencebetween is cyclic," definition causedof paths isanti cyclic,! by causedand routing" of "non-anti-reflexiveanti by policies! routingand! v arenon-anti-reflexive" policies5...6... the"v00non-anti-reflexive! arerelations! the "non-anti-reflexive" seerelations!relations.! XXx," relations.see For hereby a! XXx, formal For we hereby a justformal in- we just in- anding sequence rigorousing definitionsequence of preference of of preferenceanti (!)Pnodesand and1,Pnon-anti-reflexiveanti composition (2!,...,P and) andPnodes1non-anti-reflexive the composition,Pn (2,Q paths!,...,P and)1 relations,QP the1 announced (2!n,...,Q,Q) pathsvQ createdseerelations2 1n,Q XXx,vP1n announced:2 amongP,,...,Q created by weherebyn!Q themn say a... setamong wen that,P! of justcanby wen paths... a the in-them sayset be... ofrelation expressed. thatQ" paths can2 ...v the2 bevP is3..." relation2 expressed. non-anti-reflexivevQ0 A2Q particular1 P is2 non-anti-reflexiveP1 AQ particular1 type relationP of1 relations type if(3.3) relation the of relations if(3.3) the Γ and rigorous definitionv of3v9...v0and v7...ingv0 sequenceing sequence! of preferencerelations! of! preference see (!!) XXx, and! composition (hereby!) and! we composition! just (!!) in- relations! ! (!)! createdrelations! ! among created! a setamong! of paths a set of paths 2 v3v9...v0 v7...Pv10,P2,...,PP ,Pn,Q,...,P1,Q2",...,Q,Q ,Q!ntroduce",,...,Q we" say! that a,! wetroduce fair" theand say" definition relation thatrigorous! a! theand fair" is relation definition rigorousnon-anti-reflexive" definition based! ! is definition non-anti-reflexiveof" onantiv basedan2v!and8... example relationv of3vvnon-anti-reflexiveanti on09...v0 anand ifv relation the for6... examplevnon-anti-reflexivevv the0 if the shakerelations for the of seerelations clarity. shake XXx, see hereby of If XXx, clarity. there we hereby just is If in-an we there alternat- just is in- an alternat- ! vtroduce1 The! conditionatroduce fair1 The definition2 acondition to fairP guarantee1 !n definition basedQn1 toPv!12guarantee onsequence: existence2P!n basedanQ!n example...! onn existence ofP!sequencen anis multipath...! forcyclic, exampleP...!,P the ofQ!2,...,P shakemultipath...v solution! is for3vP!4... cyclic,2 the ofQv!,Q02 clarity. shakeisQ solution!1 that,QP!2" of,...,QP no! If1 clarity.Q isthere subset1 that!!", isP nowe Ifof1 an" there subsetthe say alternat-! paths that(3.3)! is of an" the7... the alternat-" relation0 paths(3.3)! ! " is non-anti-reflexive" ! ! " ! relation if the v2: ! ! v3v4...v0 betweenv v v pathsbetween1 The caused2 conditionpathsP1,PThe byn2,...,P caused condition routingto1P guarantee1 2n,QQ byn1 to policiesP,Qv12 routingguaranteen: existence2P,...,QnQn ... aren policiesexistence ofP,n we multipath... the say...non-anti-reflexive ofthatQ are2 multipath...v solution the3v theP4...2 relationQv0non-anti-reflexive2 isQ solution1 thatP is2 P non-anti-reflexiveno1 Q is subsetrelations.1 thatP no of1 subsettherelations. paths(3.3) For relation of the a formalpaths if(3.3) the For a formal v3v10...v0 v8...sequencev0 sequence is cyclic, is cyclic, 3 10... 0 v8...troducev0 v2 atroduce fair definition a fair!! definition based! on! basedan!! example! on! an! for example! the"! shake! for! the of!" clarity. shake! ! of! If clarity. there! is If an there alternat- is an alternat- Γ1 propagateding sequencepropagated throughout of preference throughout the network (!)ing and the createsnetworksequence composition! aing non-anti-reflexive createssequence sequence of (") preferencea relations non-anti-reflexivesequence is cyclic," of relationcreated preference is ( cyclic,!) among among and relation composition (avthem,3 set amongv)10... ofand causedv paths0 them, compositionv8...v ( caused!0 ) relations ( ) createdrelations among created a setamong of paths a set of paths v6 ing sequence of preference! and rigorous (!) andand compositionpropagateding definition rigorous! sequencepropagateding throughout ( sequence! ofdefinition of) preference relationsanti throughoutthe ofand preference networkcreated of (!non-anti-reflexive)anti and the! among createsnetwork compositionand (!) and anon-anti-reflexive a set non-anti-reflexive creates composition of (" paths) a relations non-anti-reflexiverelations (")!relationcreated relations seerelations among among XXx, createdrelation athem, see hereby setamong among ofXXx, caused paths a them,we set hereby of justcaused paths in- we just in- v6 v7 v6 v9...v0 v!3v9...v0! v7...v0! ! v7 v9...Pv10,P2,...,PP1,Pn2,Q,...,P1,Q2n",...,Q,Q1,Q!nP2,,...,Q1 we,P" say2,...,Pn that,! wePv the7 say,P"n relation,Q that,...,P1! the,Q is relation2 non-anti-reflexive",...,Q",Q is!,Q! non-anti-reflexiven",,...,Q we" sayrelation! that,! we if relation" the thev say9..."v relation0 that if! the! the" is relation non-anti-reflexive" ! ! is non-anti-reflexive" ! relation if relation the if the v5 v5 ! ! P Q v3: P"v4vThe11...... !v0 condition...1"PThe1,P2Q!2,...,P conditionv to1P v1P guarantee1",P2...nnv2,Q,...,P0 Q1!,Qn1 toPv2n,...,Q12 guaranteeP,Q": existence2P1n,QQn!2n,,...,Q we...n sayexistence ofPn thatmultipath,... we the... say relation ofQ that2 multipath...vsolution the3v isP relation4... non-anti-reflexive2 Qv02 isQ issolution1that non-anti-reflexiveP2 P no1 relationQis subset1 that ifP no relationof1 the subsetthe if paths(3.3) the of the paths(3.3) v3: v4v11...v0! The! v1condition v2...vThe0 condition to guarantee1 ! tonP! guarantee1troduce existence!vn5Q!n ! existenceof aP!troducen fair multipath! ...! definition of!2 multipatha... solution! fair!2!Q! definition based2 is solution!1 thatPP!2!" noon!1Q! isQ subset basedanv that1!3!: exampleP"v noP of4!1v" on11... subset the...!!v an paths0(3.3)! offor example..." the!" the paths!Q(3.3)! shakev!1for vP"2...!v the" of0!Q! clarity. shake!P"! of! If! clarity. there! is(3.3) If an there alternat- is an alternat- v8 v10...sequencevv80 sequence is cyclic, is cyclic, v10...vThe0 conditionThe condition to guarantee1 ! v ton3P!v guarantee1 existence10...!nQv!n0! existence ofP!vn multipath8...v!v...!0v of!2 multipath... solution! !2 Q!2 is solution!1 thatP!2 no!1 isQ subset that1 ! noP of1 subset the paths of the paths(3.3) ∆4 v0 propagatedpropagated throughout throughout the networkpropagatedsequence the creates networkpropagated asequence is non-anti-reflexivecreates throughout cyclic,sequencev8 a non-anti-reflexive issequence is throughout cyclic, the relationnetwork is cyclic,! among therelation creates network them,! among caused a non-anti-reflexive them,creates10... caused"0 a non-anti-reflexive" relation among relation them, among caused them, caused v0 ∆2 v6 ing sequencev0 ingpropagated sequence of preferencepropagated throughout of preference ( throughout! the) and network compositionthe (! creates network) and a composition non-anti-reflexivecreates (!) a relationsnon-anti-reflexive (! relation) createdrelations among relation among created them, among caused a them, setamong of caused paths a set of paths ! " ! " " ! ! " " ! ! " " ! ! " ! v11 v4: vv111v5...v0 v11...v0 ! ...v2...v0 P1 ! QnvvP!741: Pvn1Qv!5...n ...v0 P!nv...11...... !v0Q2 ...!v2...P2vQ0!2Q1 P!2"P1 Q1! "P1" ! (3.3)!v9..."v"0 (3.3)! ! " " ! ! " ! ! !The conditionThe condition to guarantee to guaranteeP existencev!111,P2!,...,P existence ofP multipath1!,Pn2,Q of,...,P! multipath1 solution,Q!2n",...,Q,Q is solution!P that11,Q!! non2Q,,...,Q is subsetn wevP that!41!": P! sayvn no1 ofQv!n5...n subsetthe that,...v! we0 pathsP!n the ofv... say!11... the"...! relationv that0Q paths!2 ...!v! the2...P!2v isQ0 relation! non-anti-reflexive2Q"!1 P!2 P!1 is!Q non-anti-reflexive1 ! P1 relation(3.3) (3.3) if relation the if the v9 v5 The conditionP TheQ condition to guaranteevP3: P"v to4Qv guarantee existence11...... !v0P existence of..." multipath...Q of! multipathv solution...1 vP"2...Qv is solution0 thatQ! noP is subsetP that" Q no of subsetthe! pathsP of the paths(3.3) v9 propagated throughout the networkThe creates condition a non-anti-reflexivevThe9 condition to guarantee1 ! relation ton ! guarantee1 existence among!n !n them,! existence of! causedn multipath! ! of!2 multipath solution! !2 !2 is solution!1 that!2 no!1 is subset that1 ! no of1 subset the paths of the paths(3.3) v10 v10 propagated throughoutsequencev the8 networkv10sequence is creates cyclic,propagated a non-anti-reflexive ispropagated throughout cyclic, throughout the relationnetwork the among creates networkv them, a10... non-anti-reflexivecreatesv caused0 a non-anti-reflexive relation among relation them, among caused them, caused Γ4 Γ propagatedpropagated throughout throughout the network the creates network a non-anti-reflexivecreates a non-anti-reflexive relation among relation them, among caused them, caused v4 3 v4 The conditionThe condition to guaranteev4 to guarantee existence existence of multipath of multipath solution is solution that no is subset that no of subsetthe paths of the paths ∆ The conditionThe condition to guarantee to guarantee existence existence of multipath of multipath solution is solution that no is subset that no of subsetthe paths of the paths 3 propagatedpropagated throughout throughout the network the creates network a non-anti-reflexivecreates a non-anti-reflexiveP " relationQ v!4: amongP" relationv1v"5... them,...v! among0 ! causedv... them,"11..."v caused0Q! ...!v2...P"v0" Q! ! P" ! v11 The conditionpropagatedThe condition topropagated1 throughoutguarantee! n to throughoutP! the1 guarantee existence! networknQ!n the! creates network existence ofP!n multipath! a non-anti-reflexivecreates...! of! a2multipath non-anti-reflexive... solution! !2 Q! relation2 is solution!1 thatP! among relation2 ! no1 Q isthem, subset among1 that! causedP no them,of1 subsetthe caused paths(3.3) of the paths(3.3) v3 v3 v9 v3 v10 propagatedpropagated throughout throughout the network the creates network a non-anti-reflexivecreates a non-anti-reflexive relation among relation them, among caused them, caused (a) (b) (c) (d) v4 The conditionThe condition to guarantee to guarantee existence existence of multipath of multipath solution is solution that no is subset that no of subsetthe paths of the paths propagatedpropagated throughout throughout the network the creates network a non-anti-reflexivecreates a non-anti-reflexive relation among relation them, among caused them, caused Figure 6.1: Unipath Disputev3 Wheel 31 Stability Analysis

In policy-based scenarios where a path vector protocol is running, paths propagate from one node to another as they are chosen in the ranking procedure as most preferred and an- nounced to the routers with peering sessions. When a path P is preferred over a path Q according to the preferences of a node, that relationship of preference can be denoted like,

P C Q (6.1)

If a path P is announced and it is chosen as most preferred by an arbitrary number of nodes vi+n, . . . , vi+1 in the network, the assigned path to vi+n is a propagated version of P , i.e. P 0 = vi+n, . . . , vi+1, vi, . . . , v0 = (vi+n, . . . , vi+1)P and its relationship of composition with P can be expressed like, J P  P 0 (6.2)

Using these two simple concepts different relations among the policies of the different nodes and the paths announced by them can be denoted. A particular type of relations be- tween paths caused by routing policies are the reflexive relations. Hereby, just a fair definition is introduced for the shake of clarity. For a formal and rigorous definition of anti-reflexive and reflexive relations see [5]. If there is an alternating sequence of preference (C) and com- J position () relations created among a set of paths P1,P2,...,Pn,Q1,Q2,...,Qn, it is said that the relation is a reflexive relation if there is a path for which the sequence is cyclic, e.g.

J C J C J C J C P1  Qn  Pn  ...  ...  Q2  P2  Q1  P1 (6.3)

Reflexive relations cannot be fulfilled simultaneously. For instance the path Qn composed by an arbitrary path and P1 is more preferred than P1, which is a contradiction since when Qn selected P1 is not selected and Qn is feasible only if P1 is selected. Hence, the protocol cannot find a solution and it is likely to oscillate.

The first reflexive relations depicted in the literature for policy-based routing are the BGP dispute wheels shown in [12]. Although the analysis in [12] shows the stability results using an abstraction of the ranking function, which covers all the different rankings that can be configured for ASSEMBLER, the analysis does not cover the case in which multiple paths are ranked as most preferred instead of just one. The framework in [5] considers multiple equally preferred paths and provided that our ultimate goal is to analyze the stability in mixed environments in which multipath nodes coexist with unipath nodes, it is interesting to study the stability of the protocol under that framework.

The condition used in the framework to guarantee existence of multipath solution is that the preferences assigned by the ranking functions do not create a reflexive relation among the paths propagated throughout the network [5]. Since the generation of a dispute wheel involves a specific relation among the ranking functions, by changing the ranking functions and announcing additional paths, it is expected that the relation among them changes as well. An stable state may no longer be reachable, or it is under a different solution. Before addressing the formal analysis of the problem, two examples are provided to show that the propagation of additional paths can provide a network running ASSEMBLER with an stable solution in unstable unipath scenarios, whereas in stable cases it can activate a dispute wheel.

Example 1 In Fig.6.1 a network is depicted in which every node selects only one path like BGP. The path ranked in first position by each node is displayed with a solid arrow. Feasible paths lower rank are displayed in dashed arrows. The preferences of each node are shown 17 17 3.4. Stability3.4. Analysis Stability Analysis 17 17 3.4. Stability3.4. Analysis Stability Analysis 32 Chapter 6 17 17 3.4. Stability3.4. Analysis Stability Analysis 17 17 Stability Analysis some unipathsome configuration unipath configuration guidelines guidelines must be slightly must be modified slightly becausemodified they because no longer they no can longer can 3.4. Stability3.4. Analysis Stability Analysis be directlybe used directly in multipath used in multipath scenarios. scenarios.some unipathsome configuration unipath configuration guidelines guidelines must be slightly must be modified slightly becausemodified they because no longer they no can longer can be directlybe used directly in multipath used in multipath scenarios. scenarios. 17 17 3.4. Stability3.4. Analysis Stability Analysis 17 17 someThe unipath stabilitysomeThe configuration unipath proof stabilityis configuration presented proof guidelines is presented after guidelines must the be guidelines after slightly must the be guidelines modified and slightly it isstructuredmodified because and it is they becausestructured as follows:no longer they as after nofollows: can longer after can 3.4. Stability3.4. Analysis Stability Analysis be directlybe used directly in multipath used in multipath scenarios. scenarios.someThe unipath stabilitysomeThe configuration unipath proof stabilityis configuration presented proof guidelines is presented after guidelines must the be guidelines after slightly must the be guidelines modified and slightly it17 isstructuredmodified because and it is they becausestructured as follows:no longer they as after nofollows: can longer after can in the table at Fig.6.1d. In Fig.6.1athe analytical thethe analytical node modelv of1 modelthereceives protocol of the three is protocol presented, paths is presented, the throughsynchronous thevsynchronous2,convergence v5 and convergencev6 of the proto- of the proto- 17 col for anycol of for its any flavours, of its in flavours, absence in of absence conflictingthebe of directlyconflicting analytical routingbethe used directly policies analytical model routing in multipath used of between policies modelthe in protocol multipath scenarios. ASesof between the is protocol3.4.scenarios. presented,shown. ASes Stability is3.4. presented,shown. the Analysis Stabilitysynchronous the Analysissynchronousconvergenceconvergence of the proto- of the proto- 17 17 respectively. The three paths are of the same AS length and v chooses the path v v . . . v 3.4. Stability3.4. Analysis Stability Analysis Ifsome convergenceThe unipath stabilityIfsome convergenceThe configuration isunipath proofstability not achieved, is configuration is presented proof not guidelines achieved, then is presented after it guidelines mustis1the shownthen be guidelinesafter itcol slightly mustisthat forthe shown the any beguidelines modifiedandcol conflictingslightly of that forit its is the any flavours, structured modifiedbecauseand conflicting2 of relation it7 its is in flavours, they structured because absenceas among0 relation follows: no longer in they of policies absenceas amongconflicting afterfollows: no can longer of policies conflicting after routing can policies routing between policies ASes between is shown. ASes is shown. someThe unipath stabilitysomeThe configuration unipath proof stability is configuration presented proof guidelines is presented after guidelines must the be guidelinesafter slightly must the beguidelines modifiedand slightly it is structured modifiedbecauseand it is they structured because as follows: no longer they as afterfollows: no can longer after can with the lowest BGP_ID. The nodeactuallythebe directly analyticalv2 exists.theactuallybehas used directly analytical modelthree Afterwards, in exists. multipath usedof paths model the Afterwards, in protocol it multipath scenarios.ofis of proven theequal is protocol it presented,scenarios. is that path proven if is nodesIf length presented, convergence thatthe followsynchronous if throughIf nodes convergence the the is followsynchronous guidelines notconvergencev7 achieved,, the v is8 guidelines notand presentedconvergence achieved, then of the it presented in is proto- Sec-shownthen of the it in isthat proto- Sec-shown the conflicting that the conflicting relation among relation policies among policies actuallythebe directly analytical exists.theactuallybe used directly analytical model Afterwards, in exists. multipath usedof model the Afterwards, in protocol it multipath scenarios.is of proven the is protocol it presented,scenarios. is that proven if is nodes presented, thatthe followsynchronous if nodes the the followsynchronous guidelinesconvergence the guidelines presentedconvergence of the presented in proto- Sec- of the in proto- Sec- v3. It is not aware yet of the pathtioncolsome for XXx,v unipath3 anyvcoltionsome9 it of. for.is XXx,configurationits .unipath not v any flavours,0 possibleand it of is itsconfiguration notchooses inflavours, guidelines toabsence possible have in to conflicting guidelinesof toabsence must go conflicting have through be conflicting of slightly policiesmust conflicting routing bev modified7 slightlyand policiesusing policies routing themodifiedbecause protocol the and between policies lowestthe they should because protocol ASes between no longerbe is they should shown.able ASes no can to longer be is shown. able can to If convergenceThe stabilityIf convergenceThe is proof notstability achieved, is is presented proof not achieved, then is presented after it is thenthe shown guidelines after ittioncolsome is that for theshown XXx, unipath the any guidelinescoltionsome and conflicting it ofthat foris itXXx,configurationitsunipath not theis any flavours, structured and possible conflicting it of relation is itsconfigurationit not is inflavours, guidelinesstructured toabsence possibleas among haverelation follows: in conflicting guidelinesof policiesto absence mustas among conflicting have after follows: be conflicting of slightlypolicies policiesmust conflicting after routing be modified slightlyand policies policies routing themodifiedbecause protocol and between policies the they should because protocol ASes between no longerbe is they should shown.able ASes no can to longer be is shown. able can to BGP_ID criteria. Node v3 is configuredfindbe directly a stablefindbe used directlysolution. ain stable in a multipath similar used Insolution. order in multipath scenarios. way, to In complete order it scenarios. chooses to completethe stabilityvThe9 theasstability proof, stability next-hopThe another proof stability proof, sinceis demonstration, another presented proof it is demonstration, presented after which the guidelines after which the guidelines and it is structured and it is structured as follows: as after follows: after the analyticalthe analytical model of model the protocol of the is protocol presented, isfindIfbe presented, convergence directlythe a stablesynchronousIffindbe used convergencedirectlysolution. the a isstablesynchronous in not multipathconvergence used achieved,In solution. orderis in not multipath scenarios.convergence to achieved, In then completeof order the it scenarios. is proto- to then shown completeofthe the it stability is that proto- shown the the proof, stability conflicting that another the proof, conflicting relation demonstration, another among relation demonstration, policies which among policies which reliesactually on exists. theactuallyreliesGeneral onAfterwards, exists. the AsynchronousGeneral Afterwards, it is Asynchronous proven Convergence it is that proven ifConvergence nodes that Theorem follow if nodesXXx, Theoremthe follow guidelines is presentedXXx, the guidelines presented is presented such presentedthat in Sec- suchthe that in Sec- thesynchronous is not aware of the path v4v11 col. . . for v0 any.col The of for its anynode flavours, of itsv4 in flavours,receives absence in of absence a conflicting pathreliesactuallythe of through conflicting analytical routing on exists. theactuallyreliesthevGeneral policies analytical11 model routing onAfterwards,but exists. the of between AsynchronousGeneral policiesit model the Afterwards, is protocol it not is ofASes between Asynchronous proven the isConvergence protocol it presented,shown. is thatASes proven if isConvergence nodes presented,shown. that the Theorem follow if nodes theXXx, Theoremthe followsynchronous guidelines isconvergence presentedXXx, the guidelines presented isconvergence presented such of the presentedthat in proto- Sec- suchthe of the that in proto- Sec- the convergencetionThe XXx, stabilitytionconvergence it of isThe XXx,the not proof stability real possible it ofisis (asynchronous) notthe presented proof toreal possible have is (asynchronous) presented after conflicting to protocol have the guidelines after conflictingcol policiesis protocol forguaranteed,the any guidelines andcol and policies is of for guaranteed,it its the is as any flavours, structured protocoland well. and of it its the is as in flavours,structuredshould protocol well. absenceas follows: be in should ofabsenceas able conflicting after follows: to be of able conflicting after routing to policies routing between policies ASes between is shown. ASes is shown. aware of the path v v . . . v , thereforeIf convergenceIf it convergence chooses is not achieved,v is noteven achieved, then though it is thenshown itsconvergencetion it isthat highestThe shownXXx, the stabilitytionconvergence conflicting it that ofpreference isThe XXx,the not the proof stability real possible conflictingit relationof isis (asynchronous) notthe presented proof is toreal possible to among have relation is (asynchronous) presented after conflicting to policies protocol among have the guidelines after conflicting policies policiesis protocol guaranteed,the guidelines and and policies is guaranteed,it the is as structured protocoland well. and it the is asstructuredshould protocol well. as follows: be should asable after follows: to be able after to 1 5 0 findthe analytical a stablefindthe solution. analytical a model stable of solution.In model the order11 protocol of to In thecomplete order is protocol presented, to complete the is stabilityIf presented, convergence the thesynchronous proof, stabilityIf convergence the anotherissynchronous notproof,convergence achieved, demonstration, anotheris notconvergence achieved, then ofdemonstration, the it iswhich proto- thenshown of the it isthatwhich proto- shown the conflicting that the conflicting relation among relation policies among policies actually exists.actually Afterwards, exists. Afterwards, it is proven it is that proven iffindthe nodes that analytical a stable follow iffindthe nodes solution. analytical athe model stable follow guidelines of solution.In themodel the order guidelines protocol presented of to In thecomplete order is protocol presented presented, in to Sec- complete the is stability presented, in the Sec- thesynchronous proof, stability the anothersynchronous proof,convergence demonstration, anotherconvergence ofdemonstration, the whichproto- of the which proto- use v1v5 . . . v0. In addition, v4reliescolfilters for on anyanyrelies thecol ofGeneral for itsAS on any flavours, the path of AsynchronousGeneral its containing in flavours, absence Asynchronous in Convergence of absencev conflicting2. Convergenceactually of conflicting Theorem routing exists.actually policies TheoremXXx, routing Afterwards, exists. is between presentedpoliciesXXx, Afterwards, it is isASes between presented provensuch is it that shown. is ASesthat provensuch the if is nodes that shown. that the follow if nodes the follow guidelines the guidelines presented presented in Sec- in Sec- tion XXx,tion it is XXx, not possible it is not to possible have conflicting to have conflictingreliescol policies for on anyrelies thecol and ofpoliciesGeneral for its the on any flavours, the protocol and of AsynchronousGeneral its the in flavours, should protocol absence Asynchronous bein Convergence of should absence able conflicting to be Convergence of able conflicting Theorem routing to policies TheoremXXx, routing is between presentedpoliciesXXx, is ASes between presented such is that shown. ASes such the is that shown. the convergenceIf convergence3.4.1convergenceIf On convergence of3.4.1 Dispute isthe not real On achieved, of (asynchronous) Dispute theis Wheels not real achieved, then (asynchronous) Wheels in it Unipath is protocol thenshown in ittion Unipath isthat protocol and guaranteed, shown XXx, the Multipathtion conflicting itis that and isguaranteed, XXx, not theas Multipath well. possible conflicting it relation Scenariosis not as well. to possible among relationhave Scenarios conflicting policies to among have conflicting policies policies and policies the protocol and the should protocol be should able to be able to find a stablefind solution. a stable solution.In order to In complete order to complete theconvergenceIf stability convergence3.4.1 the proof,convergenceIf stabilityOn convergence of3.4.1 Dispute isthe another not proof, real On achieved, of (asynchronous) demonstration, Dispute theis Wheelsanother not real achieved, then (asynchronous) demonstration, Wheels in it Unipath is protocol which thenshown in it Unipath isthat protocol which and guaranteed, shown the Multipath conflicting is that and guaranteed, theas Multipath well. conflicting relation Scenarios as well. among relation Scenarios policies among policies actually exists.actually Afterwards, exists. Afterwards, it is proven it is that proven if nodesfind that a stablefollow if nodesfind solution. the a stablefollow guidelines solution.In the order guidelines presented to In complete order presented in to Sec- complete the stability in Sec- the proof, stability another proof, demonstration, another demonstration, which which In Fig.6.1b, v2 becomes awarerelies of on the therelies pathGeneral on the through AsynchronousGeneralv Asynchronous3 and Convergence changes Convergenceactually its Theorem assignment, exists.actually TheoremXXx, Afterwards, exists. is presentedXXx, forcing Afterwards, it is is presented proven such it that is that proven such the if nodes that that the follow if nodes the follow guidelines the guidelines presented presented in Sec- in Sec- tionThe XXx, goaltion itTheof is XXx, notthis goal possiblesection it ofis notthis is to possible section to have introduce conflicting is to to have introduce a discussion conflictingrelies policies aon discussion theaboutrelies and policiesGeneral the onthe protocolthe aboutimpact and AsynchronousGeneral the the of should protocol impact multipath Asynchronous be Convergenceof should able multipath solu- to be Convergence able solu- Theorem to TheoremXXx, is presentedXXx, is presented such that such the that the convergenceconvergence of the real of (asynchronous) the real (asynchronous) protocoltion is protocolThe guaranteed, XXx, goaltion it isTheof isguaranteed, XXx, notthis as goal well. possiblesection it ofis notthis as is to well. possible section to have introduce conflicting is to to have introduce a discussion conflicting policies a discussion about and policies the the protocol aboutimpact and the the of should protocol impact multipath be of should able multipath solu- to be able solu- to v1 to change its path as well. Nodetionsfind3.4.1 a inv stable1 thedoestionsfind On stability solution.3.4.1 a inDispute not stable the On ofselect stability solution.In policy-based Dispute order Wheels the of to In policy-basednew complete order Wheels in routing path Unipath to complete the protocols. of in routingconvergence stabilityv Unipath2 andbecause the protocols. Before Multipathproof,convergence stability and of discussingis anotherthe Before longerMultipath proof, real ofScenarios (asynchronous)demonstration, discussingtheanother the than real variety Scenarios (asynchronous)demonstration, the ofwhichprotocol variety solu- of is protocolwhich solu- guaranteed, is guaranteed, as well. as well. tionsfind3.4.1 a in stable thetionsfind On stability solution.3.4.1 a inDispute stable the On of stability solution.In policy-based Dispute order Wheels of to In policy-based complete order Wheels in routing Unipath to complete the protocols. in routing stability Unipath and the protocols. Before Multipathproof, stability and discussing another Before Multipathproof, Scenarios demonstration, discussinganother the variety Scenarios demonstration, the ofwhich variety solu- of which solu- the path through v5. However,tionsreliesv3 becomes and ontions instabilitiesthe General and aware instabilities in AsynchronousGeneral unipath of the in Asynchronous andpath unipath multipath Convergence through and multipath Convergenceandv theTheorem4 and relationships and changes TheoremXXx,the relationships is between presented as well. them, between such some that them, no- the some no- relies on the tions andtions instabilities and instabilities inXXx, unipath is in presentedand unipath multipath and such multipath and that the the relationships and the relationships between them, between some them, no- some no- tationThe is goalintroducedtationThe of is this goal introduced hereby. section of this The is hereby. section to notation introduce The is to comes notation introduce a discussionrelies from comes aon the discussion thereliesabout framework fromGeneral on the the theaboutimpact framework definedAsynchronousGeneral the of impact inmultipath Asynchronousdefined XXxChikin. Convergenceof inmultipath solu- XXxChikin. Convergence solu- Theorem TheoremXXx, is presentedXXx, is presented such that such the that the The path assignment is again modifiedconvergenceconvergence in of Fig.6.1c. the real of (asynchronous) the Node real (asynchronous)v2 looses protocol its is protocol pathThe guaranteed, goalv2 isvThe of guaranteed,3v this as9 goal well.. section . . of v0 this asas is well. section to it introduce is to introduce a discussion a discussion about the about impact the of impact multipath of multipath solu- solu- tions3.4.1 in thetions On stability3.4.1 in Dispute theOn of stability policy-based Dispute Wheels of policy-based Wheels in routing Unipath protocols.in routingtationconvergence Unipath and is protocols. Before introduced Multipathtationconvergence ofand discussingisthe Before introduced Multipathreal hereby. of Scenarios (asynchronous) discussingthe the The real hereby. variety notation Scenarios (asynchronous) the The ofprotocol variety solu-comes notation of from is protocol solu- comes guaranteed, the framework from is guaranteed, theas well. framework defined as well. in defined XXxChikin. in XXxChikin. is longer than v v . . . v and v prefers the path announced by v . Intions the3.4.1 in next thetions On stability3.4.1 step, in Dispute theOn of the stability policy-based Dispute nodes Wheels of policy-based Wheels in routing Unipath protocols.in routing Unipath and protocols. Before Multipath and discussing Before Multipath Scenarios discussing the variety Scenarios the of variety solu- of solu- 2 7 0 tions4 In and policy-basedtions instabilitiesIn and policy-based instabilities scenarios in unipath scenarios where in and unipath multipatha path where and1 vector multipath a and path protocol the vector relationships and is protocol the running, relationships between is paths running, propagatethem, between paths some them, propagate from no- some from no- The goalThe of this goal section of this is section to introduce is to introduce a discussiontionsIn and policy-baseda discussiontions instabilities aboutIn and policy-based the instabilities aboutscenarios impact in unipath the of scenarios whereimpact multipath in and unipath multipatha of path where multipath solu- and vector multipath a and path protocol solu-the vector relationships and is protocol the running, relationships between is paths running, propagatethem, between paths some them, propagate from no- some from no- go back to the initial assignmentonetation shown3.4.1 node is introducedtationone to On in another3.4.1 node DisputeFig.6.1a is introduced to hereby.On as another they Dispute Wheels completing The are hereby. as notation chosen they Wheels in The are Unipath in comes notationa chosen the cycle in ranking from Unipath in andcomesThe in the the the goalprocedure Multipath ranking framework from oscillation. andThe of thethisgoal procedure Multipath as framework section definedmost ofScenarios thisThe preferred is as sectionin to defined mostScenarios XXxChikin. introduce preferredis and in to XXxChikin.an- introduce a discussion and an- a discussion about the about impact the of impact multipath of multipath solu- solu- tions in thetions stability in the of stability policy-based of policy-based routing protocols. routingonetation3.4.1 node is protocols. Beforeintroducedtationone to On another3.4.1 node Dispute is discussing introduced Before to hereby.On as another they Dispute discussingWheels the The are hereby. as variety notation chosen they Wheels in the The areof Unipath invariety comessolu- notation chosen the in ranking offrom Unipath in andcomes solu- the the procedure Multipath ranking framework from and the procedure Multipath as framework definedmost Scenarios preferred as in defined mostScenarios XXxChikin. preferred and in XXxChikin.an- and an- reflexive relation can be expressednounced in to thisnounced the case routers to asthe with follows, routers peering with sessions. peering sessions.Whentionsa in path the Whentions stabilityP = ina path thevi,v of stabilityPi policy-based1=, .v .i .,v of, vi0 policy-based1is, .preferred routing . . , v0 is protocols. routing preferred protocols. Before discussing Before discussing the variety the of variety solu- of solu- tions andtions instabilities and instabilities in unipath in and unipath multipath and multipathnounced and the relationships tonounced and the the routers relationships to the between with− routers peering them, between with− sessions. some peering them, no- sessions.When somea no- path WhenP = a pathvi,vPi 1=, .v .i .,v , vi0 1is, .preferred . . , v0 is preferred overInThe a policy-based path goaloverQInThe of a= policy-based thispathv goali,v scenarios sectionQj of,v=j thisv1i is,,v scenarios where.sectionto .j .,v ,introduce vj0 a1according is, path where . to . . ,introduce vectora v0 discussion ationsaccording pathto protocol the and vector a preferences discussiontions instabilitiesabout to is protocol the andrunning, the preferences instabilitiesabout impactof in isvi pathsunipath running,, the that of impactof propagatemultipath relationship in andv pathsunipathi, that multipath of propagate multipath relationshipsolu- from and of multipath and fromsolu- the of relationships and the relationships between− them, between− some them, no- some no- tation is introducedtation is introduced hereby.− The hereby. notation− The comes notationover fromInThe comes a policy-based path the goalover framework fromQInThe of a= policy-based thispath thev goali,v scenarios frameworksectionQj defined of,v=j thisv1i is,,v scenarios where.section into .j defined.,v XXxChikin., introduce vj0 a1according is, path where . into . . XXxChikin.,introduce vectora v0 discussion aaccording pathto protocol the vector a preferences discussion about to is protocol the running, the preferences about impactof isvi paths running,, the that of impactof propagatemultipath relationshipv pathsi, that of propagate multipath relationshipsolu- from of fromsolu- of preferenceonetions node in theonepreferencetions tocan stability another node bein the denoted to ofcan as stability another policy-based they be like, denoted are of as policy-basedchosen they like, routing are in chosen the protocols. routing rankingtation in the is protocols. Before procedureintroduced rankingtation discussing is Before procedureintroduced as hereby. most discussing− the Thepreferred as hereby. variety most notation− the Thepreferred andof variety solu- comes notation an- and of from solu- comes an- the framework from the framework defined in defined XXxChikin. in XXxChikin. preferenceonetions node in theonepreferencetions tocan stability another node bein the denoted to ofcan as stability another policy-based they be like, denoted are of as policy-basedchosen they like, routing are in chosen the protocols. routing ranking in the protocols. Before procedure ranking discussing Before procedure as most discussing the preferred as variety most the preferred andof variety solu- an- and of solu- an- nouncedtions and tonouncedtions instabilities the and routers to instabilities the in with unipath routers peering inwith and unipath sessions. multipath peering andP ! sessions. When multipath andQ theP a path relationships! When andQP the a= path relationshipsvi,v betweenPi 1=, .v .i .,v , them, between vi0 1is, . preferredsome . . , them, v(3.1)0 no-is preferred some(3.1) no- In policy-basedIn policy-based scenarios scenarios where a path where vector! anouncedtions path protocol and vector! tonouncedtions instabilities the is protocol and routersrunning, to instabilities the in is with− unipath pathsrouters running, peering propagate inwith and− unipath paths sessions. multipath peering propagate from andP ! sessions. When multipath andQ fromtheP a path relationships! When andQP the a= path relationshipsvi,v betweenPi 1=, .v .i .,v , them, between vi0 1is, . preferredsome . . , them, v(3.1)0 no-is preferred some(3.1) no- over a pathoverQ a= pathvi,vQj,v=j v1i,,v . .j .,v , vj0 1according, . . . , v0 according toIn the policy-based preferences to the preferences scenariosof vi, that of where relationshipvi, that a path relationshipvector of! protocol of! is running,− paths propagate− from tation is introducedtationJ is introduced hereby.− The hereby.C notation− The comes notationover fromJ comes a paththeover framework fromQIn a= policy-basedpath thevi,v frameworkQj defined,v=j v1i,,v scenarios . in .j defined .,v XXxChikin. , vj0 1according, .where in . . XXxChikin., v0 aaccording topath the vector preferences to protocolthe preferences of isvi running,, that of relationshipvi paths, that propagate relationship of from of v v one. . . node v one tov another nodev v to as. another . they . v are as chosen theyv v are in. chosen . the . vtation ranking inthe is introduced proceduretation ranking is introduced procedure ashereby. most− The preferred hereby.as mostnotation− The preferred and comes notation an- andfrom comes an- the framework from the framework defined in defined XXxChikin. in XXxChikin. 1 5preference0 preferencecan4 be1 denoted5can be like, denoted0  4 like,11 preferenceone0  nodepreferenceone tocan another node be denoted tocan as another they be like, denoted are as chosen they like, are in chosen the ranking in the procedure ranking procedure as most preferred as most preferred and an- and an- nouncedAfterwards, tonounced theAfterwards, routersP tois the announced with routersP peeringis announced with and sessions. peering it is andchosen! sessions.When it is as a chosen pathmost! WhenP preferred as= apath mostvi,vPi preferred by1=, . anv .i .,v , arbitrary vi0 by1is, . preferred an . . , num-arbitrary v0 is preferred num- P = v ,v , . . . , v J In policy-basedC In policy-based scenariosJ scenarios where a path whereP vector! anouncedQ pathCAfterwards, protocolP vector! tonouncedQ the isAfterwards, protocol running, routersP tois the announced is with− paths running,routersP peeringis propagate announced with− and paths sessions. peering it propagate is(3.1) fromP andchosen! sessions.WhenQ it is(3.1) from asP a chosen pathmost! WhenQ preferred as apath mosti Pi preferred by1= anvi,v arbitraryi0 by1is, . preferred an . . , num-arbitrary(3.1) v0 is preferred num-(3.1) v3v4v11berover. of. a. vnodes path0beroverQ ofv ai=3+ nodespathvnv,v9i,v.i+Q .jn,vv .i= v+j10n,v .1,vi .,,v .i+ ,.jv v.n,v2 ,i+1 vj103,1 .accordingvin, .9 . the.,. .v . ,i+1 v. network.0 v0accordingin to theIn the policy-based network. preferences The toIn assigned the policy-based preferences The scenarios of pathassignedvi, that to scenarios whereof relationshipvi pathv+in, that ais to path whereP relationshipv"i+ vector= ofn! ais pathP protocol" vector= of! is protocol running, is− paths running, propagate− paths propagate from from  one nodeone to another node to as another they− −  are as chosen they− − are in chosen theberover ranking in of a the nodespath procedureberover rankingQ ofv ai=+ nodespathnv procedure,vi as,vi+Qj mostn,vvi=+j1n,v .1,v preferredi .,as,v .i+ ,.j most v.n,v ,i+1 vj10,1 .according preferredin and, . . the., .v an- ,i+1 v network.0 accordingin and to the the an- network. preferences The to assigned the preferences The of pathassignedvi, that to of relationshipvi pathv+in, thatis toP relationshipv"i+= ofn is P " = of vpreferencei+n,vi+nvpreferenceican+1n, .,v .be .i+ , denoted vni+1can1,,v . be.i ., like, , . denoted v .i .+1 , v,v0 =(i, like, . .v .i ,+ vn0,v=(i+onenvi+1 noden, .,v . .i+ ,one to vni+1 another1 node,) .P . .and , to v asi+1 another its they−) relationshipP− and are as its chosen they− relationship− of are incom- chosen the ranking of incom- the procedure ranking procedure as most preferred as most preferred and an- and an- C nounced tonounced− theJ routers to− the with routers peering withC sessions. peering sessions.Whenvpreferencei+−n,v ai path+ Whennvpreferenceican+1−Pn, .,v= .abe .i path+ ,v denoted vnii,v+1can1Pi,,v .1 be=.i, ., like,. , .v denoted .v .i .i .,v+1 , , v vi0,v0 1is=(i,, like, . preferred . . .v . .i , ,+ v vn00,vis=(i+ preferrednvi+1n, .,v . .i+ , vni+11,) .P . .and , vi+1 its) relationshipP and its relationship of com- of com- v vAfterwards,. . . v Afterwards,Pv isv announcedv P. .is . v announced andv it isv and chosen.! .nounced . it v is as chosen most! tonounced− the preferred as routers most to− thewith− preferred by routers an peering arbitrary with− by sessions. an peering arbitrary num- sessions.When− num- a path When−P = a pathvi,vPi 1=, .v .i .,v , vi0 1is, . preferred . . , v0 is preferred  position2 7 withposition0 Pcanwith1 be2P then7can expressed be0 then expressed like,1 P5 ! Q like,Afterwards,0P ! QAfterwards,P is announcedP(6.3)is announced and it is(3.1) and chosen! it is(3.1) as chosen most! preferred as most− preferred by an arbitrary by an arbitrary num- num- over a pathoverQ a= pathvi,vQj,v=j v1i,,v . .j .,v , vj0 1according, . . . , v0 positionaccording to the preferenceswithposition toP thecan preferenceswith be of P thenvi,can that expressed be of relationship thenvi, that expressed like, relationshipP of! Q like,P of! Q − (3.1) (3.1) ber of nodesber ofvi+ nodesn,vi+vni+1−n,,v . .i .+ ,n vi+11−, .in . . , the vi+1 network.overin the a path network. TheoverQ assigned a= pathv Thei,vQj assigned,v path=j v1i to,,v . .vj .,vi path+ , vjn0 is1according to, .P .v ."i+ ,= vn0 isaccording toP the" = preferences to the preferences of vi, that of relationshipvi, that relationship of of − − ber of nodesber ofvi+ nodesn,vi+vni+1−n,,v . .i .+ ,n vi+11−, .in . . , the vi+1 network.in the network. The assigned The assigned path to vi path+n is toPv"i+=n is P " = preferencepreferencecan be denotedcan be like, denoted like, " " − − vi+n,vi+vni+1n,,v . .i .+ ,n vi+11,,v . .i ., , . v .i .+1 , v,v0 i=(, . .v .i ,+ vPn0,v=(i+preferencePvni"+1nP,,v . .i .+preference ,Pn vcani"+11, be) .P . . denotedand , vcani+1 its be)P relationship like, denotedand its relationship like, of com-(3.2)" of com-(3.2)" − − !!vi+−n,vi!+!vni+−1n,,v . .i .+ ,n vi+11,,v . .i ., , . v .i .+1 , v,v0 i=(, . .v .i ,+ vPn0,v=(i+Pvni"+1nP,,v . .i .+ ,Pn vi"+11,) .P . .and , vi+1 its)P relationshipand its relationship of com-(3.2) of com-(3.2) Afterwards,Afterwards,P is announcedP is announced and it isP and chosen! Q it is asP chosen! most−Q preferred as most− preferred by an arbitrary by an arbitrarynum-(3.1)!! − num-(3.1)!! − position withpositionP canwith beP thencan expressed be then expressed like, position like,Afterwards,withpositionAfterwards,P canPwithis be announcedP thencanP expressedis be announced then and expressedlike, it isP and chosen! Q like, it is asP chosen! mostQ preferred as most preferred by an arbitrary by an arbitrarynum-(3.1) num-(3.1) If the K-BESTRO selectionber algorithm of nodesber ofvi is+ nodesn tuned,vi+nvi+1 ton,,v . . selecti .+ , vni+11, .2in . . equal-length ,the vi+1 network.in the network. The AS assigned path, The the assignedpath sce- to vi path+n is toPv"i+=n is P " = − − ber of nodesber ofvi+ nodesn,vi+nvi+1n,,v . .i .+ , vni+11, .in . . ,the vi+1 network.in the network. The assigned The assignedpath to vi path+n is toPv"i+=n is P " = " " − − nario becomes stable. Every nodevi+Ton chooses,v indicatei+nvi+1nTo,,v .that .the indicate .i+ , vnPi path+11,is,v . that. ai . through, , propagated . v .Pi .+1 , vis,v0 =( ai, propagated.one .v version .i ,+ vPn neighbor0,v=(i+P ofnv version"iP+1Pn,.,v . In . node .i+ ,orderP of vn"i+1P1,. on) to.P In . . assignandthe , order vi+1 its dashed) toP relationshipP assignandto v itsP relationship, P oftomustcom-v ", P of mustcom-" Afterwards,− Afterwards,P is−" announcedP is" announced and it is and chosen! v iti+−Ton is as,v chosen indicatei! most+nvi+−1nTo, preferred as,v.that . indicate .i+ most , vnPi+11" ,is preferred,v .by that. ai ., , propagatedan" . v .Pi .+1 arbitrary," vis,vi0 by+=( ain, propagatedan." .v version .i ,arbitrarynum-+(3.2) vPni0+,v=(ni+P ofnv version"iP+ num-(3.2)1Pn,.,v . In . .i+ ,orderP of vn"i+1P1,.) to.P In . . assignand , order vi+1 its) toP relationshipP" assignandto v itsP relationship,"P oftomust(3.2)com-v , P of(3.2)mustcom- P Afterwards,− Afterwards,P is− announcedP is announced and it is and chosen! it− is as chosen! most− preferred as most preferred by an arbitraryi by+n an arbitrarynum-i+n num- beposition assignedwithbeposition to assignedvican. Therefore,with be toP thenvican. Therefore, expressedP be" is then feasible expressedP like," is as feasible longposition like, as asP longwithis. asP Pcanis. be then expressed like, circle and a path through an outerber of neighbor nodesber ofvi+ nodesn except,vi+nvi+1n for,,v . .i .+ ,v vn4i+1.1, For .in . . ,the instance,vi+1 network.bein assigned thev network. The1bepositionselects to assigned assignedvi. Therefore,with The the toP assignedpathvican paths. Therefore, toP be"visi thenpath+ feasiblen is expressed toPP"vis"i as+= feasiblen longis like,P as" asP= longis. as P is. − − ber of nodesber ofvi+ nodesn,vi+nvi+1n,,v . .i .+ , vni+11, .in . . ,the vi+1 network.in the network. The assigned The assignedpath to vi path+n is toPv"i+=n is P " = v ,v v ,,v . . . , v ,,v . . ., , . v . . , v,v=(, . .v . , v ,v=(v ,,v . . . , v ,) .P . .and , v its−) relationshipP − of com- com- through v5 and v2. Fig.6.2a showsi+n i the+ni+1 finaln i+n pathi+11 assignment.i i+1 0 i i+n Node0 "i+vin+i+vn1,vn4 iranks+"i+nvin+i1+1n,1,v . only. .i+ , vni+11i the+1,,v . .i ., ,path . vand .i .+1 , v its,v0 =(i relationship, . .v .i ,+ vn0,v=(i+ ofnvi+1n,,v . . .i+ , vni+11,) .P . .and , vi+1 its) relationshipP and its relationship of com- of com- To indicate− To that indicateP− " is that a propagatedP " is a propagated versionP ! ofP version−"PP. In! orderofP−"P . to In assign order toP " assignto vi+Pn," Ptomust(3.2)vi+n", P must(3.2)" positionUsingwithposition thisUsingP twocanwith simple this beP then twocan concepts expressed simple be then differentconcepts expressed like, relationsdifferent like,UsingTo indicate betweenrelations this− ToUsing two that indicate the between simple thisP− policies" is two that a conceptsthe propagated simpleP of policies" is the a differentconcepts propagateddifferent version ofP the! relationsdifferent different ofP version−"PP. In! betweenrelations orderofP−"P . to In the between assign order policies toP the" assignto of policiesv thei+Pn, different" Pto ofmust(3.2)vi the+n, differentP must(3.2) through v11 since the advertisementbe assigned frombe toassignedvv.1 Therefore,contains to v . Therefore,P vis2, feasible soP thatis as feasible longvposition4 assigns as asP longwithis.positiona asP lowerPcanwithis. beP then prefer-can expressed be then expressed like, like, nodes andnodes the pathsi and the announced pathsi announced" by them" can by themnodes be assignedexpressed. can andbenodes be the toassigned expressed.v paths Ai and. Therefore,particular the announcedto v paths Ai. Therefore, particular typeP announced" is by of feasible them relations typeP " is can by as of feasible longthem berelations expressed. as can asP longis. be expressed. as AP particularis. A particular type of relations type of relations ence. between pathsbetween caused paths by caused routing by policies routing are policies" the non-anti-reflexive are" the non-anti-reflexiverelations.relations. For a formal For a formal P !betweenP " P ! pathsbetweenP " caused paths by caused routing by policies routing(3.2)P are policies" theP non-anti-reflexive(3.2) are" the non-anti-reflexiverelations.relations. For a formal For a formal To indicateTo that indicateP " is that a propagatedP " is a propagated version of versionP . In order of P . to In assign orderP toP " assignto vi+Pn," Ptomustvi+!n, P" PmustP ! P " P v P (3.2) (3.2) andUsing rigorousand thisUsing definition rigorous two simple this definition of twoanti concepts simpleand ofnon-anti-reflexiveanti conceptsdifferentand non-anti-reflexive differentrelationsandUsingTo rigorousrelations indicate relationsbetweenand thisUsing definitionTo rigorous two that indicateseerelations thebetween simple this XXx, policies" definition ofis two that a seeanticoncepts herebythe propagated simpleP ofXXx,and policies" is ofthe we anon-anti-reflexiveanti conceptsdifferent hereby differentpropagated just versionofand thein- wenon-anti-reflexive differentrelations different of just version. in-relations In relationsbetween order of P . to seerelations In thebetween assign order XXx, policies to see herebythe" assignto ofXXx, policies thei+ wePn hereby, different" justto ofmustv thein-i+ wen, differentP justmust in- be assignedbe to assignedvi. Therefore, to vi. Therefore,P " is feasibleP " is as feasible long as asP longis. asv P is. P P Finally, if ASSEMBLER neithertroducenodes and a constrainsnodestroduce fair the definition paths and a fair the announced the definition based paths path onannounced by anbasedlength them example on canby nor an themfor exampletroducenodes be the the assignedexpressed. can amount shakeand a fornodestroduce be fair the tothe assignedexpressed. of definition paths A clarity.andi shake of. a particularTherefore, fair paths,the announced to of definition If basedv paths A clarity.i there. particularTherefore, thetype onannounced" isis by If anbased ofan feasible there them examplerelations alternat- typeP on" isis canby anas of an feasiblethemfor long example relationsbe alternat- the expressed. as can as shake longfor beis. theexpressed. of as A clarity. shakeP particularis. of If A clarity. there particular type is If ofan thererelations alternat- type is of an relations alternat- betweenTo indicate pathsbetween caused that pathsP byis caused arouting propagated! by policies routing! version are policies thebetween of non-anti-reflexiveP" are. In pathsthebetween ordernon-anti-reflexive" caused to paths assignrelations. by causedP routing!torelations. byv For policies routing, a!P formalmust Forare policiesthe a formalnon-anti-reflexive" are the non-anti-reflexive" relations.relations. For a formal For a formal ing sequenceingTo sequence of preferenceindicate" of that preference (!P)" andis a composition propagated(!) and compositioning version (! sequenceTo) relationsindicateing of (!PTo sequence of). that preferencecreatedindicate relations In orderP " ofis among that toa preference created (propagated"! assignP)" andaisi set+ among aPn composition propagated(" of!to version) paths andv ai+ setn composition, of ofP version ( paths!Pmust). relations In order of (!P). to created relations In assign order among toP created" assignto v ai set+ amongPn," ofPto pathsmustv ai+ setn, ofP pathsmust stable path assignment displayedandUsing in rigorous Fig.6.2band thisUsing definition rigorous two can simple this definition ofbe twoanti concepts achieved. simpleand of non-anti-reflexiveanti conceptsdifferentand non-anti-reflexive relationsdifferentUsingrelations betweenrelations thisUsing tworelations see the between simple this XXx, policies two seeanti conceptshereby the simple XXx, of policies the wenon-anti-reflexiveanti conceptsherebydifferent different just of in-the wenon-anti-reflexive relationsdifferent different just in- betweenrelations the between policies the of policies the different of the different Pbe1,P assigned2,...,PPbe1,P to assignednv2,Q,...,Pi.1 Therefore,,Q to2n,...,Qv,Qi.1 Therefore,,QPn"2,is,...,Q we feasible sayPn" that,is we as feasible the longP sayandbe relation,P assigned that asrigorous as,...,PP the longandPbeis. is,P relationto assigned non-anti-reflexivedefinitionrigorous asv,Q,...,PiP. Therefore,,Qis. is to definition non-anti-reflexive,...,Q ofv,Q. Therefore,,QP "and relation,is,...,Q of we feasible sayP ifand" that, relationis the we as feasible the long say relation if that as the asPrelations the longis. is relation non-anti-reflexive as Prelations seeis. is XXx, non-anti-reflexive see hereby XXx, relation we hereby just if relation the in- we just if thein- troducenodes and atroducenodes fair the definition paths and a fair the announced definition based paths on announced by based anthem example on canby an forthem examplenodes be1 the expressed.2 can shakeand fornodes be1 the the expressed.ofn2 pathsA clarity. shakeand1 particular the announced2 ofn If paths Aclarity.i there1 particular typen announced2 is byIf ofan there them relationsalternat- typen is canby anof them bealternat-relations expressed. can be expressed. A particular A particular type of relations type of relations sequencesequence is cyclic, is cyclic, sequencetroduce atroducesequence is fair cyclic, definition a isfair cyclic, definition based on based an example on an for example the shake for the of clarity. shake of If clarity. there is If an there alternat- is an alternat- between pathsbetween caused paths by caused routing! by policies routing! are policies thebetween"non-anti-reflexive are pathsthebetween"non-anti-reflexive caused pathsrelations. by caused routing! relations. by For policies routing a! formal Forare policies the a formal"non-anti-reflexive are the "non-anti-reflexiverelations.relations. For a formal For a formal ingUsing sequenceing this sequenceUsing of two preference simple this of two preference concepts (! simple) and compositionconceptsdifferent (!) and compositionrelationsdifferenting (!Using sequence) relations betweenrelationsing this (! sequenceUsing of two) preferencecreatedrelations the between simple this policies oftwo among preference created conceptsthe (! simple of) policies anda the amongset compositionconceptsdifferent (different of!) of paths and a the set compositionrelationsdifferentdifferent of ( paths!) relations betweenrelations (!) createdrelations the between policies among created the of policies a the amongsetdifferent of of paths a the setdifferent of paths and rigorousand definition rigorous definition of anti andv of2vnon-anti-reflexiveanti7...v0andvnon-anti-reflexive5...v0 relations seerelations XXx, see hereby XXx,v2 wev7... hereby justv0 in- we just in- P ,P ,...,PP ,P,Q,...,P,Q ",...,Q,Q ,Q! ,",...,Q we" say! that,! we" the sayand" relation that rigorous! ! theand is" relation non-anti-reflexivedefinition rigorous" ! is! non-anti-reflexive"definition" of anti!!and relation" of "non-anti-reflexiveanti! ifand relation! thevnon-anti-reflexive"5..."v0 if! therelations! " " seerelations! ! XXx," see hereby! XXx, we hereby just in- we just in- nodes1 2 andnodes1 then2 paths and1 P the1 announced2!n pathsQ1n vP!1n announced:2P! bynQ! themn...!n P! canbyn...! themPnodes be...!1,P expressed.Q! can22,...,P and...v!2Pnodes bevP1! the3...,P2 expressed.nvQ!2,Q0 paths A2,...,P andQ!1 particular1,QPP! the1 announced22!nP,...,Q!,Q paths1 AQQ1n particular1,QvP type!!1n announced:2P,!,...,QP by wen of1Q! themn sayrelations... type!n(3.3) that,P! canby wen of...! the them say berelations...! relation expressed. that(3.3)Q! can2 ...v! the2 bevP is!3... relation2 expressed. non-anti-reflexivevQ!0 A2Q! particular1 P! is2 non-anti-reflexiveP!1 AQ particular1 type! relationP of1 relations type if(3.3) relation the of relations if(3.3) the troduce atroduce fair definition a fair definition based onv basedan2v8... examplev on0 anv for6... exampletroducev the0 shake a fortroduce fair the of definition clarity. shake a fair of definition If based clarity. there on isv If basedan2 anv there8... example alternat-v on0 is an anv for example alternat-v the shake for the of clarity. shake of If clarity. there is If an there alternat- is an alternat- v1 sequencebetween pathssequencebetween is cyclic, caused paths isv cyclic,1 by caused routing by policies routing are policies thesequencebetweennon-anti-reflexive are pathssequencethe is cyclic,non-anti-reflexive caused is cyclic,relations. by routingrelations. For policies a formal Forare6... the a formal0non-anti-reflexivenon-anti-reflexiverelations. For a formal v2 ! ! " between" v2 paths caused! by routing! policies" are the " relations. For a formal anding sequence rigorousanding definitionsequence rigorousof preference definition of of preferenceanti (!)and and of non-anti-reflexiveanti composition (!)and andnon-anti-reflexive compositioning (! sequence) relationsing (! sequence of) preferencecreatedsee relations XXx, of among preference createdsee hereby (! XXx,) anda setamong we composition hereby ( of! just) paths and a in- set we composition of just ( paths!) in- relations (!) createdrelations among created a setamong of paths a set of paths v3v9...v0 v7...andv0 rigorousand definition rigorous definition of anti andv of3vnon-anti-reflexiveanti9...v0and vnon-anti-reflexive7...v0 relations seerelations XXx, see hereby XXx, we hereby just in- we just in- P1,P2,...,PP1,Pn2,Q,...,P1,Q2"n,...,Q,Q1,Q!n2",,...,Q we" say!n that,! we" the sayP"1 relation,P that!2,...,P! theP" is,P relation non-anti-reflexiven",Q,...,P!1,Q! is2" non-anti-reflexive,...,Q",Q ,Q!!n relation",,...,Q we" say! if that, relation! the we" the say" relation if that! the! the" is relation non-anti-reflexive" ! ! is non-anti-reflexive" ! relation if relation the if the troduceThe conditionatroduce fairThe definition acondition to fairP guarantee1 ! definition basedQn toPv!12guarantee on: existenceP!n basedanQ!n example...! on existence ofP!n an multipath...! for exampletroduce...! theThe ofQ!2 shakemultipath...v solution! conditiona for3troducev fairP!14...2 theThe ofQv! definition202 clarity. shakeisQsolution! acondition to1 that fairPP! guarantee12 of!Pnno! definition If based1Q clarity.Q isthere subsetn1 to1 thatPv!!12guarantee on: existence2 isP!P no Ifnof basedan1 anQ! there subsetthen example alternat-...! onpathsn existence(3.3) ofP is!n anof multipathan...! forthe example alternat-...! the paths(3.3) ofQ!2 shakemultipath...v solution! for3vP!4...2 the ofQv!02 clarity. shakeisQ solution!1 thatP!2 ofP no! If1 clarity.Q isthere subset1 that! isP no Ifof1 an there subsetthe alternat- paths(3.3) is of an the alternat- paths(3.3) v3v10...v0 v8...v0 v3v10...v0 v8...v0 propagatedsequencepropagatedsequence is throughout cyclic, is throughout cyclic, the network! the creates network! a non-anti-reflexive createspropagatedsequence" a non-anti-reflexivepropagatedsequence is throughout cyclic," relation is throughout cyclic, the among relation network! them,the among creates network! caused them, a non-anti-reflexive creates caused" a non-anti-reflexive" relation among relation them, among caused them, caused v6 ing sequenceing sequence of preference of preference (!) and compositionv (!6) and compositioning (! sequence) relationsing (! sequence of) preferencecreated relations of among preference created (!) anda setamong composition ( of!) paths and a set composition of ( paths!) relations (!) createdrelations among created a setamong of paths a set of paths v7P ,P ,...,P ,Q ,Q ,...,Q vv79...v0 v9...v0 v5 1 2 P1,Pn2,...,P1 2n",Q1,Qv!n:2,",...,Qv we5v" say!vn that,! we" the sayP1" relation,P that!2,...,Pv! theP v1 is",P relation non-anti-reflexivenv"2,Q,...,P!1,Q is!2 non-anti-reflexiven",...,Q",Q1,Q!!n2 relation,",...,Q we" say!n if that, relation! the we" the say" relation if that! the! the is" relation non-anti-reflexive" ! is! non-anti-reflexive" ! relation if relation the if the The conditionThe condition toP guarantee1 ! Q tonP! guarantee31 existenceP!n4Q!11...n ...! existence0 ofP!n multipath...! ...!The ofQ!2 multipath... solution condition!1 P!2...2TheQ!02 isQ solution condition! to1 thatPP! guarantee12!P no!1Q isQ subset ton that1vP! guarantee3!1: existenceP!v noP ofn41Qv! subset the11...n ...!v pathsexistence(3.3)0 ofP! ofn multipath...! the...! paths of(3.3)Q!2 multipathv... solution!1 vP!2...2 Qv!02 isQ solution!1 thatP!2 P no!1 isQ subset that1 ! noP of1 subset the paths(3.3) of the paths(3.3) sequencev8 sequence is cyclic, is cyclic, v10...sequencev8v0 sequence is cyclic, is cyclic, v10...v0 v0 propagatedpropagated throughout throughout the network the creates networkv0 a non-anti-reflexivecreatespropagated a non-anti-reflexivepropagated throughout relation throughout the among relation network thethem, among creates network caused them, a non-anti-reflexivecreates caused a non-anti-reflexive relation among relation them, among caused them, caused P " Q v!4: P"v1v"5...... v!0 ! v..."11..."v0Q! ...!v2...P"v0" Q!P! "P"Q v!4!: P"v1v"5...... v!0 ! v..."11..."v0Q! ...!v2...P"v0" Q! ! P" ! v11 The conditionThe condition to1 guarantee! n toP!1 guarantee existencev!11nQ!n ! existence ofP!n multipath! ...!The of!2 multipath... solution! condition!2TheQ!2 is solution! condition to1 thatP!1 guarantee2!! no1 Q is subsetn to1 thatP!!1 guarantee existence!P non of1Q! subsetthen ! paths existence(3.3) ofP!n of multipath! the...! paths(3.3) of!2 multipath... solution! !2 Q!2 is solution!1 thatP!2 ! no1 Q is subset1 that! P no of1 subsetthe paths(3.3) of the paths(3.3) v9 v9 v10 propagatedpropagated throughout throughout the network the creates networkv10 a non-anti-reflexivecreatespropagated a non-anti-reflexivepropagated throughout relation throughout the among relation network thethem, among creates network caused them, a non-anti-reflexivecreates caused a non-anti-reflexive relation among relation them, among caused them, caused v4 The conditionThe condition to guaranteev4 to guarantee existence existence of multipathThe of multipath solution conditionThe is solution condition to that guarantee no is subset to that guarantee existence no of subsetthe paths existence of of multipath the paths of multipath solution is solution that no is subset that no of subsetthe paths of the paths propagatedpropagated throughout throughout the network the creates network a non-anti-reflexivecreatespropagated a non-anti-reflexivepropagated throughout relation throughout the among relation network thethem, among creates network caused them, a non-anti-reflexivecreates caused a non-anti-reflexive relation among relation them, among caused them, caused v3 v3 (a) 2-Best Equal Path Length (b) Multiple Unipath Solutions

Figure 6.2: Examples of the effect of relaxing the maximum number of paths

Example 2 In this second example we want to show that a particular configuration for which there is unipath solution but no multipath. Nodes v1 and v2 are running ASSEMBLER 17 3.4. Stability Analysis 17 3.4. Stability Analysis some unipath configuration guidelines must be slightly modified because they no longer can be directly used in multipath scenarios. some unipath configuration guidelines must be slightly modified because they no longer can be directlyThe stability used in proof multipath is presented scenarios. after the guidelines and it is structured as follows: after the analytical model of the protocol is presented, the synchronous convergence of the proto- col forThe any stability of its proofflavours, is presented in absence after of conflicting the guidelines routing and policies it is structured between as ASes follows: is shown. after theIf convergence analytical model is not of achieved, the protocol then is it presented, is shown that the thesynchronous conflictingconvergence relation among of the policies proto- actuallycol for any exists. of its Afterwards, flavours, in it absence is proven of that conflicting if nodes routing follow policies the guidelines between presented ASes is shown.in Sec- tionIf convergence XXx, it is is not not possible achieved, to have then conflictingit is shown policies that the andconflicting the protocol relation should among be policies able to findactually a stable exists. solution. Afterwards, In order it is to proven complete that the if nodes stability follow proof, the another guidelines demonstration, presented in which Sec- reliestion XXx, on the itGeneral is not possible Asynchronous to have Convergenceconflicting policies Theorem andXXx, the protocol is presented should such be that able the to findconvergence a stable solution. of the real In (asynchronous) order to complete protocol the stability is guaranteed, proof, another as well. demonstration, which relies on the General Asynchronous Convergence Theorem XXx, is presented such that the convergence of the real (asynchronous) protocol is guaranteed, as well. 3.4.1 On Dispute Wheels in Unipath and Multipath Scenarios

The3.4.1 goal On of Dispute this section Wheels is to introduce in Unipath a discussion and Multipath about the impact Scenarios of multipath solu- tions in the stability of policy-based routing protocols. Before discussing the variety of solu- tionsThe and goal instabilities of this section in unipath is to and introduce multipath a discussion and the relationships about the impact between of them,multipath some solu- no- tationtions in is the introduced stability hereby. of policy-based The notation routing comes protocols. from the Before framework discussing defined the in variety XXxChikin. of solu- tions and instabilities in unipath and multipath and the relationships between them, some no- tationIn is policy-based introduced hereby.scenarios The where notation a path comes vector from protocol the framework is running, defined paths inpropagate XXxChikin. from one node to another as they are chosen in the ranking procedure as most preferred and an- nouncedIn policy-based to the routers scenarios with peering where sessions. a path vector When protocol a path P is running,= vi,vi paths1, . . . , propagate v0 is preferred from − oneover node a path toQ another= vi,v asj,v theyj 1 are, . . . chosen , v0 according in the ranking to the preferences procedure as of mostvi, that preferred relationship and an- of − preferencenounced tocan the be routers denoted with like, peering sessions. When a path P = vi,vi 1, . . . , v0 is preferred − over a path Q = vi,vj,vj 1, . . . , v0 according! to the preferences of vi, that relationship of − P ! Q (3.1) preference can be denoted like, P ! Q (3.1) Afterwards, P is announced and it is chosen as most preferred by an arbitrary num- ber of nodes vi+n,vi+n 1, . . . , vi+1 in the network. The assigned path to vi+n is P " = − vi+nAfterwards,,vi+n 1, . . .P , visi+1 announced,vi, . . . , v0 and=( itvi+ isn,v choseni+n 1 as, . . most . , vi+1 preferred)P and its by relationship an arbitrary of num-com- − − positionber of nodeswith Pvi+cann,v bei+n then1, expressed. . . , vi+1 in like, the network. The assigned path to vi+n is P " = − vi+n,vi+n 1, . . . , vi+1,vi, . . . , v0 =(vi+n,vi+n 1, . . . , vi+1)P and its relationship of com- − − " position with P can be then expressed like,P ! P " (3.2)

" P ! P " (3.2) To indicate that P " is a propagated version of P . In order to assign P " to vi+n, P must be assigned to vi. Therefore, P " is feasible as long as P is. To indicate that P " is a propagated version of P . In order to assign P " to vi+n, P must be assignedUsing this to v twoi. Therefore, simple conceptsP " is feasible different as longrelations as P betweenis. the policies of the different nodes and the paths announced by them can be expressed. A particular type of relations betweenUsing paths this two caused simple by routing concepts policies different are relations the non-anti-reflexive between the policiesrelations. of For the adifferent formal nodesand rigorous and the definition paths announced of anti and bynon-anti-reflexive them can be expressed.relations A see particular XXx, hereby type of we relations just in- betweentroduce a paths fair definition caused by based routing on anpolicies example are for the thenon-anti-reflexive shake of clarity.relations. If there33 is For an a alternat- formal anding sequence rigorous definition of preference of anti (!)and andnon-anti-reflexive composition (!") relationsStability created see Analysis XXx, among hereby a set we of just paths in- troduce a fair definition based on an example for the shake of clarity. If there is an alternat- P1,P2,...,Pn,Q1,Q2,...,Qn, we say that the relation is non-anti-reflexive relation if the sequenceing sequence is cyclic, of preference (!) and composition (!") relations created among a set of paths P1,Pv21 ,...,Pn,Q1,Q2,...,Qv2 n, we say that the relation is non-anti-reflexive relation if the v3 " ! v1:" ! ... " ! " ! sequence is cyclic, P1 ! Qn ! Pn !v2...v3 ! ... ! Q2 ! P2 ! Q1 ! P1 (3.3)

v3 " ! v2:" ! ... " ! " ! P1 ! Qn ! Pn !v1...v3 ! ... ! Q2 ! P2 ! Q1 ! P1 (3.3) The condition to guarantee existence of multipath solution is that no subset of the paths v3 propagated throughout the network creates a non-anti-reflexive relation among them, caused The condition to guarantee existence of multipath solution is that no subset of the paths propagatedFigure 6.3: Scenario throughout with unipath the network but not multipath creates asolution. non-anti-reflexive relation among them, caused to select maximum 2 paths with a AS path length difference of one. Assume that, v1 and v2 give the same preference to their peering connection than to the connection to the customer that is multihomed to them. The policies in that case can be expressed as in the table on the right side of Fig.6.3. The reflexive relation is in this case,

J C J C v1v0  v2{v1, v0}  v2v0  v1{v2, v0}  v1v0 (6.4)

The reflexive relation is created due to the fact that both nodes prefer the aggregated of the di- rect and indirect paths, to only the direct path. Using equal-length multipath there is solution, in which ASSEMBLER assign only direct paths to each node.

Those two examples lead to the conclusion that the propagation of additional paths can either stabilize a network running ASSEMBLER or activate a reflexive relation. Therefore, it is necessary to analyze the stability of ASSEMBLER, since the stability results inferred for BGP do not apply.

6.0.2 Synchronous Model of Path ASSEMBLER

Let G = hV,Ei be a topology graph, where V is the set of vertex and E is the set of edges of the graph and v0 ∈ V denotes the origin of a prefix advertisement. Let P(vi, v0) be the set of reachable paths between vi and v0 in G, i.e. any path that can be physically constructed from vi to v0. Now P(P(vi, v0)) is defined as the super-set of the subsets in P(vi, v0), so to speak, every element Φvi ∈ P(P(vi, v0)) is an arbitrary collection of elements in P(vi, v0). A multipath assignment over G is defined as,

Φ = {Φvi ∈ P(P(vi, v0)), ∀vi ∈ V } (6.5)

Additionally, a partial multipath assignment is defined as,

Φ = {Φvi ∈ P(P(vi, v0) ∪ ∅), ∀vi ∈ V } (6.6)

that is, for some nodes the assignment may be empty. The protocol is modelled as a fixed- point iteration of a distributed synchronous Bellman-Ford mapping F(Φ) over the multipath assignment Φ.

The protocol starts growing from the initial iteration, in which the origin v0 announces the path containing itself to its neighbors, and it increases the path assignment until reaching an assignment Φ that verifies the fixed-point equation,

Φ = F(Φ) (6.7) 34 Chapter 6 Stability Analysis

In the ASSEMBLER mapping, nodes exchange announcements constructed as depicted in Section 4.2. The most recent advertisement received by node vi from node vj at iteration k is denoted as adv(vj → vi)[k] (6.8)

Hereafter, we use indistinctly the terms path and assembled path, (a path is a particular case of an assembled path in which only one element is assembled). The set of available paths for a node vi at iteration k is the set,

Φvi[k] = {adv(vj → vi)[k 1], ∀vj ∈ peers(vi)} (6.9) −

The ASSEMBLER mapping is defined locally at node vi as the operation of selecting the

K best assemblable paths in Φvi[k] ∈ P(P(vi, v0)) according to the K-BESTRO algorithm (Section 4.1) and propagate the ASSEMBLER advertisement corresponding to them. The analysis assumes that each node defines a different K-BESTRO configurations and the fol-

lowing notation is used to differentiate among K-BESTRO procedures, so that Fvi (Φvi[k]) denotes the procedure at node vi. The mapping can be also defined as the set of local opera- tions at each vertex during the kth iteration like,  F(Φ[k]) = Fvi (Φvi[k]), vi ∈ V (6.10) Before advertisement the candidate paths are ranked. The rank value of a path is denoted like

λ(θ) and all the paths in the selected set has a ranking value of λmax(Φvi[k]) .

Given the set of advertisements in Φvi and the ranking procedure at vi, Fvi (which es- tablishes ranking values λ, its maximum for a given set of announcements λmax), the set of most preferred paths at iteration k can be defined as,

βvi[k] = Fvi (Φvi[k]) (6.11) where,

βvi[k] = {θ ∈ Φvi[k] / λ(θ) = λmax(Φvi[k])} (6.12)

Each node vi, its candidate set Φvi is updated with the advertisements adv(vj → vi)[k] ≡

βvi,[k] from the peering routers, such that the updated overall path assignment is defined like

Φ[k+1] = {Φvi,[k+1], ∀vi ∈ V } (6.13)

and according to Eq.6.10 we get to the iterative equation

Φ[k+1] = F(Φ[k]) (6.14)

As the synchronous execution of the mapping goes on, the paths in Φvi[k] change dynami- cally. Before expressing those dynamic changes and the evolution of the mapping in a formal way, first we have to define the concepts of feasible and stabilized set of paths.

The concept of feasible multipath assignment is rather intuitive if we define the set of all possible multipath assignments as the following Cartesian product (including partial assign- ments), Y χ = P(P(vi, v0) ∪ ∅) (6.15) vi V ∈ 35 Stability Analysis

Therefore a multipath assignment Φ = (Φv0 , Φv1 ,..., Φvn ) ∈ χ provides to each vertex vi a set of paths Φvi to reach the origin. In addition, for each vertex in V , we define the following,

Definition 1 Given two vertex vi, vj ∈ V such that (vi vj) ∈ E, the assignment Φvi is said to be consistent with Φvj if ∀ρ ∈ Φvi of the form ρ = (vi vj)θ, it holds that θ ∈ Φvj and vi 6∈ θ (i.e. to ensure loop-freeness).

It seems clear from the definition of χ that not all the components of Φ ∈ χ are nec- essarily consistent with each other. Since our protocol handles only local information, the paths that it is able to construct must be consistent for all the vertex in the path. Hence, the definition of a feasible multipath assignment for our protocol can be expressed like,

Definition 2 A multipath assignment Φ ∈ χ is said to be feasible if ∀vi, vj ∈ V and

(vi vj) ∈ E then Φvi is consistent with Φvj .

Before defining the concept of stabilized multipath assignment, the following relations between multipath feasible assignments must be defined,

Definition 3 Let Φ, Φ0 ∈ χ be two feasible partial multipath assignments then, Φ0 contains Φ Φ ⊆ Φ ⊆ ∀ ∈ Φ Φ Φ ⊆ Φ , i.e. 0, if Φvi Φv0 i vi V and ( 0, if 0 and Φvi ( Φv0 i for some vi ∈ V .

Definition 4 Given a partial feasible multipath assignment Φ, the set Ψ(Φ) defined as,

Ψ(Φ) = {Φ0 ∈ χ / Φ ⊆ Φ0} (6.16) is the set of feasible assignments which contain the path assignment Φ.

Definition 5 An assignment Θ[k] is said to be stabilized if for all the sets of feasible sets containing Θ[k], i.e. ∀Φ ∈ Ψ(Θ[k]), it holds that

Φ ⊇ Θ[k] implies F(Φ) ⊇ Θ[k] (6.17)

The latter means that for any feasible assignment Φ containing Θ[k], an iteration of AS- SEMBLER over the assignment Φ does not remove any path in Θ[k]. Therefore, any path θ ∈ Θ[k] is part of the fixed-point solution of Eq.6.7 for the function F and its ranking value verifies that

λ(θ) = λmax(Ψ(Θ[m])) ∀m ≥ k (6.18)

th Definition 6 Let C[k] be the set of converged nodes at the k iteration. Any node vi ∈ C[k] verifies that the set βvi[k] belongs to the stabilized assignment at iteration k, what means that vi converged at iteration k or before. Being m ≤ k the iteration at which vi converged, then

∀n > m, βvi[n] = βvi[m] and, therefore adv(vi → w)[n] = adv(vi → w)[m]. 36 Chapter 6 Stability Analysis

Definition 7 Let D[k] ⊆ V − C[k] be the set of nodes which are direct peers of converged nodes, then, ∀v ∈ D[k], ∃u ∈ C[k] and e = (v u) ∈ E.

6.0.3 Path ASSEMBLER Convergence

In this section, the anti-reflexive property of routing policies, the notation and the protocol model depicted in the previous section are combined to asses the stability of ASSEMBLER. The analysis begins with the progress condition of the protocol. We show that if the progress condition does not hold for some iteration of the protocol, then the policies create a reflexive relation among the advertised paths. Afterwards, we prove that the progress condition implies that at each iteration the protocol gets closer to the fixed-point solution. Finally, the safety of the protocol is shown at the final theorem stated in this section.

Lemma 1 (Progress Condition) Let S be the set of routing policies of nodes in G. If any relation among policies in S is anti-reflexive and the current overall stabilized assignment Θ[k] is not a fixed-point of the mapping, then there is an assignment Θ[k+1] such that,

1. Θ[k+1] ) Θ[k]

2. Θ[k+1] is also stabilized

3. ∀Φ ∈ Ψ(Θ[k]), then F(Φ) ⊇ Θ[k+1]

Proof: If Θ[k] is stabilized it means that F(Θ[k]) ⊇ Θ[k] and Fv0 (Θ[k]) = {v0}, k ≥ 0. In order to increase the multipath stabilized assignment there must be at least one node v ∈ D[k] peer of u ∈ C[k] such that,

1. By definition 6 u has a stabilized set Θu[k], then ∀θ ∈ Θu[k] it holds θ ∈ Θ[k]

2. Given ρ = (v u) ∈ E, α = adv(u → v)[k], it holds that

λ(ρα) = λmax(Ψ(Θv[m])) ∀m ≥ k (6.19)

hence ρα ∈ Θvi[k+1] (i.e. ρα is stabilized since no path with higher rank will replace it in later iterations).

3. At iteration k, ρα ∈ Θ[k+1] = F(Θ[k]) and ρα 6∈ Θ[k]

If such a node v exists then the proof of the lemma is completed since by construction F(Θ[k]) ) Θ[k]. By definition 5 and Eq.6.19, ρα will not be removed in further iterations, therefore F(Θ[k]) is stabilized.

Now we show that if that node v ∈ D[k] does not exist then the anti-reflexivity property does not hold over the policies in S. If v does not exists then no node v1 ∈ D[k] is be able to find a direct path Γ1 constructed like above Γ1 = ρα for any peer node u ∈ C[k] and being λ(Γ1) = λmax(Ψ(Θv[m])) ∀m ≥ k. Therefore, v1 prefers more a path ∆1 that is not through a converged peer. Using the preference and composition relations, we can express the policy relation between Γ1 and ∆1 like,

C ∆1  Γ1 (6.20) 37 Stability Analysis

Then if ∆1 is not at one hop to a converged vertex, then ∆1 must come from a propagated version of a direct path of some node v2 ∈ D[m], therefore it can be constructed like ∆1 = Π2Γ2. Path Π2 is an arbitrary path passing through nodes in V − C[m] and Γ2 = ρ0α0, with ρ0 = (v2 u0) ∈ E and α0 = adv(u0 → v2)[m], is a direct path of v2. In terms of policy relations the latter can be expressed like,

J C Γ2  ∆1  Γ1 (6.21)

Using the same reasoning, v2 is not choosing any direct path Γ2, otherwise the path ∆1 would become stabilized and the stabilized paths assignment would grow. Therefore the set

βv2[m] is formed by at least one path ∆2 which is not direct and goes through a direct path announced by some node v3 ∈ D[m]. The same procedures repeats for v3 and we get to the relation, J C J C Γ3  ∆2  Γ2  ∆1  Γ1 (6.22)

The relation keeps repeating for every element in D[m] until it hits v1 again, producing a circular relationship of policies that cannot be fulfilled simultaneously,

J C C J C J C Γ1  ∆n  ...  Γ3  ∆2  Γ2  ∆1  Γ1 (6.23)

The latter relation implies that Γ1 is less preferred than a path which is composed by an arbitrary path and Γ1, which is a contradiction. The latter completes the proof by showing that if the protocol gets stuck, then a reflexive relation exists among the policy relations.

Lemma 2 If a path θ = vivi 1 . . . v0 does not appear infinitely often in the multipath set − βvi of vi, then there is an iteration k after which any path of the form ρθ disappears from the network.

Proof: Given a vertex vi, θ = ρ0θ0 with ρ0 = vivi 1 . . . vj+1vj and θ0 = vjvj 1 . . . v1v0, − − if θ = ρ0θ0 does not appear in βvi[m] ∀m ≥ k it means that there is at least one node vj 0 ≤ j ≤ i, for which there is a path θ00 ∈ P(vj, v0) such that λ(θ00) > λ(θ0), therefore θ0 is not part of adv(vj → vj+1)[k] after iteration k. At iteration k + 1 the nodes w ∈ peers(vj) cannot use the path θ0 any longer. The process repeats at each iteration along the next-hop in the path ρ0 until ρ0θ0 disappears. Thus, vi cannot announce θ any longer and eventually ρθ also disappears.

Lemma 3 The successive iterations of the ASSEMBLER mapping F(Θ[0]), F(Θ[1]), ... , F(Θ[k]), over the stabilized partial assignments reduce at each step the set of feasible path assignments Ψ, i.e. Ψ(Θ[0]) ) Ψ(Θ[1]) ) ··· ) Ψ(Θ[k]).

Proof: Since we are using a synchronous model, we can assume that changes made by the mapping at vi are propagated to the peers of vi in the next iteration. At iteration zero, the set of feasible paths is equal to the super-set Ψ(Θ[0]) whose elements are any feasible set Φ defined by Eq. 6.16. Since the mapping evolves by repeatedly applying Lemma 1, then all those paths with lower rank than stabilized paths in the current iteration are not announced anymore. Then, by Lemma 2 lower ranked paths and those constructed upon them eventually disappear. In other words, following iterations of the mapping will not propagate them throughout the network and they are not feasible paths anymore. Those paths are removed from the set of feasible sets at that iteration, proving Lemma 3. 38 Chapter 6 Stability Analysis

Theorem 1 (Safety) Given a network graph G = hV,Ei, given the set of policies S defined by each vertex in V , a synchronous distributed Bellman-Ford mapping F(Θ[k]) iterating over the path assignment Θ[k] which is initially defined as, ( {v0}, i = 0 Θv [0] = (6.24) i ∅, i =6 0

If every policy relation over S is anti-reflexive then the mapping is able to grow the path assignment at each iteration until the fixed-point of the following equation is reached at some iteration m, Θ[m] = F(Θ[m]) (6.25)

Thus, it can be stated that in absence of reflexive policies the protocol is able to syn- chronously converge.

Proof: By applying Lemma 1 at each iteration, in absence of conflicting policy relations, the mapping is always able to increase the path assignment with at least one path such that the new assignment F(Θ[k]) ) Θ[k] is also stabilized. By Lemma 3, as the mapping is consolidating stabilized paths at each vertex, the set of feasible paths Ψ is decreasing, until the highest ranked paths feasible at each node are announced. Hence, there is one iteration

k at which the only feasible set of paths at a certain node vi is the set Φvi[k] ∈ P(P(vi, v0))

formed by elements that verify the equation λ(θ) = λmax(Ψ(Φvi[m])), ∀θ ∈ Φvi[k] and ∀m ≥ k. Since the mapping does not remove paths from a stabilized assignment and it cannot find higher ranked paths at any node, the next iteration k + 1 will have as outcome the same path assignment. Therefore, we can say that the fixed-point has been hit at iteration k.

6.0.4 Asynchronous Convergence

Despite using a reliable transport protocol, which guarantees ordered message delivery among nodes, the execution of our protocol is not free from communication delays since data synchronization is not enforced between peers. Therefore it may happen that the set of paths

used by a node vi to compute its multipath set at time t (i.e. Φvi[t])),

Φvi[t+1] = Fvi (Φvi )[t] (6.26)

it is a distorted version due to delay in the propagation of some of the announcements

coming from peers. A distortion due to a delay of t − τvi,vj (t) between peers vi and vj, with

0 ≤ τvi,vj (t) ≤ t, makes vi perceive,

Φ = F (Φ ) (6.27) vi[t+1] vi vi [τvi,vj (t)]

Since the iteration over the fixed point is now distorted by t − τvi,vj (t) we cannot ensure convergence of the fixed-point iteration anymore. According to the general results in [2], it is possible to ensure convergence for a totally asynchronous distributed fixed-point iteration if,

1. The propagation of information happens infinitely often. In other words, it can be assumed that after a certain time t0 > t all the announcements adv(vj → vi)[t] have been propagated and renewed at peer nodes. 39 Stability Analysis

2. Synchronous Condition: The protocol creates at each iteration k = 0, 1, . . . , m, a se- quence of sets, X[0] ) X[1] ) ··· ) X[n 1] ) X[n] ) ... (6.28) − and it holds F(x) ∈ X[k+1], ∀x ∈ X[k] (6.29)

3. Box Condition: For each iteration k = 0, 1, . . . , m and each node vi, i = 0, 1, . . . , n,

there exist sets of elements Xvi[k] such that the set of elements X[k] can be expressed as the Cartesian product, Y X[k] = Xvi[k] (6.30) i

The box condition implies that different elements in Xvi[k] can be exchanged without affecting the final result of the iteration, so to speak, the order in which the Bellman- Ford mapping allocates paths does not affect to the evolution of the mapping.

If those three condition hold, the following theorem can be proven,

Theorem 2 (Asynchronous Convergence) Given a network graph G = hV,Ei with n nodes, the set of policies S defined by each node and a distributed ASSEMBLER mapping F(Θ[k]) iterating over the path assignment Θ[k], if every policy relation over S is anti- reflexive then the mapping is able to asynchronously converge to a multipath assignment of paths over G.

Proof The condition (1) is guaranteed since ASSEMBLER uses a reliable transport proto- col to exchange the information and every node advertises its neighbors with every change in the selected multipath set. Condition (2) is guaranteed by Theorem 1. In addition, replac- ing X[k] by Ψ(Θ[k]), the set of feasible sets containing Θ[k], both Eq.6.28 and 6.29 can be rewritten as follows. Lemma 3 proves that the protocol creates the sequence,

Ψ(Θ[0]) ) Ψ(Θ[1]) ) ··· ) Ψ(Θ[k]) ... (6.31)

which can be easily identified with the sequence in Eq.6.28. Moreover, it can be stated by definition 4,

Θ[k] ⊆ Φ, ∀Φ ∈ Ψ(Θ[k]) (6.32) and by definition 5, applying an iteration of the algorithm on both sides, if Θ[k] is stabilized then, F(Θ[k]) ⊆ F(Φ) ⇒ Θ[k+1] ⊆ F(Φ) (6.33) again, by definition 4, F(Φ) ∈ Ψ(Θ[k+1]), ∀Φ ∈ Ψ(Θ[k]) (6.34) so that Eq.6.29 can be rewritten as well identifying X[k] ≡ Ψ(Θ[k]) and Φ ≡ x.

Finally, in order to complete the proof of Theorem 2, the box condition must be verified.

The stabilized assignment can be also expressed like Θ[k] = (Θv0[k], Θv1[k],..., Θvn[k]). Lemma 1 guarantees that at least one node vi that increases its feasible assignment, so that Φ Θ Φ there is a set v0 i[k] such that vi[k] ( v0 i[k]. As there can be more than one, the following super-set Ψvi[k] can be defined as the set of feasible assignments that contain the stabilized 40 Chapter 6 Stability Analysis

assignment Θ Φ(p) , i.e. Ψ = {Φ(p) , ∀p}. Provided that all the assignments vi[k] ( vi[k] vi[k] vi[k]

within the super-sets, Ψvi[k], ∀i, are feasible, therefore the set of feasible sets containing Θ[k], can be rewritten as the following cartesian product,

Ψ(Θ[k]) = Ψv0[k] × Ψv1[k] × · · · × Ψvn[k] (6.35)

which can be arranged to resemble Eq.6.30 as follows, Y Ψ(Θ[k]) = Ψvi[k] (6.36) 0 i n ≤ ≤

and the box condition is proven. Provided that the three conditions are verified for the protocol, by the General Asynchronous Convergence Theorem (Proposition 2.1 in [2]) it can be stated that the protocol is able to converge asynchronously.

6.0.5 Stable Multipath Policy Guidelines

The work in [10] studies how to define routing policies in the Internet to avoid instabilities inGBP. The resulting guidelines are based on the business relations between ASes. Using the stability condition derived above, the guidelines in [10] can be reformulated for multipath. They can be proven stable if no reflexive relation can be constructed in the network.

Two guidelines are presented. We do not claim that the resulting routing policies are the only policies ASes can follow to achieve stability. Instead, we choose them because an AS needs to know its commercial relation with each neighbors to define its policy and the guidelines cover the most common business relations found on the Internet. The following two assumptions must hold for both guidelines,

Assumption 1 An AS advertises paths coming from its provider only to its customers. Paths coming from peers only to its customers and finally, paths coming from its customers to other customers, peers and providers.

Assumption 2 A customer AS cannot be an indirect provider of one of its direct providers.

Now, we present the guidelines. Basically the proofs fail to construct reflexive relations when the policies are defined according to the guidelines.

Guideline 1 If every AS assigns a higher local preference to the paths received from its customers than to paths received from its peers and they assign higher preference to paths coming from its peers than to paths coming from its providers, then ASSEMBLER is able to converge.

In addition, convergence is possible regardless of the size and characteristics of the paths in the multipath set.

Proof The proof fails to construct the reflexive relation in Eq.6.3. Without loss of generality the proof refer to the scenario in Fig.6.1. First we assume that Γ1 comes from a customer of 41 Stability Analysis

v1, then according to Guideline 1, ∆1 must come from another customer, otherwise it cannot have higher preference. Then ∆1 is of the form ∆1 = Π2Γ2 (in Fig.6.1 Π2 is just a link between v1 and v2 but in general is an arbitrary path). Since v1 is a provider of v2, according to Assumption 1, the latter can only advertise paths from its customers to v1 (notice that if intermediate nodes between v1 and v2 exists the situation is the same). If ∆2 has higher preference than Γ2 and v2 is following the Guideline 1, then it must come from another customer of v2. The same reasoning apply for v3. Now, at v4, the path ∆4 through v1 should be preferred over the path Γ4. That can only happen if ∆4 comes from a customer of v4, however if v1 is a customer of v4 and v4 is in the chain of customers from v1 it means that Assumption 2 is broken.

In the second case, we assume that Γ1 comes from a peer of v1. Therefore, ∆1 must come from either a peer or a customer of v1 (Assumption 1). In both cases, it means that Γ2 and ∆2 come from customers of v2, otherwise they cannot be advertised to a peer or a provider. The chain of customers continue until v4. A reflexive relation would be constructed if v1 is a customer of v4. If v2 is a peer of v1, then ∆1 cannot be advertise to v4 as it is a v1 provider. If v2 is a customer of v1 we are in the previous case.

In the last case, v1 learns Γ1 from a provider, then v2 can by a customer, peer or provider of v1. If v2 is a provider of v1 and Γ2 and ∆2 are advertised to v1 since they come from customers or peers of v2, the chain continues like in the two previous cases. Otherwise, if Γ2 and ∆2 come from providers of v2, then the chain of providers continue to v4. If Γ4 comes from a customer of v4 then according to Guideline 1 ∆4 must come from a customer, however v1 does not advertise its provider v4 with paths from other providers, therefore ∆4 is not announced and the reflexive relation is broken. If Γ4 comes from a peer the situation is the same. Only if Γ4 comes from a provider, ∆4 can be more preferred, therefore if v1 is the provider of v4 then ∆4 is advertised, however in that case v4 becomes the indirect provider of one of its direct providers (through v3 and v2) and Assumption 2 is broken.

Guideline 2 An AS can assign the same local preference to paths coming from a customer or a peer ASes. Yet, stability can be only guaranteed if paths from peers and customers are not selected together in the multipath set.

Proof Again we try to construct a reflexive relation among the propagated paths. The sce- nario in Fig.6.1 is used again. First we assume that Γ1 comes from a customer of v1, then according to Guideline 2, ∆1 can now come either from another customer or a peer, otherwise it cannot have higher preference. Therefore, under Guideline 2, the first and second cases of the proof of Guideline 1 can be combined. Due to Assumption 1, the most complex case is an alternation of peers and customers links, since two consecutive peering links cannot hap- pen. If we follow this alternation from v1 till v4, a reflexive relation can happen only in two cases. First, if v3 and v4 are peers, then Γ4 and ∆4 must come from customers of v4 which implies that Assumption 2 must be broken to create the reflexive relation. Second, if v3 is a provider of v4 then path Γ4 must be then from a customer of v4. Now v1 and v4 can be peers and ∆4 can be still be more preferred than Γ4. However, in that case v4 is again breaking the hierarchy of the network, acting as a particular case of provider defined as peer-provider [10].

So far the proof states the stability under equal preference for peers and customers, how- ever nothing is said about the mixture. The proof is completed with the counter-example. In Fig.6.3 ASSEMBLER oscillates if the policy allows to select a path from a customer and another from a provider at the same time. That is, an AS can choose to send the traffic among 42 Chapter 6 Stability Analysis

paths through customers or a peers, but a router must not mix paths from both types. Notice that the corresponding guideline for BGP (Guideline 5.2, [10]) does not make this distinction because BGP selects only one path. Chapter 7

Implementation of an ASSEMBLER-Capable Router

In this chapter the implementation of a multipath software router running ASSEMBLER as multipath inter-domain protocol is presented. Given that our protocol inherits a large functionality from BGP, the implementation sets off from an implementation of BGP. The implementation is intended as a proof of concept to show that multipath capabilities can be added to border routers introducing little modifications in the software. Furthermore, the implementation is part of a testbed developed within the scope of the Trilogy Project [1] and the integration of multipath routing with the implementation of MPTCP [9].

There are already some available open source implementations of BGP. Some of them are stand-alone BGP daemons like OpenBGPd [3]. Other BGP implementations are part of complete routing suites. Those packages offer the possibility of analyzing the interactions between different routing protocols, which can be a highly interesting characterization. The two most widely referenced open source routing suites (that include BGP) are Quagga [17] (formerly Zebra) and XORP [14]. The routing software package Quagga encloses several routing modules, which can be launched simultaneously. One of the Quagga best strengths is its implementation efficiency (memory consumption of the Quagga BGP daemon is about 20MB at initialization time). Among its drawbacks to be used as base for multipath ex- tensions, there are its scarce documentation for developers, the BGP implementation is not very modular and it uses relies directly in the OS kernel to install and remove routes in the data-forwarding plane, thus in case a special forwarding plane architecture is needed, kernel libraries must be modified directly.

An alternative to Quagga is XORP. The routing project XORP is a truly modular routing package to create software routers. In contrast to Quagga, XORP offers certain flexibility to choose which forwarding plane management interface to use, either the kernel forwarding plane, CLICK [20] and supports distributed forwarding planes. Among XORP disadvantages, it can be found that in exchange for modularity, its code tends to be more inefficient (for in- stance the BGP daemon has a memory footprint of about 100MB at initialization time). De- spite Quagga’s efficiency, XORP has been chosen as the preferred routing suite to introduce the multipath extensions thanks to its modularity, flexibility, integrated support for CLICK and extensive documentation.

The implementation of XORP is running in the control plane of the router and a multi- 44 Chapter 7 Implementation of an ASSEMBLER-Capable Router

path data-fowarding plane has been implemented using CLICK. The data plane is using the CLICK setting presented in [20] and the routing table module has been modified to add mul- tiple next-hops per prefix and use them to balance traffic flows through different next-hops preserving the packet ordering.

The extensions for XORP and CLICK are integrated in virtualized appliances and a testbed is settled to run the experiments. The details of the testbed are presented in the next section. The extensions of XORP and CLICK are presented afterwards. Finally, the section concludes with the results obtained over a small sample topology.

7.1 The Evaluation Testbed

The architecture of the testbed is illustrated in Fig.7.1. The testbed is deployed among several physical PCs, which provide the hardware layer. Each physical host can be devoted to either router emulation or network emulation tasks. A routers emulator is able to multiplex several virtual routers in a physical host. The network emulation is performed in different hosts. All the PCs are connected through Gigabit Ethernet LAN.

Fig.7.1 also depicts the different layers involved. On top of the hardware layer there is the virtualization layer, which has in turn two sublayers, the router virtualization sublayer and the network virtualization sublayer. One physical host provides the functionality of the network virtualization sublayer. That host is running a network emulation software. Each virtual router sends and receives their traffic over a VLAN towards the network emulator. When a packet gets into the emulator, the latter creates the illusion that packets are forwarded across a real network, delaying, dropping, mixing them in queues, etc. Afterwards, the emulator sends the packets back to the destination host using another VLAN.

On the other hand, the router virtualization layer handles the virtual machines that emulate routers and the virtual network interfaces of those routers, which are bridged to the physical network interfaces.

In each virtual machine emulating a router, there is a Linux Debian appliance running CLICK and XORP. CLICK processes and forwards packets at the kernel level and it has been used to implement the multipath Forwarding Information Base (FIB). XORP is used the control plane of the routers and CLICK in the data-forwarding plane. In the control plane the routing protocols are executed. In this testbed only BGP and ASSMBLER are running, although all the protocols in the XORP bundle are supported. The rest of the section describes the multipath extensions for XORP and CLICK. The modifications to the RIB process are described first. Afterwards, multipath extensions to the unipath BGP daemon to implement ASSEMBLER are shown and finally the implementation of a multipath FIB using CLICK is detailed.

7.2 The Control Plane

Four processes from the XORP suite are running at each router. Depending on whether a router is multipath capable or not, either the ASSEMBLER process or the BGP daemon is running respectively. The Static Routes module is running such that the prefixes to be announced to other routers can be configured. In addition, the RIB process merges the infor- 45 7.2. The Control Plane

Routers Network Routers Emulator Emulator Emulator mXORP ... mXORP mXORP mCLICK ... mCLICK mCLICK Linux OS ... Linux OS NetPaths Linux OS Router Network Router Virt. Virt. Virt. Sublayer Sublayer Sublayer

Hardware Layer HW Layer HW Layer

LAN

Figure 7.1: Different layers involved in the virtual testbed. mation from the different routing protocols and update each routing process when changes, such as connectivity, failure occurs. The RIB process does not directly install the most opti- mum paths in the FIB, instead it relays those paths to a forth process, the forwarding engine abstraction(i.e. FEA). The FEA process is the process which acts as interface to the data- forwarding plane for the RIB.

All those processes are intercommunicated using a RPC-like (i.e. remote procedure call) mechanism. The methods that can be called upon one process are defined by an interface in a language called XRL (i.e. eXtensible Router Language). Automatic tools generate the client object and the stub code to achieve the inter-process communication. In XORP the modifications are carried out at three different levels: the BGP daemon, the RIB process and at the FEA process. The multipath extensions for the BGP daemon to obtain an implemen- tation for ASSEMBLER are described first. The redefinition of the XRL for the RIB and the multipath extensions for its modules are defined afterwards. Finally, in order to complete the top-down analysis, the implementation of a plug-in to let the FEA process handle a CLICK- based data-forwarding plane is detailed. Before getting into the details of the ASSEMBLER implementation, we briefly review the normal operation of the XORP BGP Daemon which serves as a base for the ASSEMBLER implementation.

7.2.1 The Standard BGP Daemon

This section describes the architecture and modules of the unipath BGP process. In order to have a general overview and identify the different modules, we always refer to Fig.7.2. Every module labelled as PeerHandler handles a BGP session with a peer router. Every time a BGP Update message is received, the PeerHandler creates one or several internal messages of type ADD ROUTE, DELETE ROUTE or REPLACE ROUTE. For instance, when a peer router announces to the router in Fig.7.2 that a new destination is reachable through it for the first time, an ADD ROUTE message crosses the chain of modules from the PeerHandler until the module DecisionTable is reached. Otherwise, it gets filtered at some module in between.

Once in the DecisionTable, the route contained within the ADD ROUTE message under- 46 Chapter 7 Implementation of an ASSEMBLER-Capable Router

goes the BGP tie-breaking process along with other routes announced by other peer routers for the same destination. The DecisionTable queries the different RIBInTable on each branch to obtain those alternative routes. If the route becomes the new winner then two new internal messages are propagated downstream through every output branch thanks to the FanOutTable module, which replicates the ADD ROUTE. A DELETE ROUTE is propagated to delete any stored/cached state of the previous winner.

Afterwards an ADD ROUTE message establishes state for the new winning route, which in turn upon reaching the PeerHandler module is included into an external BGP Update mes- sage. The latter will be sent to each peer router for which a branch exists. Nevertheless, there is an additional branch, which is not connected to any peer, it ends in a IpcRIBHandler module, instead. That special module acts as the interface with the control process of the RIB.

In addition to the announcement or withdrawn of routes towards a given prefix, peering session establishment is another common event in BGP. No sooner than the TCP connection is established, the router starts dumping its current routing information to its new (or recov- ered) peer. To that extent, the BGP daemon of XORP defines and additional module called DumpTable. The DumpTable is in charge of feeding the new router in background while consistency is kept as new routes arrive in the meantime to the different RIBInTables.

7.2.2 Path ASSEMBLER Extensions in XORP

In this section, the ASSEMBLER modifications carried out over the XORP BGP daemon modules are detailed. From the analysis of the unipath XORP BGP process, several key modules can be identified in order to accomplish the ASSEMBLER extensions.

Before addressing the specific modifications of each module, there is a modification re- quired at every module (with the exception of the PeerHandler module). The modules in Fig.7.2 implement an interface called RouteTableBase. The RouteTableBase interface de- fines the operations that one module can call upon another. For instance, the interface defines operation to relay paths between connected modules. The current definition of the interface allows to call an add, delete or replace operation over just one path at a time. Operations called afterwards regarding the same network prefix override the state generated in the mod- ule by previous operations. Therefore, the first modification is the addition of a new operation on the interface, so-called ROUTE_MADD, such that multiple paths can be relayed between modules. Now, we present the modifications module by module as they appear downstream in Fig.7.2 (i.e. from left to right).

PeerHandler The first module that we come across in the picture is the PeerHandler mod- ule. One of the most remarkable features of the design of ASSEMBLER is the backwards compatibility with BGP. As a direct consequence, the messages received or generated by a multipath capable router must allow unipath routers to process them. The only subtle differ- ence in a message generated by a multipath capable router running ASSEMBLER is that the AS_PATH attribute is generated following particular rules, but that does not modify the way it is processed by the router receiving the announcement. Therefore, the module interacting with routers for which a BGP session is established does not require any modification.

RIBInTable, CacheTable, In/OutFilterTable and NextHopLookupTable Since a router receives one announcement/path per network prefix from each neighbor, the RIBInTable does 47 7.2. The Control Plane

not require any further modifications to implement ASSEMBLER. Nevertheless, the mod- ule has been extended to return all the most recent paths received from the same peer upon quering from the DecisionTable. The CacheTable functionality remains the same and the definition of new filtering policies for multipath scenarios is out of the scope of this work, thus the FilterTable implementation is not extended. The next-hop resolution is exactly as in the unipath case. Again, in order to make the modifications of the XORP BGP process as general as possible, if another protocol propagates several paths for the same network prefix, these modules run through each each path in the multipath set executing their functionality over each path independently. For instance, if one of the paths in the multipath set is not re- solvable by the NextHopLookupTable, then only that path is removed from the set propagated downstream.

DecisionTable The DecisionTable carries out the ASSEMBLER path selection process. Every time a new ROUTE_ADD, ROUTE_DELETE or ROUTE_REPLACE operation is going on the decision process is carried out in order to check whether the new information discloses a more optimum set of paths to forward traffic. The DecisionTable module retrieves all the available alternative paths querying the different RibInTable modules at each branch.

In contrast to the unipath case, the rules in the decision process are those depicted in Section 4.1. Therefore the outcome of the decision process is not a single best path, but the set of most preferred paths. Nevertheless, the paths in the selected set are not an- nounced independently to router peers. An special announcement is constructed as depicted in Section 4.2. The AS_PATH attribute is constructed by taking the path with the shortest AS_PATH_LENGTH and adding an AS_SET. The AS_NUMBERs included in the AS_SET are those in the remaining paths in the multipath set which are not already in the shortest path.

In addition to removing some tie-break rules and adding the functionality to construct the multipath set, now the DecisionTable makes two calls upon the next module, the FanOut- Table. The first call the default ROUTE_ADD operation, which supports only one path with the constructed aggregated AS_PATH and a flag in the attributes of the internal message indicating that the path is intended to be announced to the peers. The second call con- tains the whole multipath set with each path passed independently in the call. This time a ROUTE_MADD operation is called. In that way, the DecisionTable indicates the FanOut- Table that the set of paths relayed to it in the call should be relayed to the branch that connects the ASSEMBLER process to the RIB process. The paths are passed separately, such that sev- eral next-hops will be installed in the data-forwarding plane, enabling multipath forwarding of traffic.

FanOutTable The path selection procedure is carried out once per operation. The result of the selection procedures must be announced to the peers in order to guarantee path con- sistency and loop-freeness. Every established session with another peer creates a branch of modules like in Fig.7.2. The FanOutTable module is responsible for duplicating the output of the selection procedure to every branch so that all the peers are eventually updated with the new state of the router.

In addition, given the peculiarities of ASSEMBLER the FanOutTable distinguishes be- tween the information to create the announcements to other peers and the information to create the new state in the FIB (which has to be relayed to the RIB process first). Therefore, the FanOutTable receives a ROUTE_ADD operation carrying the aggregated AS_PATH cre- ated by the DecisionTable module and a ROUTE_MADD operation containing the expanded version of the previous AS_PATH, i.e. the set of paths aggregated in the AS_PATH of the first 48 Chapter 7 Implementation of an ASSEMBLER-Capable Router

call. The ROUTE_ADD operation duplicates the internal message to every branch connect- ing to a peer router whereas the ROUTE_MADD operation forwards the internal message through the branch connecting to the RIB process.

RIBOutTable The RIBOutTable is responsible for coordinate the inner operations and the announcements towards peer routers. The RIBOutTable receives operation and schedules them in a queue. Each element in the queue comprises an operation object, which in turn contains a path attribute. When the PeerHandler module notifies the RIBOutTable that it is idle, the RIBOutTable pops messages from the queue and the PeerHandler creates the announcement. This module has been extended to support the multiple paths passing in the branch connecting to the RIB. Each element in the waiting queue is an operation object which is able to store now a linked list of paths instead of a path element.

IPCRIBHandler The RIB process works based on transactions. The next transaction does not start until the ongoing is committed. The operations meant for the following transaction are withheld in the RIBOutTable. The main modification in this part of the module chain is not the modification in the module itself, since it consist in changing the type of object called to relay routes to the RIB process, from the RPC (i.e. remote procedure call) client class XrlRibV0p1Client to XrlRibV0p2Client. RPCs are called XRLs in XORP. The key modification here is the definition of a new interface to allow other processes to interact with the RIB process.

DumpTable As soon as a new peer session is established the router dumps all its most preferred paths for each destination to the peer so it can get a quick vision of the network. In XORP BGP, the DumpTable is in charge of dumping the paths. The DumpTable polls every RIBInTable looking for the paths marked as the most preferred.

Nevertheless, ASSEMBLER creates artificial AS_PATHs. Those paths are not in the RIBInTable modules since they are not paths announced by a peer, but internally created. To that extent, the AuxiliarTable path storage module is added to the architecture in Figure 4. The AuxiliarTable simplifies the implementation of the DumpTable. Now the DumpTable does not need to synchronize with all the RIBInTables, synchronization with the AuxiliarTable is enough to dump the current routing information to new peers.

7.2.3 Modifying the RIB and FEA Processes

In addition to the routing protocols, XORP runs the RIB and FEA processes. The RIB merges the routing information acquired by each routing protocol. The outcome of the merg- ing process is passed to a third process, the FEA. The FEA allows XORP to decouple from a specific data-forwarding plane. Thanks to that abstraction XORP can use different data- forwarding planes, such as the Linux routing module, CLICK or even a distributed data- forwarding plane among several PCs running XORP.

The routing protocols can add, delete or replace routes in the forwarding plane executing a transaction against the RIB process interface. Paths are added or removed from the data- plane by means corresponding transactions and no changes are made effective in the routing table until the transaction has been committed. A new transaction has been added to the RIB interface. The new transaction allows a routing protocol to add or delete several paths simultaneously. 49 7.3. The Data-Forwarding Plane

The FEA has an interface with common abstract operations over the data-forwarding plane. That interface is used by the RIB process to execute the changes in the data-plane. Once the operation is called from the RIB, the FEA translates the abstract operations into the specific set of operations to be executed by the data-forwarding plane used by XORP.

In order to know which are the operations to be carried out over the data-plane, for each type of data-plane there is a plug-in that allows the FEA module of XORP to interact with the FIB. We have extended the current plug-ins for CLICK to support multi-path addition and removal of paths. The details of this new plug-in are explained later on along with the implementation of a multi-path FIB using CLICK.

Figure 7.2: Module diagram of XORP BGP daemon.

7.3 The Data-Forwarding Plane

The data-forwarding plane of one of the ASSEMBLER-capable virutal routers is imple- mented using CLICK. CLICK comes with a default data-forwarding plane implementation that can be found in [20]. CLICK allows its modules to have hooks to interact with exter- nal processes. Hooks are nothing but inter-process communication pipes as those commonly used in Linux-based operating systems.

Depending on the type of hook, external processes can read, write or both, information that CLICK modules can parse/publish. In our case, we set off from the default implemen- tation of the routing table (so-called LookupTable) and a new hook has been added. The new hook allows to progressively add entries to the routing table for the same network prefix without overwriting previously added entries (the default add operation replace the current entry by the new one). The new hook appears in the folder corresponding to the routing table module in the CLICK file system as a file called madd (for further details about the CLICK file system check [20]).

The FEA plug-in handling CLICK dumps to the CLICK routing table the routing infor- mation extracted from the multipath set chosen by the ASSEMBLER process. The plug-in transform the routing information from the data format used by XORP into a format that can be parsed by the CLICK module. Afterwards, the plug-in initiates a transaction. The trans- action consist in two different steps. First, the FEA flushes the previous entries for a given network prefix. Second, it iteratively writes the information of each of the paths into the madd file introduced earlier. After, the FEA closes and commits the transaction. The deletion step is used simply because, it simplifies the installation of paths.

The multipath addition creates a routing table like the one in Fig.7.3, where for each network prefix we can find several < nexthop, interface, metric > tuples. The difference 50 Chapter 7 Implementation of an ASSEMBLER-Capable Router

between the appearance of multiple entries per prefix in the routing table in the unipath and the multipath case, is that the additional entries in the unipath are used as backup, such that only the first entry matching the destination network prefix of the packet and the others are used in case the first path becomes unavailable. In the multipath case, the multiple entries can be used to forward packets towards the same destination concurrently.

Many forwarding policies can be defined, however some of them like random forwarding or round-robin increase packet reordering which can be harmful, specially using a transport protocol such as TCP or MPTCP, which guarantee ordered delivery of packets. Therefore, it is desirable to have a forwarding algorithm, which guarantees that at least certain sequences of packets follow the same path, minimizing reordering.

IP Prefix Next-Hop If. TCP/IP 16.11.3.0/30 10.0.2.5 - 2 Header IP Prefix Lookup 10.0.1.0/24 10.0.2.5 - 1 10.0.3.10 - 2 10.0.4.231 - 3 Hash Function 12.1.0.0/16 10.0.2.1 - 1 10.0.2.5 - 1

Figure 7.3: Multipath FIB.

Therefore, we propose the following forwarding algorithm, a hash function is computed out of the so-called 5-Tuple of TCP/IP fields (see RFC [16]), which comprises the source IP address, destination IP address, source TCP port, destination TCP port and TCP sub-flow identifier (only if MPTCP is enabled). The 5-Tuple uniquely identifies each flow . The outcome of the hash function is then mapped to an output interface such that the same TCP flow is always forwarded to the same next-hop.

There are different ways to map flows with the interfaces. The simplest is to compute the result of the hash function and compute that number modulo the number of available paths in the routing table for a certain prefix. Another solution is to assign equally distributed identifiers to each path in the output range of the hash function and forward to the closest higher identifier to the hash function value over the TCP 5-Tuple. Figure 5 shows a diagram of the forwarding mechanism. The packet header information is passed to both the entry selector module (HashFunction in the figure) and to the lookup algorithm. When the lookup algorithm finds and entry in the routing table, it retrieves the number of alternatives and along with the result of the hash function computes the next-hop that will be used to forward the packet.

7.4 Disclosed Path-Diversity

The network depicted in Fig.7.4 is running ASSEMBLER. The testbed introduced in previous sections emulates the network. Even in this small topology is interesting to see how much path diversity ASSEMBLER can disclose. As discussed during the introduction, as more path-diversity is introduced and operated, the adaptation and responsiveness of multipath inter-domain protocols to network changes should improve. 51 7.4. Disclosed Path-Diversity

Costumer-Provider 1 2 Peering

3 4 5

6 7 8

9 10 13

11 12

Figure 7.4: Sample Topology

The setup of the experiment is as follows. Every node depicted in Fig.7.4 is an AS- SEMBLER router. Each router belongs to one AS and it announces one network prefix of the format 150.X.0.0/16 where X is the AS number the router belongs to. Once the advertisement is propagated from the router, the network converges and for the amount of paths from each of the remaining nodes to the advertising node is calculated. Each pair origin-destination is called a case. The process is repeated for each router such that all the prefixes are propagated and the amount of paths for each case measured.

The results in Fig.7.5 show the path diversity that ASSEMBLER is able to expose in the sample topology in Fig.7.4. The figure is the cumulative distribution function of the number of different paths for each case. Two paths are considered different if they differ at least in an intermediate node or link. The chart shows that in about 60% of the cases, a node is left only with one path towards the destination AS. That percentage should decrease if the connectivity between ASes is not represented as a single link and each AS is a single router. On the other hand, about 40% of the cases a node is using at least an additional end-to-end path towards a destination, which is an acceptable results for such a small topology. Moreover, roughly in 20% of those cases nodes get 2 or 3 additional paths. Larger amount of additional paths are rare and only happen in a reduced number of cases.

Dealing with the improved reliability introduced by the network diversity, Fig.7.6 shows the probability distribution function of having an additional path towards a destination after one and two node in the shortest path fail simultaneously. The experiment setup is similar to the previous one. The difference is that whereas in the previous case every pair origin- destination was considered a case, now for every pair there are several cases, one per each node on the shortest path that is assumed to be down to compute the addition paths to over- come the failure. Cases in which there is one path between the origin and destination are not considered.

The results show that even for a small topology a certain fast recovery can be achieved compared to the unipath case. In 65% of the cases, in which a node has at least one alternative path to another node, connectivity between the two ASes is possible without waiting for the ASSEMBLER process to reconverge. As expected, as the number of simultaneous failures increases the connectivity drops dramatically, even if multi-path is enabled. In the sample topology, if two nodes in the shortest path fail simultaneously, in less than 10% of the cases nodes have an available end-to-end path without executing the BGP selection process. 52 Chapter 7 Implementation of an ASSEMBLER-Capable Router

Figure 7.5: CDF of alternative paths available at each node.

Figure 7.6: PDF of alternative paths after 1 and 2 nodes have failed. Chapter 8

Related Work and Conclusions

This chapter compares the most relevant BGP-compatible multipath inter-domain routing proposals with ASSEMBLER and presents the conclusions of this work. Alternative proto- cols that require a global upgrade of the network (see for instance [11]) are not considered in this discussion. The first set of solutions comparable to ASSEMBLER achieve backwards compatibility using BGP to exchange the primary path (ensuring backwards compatibility) and they use a parallel protocol or BGP extension to advertise additional paths. This is the case of R-BGP, which advertise failover paths [21] to achieve fast recovery. BGP Add-Paths [32] is another solution in which routers add a new BGP capability to incrementally advertise extra paths. Finally, MIRO [36] relies also on an additional negotiation of paths. Although, they are compatible with BGP, these solutions require that two or more neighbor ASes coor- dinate to deploy multipath border routers. ASSEMBLER does not require of such an incre- mental/additional negotiation of paths and a coordinated deployment between neighbor ASes is not required.

Another set of multipath inter-domain protocols compatible with BGP do not modify the BGP protocol at all and no additional/incremental advertisement of paths is performed. The first solution is the Multipath-BGP proposed by the manufacturers Cisco and Juniper [6, 18], in which all the considered paths must share the same attributes except for the BGP_ID and interface address of the announcing border router. This type of multipath is intended for a set of particular settings in which several physical connections exists between two ASes. Hence, the multipath set yielded is constrained to have the same AS_PATH attribute in all the routes. As shown in Section 4.1, the K-BESTRO path selection algorithm and the assembling tech- nique used by ASSEMBLER does not constrain the path diversity and any set of aggregatable paths can be used concurrently, even if they have different egress ASes.

Some other BGP-compatible protocols propose to advertise one path and use the different alternatives received through BGP to forward the traffic without advertise them. For instance, the inter-domain flavours of Routing Deflections [37] and Path Splicing [25] forward traffic among the available alternative BGP paths according to a tag in the packet header. Thus, since BGP advertises only one of those paths, the loop-freeness of the multiple BGP routes cannot be guaranteed, since routes that are not propagated to further ASes are used in prac- tice to forwards the packets. Therefore the control plane information in neighbor ASes is inconsistent with how the traffic is forwarded. The authors in [25] argue that if the common routing policies presented in [10] are followed, no routing loops are possible. In addition, they propose some additional mechanisms to overcome that limitation like deflection coun- 54 Chapter 8 Related Work and Conclusions

ters or include the AS number of the ASes crossed before in the packet header. An alternative that solves the loop-freeness problem is to propagate the longest available path like in LP- BGP [31], however longer paths are likely to suffer a penalization when compared with other paths at a legacy router.

Thanks to the advertisement scheme presented in Section 4.2, ASSEMBLER is able to advertise information that is consistent with the forwarding of traffic currently used. Using ASSEMBLER, the forwarding mechanisms of Routing Deflections and Path Splicing can be used while preserving loop-freeness and without propagating paths that are likely to be considered worse by legacy routers like in [31], since the AS path length attribute in the advertisement is equal to the shortest path within the multipath set.

8.1 Conclusions

In this work, ASSEMBLER a novel protocol for multipath inter-domain routing has been presented. It is the first inter-domain routing protocol that features both, flexible multipath routing and backwards compatibility with BGP, without limiting the path diversity or using another protocol in parallel. Thanks to its design, ASes can benefit from multipath capabil- ities upgrading progressively their network equipment inside the AS and no coordination or global upgrade is required to take advantage of multiple inter-domain paths.

The characteristics of the multipath set provided by ASSEMBLER can be flexibly tuned using a few parameters to fully exploit the available path diversity or constrain the amount of paths installed in the data-plane (avoiding an exponential growth of the routing tables).

The ASSEMBLER announcements are regular BGP updates generated with an special al- gorithm such that advertised updates gather information from multiple paths in one message. Those updates can be processed by legacy routers, they are not penalized when compared to regular BGP paths and loop-freeness is maintained. The deployment in a real AS can be carried out progressively and current routing policies and traffic engineering techniques are supported by ASSEMBLER. It can be combined with multipath forwarding techniques to split the traffic amount those installed paths.

The stability of the protocol has been proven in absence of conflicting configurations. Whenever the ASes can simultaneously fulfil their preferences with the available paths, the protocol is able to converge. Two guidelines have been introduced in order to guarantee global stability for every possible configuration of ASSEMBLER.

The adoption of ASSEMBLER as multipath inter-domain routing should provide ISPs with more flexible routing configurations, simplified and dynamic traffic engineering tech- niques and decrease inter-domain churn.

8.2 Future Work

In order to complete and extent the research work initiated by this thesis, there are some open questions that can serve as the base for future research work. Of special interest are the inter- operability aspects. For instance, large ISP do not have a planar architecture for their border routers. The reason behind this is the scalability of the system, since a planar architecture im- 55 8.2. Future Work

plies that the full-mesh of iBGP between border routers grows exponentially with the size of the network and the system may get to a saturation point in terms of connections and internal churn. ISPs avoid that situation by introducing special nodes called route reflectors which help to reduce the amount of existing connections creating a hierarchy. Typically the portion of border routers per route reflectors is 10 to 1, the border routers keep only one connection to the route reflector and the full-mesh is only created among the reflectors, which reduces the scalability problem. It could be interesting to perform an analysis of the possible effects that introducing ASSEMBLER inside and AS may have if legacy route reflectors are present.

Another interesting interoperability analysis could be the interactions with BGP Add- Paths, such that the architecture of the AS would be inter-domain routers, internally commu- nicated using Add-Paths and using ASSEMBLER to communicate with external ASes that do not support multipath. A transition analysis between these technologies could be interesting as well.

The design of traffic engineering techniques that exploit the advantages of using multipath routing is also an open question. It is an intuition of the author that multipath should provide more flexible and finer granularity in the handling of traffic, reducing in some cases the inter- domain churn and achieving fast convergence when local failures occur.

Finally, another interesting future research line could be the design of new network ser- vices using the additional paths. Since it was not possible before, there is not applications of multipath inter-domain routing appearing in the literature or in hands-on practical manuals. Actually the basis for this research line has been already initiated by the author and the su- pervisor of the thesis, getting promising positive feedback from several ad-hoc conversations with former partners of the FP7 Trilogy Project and in the Routing Research Group (RRG) of the IETF. Therefore the real potential of multipath routing in inter-domain scenarios remains as an open question.

This research work is planned to be presented as a paper publication along with more empirical results, specially from simulations. This prospective publication is work in progress by the time writing. Bibliography

[1] 7th Frame Programme. Trilogy project: Architecting the future internet. http:// trilogy-project.org/.

[2] D. Bertsekas and J. Tsitsiklis. Parallel and distributed computation. 1989.

[3] H. Brauer and C. Jeker. OpenBGPd.

[4] M. Caesar and J. Rexford. BGP routing policies in ISP networks. Network, IEEE, 19(6):5–11, 2005.

[5] C. Chau. Policy-based routing with non-strict preferences. In Proceedings of the 2006 conference on Applications, technologies, architectures, and protocols for computer communications, pages 387–398. ACM, 2006.

[6] Cisco. Bgp best path selection algorithm. http://www.cisco.com/en/US/ tech/tk365/technologies_tech_note09186a0080094431.shtml.

[7] Cisco. How does load balancing work? http://www.cisco.com/en/US/ tech/tk365/technologies_tech_note09186a0080094820.shtml.

[8] Cisco. Load sharing with bgp in single and multihomed environments: Sample config- urations. http://www.cisco.com/en/US/tech/tk365/technologies_ configuration_example09186a00800945bf.shtml.

[9] A. Ford and C. Raiciu. M. Handley," TCP Extensions for Multipath Operation with Multiple Addresses", draft-ietf-mptcp-multiaddressed-03 (work in progress), 2010.

[10] L. Gao and J. Rexford. Stable Internet routing without global coordination. IEEE/ACM Transactions on Networking (TON), 9(6):681–692, 2001.

[11] P. Godfrey, I. Ganichev, S. Shenker, and I. Stoica. Pathlet routing. In Proceedings of the ACM SIGCOMM 2009 conference on Data communication, pages 111–122. ACM, 2009.

[12] T. Griffin, F. Shepherd, and G. Wilfong. The stable paths problem and interdomain routing. IEEE/ACM Transactions on Networking (TON), 10(2):232–243, 2002.

[13] F. Guo, J. Chen, W. Li, and T. Chiueh. Experiences in building a multihoming load balancing system. In INFOCOM 2004. Twenty-third AnnualJoint Conference of the IEEE Computer and Communications Societies, volume 2, pages 1241–1251. IEEE, 2004.

[14] M. Handley, O. Hodson, and E. Kohler. XORP: An open platform for network research. ACM SIGCOMM Computer Communication Review, 33(1):53–57, 2003. 57 Bibliography

[15] J. He and J. Rexford. Toward internet-wide multipath routing. Network, IEEE, 22(2):16–21, 2008.

[16] C. Hopps. Analysis of an Equal-Cost Multi-Path Algorithm, IETF. Technical report, Internet RFC 2992, Novembre 2000.

[17] K. Ishiguro, T. Takada, Y. Ohara, A. Zinin, G. Natapov, and A. Mizutani. Quagga routing suite.

[18] Juniper. Configure bgp to select multiple bgp paths. http://www.juniper. net/techpubs/software/junos/junos53/swconfig53-ipv6/html/ ipv6-bgp-config29.html.

[19] S. Kandula, D. Katabi, S. Sinha, and A. Berger. Dynamic load balancing without packet reordering. ACM SIGCOMM Computer Communication Review, 37(2):51–62, 2007.

[20] E. Kohler, R. Morris, B. Chen, J. Jannotti, and M. Kaashoek. The Click modular router. ACM Transactions on Computer Systems (TOCS), 18(3):263–297, 2000.

[21] N. Kushman, S. Kandula, D. Katabi, and B. Maggs. R-BGP: Staying connected in a connected world. In Proc. NSDI, pages 341–354, 2007.

[22] T. S. A. Labs. Best practices for network interconnection. NANOG 43.

[23] T. McGregor, S. Alcock, and D. Karrenberg. The RIPE NCC internet measurement data repository. In Passive and Active Measurement, pages 111–120. Springer, 2010.

[24] D. Meyer. University of oregon route views archive project. at http://archive. route- views. org.

[25] M. Motiwala, N. Feamster, and S. Vempala. Better interdomain path diversity with BGP path splicing, 2007.

[26] B. Quoitin, C. Pelsser, L. Swinnen, O. Bonaventure, and S. Uhlig. Interdomain traffic engineering with BGP. Communications Magazine, IEEE, 41(5):122–128, 2003.

[27] B. Ramamurthy, G. Rouskas, and K. Sivalingam. Next-Generation Internet: Architec- tures and Protocols. Cambridge Univ Pr, 2011.

[28] Y. Rekhter, T. Li, and S. Hares. RFC 4271: a Border Gateway Protocol 4 (BGP-4). Internet Engineering Task Force, Tech. Rep, 2006.

[29] E. Rosen, Y. Rekhter, et al. Bgp/mpls vpns, 1999.

[30] R. Teixeira, A. Shaikh, T. Griffin, and G. Voelker. Network sensitivity to hot-potato disruptions. In ACM SIGCOMM Computer Communication Review, volume 34, pages 231–244. ACM, 2004.

[31] I. van Beijnum, J. Crowcroft, F. Valera, and M. Bagnulo. Loop-freeness in multipath BGP through propagating the longest path. In Communications Workshops, 2009. ICC Workshops 2009. IEEE International Conference on, pages 1–6. IEEE, 2009.

[32] V. Van den Schrieck, P. Francois, and O. Bonaventure. BGP add-paths: the scal- ing/performance tradeoffs. Selected Areas in Communications, IEEE Journal on, 28(8):1299–1307, 2010.

[33] Y. Wang, I. Avramopoulos, and J. Rexford. Design for configurability: rethinking inter- domain routing policies from the ground up. Selected Areas in Communications, IEEE Journal on, 27(3):336–348, 2009. 58 Chapter 8 Bibliography

[34] Y. Wang, M. Schapira, and J. Rexford. Neighbor-specific BGP: more flexible routing policies while improving global stability. In Proceedings of the eleventh international joint conference on Measurement and modeling of computer systems, pages 217–228. ACM, 2009. [35] D. Wischik, M. Handley, and M. Braun. The resource pooling principle. ACM SIG- COMM Computer Communication Review, 38(5):47–52, 2008. [36] W. Xu and J. Rexford. MIRO: multi-path interdomain routing. In Proceedings of the 2006 conference on Applications, technologies, architectures, and protocols for com- puter communications, pages 171–182. ACM, 2006. [37] X. Yang and D. Wetherall. Source selectable path diversity via routing deflections. In Proceedings of the 2006 conference on Applications, technologies, architectures, and protocols for computer communications, pages 159–170. ACM, 2006.