AARNet's experiences using MPLS for protection Internet2/NLANR Joint Techs Meeting Boulder, CO, USA 2002-07-28 Glen Turner, Network Engineer Australian Academic & Research Network [email protected] http://www.aarnet.edu.au/ Topics MPLS overview Protection technology AARNet's experiences with MPLS Other interesting stuff if we have time Coverage MPLS is a big topic with multiple implementation choices at almost every turn Only discuss some of the technolgy choices ● MPLS generic tagging, not ATM tagging ● RSVP and not LDP ● OSPF and not IS-IS Coverage Discuss the use of MPLS for protection, not discussing some important uses of MPLS ● VPNs (and thus BGP) ● GMPLS, the integrated control layer for switching technologies “How to speak Australian” ● words with “or” à “our”, “z” à “s” SONET à SDH (slight framing difference) T1 à E1 (E1 is 2Mbps) Topic MPLS overview Label switching Control plane protocols, RSVP Routing protocols, OSPF MPLS aims Scalable IP traffic engineering ● Avoid need for full IP network knowledge at core Virtual private network service ● By providing label switch paths exclusive to a customer This presentation focuses on traffic engineering ● Only beginning to experiment with VPNs MPLS is a layer 2½ protocol 7 Application 7 Application 6 Presentation 6 Presentation 5 Session 5 Session 4 Transport 4 Transport 3 Network 3 Network 2 Link 2½ MPLS 1 Physical 2 Link 1 Physical Advantages of layer 2½ No complex next hop algorithm ● IP address lookup is expensive – Closest matching prefix versus table lookup ● IP next hop algorithm gets more complex with each new service – Policy routing – Multicast Want GbE switch prices not GbE router prices New behaviours only effect edge routers Advantages of layer 2½, cond No need to follow IP routing ● The shortest path may not be the best path ● Want policy – For traffic engineering ● Bandwidth ● Diverse routers and paths – For arbitrary customer requirements ● eg: Australian Army doesn't want to be routed over links not owned by Australian-controlled telcos Advantages of layer 2½, cond Why MPLS for policy and not BGP? ● BGP is globally visible – Scalability: Does outer Mongolia need to know of an interface failure in outback Australia? – Can lose connectivity due to dampening, which is essential due to global visibility ● Not all reasonable policies can be expressed in BGP Disadvantages of layer 2½ Another set of control protocols ● ATM: OAM, ILMI, PNNI ● 802.1Q VLANs: Virtual LAN reservation protocol ● SDH/SONET MPLS uses IP as its control and routing protocol Layer 2.5 and protection Network layer protection requires a network-layer repsonse ● Limited by convergence time of routing protocol ● Fast convergence and global visibility do not mix – BGP rate limiting is an expression of this Layer 2.5 and protection Link layer protection requires a link layer response ● These often have constrained topologies – SDH/SONET rings – 802.1D and parallel links ● They often inefficiently use protection bandwidth ● They often treat all network traffic as equally valuable ● Lack of network topology: poor decisions Layer 2.5 and protection Allow network layer to establish pre- routed fallback path ● Full topology awareness Allow link layer to switch to fallback path ● Not globally visible ● Fast convergence This could get messy upon multiple failures ● Run interior routing protocol afterwards Forwarding equivalence class Another view of IP routing ● Step 1: Determine forwarding equivalence class from IP header (or more) – Standard: Destination IP address – Advanced: source IP address, multicast group, DSCP, TCP port, increasing bizaare ● Step 2: Lookup FEC forwarding table to determine output interface (ie: switch the packet) Forwarding equivalence class, cond IP router calculates forwarding equivalence class at every hop ● Expensive – either in CPU time or hardware ● Extensive – IP forwarding table is big with frequent updates ● Difficult to alter for new behaviours – ASIC designers may have not anticipated the change (reverse path lookup, source-specific multicast) Forwarding equivalence class, cond MPLS switching ● Determine forwarding equivalence class at ingress ● Tag packet with a fixed-length label for this forwarding equivalence class ● Switch using the label at every other hop to egress – Tags are designed for hardware manipulation Labels are not globally unique Even one router can run multiple “label spaces” – eth0, eth1 in LS1 – eth2, eth3 in LS2 Edge routers need distinct IP routing tables for each label space ● The key to MPLS VPNs ● We often want multiple routing tables and settle for policy routing instead MPLS tag A 32-bit header in front of the packet Tag contains just enough information for forwarding and queuing ● Unlike IPv4/IPv6 header, which carries a lot more Tag has hardware-friendly structure MPLS tag, fields Label ● Determines next-hop interface Experimental (QoS) ● Determines output interface queuing S for “last of stack” ● S=1 on last header Time to live ● Discard upon zero, otherwise decrement MPLS tag, stacking Tag Tag Tag Network-layer packet S=0 S=0 S=1 An MPLS tagged packet can be tagged again (“stacked”) ● Allows Provider-Provider connections to maintain customer tags ● Simplifies design considerably ● Avoids need for global label space MPLS tag, stacking and MTU The tag may reduce the size of the path maximum transmission unit (PMTU) ● TCP/IP stacks don't cope well with change of PMTU – PMTU at establishment of TCP determines TCP MSS ● Best to ensure that main and protect paths have identical tag depths Or may not, if the link layer will let us flex the rules MPLS operation mpls-path.dia MPLS operation, cond Label switch router Incoming packet, look up incoming label map, which contains ● Incoming label ● MPLS opcode: PUSH, POP, etc ● Forwarding equivalence class ● Link to outgoing next hop label entry MPLS operation, cond Label switch router Incoming packet operations ● Extract label from top tag ● Lookup incoming label map ● Execute MPLS opcodes to manipulate tags ● Forward packet to outgoing processing MPLS operation, cond Label switch router Outgoing packet, look up next hop label entry, which contains ● Outgoing label ● Outgoing interface ● Perhaps, outgoing per-hop queuing behaviour MPLS operation, cond Label switch router Outgoing packet operations ● Look up next hop label entry ● Create new tag containing outgoing label ● PUSH tag onto label stack ● Add to transmit queue on outgoing interface – queuing discipline may depend upon ● Value in next hop forwarding entry ● Value determined from Exp bits, a lá IP DSCP and weighted fair queuing + RED MPLS operation, cond Ingress label edge router Incoming packet, look up forwarding equivance class to next hop label entry (FTN), which contains ● forwarding equivalence class ● next hop label entry MPLS operation, cond Ingress label edge router Incoming packet operations ● Determine forwarding equivalence class using “standard” IP forwarding – Basic: lookup destination IP address in IP forwarding table – Advanced: policy routing, multicast routing, QoS routing, ... ● Use FEC to lookup forwarding equivalence class to next hop label entry table ● Process next hop label entry MPLS operation, cond Egress label edge router Next hop label entry shows this router as the penultimate hop Protocol-dependent actions to simulate label switch routers being real routers ● Decrement IP TTL ● Generate any ICMP which would have occurred Forward the packet using the standard IP algorithm Faking ICMP gives interesting results Traceroute from Glen's home to www.internet2.edu 1 sadial.sa.csiro.au 119.657 ms 129.673 ms 100.004 ms 2 sa.gw.csiro.au 119.944 ms 129.829 ms 110.382 ms 3 lis255.atm1-0.central.saard.net 131.917 ms 119.858 ms 109.980 ms 4 sa-nsw.atm.net.aarnet.edu.au 139.715 ms 149.829 ms 140.002 ms 5 vlan916.gbe3-0.sccn1.broadway.aarnet.net.au 149.941 ms 149.773 ms 149.968 ms 6 pos1-0.sccn1.manoa.aarnet.net.au 349.907 ms 279.791 ms 289.963 ms 7 pos2-0.sccn1.seattle.aarnet.net.au 279.866 ms 329.880 ms 279.904 ms 8 Abilene-PWAVE.pnw-gigapop.net 279.870 ms 351.155 ms 328.555 ms 9 dnvr-sttl.abilene.ucaid.edu 339.933 ms 339.861 ms 329.944 ms 10 kscy-dnvr.abilene.ucaid.edu 349.847 ms 339.622 ms 350.053 ms 11 ipls-kscy.abilene.ucaid.edu 339.756 ms 339.932 ms 339.903 ms 12 clev-ipls.abilene.ucaid.edu 339.884 ms 349.808 ms 339.963 ms 13 nycm-clev.abilene.ucaid.edu 349.752 ms 349.857 ms 339.969 ms 14 border-abilene-oc3.advanced.org 360.135 ms 359.857 ms 379.851 ms 15 www.internet2.edu 379.865 ms 359.838 ms 359.950 ms Architectural issues There is a lot of complexity at the edge ● Especially in the egress router But we want the edge to be cheap, as there is a lot of it There are no MPLS applications ATM has applications ● (Today's bizaare but true fact) Links between 3G base stations and switching points is the most recent application to treat ATM as a transport layer Even ethernet has applications ● DEC Local Area Transport There are no MPLS applications MPLS exists only to carry other protocols ● The label edge routers must support the protocol ● This isn't new – All routers have to support the network layer protocol they are routing Model is strained somewhat by abuse of MPLS to carry ethernet frames Configuring a label switch router Linux Both eth0 and eth1 in label space 1 ● mplsadm -L eth0:1 mplsadm -L eth1:1 Configuring a label switch router Linux Configure label switching ● mplsadm -A -I gen:10:1 -O gen:20:ipv4:10.3.0.2 -B mplsadm -A -I gen:21:1 -O gen:11:ipv4:10.2.0.1 -B – -A -B: add and bind – -I: incoming