Carrier Ethernet Operations, Administration & Maintenance

Santanu Dasgupta

1 Acknowledgement

. Jose Liste . Sudarshan Pathalam

2 Agenda

. OAM Overview . Ethernet OAM – Protocol Overview IEEE 802.3ah – Link OAM, also known as EFM IEEE 802.3ag – Connectivity Fault Management (CFM) ITU Y.1731 – Fault & Performance Management MEF E-LMI – Configuration Management . Ethernet Service Activation Testing . OAM Interworking . Sample Fault Management Scenarios . Ethernet OAM Deployment Use Cases . Summary

3 Operations, Administration & Maintenance

. F – Fault Management . C – Configuration Management . A – Accounting . P – PerformanceFC ManagementA PS . S – Security Management

OAM Protocols and Mechanisms helps operator to achieve some of the FCAPS functionality

OAM capability is one of the key differentiator to make a SP network truly “Carrier Grade”

4 Problem Scope A few possible scenarios

Excessive Uni-directional D-LDP session Encoding Errors Link failure Duplex SP Network mismatch Access Core VC failure Customer Excessive Equipment FCS Errors N-PE 1 N-PE 3 U-PE D C-VLAN to EVC U-PE A MPLS CE Cross-connectedCE mismatch Unexpected LSPSONET/SDH Service Native Speed Ethernet Endpoint Failure mismatch PW CE U-PE B N-PE 4 CE N-PE 2 MPLS U-PE C AC Ethernet UNI Ethernet UNI Physical failure link failure PW failure Physical VLAN to VFI link failure xconnect failure UNI VFI port failure failure Physical U-PE P-Router link failure device failure failure

5 OAM: The Concept

. Operations, Administration, Maintenance & Provisioning: − fault indication − performance monitoring − security management − diagnostic functions − configuration & service provisioning

. OAM covers both Nßà S and WßàE interfaces

Management Plane N

(NMS / EMS) ßà S

Network Plane (Elements)

W ßà E 6 Scope of Ethernet OAM

. Operations, Administration, Maintenance & Provisioning: − fault indication − performance monitoring − security management − diagnostic functions − configuration & service provisioning . OAM covers both Nßà S and WßàE interfaces

Management Plane N

(NMS / EMS) ßà

Primary Focus of Ethernet OAM protocols is on W ßà E interactions S (across NEs)

Network Plane (NEs)

W ßà E

7 Focus Areas of Ethernet OAM

UNI UNI Access Aggregation Edge Core Edge Aggregation Access

U-PE PE-AGG N-PE N-PE Core Core PE-AGG U-PE

EPL EVPL EP-LAN EVP-LAN EP-TREE EVP-TREE AC EPL AC EVPL CE Services & Network Infrastructure

Fault Management Config Management Performance Management

Fault Detection Frame Loss Measurement Fault Notification Service Provisioning Delay Measurement Fault Verification Delay Variation Measurement Fault Isolation Availability Measurement Fault Recovery

8 Ethernet OAM Building Blocks

Performance Fault Management Management

Performance Performance Fault ManagementManagement Management

Configuration ConfigurationManagement Management E-LMI Configuration E-LMI IP SLAs Management

802.1ag / Y.1731

802.3ah Fault

Performance Fault Management ManagementManagement

Configuration Management

. IEEE 802.1ag: Connectivity Fault Management (CFM) . IEEE 802.3ah: Ethernet Link OAM (EFM OAM) . ITU-T Y.1731: OAM functions & mechanisms for Ethernet based networks . MEF E-LMI: Ethernet Local Management Interface . IP SLA’s: Performance Management using CFM and Y.1731 mechanisms

9 Carrier Ethernet OAM Protocol Positioning

Connectivity Customer AggregationE-AccessLMI Fault ManagementCore Aggregation Customer Y.1731 Performance Business Management Business MPLS OAM

Ethernet Ethernet MPLS/IPEthernet MPLS/IP MPLS/IP Link OAM Residential Residential

UNI NNI NNI NNI UNI

. E-LMI – User to Network Interface (UNI) . Link OAM – Any point-point 802.3 link . CFM, Y.1731 PM – End-to-End UNI to UNI . MPLS OAM – within MPLS cloud 10 IEEE 802.3ah (Clause 57) Link OAM

11 Link OAM (IEEE 802.3ah, Clause 57)

. Provides mechanisms useful “ ” for monitoring link operation , OSI Model such as: LAN Application Link Monitoring CSMA/CD Remote Failure Indication Layers Presentation Remote Loopback Control Higher Layers Session . Defines an optional OAM sublayer LLC . Intended for single point-to-point Transport OAM (Optional) IEEE 802.3 links Network MAC . Uses “Slow Protocol”1 frames called OAMPDUs which are Data Link Physical Layer never forwarded by MAC clients Physical . Standardized: IEEE 802.3ah, clause 57 (now in IEEE 802.3-2005)

(1) No more than 10 frames transmitted in any one-second period 12 IEEE 802.3ah Key Functions

. OAM Discovery Discover OAM support and capabilities per device . Link Monitoring Basic error definitions for Ethernet so entities can detect failed and degraded connections . Fault Signaling Mechanisms for one entity to signal another that it has detected an error . Remote MIB Variable Retrieval Ability to read one or more MIB variables from the remote DTE . Remote Loopback Used to troubleshoot networks, allows one station to put the other station into a state whereby all inbound traffic is immediately reflected back onto the link

13 IEEE 802.3ah OAM Events

. Set of events that may impact link operation . Critical Link events Link fault—fault in the Rx direction of local DTE Dying gasp—unrecoverable local failure condition Critical event—unspecified critical event . Link events Errored Symbol Period Event Errored Frame Event Errored Frame Period Event Errored Frame Seconds Summary Event

14 IEEE 802.3ah Remote Loopback

. Fault localization and intrusive link performance testing . Loopback Control OAMPDU is used to control a remote OAM client . Traffic sent from master loopback port is loopback by slave port, except Pause and OAMPDU Slave drops traffic from LLC on looped interfaces Returned traffic dropped at Master OAM X to prevent MAC X OAM moves MAC MAC Slave loops service PHY frames without PHY swapping MAC addresses

Master OAM Slave OAM

15 IEEE 802.1ag: Connectivity Fault Management (CFM)

16 CFM Overview

. Family of protocols that provides capabilities to detect, verify, isolate and report end-to-end Ethernet connectivity faults . Employs regular Ethernet frames that travel in-band with the customer traffic Devices that cannot interpret CFM Messages forward them as normal data frames . CFM frames are distinguishable by Ether-Type (0x8902) and dMAC address (for multicast messages) . Standardized by IEEE in 2007 IEEE std. 802.1ag-2007

. Nested Maintenance Domains (MDs) that break up the responsibilities for network administration of a given end-to-end service . Maintenance Associations (MAs) that monitor service instances under a given MD . Maintenance Points (MPs) that generate and respond to CFM PDUs . Protocols (Continuity Check, Loopback and Linktrace) used for Fault Management activities

CE CE

Operator A Operator B

Service Provider

Customer

. Defined by Operational/Contractual Boundaries e.g. Customer / Service Provider / Operator . MD may nest and touch, but never intersect . Up to eight levels of “nesting”: MD Level (0..7) The higher the level, the broader its reach . MD Name Format: null, MAC address, DNS or string-based

CE Operator A Operator B CE

. Monitors connectivity of a particular service instance in a given MD (e.g. one service traversing four MDs = four MAs) . Defined by a set of Maintenance End Points (MEP) at the edge of a domain . Identified by MAID == “Short MA” Name + MD Name . Short MA Name Format: Vlan-ID, VPN-ID, integer or string-based

MEP MEP

MEP MEP MEP MEP

. Maintenance Association End Point (MEP) . Define the boundaries of a MD . Support the detection of connectivity failures between any pair of MEPs in an MA . Associated per MA and identified by a MEPID (1-8191) . Can initiate and respond to CFM PDUs

CE Operator A Operator B CE

MEP MIP MIP MEP MEP MIP MIP MEP

MEP MIP MIP MEP MEP MIP MIP MIP MIP MEP

MIP MIP MIP MIP

. Maintenance Domain Intermediate Point (MIP) . Support the discovery of paths among MEPs and location of faults along those paths . Can be associated per MD and VLAN/EVC (manually or automatically created) . Can add, check and respond to received CFM PDUs

. CFM PDUs generated by the MEP are sent towards the Bridge’s Relay Function and not via the wire connected Bridge to the port where the MEP is configured . CFM PDUs to be responded by Relay the MEP are expected to arrive Entity via the Bridge’s Relay Function

. Applicable to switches Port A Port B

CFM PDUs

. CFM PDUs generated by the MEP are sent via the wire connected to the port where the MEP is configured Bridge / Router . CFM PDUs to be responded by the MEP are expected to arrive via the wire connected Relay to the port where the MEP Entity is configured

. Port MEP—special Down Port A Port B MEP at level zero (0) used to detect faults at the link level (rather than service) CFM PDUs . Applicable to routers and switches

Applicability of UP/DOWN DOWN MEP to UP MEP MEPs in switches: Bridge 1 Bridge 2 Bridge Bridge Bridge Bridge . DOWN MEPs are typically used Port Port Port Port for MAs spanning a single link Relay Relay Entity Entity . UP MEPs are commonly used for MAs with a wider reach (e.g. end- Monitored area to-end, beyond a single link)

UP MEP to UP MEP

Bridge 1 Bridge 2 Bridge Bridge Bridge Bridge

Port Port Port Port Relay Relay Entity Entity

Monitored area

There are three (3) protocols defined by CFM . Continuity Check Protocol Fault Detection Fault Notification Fault Recovery . Loopback Protocol Fault Verification . Linktrace Protocol Path Discovery and Fault Isolation

Catalogue and Catalogue Catalogue Terminate MEP MIP MIP MEP

1 2 3 X Continuity Check Message (CCM) . Used for Fault Detection, Notification and Recovery . Per-Maintenance Association multicast “heart-beat” messages Transmitted at a configurable periodic interval by MEPs (3.3ms, 10ms, 100ms, 1s, 10s, 1min, 10min) Uni-directional (no response required) Carries status of port on which MEP is configured . Catalogued by MIPs at the same MD-Level, terminated by remote MEPs in the same MA © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 27 CFM Protocols Loopback Protocol CE Operator A Operator B CE

S D MEP MIP MIP MEP

1 1. Loopback Message 1 (LBM) 2 2 2. Loopback Reply (LBR)

. Used for Fault Verification—Ethernet Ping . MEP can transmit a unicast LBM to a MEP or MIP in the same MA . MEP can also transmit a multicast LBM (defined by ITU-T Y.1731), where only MEPs in the same MA respond . Receiving MP responds by transforming the LBM into a unicast LBR sent back to the originating MEP

6 S 4 D 2 MEP MIP MIP MEP

1 3 5 1, 3, 5 X Linktrace Message (LTM) Y 2, 4, 6 Linktrace Reply (LTR)

. Used for Path Discovery and Fault Isolation—Ethernet Traceroute . MEP can transmit a multicast message (LTM) in order to discover the MPs and path to a MIP or MEP in the same MA . Each MIP along the path and the terminating MP return a unicast LTR to originating MEP

. ITU-T recommendation that provides mechanisms for user-plane OAM functionality in Ethernet networks. It

covers: Ethernet Service OAM MPLS Fault Management mechanisms Service OAM MPLS-TP IP / MPLS Performance Management mechanisms Tunnel OAM Tunnel OAM . Standardized by ITU-T SG 13 in May 2006 Latest published version dated Feb. 2008 after IEEE 802.1ag standardization . Frame formats (Multicast Address, Ethertype, and common OAM PDU fields) and base functionality are agreed upon across IEEE 802.1ag and Y.1731

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 31 ITU-T Y.1731 Overview . OAM Functions for Fault Management Ethernet Continuity Check (ETH-CC) (Y.1731 adds unicast CCM) Covered by IEEE 802.1ag Ethernet Loopback (ETH-LB) (Y.1731 adds multicast LBM) Ethernet Linktrace (ETH-LT) Ethernet Remote Defect Indication (ETH-RDI) Ethernet Alarm Indication Signal (ETH-AIS) Ethernet Locked Signal (ETH-LCK) In addition: ETH-TEST, ETH-APS, ETH-MCC, ETH-EXP, ETH-VSP . OAM Functions for Performance Management Frame Delay Measurement (ETH-DM) Frame Loss Measurement (ETH-LM) Frame Loss Measurement (ETH-SLM)

. Without AIS, a MEP would report Loss of Continuity (LOC) for each of its remote MEPs upon timer expiration . With AIS, if a MEP receives AIS from the network, it suppresses LOC alarms from peer MEPs

LOC 1 LOC 3 Network outage LOC 4 2 AIS defect 2

LOC 1 LOC 2 AIS mpid = 1 LOC 4 mpid = 1 AIS defect X AIS X AIS LOC 2 3 3 LOC 3 AIS LOC 4 AIS defect

4 LOC 1 4 LOC 2 LOC 3 AIS defect WITHOUT AIS WITH AIS Timer-driven Event-driven LOC notification AIS notification © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 33 Alarm Indication Signal Transport Path Failure propagation

NMS AIS Defect Link Northbound Failure Syslogs Customer Operator A Operator B Customer Equipment Bridges Bridges Equipment

MEP(7) MIP(7) MEP(5) MIP(5) MEP(3) MEP(2) MIP(3) MIP(2) SMEP/link-status

AIS

. Frame Loss Ratio – percentage (%) of service frames not delivered / Total number of service frames delivered in T time interval . Frame Delay – round-trip/one-way delay for a service frame . Frame Delay Variation – Variation in frame delay between a pair of service frame

Ethernet Ethernet Delay Measurement Ethernet Loss Measurement Synthetic Loss ETH-DM ETH-LM Measurement ETH-SLM

One-Way Two-Way Single-Ended Dual-Ended Single-Ended ETH-DM ETH-DM ETH-LM ETH-LM ETH-SLM

Scope

• One-Way delay • Two-Way delay • Unidirectional frame • Unidirectional frame • Unidirectional frame measurements measurements loss measurements loss measurements loss measurements • Also One-Way delay • Applicable to P2P • Applicable to P2P • Applicable to P2P when synchronized services only services only and MP services Specifics

• 1DM PDU • DMM / DMR PDUs • LMM / LMR PDUs • CCM PDUs • SLM / SLR PDUs • Synthetic traffic with • Synthetic traffic with • On-demand operation • Proactive operation • Based on statistical two (2) timestamps two (2) mandatory • Based on actual • Based on actual sampling • Need for Time-of-Day and two (2) optional Service Frame Loss Service Frame Loss • Exchange of synchronization timestamps • Exchange of service • Exchange of service synthetic frame frame counters frame counters counters

. Frame Delay calculated Ethernet Delay Measurement based on timestamps ETH-DM applied to synthetic traffic . Applicable to point-to- point and multipoint One-Way ETH-DM Two-Way ETH-DM services . Two (2) mechanisms defined Performance Metrics Performance Metrics One-Way ETH-DM • One-Way FD/FDV • Two-Way FD/FDV • Time-of-Day (ToD) • One-Way FD/FDV when Two-Way ETH-DM synchronization required for OW FD synchronized

Customer Service Provider Customer Equipment Equipment

NEs must be synchronized (ToD) for one-way delay MEP MEP

1DM TxTimeStampf 0 1DM tp TxTimeStampf 1DM RxTimeStampf

1DM TxTimeStampf 0 t 1DM c TxTimeStampf 1DM RxTimeStampf

One-Way Delay One-Way Delay Variation

Delay = RxTimeStampf – TxTimeStampf Delay Var.[tc] = Delay[tc] – Delay[tp]

Customer Service Provider Customer Equipment Equipment

NEs may be synchronized (ToD) for one-way delay MEP MEP

D TxTimeStampf M 0 0 DMM TxTimeStampf D M 0 RxTimeStampf M 0 0 M TxTimeStampf D TxTimeStampf Optional D RxTimeStampf Timestamps RxTimeStampf M M TxTimeStampb TxTimeStampb DMR R R RxTimeStampb 0 Two-Way Delay One-Way Delay (Forward) One-Way Delay (Backward) (RxTimeStampb – TxTimeStampf) – RxTimeStampf – TxTimeStampf RxTimeStampb – TxTimeStampb (TxTimeStampb – RxTimeStampf) DMM – Delay Measurement Message © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public DMR – Delay Measurement Reply 39 Y.1731 – Frame Loss Measurement

. Frame Loss calculated based on actual in-profile service counters Ethernet Loss Measurement . Applicable to point-to-point ETH-LM services only (with ETH-LM) . Near-End Frame Loss measurement Single-Ended ETH- Dual-Ended Loss associated with Ingress Data Frame LM ETH-LM . Far-End Frame Loss measurement Loss associated with Egress Data Frame . Per-CoS counters maintained per MEP Performance Metrics

TxFCl – in-profile data frames transmitted • Unidirectional Frame Loss towards the peer MEP • Frame Loss Ratio RxFCl – in-profile data frames received • Availability from the peer MEP

Customer Service Provider Customer Equipment Equipment

Service Frames

MEP MEP Per-CoS Service Frame counters TxFCf TxFCf TxFCl LMM 0 LMM 0 LMM RxFCl 0 0 tp Per-CoS TxFCf LMR TxFCf LMR Service Frame LMR RxFCf RxFCf counters TxFCb TxFCb RxFCl LMM TxFCl t c LMR

Frame Loss (Far-end) Frame Loss (Near-end)

(TxFCf[tc] – TxFCf[tp]) – (RxFCf[tc] – RxFCf[tp]) (TxFCb[tc] – TxFCb[tp]) – (RxFCl[tc] – RxFCl[tp])

. Y.1731 ETH-LM cannot be used for frame loss in multipoint EVCs SLM . A new protocol (SLM) based on SLR synthetic frames Statistical Frame Loss Covers P2P or MP EVCs Implementable by ALL platforms A B . ITU agreed (June 2010) to allocate code points for new PDUs (SLM / SLR) . ETH-SLM included ITU-T G. 8013 / Y.1731 (07/2011) C

SLM – Synthetic Loss Message SLR – Synthetic Loss Reply

Customer Service Provider Customer Equipment Equipment

Synthetic Frames

MEP MEP Per-Test ID Synthetic Frame counters TxFCf TxFCf SLM SLM TxFCl 0 SLM 0 RxFCl Per-Test ID tp TxFCf SLR TxFCf SLR Synthetic Frame SLR TxFCb TxFCb counters RxFCl SLM TxFCl

tc RxFCl = TxFCl SLR at Responder Frame Loss (Far-end) Frame Loss (Near-end)

(TxFCf[tc] – TxFCf[tp]) – (TxFCb[tc] – TxFCb[tp]) (TxFCb[tc] – TxFCb[tp]) – (RxFCl[tc] – RxFCl[tp])

tp – Time Previous – Start of Measurement Period SLM – Synthetic Loss Message

. Provides protocol and mechanisms used for: Notification of EVC addition, deletion or status (Active, Not Active, Partially Active) to CE Communication of UNI and EVC attributes User Network Interface to CE (e.g. CE-VLAN to EVC map) (UNI) CE auto-configuration UNI-C UNI-N Metro Notification of Remote UNI count, name and status to CE Ethernet . Asymmetric protocol based on CE Network Frame Relay LMI, mainly applicable to the UNI (UNI-C and UNI-N) E-LMI . Specification completed by MEF: http://www.metroethernetforum. org/PDFs/Standards/MEF16.doc

. Issuance of ‘Birth Certificate’ Traffic Generation/ Reception . Validation of Service Configuration Control . Validation of SLA Plane Signaling Throughput Test Methodology Latency Data Plane Loss Loopback Jitter

Bi-directional Test Operation Ethernet Data-Plane Loopback

Test Traffic TG EVC

Test Initiator Responder

Uni-directional Test Operation Start Reply / Request

Test Traffic TG EVC RL

Stop Reply / Request Test Initiator Responder TG = Traffic Generation function HW-accelarated (Platform-Dependent) RL=Responder Logic

Project Name Highlights Status

• Defines set of tests needed to run before handing Approved Project Service Activation off a service to subscriber Editor: CableLabs Testing project • Leverages ITU-T Y.1564 Targeting completion in Q2, 2013

• Defines the test PDUs that can be used to perform Approved Project the tests defined by the SAT project Editor: Verizon SAT PDU project • Defines control protocol to be used for unidirectional Targeting completion in Q4, 2013 tests - to set up the collector and/or responder at the remote end. Approved Project Latching loopback • Defines control protocol to be used for bidirectional Editor: Adtran project tests - to put remote device into latching loopback Targeting completion in Q1, 2014

. RFC2544 methodology shortcomings (as stated by Y.1564) Not Ethernet service aware Tests run as a single flow at a time 2544 1564 Tests are performed sequentially

Does not measure Frame Delay Variation Service centric

Device Centric All KPI measured in Does not verify CIR, CBS, EIR, EBS and CM 1 test

Sequential Tests Includes Jitter, CIR/ . Y.1564 advantages EIR,CBS/EBS

Missing KPI Tests all KPIs at same time Faster Execution Test to CIR to verify SLA performance (FD, FDV, FLR) Tests to EIR limit and just beyond to verify policing behavior (no SLA performance expected for yellow frames (above CIR and below CIR +EIR)

Presentation_ID © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 51 What Is OAM Interworking? Operator A Operator B Edge Router Edge Bridge ATM Switch ATM Switch . Strict OAM layering should be honored: messages should not cross layers Customer Customer Router Router Router Bridge Bridge Bridge switch Customer Customer switch Bridge . OAM Messages should not leak outside domain boundaries withinBridge a layer

. Interworking is event translations & not necessarily 1:1 message mapping

. Interworking may be inter-layer and intra-layer

Ethernet Ethernet Event10G TranslationATM inter- over SONET over MPLS Ethernetdomains RFC1483intra-layer

Service OAM Server Layers feed events into Client Layers

Network Network OAM OAM

Transport Transport Transport Transport OAM OAM OAM OAM

Ethernet Service Provider

L2 VPN Customer L2 VPN L3 CE Customer EVC NID / Access / Demarcation Device L3 CE EVC

Shadow Router NID Optional L2 VPN Customer L3 CE

EVC = Ethernet Virtual Circuit NID = Network Interface Device = CFM Maintenance Association End Point (MEP) ----- = IP-based probe

Distribution Ethernet Service Provider Router

SAE 4G GW EVC Demarc 3G RNC Probe Demarc (Optional) eNodeB (4G) Cell Site EVC Demarc NodeB (3G) Router BSC 2G BTS (2G)

Distribution Router Mobile Operator Wireline Provider / Mobile Backhaul Operator Mobile Operator Cell Site(s) MTSO

eNode B = Enhanced Node B MTSO = Mobile Telephone Switching Office EVC = Ethernet Virtual Circuit SAE GW = System Arch Evolution GW BTS = Base Transceiver Station NID = Network Interface De`vice RNC = Radio Network Controller BSC = Base Station Controller = CFM Maintenance Association End Point (MEP)

Ethernet Service Provider

VRF Internet

L3 VPN Customer EVC Aggregator L3 PE Demarc Device IP / MPLS Core Managed NID EVC L3 CE Shadow G.SHDSL Router DSLAM Optional

Managed L3 CE

IP Department Third Party Ethernet Service Provider IP Department Or Transport / Aggregation Department

EVC = Ethernet Virtual Circuit NID = Network Interface Device = CFM Maintenance Association End Point (MEP) ----- = IP-based probe © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 56 Ethernet OAM Deployment Use Case #4 Monitoring of Broadband & Wholesale Ethernet Access Service

Residential Ethernet Service Provider Subscribers VRF Internet

EVC Aggregator BNG/ BRAS SP Retail ISP DSLAM Aggregator EVC PON OLT (Bitstream Wholesale) Internet

Retail ISP Wholesale SP subscribers

Broadband Department Third Party Ethernet Service Provider Broadband Department / Or Transport / Aggregation Department Wholesale SP

EVC = Ethernet Virtual Circuit NID = Network Interface Device = CFM Maintenance Association End Point (MEP) © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 57 Ethernet OAM Deployment Use Case #5 Monitoring of Ethernet Service over DOCSIS Cable

L2 VPN Customer

Ethernet NID / Cable MSO Service Provider Cable MSO Demarcation (HFC) (HFC) EVC

L2 VPN NID / Customer Demarcation

L2 VPN Customer

EVC = Ethernet Virtual Circuit NID = Network Interface Device = CFM Maintenance Association End Point (MEP) ----- = IP-based probe

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 59 Ethernet OAM Deployment Use Case #7 Fault Management - Fault detection, Loop detection, Node monitoring, LAG port monitoring, reachability check ….

Customer Ethernet Service Provider

Ethernet NID / Ethernet Demarcation NID / Transport Or MPLS Core Demarcation Ethernet Transport

Customer NID / Demarcation

Customer

EVC = Ethernet Virtual Circuit NID = Network Interface Device = CFM Maintenance Association End Point (MEP) ----- = IP-based probe Link OAM Sessions

. Application: Service Turn-Up (out-of-service) Throughput Test - POP-Local CPE (shown) or POP-Remote CPE

Customer site Ethernet Service Provider L3 Provider POP X,Y A,B Central ip = Y Test Head mac = C Loopback Test Traffic EVC EVC VRF

ip = X ip = W Managed mac = A mac = B CPE A,B L3 PE X,Y L3 Loopback Hardware-accelerated (no CPU intervention) Ethernet Data-plane IPv4/IPv6-aware Loopback Loopback Configurable source IPv4/IPv6 of packets to loop IP address swap (implicit MAC swap)

Cisco 3800 L3 VPN “A” MPLS Core L3 VPN “B” Catalyst 3750-ME CFM

Ethernet access to IEEE 802.1ag L2VPN P2P / MP MPLS VPN (EoMPLS) (EoMPLS / VPLS) MPLS E-LMI Aggregation

Number Description 1 End-to-End Service/Path Verification Fault Verification/Isolation 2 Using E-OAM for Ethernet Access to L3VPN 3 E2E CPE Fault Notification & CPE Corrective Action using Ethernet OAM Interworking

Customer Service Provider

Ethernet MPLS Ethernet Access Aggregation Access CE 11 uPE 11 PE-Agg nPE 11 nPE 31 uPE 31 CE 31 Customer Domain

Service Provider Domain

Operator Domain

Customer Service Provider uPE11 uPE31

EthernetuPE11# ping ethernet MPLSmpid 3100Core level 4 vlan 100 Ethernet Access Access Type escape sequence to abort. CE 11 uPE 11 PE-AggSending nPE 5 11 Ethernet CFM loopback messages,nPE timeout 31 is 2 uPEseconds: 31 CE 31 !!!!! Customer Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms Domain Service Provider Domain

Operator Domain

Customer Service Provider uPE11 nPE11 nPE31 uPE31

Ethernet uPE11#traceroute ethernetMPLS Core0012.017c.3d00 level 4Ethernet vlan 100 Type escape sequence to abort. TTL 255. Per-Hop AccessTimeout is 10 seconds Access Tracing the route to 0012.017c.3d00 on Domain PROVIDER_DOMAIN, Level 4, vlan 100 Traceroute sent via GigabitEthernet0/16 CE 11 uPE 11 PE-Agg nPE 11 nPE 31 uPE 31 CE 31 ------Customer MAC Ingress Ingress Action Relay DomainAction Hops Host Forwarded Egress Egress Action Next Hop ------Service Provider B 1 nPE11 0013.5f21.cec5 Gi3/1 IngOk RlyCCDBDomain Forwarded B 2 nPE31 0007.8508.3485 RlyFDB Forwarded Gi3/1 EgrOK uPE31Operator Domain ! 3 uPE31 0012.017c.3d00 Gi1/1/1 IngOk RlyNone Not Forwarded © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 67 Operational Scenario 1 (cont.)

. Problem Statement Fault Verification and Fault Isolation of ethernet connectivity issues . Problem Solution IEEE 802.1ag (CFM) Ping and Traceroute utilities for reactive troubleshooting of service connectivity issues nPE31(config)#int gig3/1 Customer Service Provider nPE31(config-if)#shutdown uPE11 nPE11 nPE31 uPE31 X

Jan 26 03:14:10.608: %ETHER_SERVICE-6-EVC_STATUS_CHANGEDMPLS Core Ethernet: status of EVC_P2P_100 Proactive Ethernet changed to InActive Access Access Fault Notification uPE11#traceroute ethernet 0012.017c.3d00 level 4 vlan 100 CE 11 uPE 11 PE-AggType nPEescape 11 sequence to abort. TTL 255. Per-HopnPE Timeout31 is 10uPE seconds 31 CE 31 Tracing the route to 0012.017c.3d00 on Domain PROVIDER_DOMAIN, Level 4, vlan 100Customer Traceroute sent via GigabitEthernet0/16 Domain Reactive ------Fault Isolation MAC Ingress Ingress Action Relay ActionService Provider Hops Host Forwarded Egress Egress Action Next Hop Domain ------B 1 nPE11 0013.5f21.cec5 Gi3/1 IngOk RlyCCDB Forwarded B 2 nPE31 0007.8508.3485 RlyCCDB Operator Domain Not Forwarded Gi3/1 EgrDown © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 68 Operational Scenario 2

. Problem Statement Troubleshooting Ethernet access connectivity problems by L3VPN PE . Problem Solution IEEE 802.1ag CFM with Outward-facing / Down MEPs at L3VPN PE

Customer Service Provider

Ethernet MPLS Access Aggregation CE 11 uPE 11 PE-Agg nPE 11 nPE 21 L3VPN PE Customer Domain

Service Provider Domain

Operator Domain

. Problem Statement Troubleshooting Ethernet access connectivity problems by L3VPN PE . Problem Solution IEEE 802.1ag CFM with Outward-facing / Down MEPs at L3VPN PE

L3VPN-PE#show running-config | begin GigabitEthernet3/0/0 Customerinterface GigabitEthernet3/0/0 Service Provider description L3VPN PE to nPE21 gig3/3 ethernet cfm mep level 4 outward domain PROVIDER_DOMAIN mpid 2450 vlan 450 ethernet cfm mep level 4 outward domain PROVIDER_DOMAIN mpid 2350 vlan 350 ! interface GigabitEthernet3/0/0.350 description To CE31 encapsulation dot1Q 350 ip vrf forwarding BLUEEthernet MPLS Core Ethernet ip address 1.1.1.1 255.255.255.0Access Access ! interface GigabitEthernet3/0/0.450 descriptionCE 11 To uPECE21 11 PE-Agg nPE 11 nPE 21 uPE 21 L3VPN PE encapsulation dot1Q 450 Customer ip vrf forwarding RED Domain ip address 1.1.1.1 255.255.255.0 Service Provider

L3VPN-PE#show ethernet cfm maintenance-points remote Domain Can only Ping/Traceroute to remote MEPs marked with *

MPID Level Mac Address Vlan PortState InGressPort Age(sec) Service ID Operator Domain 3350* 4 0012.017c.3d00 350 UP Gi3/0/0.350 20 customer_350_provider 2451* 4 0019.552c.0b80 450 UP Gi3/0/0.450 23 customer_450_provider © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 70 Operational Scenario 3

. Problem Statement End to End Fault Notification of service status to Customer Equipment . Problem Solution IEEE 802.3ah to IEEE 802.1ag (CFM) Inter-Working (IW) for propagation of EVC status and CFM to E-LMI IW for notification to CE

Customer Service Provider

Ethernet MPLS Ethernet Access Aggregation Access CE 11 uPE 11 PE-Agg nPE 11 nPE 31 uPE 31 CE 31 Customer Domain

Service Provider Domain

Operator Domain

E-LMI CFM E-LMI MPLS Core Link OAM Ethernet Ethernet Link OAM Access Access CE 11 uPE 11 PE-Agg nPE 11 nPE 31 uPE 31 CE 31 Customer Domain

Service Provider Domain

Operator Domain

Ethernet MPLS Core Ethernet Access Access

CE 11 uPE 11 PE-AggCE31(config nPE)# 11int fast 0/0 nPE 31 uPE 31 CE 31 CE31(config-if)#shutdown Customer Domain Jan 26 03:40:08.176: %ETHERNET_OAM-6-EXIT_SESSION: The client on interface Fa0/0 has left the OAM session. Service Provider Jan 26 03:40:10.180: %LINK-5-CHANGED: Interface FastEthernet0/0, changedDomain state to administratively down Jan 26 03:40:11.180: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down Operator Domain

Jan 26 03:40:08.176: %ETHERNET_OAM-6-RFI: The client on interface Fa1/0/1 has received a remote failure Indication from its remote peer (failure reason = remote client administratively turned off) Ethernet MPLS Core Ethernet Jan 26 03:40:08.184: %ETHER_SERVICE-6-EVC_STATUS_CHANGEDAccess : status of EVC_P2P_100 Accesschanged to InActive Jan 26 03:40:09.191: %ETHERNET_OAM-6-EXIT_SESSION: The client on interface Fa1/0/1 has left the OAM session. CE 11 uPE 11 PE-Agg nPE 11 nPE 31 uPE 31 CE 31 uPE31#show ethernet service evc Customer Identifier Type Act-UNI-cnt Status Domain EVC_P2P_100 P-P 1 InActive uPE31#show ethernet lmi evc Service Provider UNI Id: CE31_UNI Domain St Evc Id CE-VLAN ------?I EVC_P2P_100 100 Operator Domain

Operational Scenario 3 (cont.)

Jan 26 03:40:36.093: %ETHER_SERVICE-6-EVC_STATUS_CHANGED: status of EVC_P2P_100 changed to InActive

uPE11#show ethernetEthernet cfm maintenance-point remoteMPLS Core Ethernet MPID Level Mac AddressAccess Vlan PortState InGressPort Age(sec) Service AccessID 3100 4 0012.017c.3d00 100 DOWN Gi0/16 21 customer_100_provider CE 11 uPE 11 PE-Agg nPE 11 nPE 31 uPE 31 CE 31 uPE11#show ethernet service evc Customer Identifier Type Act-UNI-cnt Status Domain EVC_P2P_100 P-P 1 InActive Service Provider uPE11#show ethernet lmi evc UNI Id: CE11_UNI Domain St Evc Id CE-VLAN ------I EVC_P2P_100 100 Operator Domain

Operational Scenario 3 (cont.)

Ethernet MPLS Core Ethernet Access AccessCE11# show ethernet lmi evc UNI Id: CE11_UNI CE 11 uPE 11 PE-Agg nPE 11 nPE 31 uPE 31 CE 31 Proactive E2E St Evc Id CE-VLAN ------Customer Fault Notification I EVC_P2P_100 100 Domain

Key: St=Status, A=Active, P=Partially Active, I=Inactive, *=Default EVC, ?=LinkService Down Provider Domain Proactive CPE CE11#show ip interface brief Interface IP-Address OK? Method Status Protocol Action Operator Domain GigabitEthernet0/0.100 100.100.100.11 YES manual down down

. You can perform fault, performance and configuration management with E-OAM for Ethernet based Services . There are multiple suites of protocols from various standard bodies (& vendors) that work in different bucket . And they do interwork to give you a comprehensive troubleshooting and maintenance platform.

Acronym Acronym AIS Alarm Indication Signal MEF Metro Ethernet Forum CCM Continuity Check Message MEN Metro Ethernet Network CCMDB CCM Data Base (see CCM) MEP Maintenance Association End Point CE Customer Edge MEPID MEP Identifier (see MEP) CFM Connectivity Fault Management MHF MIP Half Function (see MIP) EFM Ethernet in the First Mile MIB Management Information Base E-LMI Ethernet LMI (see LMI) MIP Maintenance Domain Intermediate Point E-OAM Ethernet OAM (see OAM) MP Maintenance Point EVC Ethernet Virtual Connection Operations, Administration and OAM Maintenance Institute of Electrical and IEEE Electronics Engineers PDU Protocol Data Unit ITU International Telecommunication Union PE Provide Edge LBM Loopback Message RDI Remote Defect Indicator LBR Loopback Reply RFI Remote Failure Indicator LMI Local Management Interface TLV Type, Length, Value LTM Linktrace Message UNI User to Network Interface LTR Linktrace Reply UNI-C Customer side of UNI (see UNI) MA Maintenance Association UNI-N Network side of UNI (see UNI) MAID MA Identifier (see MA) VID VLAN Identifier MD Maintenance Domain VLAN Virtual LAN