Architecture and Design of Metro Ethernet Over Optical Transport Networks

Architecture and Design of Metro Ethernet over Optical Transport Networks

• Metro Ethernet Services • Metro Ethernet options • Mapping Data over SONET • Optimizing SONET for Data Transport • Topologies and Architectures

• Ethernet Private Line (EPL) Pt-Pt CAGR:58.6% • Ethernet Internet Access (EIA) CAGR: 102.4% • Ethernet Any-to-Any CAGR: 134% Yankee Group Report August 19,2003

SONET/SDH RPR Remote Regional Office 1 Headquarters DWDM/CWDM Ethernet MPLS/IP Ethernet- Connected 10 Mbps Branch Ethernet

SP Remote 100 Mbps Metro Ethernet Office 2 Ethernet Network 10 Mbps Ethernet • Delivers an Ethernet UNI to Remote Enterprises/SMB for MAN/WAN Office 3

connectivity Ethernet- Connected • SP has multiple transport options Branch

• Ethernet becomes the ubiquitous interface: single technology for LAN, MAN and WAN • Efficient packet-based infrastructure: IP friendly • Cost effective interface with flexible bandwidth offerings: 10/100/1000/10000 Mbps • Geographical independence: Ethernet over Optical, IP or MPLS

Transparent • LAN interconnect Services

• Service aggregation High Availability • Interconnect data centers High Bandwidth

• Backup and Low Latency disaster recovery Cost Effective • Connect to hosting services Manageable • Value-added services and Secure

How SPs Deliver This Is Largely Irrelevant… Metro Ethernet Is Simply a Tool in the Tool Box

Ethernet-Based Services

Layer 1 Layer 2 Layer 3

Point-to-Point Multipoint

Ethernet Ethernet Ethernet Ethernet Ethernet Relay MPLS Private Relay Wire Multipoint Multipoint VPN Line Service Service Service Service

Transparent LAN Service Similar to Leased Line

Analogous to Frame Relay Analogous to Private Line

Cisco Cisco ONS 15454 ONS 15454

CE CE

• Dedicated, transparent private line between sites • Guaranteed SLAs per connection • Built with dedicated bandwidth/wavelength • ML-series card integrates switching functionality within the ONS-15454 • Can be built with SONET, CWDM, DWDM © 2003, Cisco Systems, Inc. All rights reserved. 8 The Ethernet Relay Service (ERS)

Non-Multiplexed Interface

s T1 bp M Service Multiplexed 20 Interface Remote Office 1

20 MbpsT T3 1

Remote Office 2

1 0 • Service multiplexing allows single port to M b T1 p Acs provide service to multiple end-points (referred c es to as “point-to-multipoint) s • EVC identified by the VLAN Remote Office 3 • CE-VLAN must be the same as SP-VLAN • Service analogous to Frame Relay, functionally, it is the same – encourages a router as CE edge device, not a switch (except for remote site)

int faste 0/1 int faste 4/0

VLANs 101-110 Remote Office 2 VLANs 101-110 • Analogous to a private line in that all data transverses, unaltered across the EVC • Port-to-Port mapping, no service multiplexing allowed. Therefore all services must exist on one port (All-to-One Bundling) • Switches or routers can be deployed as CE edge devices © 2003, Cisco Systems, Inc. All rights reserved. DMAC 10 The Ethernet Multipoint Service (EMS)

• Often referred to as a Transparent LAN Service (TLS) • Service Provider cloud appears to be a switch, with UNI supporting VLAN transparency and All-to-One Bundling • While multipoint by definition, it can also be point-to-point (such© as2003, Cisco EWS) Systems, Inc. All rights reserved. 11 Ethernet-based MPLS VPNs

Connection-Oriented Connectionless VPN Topology VPN Topology VPN A VPN A VPN B VPN C VPN B VPN C VPN C VPN B VPN C VPN B

VPN A VPN A VPN A VPN A

VPN B VPN B VPN C VPN C VPN B VPN C VPN C VPN A VPN B VPN A

• Managed service opportunity for the provider (more money for the SP) • Opportunity for provider to offer “value-added” services (such as content hosting, IP Centrex) • Ethernet access provides low-cost, “customer friendly” interface • Any Layer 2 access mechanism at the UNI (interworking © 2003, Cisco Systems, Inc. All rights reserved. comes “for free”) 12 Metro Ethernet Options

Ethernet Relay Ethernet Relay Service Service

Cisco 7600 Cisco 7600 Cisco 12000 Catalyst 3550 Catalyst 3550 Cisco 12000 SONET End-to-End Dark Fiber STM-16/ Or CWDM/DWDM STM-64 ONS 15454 Ethernet Relay ONS 15454 Service • Transparent LAN-optimized • Ethernet service over an existing structured transport network • Greenfield and overbuild architectures • Single CPE • Effective for a small number of • Larger implementations customers • Build-in resiliency scheme (UPSR, • Redundancy has to be carefully BLSR, SRP) can eliminate planned due to Spanning Tree Spanning Tree and MPLS © 2003, Cisco Systems, Inc. All rights reserved. 14 Logical Hub-and-Spoke on a Physical Fibre Ring

MUX-AD-1510 IP/MPLS

MUX-8 MUX-AD-1530

MUX-8 MUX-AD-1550

East Facing GBIC MUX-AD-1570 West Facing GBIC

• Many providers will look to deploying a physical ring, but using a logical point-to-point service with guaranteed SLAs on top • Technologies such as SONET/SDH, CWDM, and DWDM provide this capability

SONET/SDH Switched Ethernet using Spanning Tree Protocol

DWDM and SDH Backbone Network

OC-N

STM1/4/16/64 STM1/4/16/64 Business Metropolitan Network Network

STM-N STM-N Metro POP DPT/RPR DWDM/CWDM

SONET/SDH Switched Ethernet using Spanning Tree • Installed base in service providers • Low cost solution over dark fiber • Evolutionary approach for the service • Perceived simplicity of Ethernet providers vs. revolutionary switching • Best choice for large scale deployments • Fairness, bandwidth, delay/jitter • Best use of fiber infrastructure dependant on location on the ring • Hierarchical bandwidth • 10-12 node limit • Best fiber utilization for dual homing • LAN switch as edge device (no MPLS, • Next-gen SONET, with VCAT, LCAS, Traffic shaping, BGP, etc) GFP, will help optimize SONET for data • 1-50 seconds convergence (standard • 50 ms convergence 802.1d or 802.1w) DPT/RPR CWDM • Shared packet ring scales bandwidth • Guaranteed bandwidth per lambda up up to 5 Gbps to 8 nodes • SONET framing provides insertion • Logical star over a physical ring point for many providers • EtherChannel or Layer 3 load balancing • Spatial reuse provides good bandwidth for redundancy utilization • 200 millisecond failover • Optimized for Layer 3 (currently) • More consistent delay/jitter and better • Large number of nodes (128) per ring access onto the ring • 50 ms convergence •Still a low-cost Ethernet switch at edge © 2003, Cisco Systems, Inc. All rights reserved. 17 Cisco COMET Multiservice over SONET/SDH (MSOS) Strategy

• Deploying advanced services is key to profitability Advanced Services • Ethernet, Video, VoIP, and SAN Interconnect key service needs

• QoS is necessary for advanced services and providing SLAs • Cisco IOS provides advanced per- QoS Engine packet, per port QoS • Packet Multiplexing is crucial for network efficiency and scale • Cisco technology innovations Packet Multiplexing to the (MPLS, DPT, CDL, EoS) enable this Transport Layer • SONET/SDH will need to become more efficient • GFP, CCAT, VCAT, and LCAS all Data-Optimized SONET/SDH offer incremental improvements © 2003, Cisco Systems, Inc. All rights reserved. 18 ONS 15454 ML-Series Ethernet Cards • ML-Series enables private line services and Layer 2/3 Packet Multiplexing into SONET/SDH • Packet processing capabilities allow creation of multipoint services • Common QoS feature set and code base with existing enterprise networks • Common management with Cisco Transport Manager, as well as SNMP and TL1 support

Shipping Ports Services Deployed Since Deployed

Tens of Layer 1 Private Line and E-Series 1999 Tens of Layer 1 Private Line and Thousands basic switched services

Line Rate Gigabit Ethernet G-Series 2002 Thousands Private Line

Integrated Layer 2/3 ML-Series 2003 Trials capabilities to provide advanced switched NEW!!! services and SLAs

Internet/LH/Data BB Point to Point, Hub & Spoke, and Resilient Packet Ring architectures Cisco 7600 Seamless transition to Enterprise A pure packet network ONS with Cisco IOS 15454

Multimillion ONS packets per 15454 second ONS Access Ring X Mix & match: performance at 15454 Catalyst, G- L1/L2/L3 Series & ML- Series solutions

ONS Enterprise C Enterprise C 15454

Frame Relay-like SONET/SDH and/or per VLAN Ethernet Services Rapid Spanning Tree Protected With QoS Enterprise B Enterprise C

QoS Actions QoS Actions Layer 3 at Ingress at Egress Functionality

Classification Policing & Marking Queue & Schedule

—

Tokens

—Burst Size Token Ar rival Rate Overflow Tokens

Packets Packets deQueued Arr iv ing

Exceed or Violat e

• Classification • Mark or • Fairly • Static IP by: Discard non- allocate routing: conformant unused Interface bandwidth OSPF, ISIS, packets EIGRP, BGP, Bridge Group • Multiple RIP, HSRP, 802.1p queues/port VRF-Lite and ACL IP Precedence • MPLS-capable DSCP

SONET/SDH Circuit provisioning via CTM, CTC, and TL1 ONS A to Z Point-Click Auto Provisioning … 15454

Internet Enterprise B Access Ring OC-12 Enterprise A InterOffice Ring OC-48 Enterprise A

Access Ring OC-12 STEP 3: Select Destination (Z) – node, slot, port Enterprise C ML-Series and IOS: STEP 1: Set Circuit Attributes – CTM or CLI for name, type, size Configuration, SNMP Enterprise B Enterprise C for Monitoring STEP 2: Select Source (A) – • Alarms node, slot, port • Performance

• Generic Framing Procedure expands on functionality of X.86 in that it provides for: Encapsulation of L2/L3 PDU client signals (GFP-F) Encapsulation/mapping of block coded client signals (GFP-T) Multiplexing of multiple client signals into a single payload

Escon/Ficon Escon/Ficon

10/100/GbE 10/100/GbE L2/L3 MSPP MSPP L2/L3 Switch Switch

Data Layer Bridge/Switch/Router in TNE Network Bridge/Switch/Router in TNE

PHY-T GFP-TGFP-F GFP-FGFP-F GFP-F GFP-T PHY-T PHY-X PHY-X 10Base 10Base Transport Layer Network 100Base 100Base ESCON ESCON 1000Base 1000Base FICON FICON 10GBase 10GBase Fiber Channel Fiber Channel 1000BaseLX 1000BaseLX 1000BaseCX 1000BaseCX

• Pro: Supports Ethernet, PPP and SAN interfaces ITU Standard • Con: Added flexibility adds complexity (not really that bad though and can be overcome by a good provisioning scheme)

• Pro: IETF endorsed It is simple Can be deployed today! • Con: Most customers want GFP

• In terms of overhead, GFP has more overhead; however, BCP uses padding and GFP uses idle patterns for synchronization; these can all add considerable overhead, thus making the two roughly equal • BCP transports Ethernet and some useless LAN protocols, while GFP transports Ethernet, PPP and SAN protocols • GFP is an ITU standard which carries more weight with IXCs, RBOCs and PTTs

10/100 EoS Card STS-12c GigE EoS Card STS-12c GigE EoS Card STS-6c STS-6c Side View STS-3c Side View STS-3c Side View STS-1 STS-1 Port 1 STS 1 STS 1 Port 2 STS 1 STS 1 STS-1 Port 1 Port 3 STS 1 STS 1 STS-3c STS-6c Port 4 STS 1 Port 1 STS 1 Port 2 STS-9c Port 5 STS 1 STS 1 STS-12c Port 2 STS-24c Port 6 STS 1 STS 1 Port 3 Port 7 STS 1 STS 1 Port 8 Port 8 STS 1 STS 1 Port 4 Port 9 STS 1 STS 1

Port 10 STS 1 STS 1 ETHERNET CARD ETHERNET ETHERNET CARD ETHERNET Port 11 CARD ETHERNET STS 1 STS 1 Gigabit Ethernet (GBIC) Ports Port 12 Gigabit Ethernet (GBIC) Ports

10/100 RJ 45 Ethernet Ports Ethernet 45 RJ 10/100 STS 1 STS 1

Example: • 100 mbps service required for a CMTS (residential cable access) and a high school LAN • Assume STS bandwidth scaling onto a shared STS-3c • Service provider preserves 155 mbps of transport bandwidth (enough to deliver 84 DS-1 services for over $20K/month or 3 DS-3 services for over $10K/month)

• Virtual Concatenation is a method of creating a payload made up of 2 or more associated SPEs transported through a network completely independently • Channels are “administrated” together; common processing of channels is limited to end points • Channels not constrained to same path (end point delay equalization required) • Channels not necessarily constrained to same transport channel (e.g., same STS-12)

SONET Paths (SPEs)

Service Bit Rate Without VC With VC

Ethernet 10 Mbit/s STS-1 (20%) VT1.5–7v (89%)

Fast Ethernet 100 Mbit/s STS-3c (67%) STS-1–2v (Approx. 100%)

Gigabit Ethernet 1000 Mbit/s STS-48c (42%) STS-3c–7v (95%)

Low Speed ATM 25 Mbit/s STS-1 (50%) VT1.5–16v (98%)

Fibre Channel 200 Mbit/s STS-12c (33%) STS-1–4v (100%)

Fibre Channel 1000 Mbit/s STS-48c (42%) STS-3c–7v (95%)

ESCON 200 Mbit/s STS-12c (33%) STS-1–4v (100%)

• A mechanism for dynamically adjusting the size of a virtually concatenated channel Allows TDM services more flexibility for handling dynamic bandwidth demands Relies on the NMS/EMS to provision the bandwidth change Allows channel size adjustment to be hitless • Currently defined for SONET/SDH Proposed for inclusion into G.709 OTN with ODUk VC • Bit-oriented protocol encapsulated in control packets • Applies to high order (STS) and low order (VT)

• For high order VCs, it is communicated in a control packet carried in bits 1–4 of the H4 Path Overhead byte Carried across a 16 frame multiframe • For low order VCs, it is communicated in a control packet in bit 2 of the Z7* Path Overhead byte Carried across a 32 frame multiframe

* aka K4 and originally V8

• Dedicated Network

DWDM and SONET Backbone • High Capacity Network

OC-N

OC48/192 OC48/12 Regional Network Business Metropolitan Network Network

Internet/LH/Data BB Point to Point, Hub & Spoke, and Resilient Packet Ring architectures Cisco 7600 Seamless transition to Enterprise A pure packet network ONS with Cisco IOS 15454

Multimillion ONS packets per 15454 second ONS Access Ring X Mix & match: performance at 15454 Catalyst, G- L1/L2/L3 Series & ML- Series solutions

ONS Enterprise C Enterprise C 15454

Frame Relay-like SONET/SDH and/or per VLAN Ethernet Services Rapid Spanning Tree Protected With QoS Enterprise B Enterprise C

ML-Series L2 Advantages: Customer A L2VPN L2 Internet Access/L2VPNs

L2 Mappings to MPLS E/FE/GE L2 and SONET Protection Stat Muxing Creates B/W Efficiencies Central Office VLANs Access Ring IP/MPLS E/FE/GE SONET/SDH Customer Switches/Routers

EoMPLS

15454 15454 ML/G-Series VLANs Cisco 7600 Access Ring IOF Ring GE/POS E/FE/GE SONET/SDH SONET/SDH Customer Switches/Routers 15454 15454 Cisco 12000

E/FE/GE VLANs E/FE/GE Internet

Customer A L2VPN Customer Switches/Routers

ML-Series L3 Advantages: L3 Internet Access for Customers L3 and SONET Protection

L3 Routing Protocol Support 12xxx IP / MPLS BGP Rate Limiting Creates B/W Efficiencies Internet TDM Customer Network PE Router DS-1/3 OSPF- Area 0 Static Route Cisco 7600 IS-IS – Level 2 OC-n Customer Router VRF-Lite EIGRP - ASN

PoS FE/GE GigE 15454 15454

SONET/SDH SONET/SDH

15454 15454 15454 15454 15454 FE/GE

Customer Routers OSPF- Area Y FE/GE OSPF- Area X FE/GE IS-IS – Level 1 IS-IS – Level 1 EIGRP – ASN EIGRP – ASN BGP- ASN BGP - ASN Customer Routers Customer Routers © 2003, Cisco Systems, Inc. All rights reserved. 40 ML-Series Managed Services For Service Providers

ML-Series Advantages: Provide L2/L3 Services Internet Access/Private Line Single Customer Ring Central Office

Cisco 7600 TDM Services Internet Voice FE/GE Services DS-1/3 OC-n

Site 1 Site 4 15454 15454 E/FE/GE E/FE/GE

SP Owned SDH/SONET Ring 15454 15454 15454

PBX PBX E/FE/GE E/FE/GE

ML-Series L2 Advantages: LAN Interconnection IP Video Distribution VoIP – QoS Guarantees

Enterprise Headquarters VLAN A -Data Voice Gateway

VLAN B To PSTN

VLAN C IP Video Server

G/ML-Series VLAN A -Data VLAN A -Data Site 2 15454 Site 3 Enterprise Owned/ VLAN B – IP Voice Leased Fiber Ring VLAN B – IP Voice

15454 15454 VLAN C -Video VLAN C -Video ML-Series ML-Series

ML-Series L3 Advantages: Internet Access/Intranet Web Access IP Routing Protocol Support Works with L2 Features

Intranet Web Servers

Enterprise Headquarters

Internet Voice Gateway Firewall To PSTN

IP Video Server

G/ML-Series IP Data IP -Data Site 2 15454 Site 3

IP Voice OSPF, EIGRP, RIP, BGP IP Voice

15454 15454 IP Video IP Video ML-Series ML-Series

© 2003, Cisco Systems, Inc. All rights reserved. 43 Summary • Cisco’s Multiservice over SONET/SDH (MSOS) Strategy enables profitability through: – Support for Advanced Services – Enhanced QoS – Integrated Packet Multiplexing – Data Optimized SONET/SDH Transport • ML-Series for the ONS 15454 Multiservice Provisioning Platform (MSPP) delivers private line and switched Ethernet services • Only Cisco Systems has the expertise in both packets and circuits