Designing Broadband Networks to Deliver Business and Triple Play Residential Services

Wei Yin Tay Consulting System Engineer

Carrier-Class Ethernet Network Concepts Broadband Services Overview Broadband Transport Architecture Subsystems Service Deployment Models Quality of Service Broadcast TV and IP Multicast Network Availability

Portal Monitoring Billing Identity Address Policy Business Services Residential Services Mgmt Definition Personal EPL EVPL Monitoring Computing Entertainment E-LAN L3VPN VoIP Service/Policy Management Plane Video/Gaming

Business

Corporate

Aggregation Distribution Business Node Node Si Corporate Aggregation Internet Node Service Si Gateway Business

Corporate Si Deep Packet Inspection Residential Si

Si Distribution Aggregation Node Node MPLS Provider Edge STB BBBB TransportTransport NetworkNetwork Access Core Network CPE Aggregation Network Ethernet/DSL MPLS, Ethernet, IP IP/MPLS

Residential services Video—Broadcast TV Video—Video on demand Voice—Voice over IP Data—High-Speed Internet Business services E-LINE E-LAN IP VPN

Service considerations User selects a broadcast channel among multiple channels (<200) BTV Service controlled by video middleware Application requirements Residential Customer Distribution Real-time traffic where majority Node of traffic is downstream Multicast Access Video Node Broadband Efficient method of distributing high RG bandwidth traffic: Network MPEG2: 3.75M to 14M Channel change time < 1500 ms End-to-end delay requirement: < 100–500 ms Network loss < 1E-07 Network jitter < 100–200ms 50ms recovery in failure scenario Very low packet loss is key

Service considerations User selects on video on Video on Demand demand (non-scheduled videos) Service controlled by video middleware Application requirements Residential Customer Real-time traffic where majority MCO-PE of traffic is downstream VoD Server ACCESS Broadband Method of call admission Node control for high bandwidth traffic (3.75M RG Network to 14M video streams) End-to-end delay requirement: < 100–500 ms Network loss < 1E-07 Network jitter < 100–200ms 50ms recovery in failure scenario Very low packet loss is key

Service considerations User makes IP, non-circuit based Residential VoIP voice calls as alternative to traditional PSTN

On-net VoIP calling Residential Customer Off-net calling (hopoff to PSTN) BRAS/MCO-PE Call Control ACCESS Broadband Application requirements Node RG Network Real-time bi-directional traffic End-to-end delay requirements: 150ms Network loss < 1E-02 Network jitter < 20–30ms < 1s recovery in failure scenario Low jitter is key

Service considerations Best effort data service (Internet, Residential VoIP e-mail, chat rooms, gaming)

Bridged or PW (EoMPLS) Residential Customer Provisioned via IP or BRAS/MCO-PE PPPoE Internet access Call Control ACCESS Broadband Node Application requirements RG Network Non-real-time bi-directional traffic Best effort

Service considerations EPL Port/VLAN based point-to-point UNI UNI business service Broadband Service multiplexing at Network UNI (VLAN)

High availability-protected Port Based UNI Fully transparent offering Service requirements Scalability for large sites Real-time traffic and stringent EVPL packet loss requirements UNI UNI SLA—CIR/PIR/burst, loss Broadband Network

VLAN Based UNI

Business VoIP Call Control Corporate L3 Edge Multicast Channels 239.2.1.1 = ch 3 Si IP 239.2.1.2 = ch 4 235.3.1.1 = ch 5 Si Si IP 2 Residential L MPLS or Internet Data Ethernet

STB

L3 Edges Are Distributed at Different Points in the Network For example, VoIP, VOD IP/TV ®, broadcast IP/TV insertion point: driven by the minimal network operational needs, the simplicity and efficiency of the IP multicast transport

Ethernet Multipoint Service (EMS) Service considerations Port/VLAN-based multipoint business service Service multiplexing at MEN UNI (ERMS) High availability-protected Fully transparent offering (EMS) Service requirements Ethernet Relay Multipoint Service (ERMS) Small-to-mid size multipoint connectivity Corporate/campus LAN extension MEN Disaster recovery

Business

Corporate

Aggregation Business Node Distribution Si Node Corporate Aggregation Internet Node Service Gateway Business Si

Corporate Si Deep Packet Inspection Residential Si

Aggregation Si Distribution Node Node MPLS Provider Edge STB BBBB TransportTransport NetworkNetwork Access Core Network CPE Aggregation Network Ethernet/DSL MPLS, Ethernet, IP IP/MPLS

Customer AggregationCore Aggregation Customer Business Business Native Native Ethernet Ethernet and IP and IP

MPLS and IP MPLS and IP MPLS

Residential Residential

MPLS/IP Ethernet/IP

Two aggregation network models Similar Layer 2 and Layer 3 BUS mechanisms Support point to point and multipoint Layer 2 and Layer 3 transport Support the same residential, business and wholesale broadband services

May use Spanning Tree Protocol (STP) domain for Aggregation the Layer 2 BUS transport Business

Si Supports virtualized Corporate Distribution Aggregation Node Layer 2 services thru native Node 802.1q and Ethernet/IP 802.1ad bridges Si Si Residential Provides optimal Layer 2 multipoint transport that is topology independent Si STB Can aggregate other access services as: Mobile RAN, Legacy ATM/FR/TDM with L2TPV3

Allows different or common administrative domains

Supports virtualized Layer 2 and Business Access Aggregation 3 services thru MPLS-based Corporate Node VPNs (EoMPLS and H-VPLS) MPLS/IP Supports traffic engineering thru Si Si PW MPLS TE mechanisms Residential Distribution Aggregation Node Can aggregate other access Node services as: Mobile RAN, Legacy ATM/FR/TDM with MPLS AToM STB Pseudo Wire (PW) used to transport Layer 2 domain across the MPLS/IP network

L3 edge point—why is this Aggregation significant? Operational L2 VPN Bandwidth efficiency Customer Scalability Centralized This influences possible L3 Edge

architectures Residential

Centralized vs. distributed L3 IP edge points Centralized = service L3 edge at a single “point” Distributed L3 Edge Distributed = service L3 edge at multiple “points”

239.2.1.1 = ch 3 Business L3 Edge 239.2.1.2 = ch 4 Corporate 235.3.1.1 = ch 5 LCO-PE MCO-PE Multicast Channels Q /.1 Si LS Si MP Eo Residential Aggregation Internet Data

Voice Call Control STB

Layer 3 Edge Point Reside at Central Location in Network Ethernet/IP L2 service VLANs used from subscriber to MCO-PE MPLS/IP Pseudowire used from LCO-PE to MCO-PE for L2 tunneling For example, Internet access insertion point: driven by the existing service model, operational structure, traffic patterns

Distributed Voice Call L3 Edge Control Subinterfaces (VoiP, Video) Multicast Channels Access Video on Demand

Si Residential IP Internet Data VoiP (L2) Aggregation Network Video (L2) Si Si

STB Data (Trunk) EoMPLS (PW) Centralized

L3 Edge (Data) Distributed L3 edge for VoIP/IP/TV Broadcast/IP/TV VOD Minimal network operational needs Allows multicast channels to be distributed at closest edge to subscribers Centralized L3 edge for data Driven by the existing service model, operational structure, traffic patterns

Policy Plane Policy Plane Edge Edge

Access PE- Edge BRAS Access PE- Edge BRAS AGG AGG

Residential Si N-PE BRAS Residential Si N-PE BRAS 1q 802.1q 2. Si80 Core W Si Core VLAN LS P 802.1q/PVC Network 802.1q/PVC EoMP Network Si(802.1q with RSTP) Si IP/MPLS IP/MPLS Access PE- Si SCE Access PE- Si SCE STB AGG STB AGG Node N-PE Node N-PE

Si Si PE- BRAS PE- BRAS AGG AGG Ethernet/IP Aggregation MPLS/IP Aggregation

PPPoE or IP subscriber sessions carried PPPoE or IP subscriber sessions carried over a VLAN (802.1q) in the Ethernet/IP over a EoMPLS PW in the MPLS/IP aggregation network aggregation network Subscriber line identity provided by the Subscriber line identity provided by the DSL Forum PPPoE line-id VSA or DHCP DSL Forum PPPoE line-id VSA or DHCP Op82 Op82

Portal Monitoring Billing Subscriber Identity Address Policy Database Mgmt Definition

Policy Plane (per subscriber)

Access Edge DSLAM Multicast PE-AGG • IGMP Filtering • IGMP Snooping IGMP Snooping Residential N-PE • IGMP Message Throttling Si • IGMP Message Suppression Service VLAN IP Unicast and Si 802.1q/PVC 802.1q Multicast Routing Si or STB RFC 2547 bis MPLS VPN DSL PE-AGG Access Si Node Si N-PE PE-AGG

This Service Model Is Same for Both Ethernet/IP and MPLS/IP Aggregation

Policy Plane Policy Plane

AccessPE- Edge AccessPE- Edge AGG AGG

Si Business PE- Si N-PE N-PE q Business AGG 2.1 MPLS VPN VLAN Corporate 80 Corporate (802.1q with RSTP) MPLS VPN 802.1q PW Si Si L3VPN Core PE Core Business Si Network Si Network Si VLAN Business802.1q PE- IP/MPLS Corporate802.1q 802.1q PW IP/MPLS (802.1q with RSTP) Si80 Si802 Corporate AGG 2.1q .1q MPLS VPN Access N-PE MPLS VPN Access N-PE Node Node Si L3VPN Si L3VPN PE- PE PE- PE AGG AGG Ethernet/IP Aggregation MPLS/IP Aggregation

DSL UNI (PVC or 802.1q) 1:1 VLAN DSL UNI (PVC or 802.1q) 1:1 VLAN mapping in DLAM mapping in DLAM L3-VPN provided by a dedicated L3 VPN L3-VPN provided by a dedicated L3 VPN PE or by N-PE PE with EoMPLS tunnels from PE-AGG

Policy Plane Policy Plane

AccessPE- Edge AccessPE- Edge AGG AGG

Si N-PE Business Si N-PE Business Corporate 802.1q Corporate 802.1q or QinQ (L2PT) PW 802.1q PW Si Si Core Core Business Si Network Si Network Si VLAN Business802.1q PE- IP/MPLS Corporate802.1q 802.1q PW IP/MPLS (802.1q with RSTP) Si PW Si Corporate DSL AGG PE- Access N-PE Access AGG N-PE Node Node Si Si PE- PE- AGG AGG Ethernet/IP Aggregation MPLS/IP Aggregation DSL UNI (PVC or 802.1q) 1:1 VLAN DSL UNI (PVC or 802.1q) 1:1 VLAN mapping in DLAM mapping in DLAM Ethernet VLANs in the metro Ethernet EoMPLS PW backhaul from PE-AGG/U-PE aggregation network EoMPLS pseudowires for ERS services for the DSL UNI and from the N-PE ERS/EWS for Ethernet UNI

Policy Plane Policy Plane

AccessPE- Edge AccessPE- Edge AGG AGG

Si N-PE Business Si N-PE Business Corporate Corporate802.1q 802.1q or QinQ VFI 802.1q VFI Si Si Core Core EoMPLS PW Business Si Network Si Network Si VLAN Business802.1q PE- VFI IP/MPLS Corporate 802.1q VFI IP/MPLS (802.1q with RSTP) Si Si Corporate DSL AGG PE- Access N-PE Access AGG N-PE Node Node Si VPLS Si VPLS PE- PWs PE- PWs AGG AGG Ethernet/IP Aggregation MPLS/IP Aggregation DSL UNI (PVC or 802.1q) 1:1 VLAN DSL UNI (PVC or 802.1q) 1:1 VLAN mapping in DLAM mapping in DLAM Ethernet VLANs in the metro Ethernet EoMPLS PW backhaul from PE-AGG/U-PE for ERS services for the DSL UNI and aggregation network VPLS from N-PE ERS/EWS for Ethernet UNI

Portal Monitoring Billing Subscriber Identity Address Policy Database Mgmt Definition

Policy Control Plane (Per Subscriber) Content Network Business Access L2/3 Edge VoD TV SIP Corporate Aggregation Node

Business Distribution Node Corporate Si BRAS Business Ethernet Access Aggregation Network Si Core Network Node Corporate MPLS, Ethernet, IP IP/MPLS Internet Si SCE Residential DSL Aggregation Access Node Node Si

Si Access Node Distribution MPLS PE Business Node STB Aggregation Node

Corporate

InternetVoice and and Business Video traffic Traffic Utilize Utilize Per-Service Per-Subscribe r Diff-ServQoS QoSModel Model in Access, in Access, Aggregation Aggregation and Coreand Core

VoIP telephony service Subscriber line: no CAC required The number of calls generally limited at the application level Aggregation network: no CAC required Generally provisioned with sufficient class bw to cope with peak during working and failure case TV broadcast service Subscriber line: CAC required Aggregation network: no CAC required Generally provisioned with sufficient class bw to cope with peak during working and failure case Video on demand service Subscriber line For fixed number of channels per subscriber line: no CAC required For variable number of channels per subscriber line/variable bandwidth cases: CAC required Aggregation network: CAC required Potential for congestion both in working and network failure cases

Topology-Unaware Off-Path CAC Topology-Aware Off-Path CAC Topology-Aware On-Path CAC • Embedded within the VoD • CAC decision outsourced • Dynamic adjustment to any server to Policy Server topology change 1 • CAC decisions not 2 • Policy server interaction 3 • Requires network level synchronized with the with the network for connection admission network topologies topology synchronization control signaling: RSVP

Integrated CAC 2 + 3 Integrated video CAC approach combines two methods COPS: Common Open VOD stream will be denied if business rules of either fail Policy Server Prioritize blocking of free VOD vs. pay VOD in network failure scenarios 1 VOD Request

Deny or Admit 5 2 2 2 RSVP-CAC 4 Policy 6 COPS Server 2 BRAS L3 3 L3 Core L3 VoD Servers L3 L3 PE-AGG DSLAM MPLS PE N-PE

Bandwidth Fast Service Extra Efficiency Convergence Separation Services

MPLS TE Security IP Multi-VRF Fast IGP (BFD) Monitoring PIM SSM, MPLS VPN L3 Fast Multicast Load Balancing SSM mapping Multicast VPN Convergence Anycast Sources

802.1q 802.1q/ 802.1ad 802.1w RSTP L2 IGMP Snooping VLAN

H-VPLS TE-FRR (Only Link) L2-VPN L2 IGMP Snooping

Transport options in the aggregation network? L2—Ethernet switching L2 emulated—H-VPLS L3—P multicast What are the main benefits of IP multicast in the aggregation network? What are the main factors contributing to channel change delay?

Portal Monitoring Billing Subscriber Identity Address Policy Database Mgmt Definition

Policy Plane (Per Subscriber)

N-PE Multicast Access Edge • IGMP Registration • PIM-SSM, SSM Mapping PE-AGG

Si N-PE Residential

Si VLAN IP unicast and Multicast routing (Retail) 802.1q/PVC (802.1q Aggregation Network) or Core Network Si Multicast VPNs (Wholesale)IP / MPLS

STB PE-AGG Si Optional IGMP Snooping DSL N-PE Access Node Si

PE-AGG DSLAM Multicast PE-AGG IGMP snooping to optimize the multicast distribution • IGMP Filtering Optimal multicast with any topology and in case of topology failure • IGMP Snooping • IGMP Message Throttling Resiliency can be provided at both L2 and L3 based on N-PE capabilities • IGMP Message Suppression

Portal Monitoring Billing Subscriber Identity Address Policy Database Mgmt Definition

Policy Plane (Per Subscriber)

N-PE Multicast Access Edge • IGMP Registration • PIM-SSM, SSM Mapping PE-AGG

VFISi Residential 802.1q VFI IP Unicast and Multicast Routing (Retail) 802.1q/PVC VFI Si or VFI 802.1q STB PE-AGG Multicast VPNs (Wholesale) Optional IGMP Snooping DSL Access Node N-PE VFISi

PE-AGG DSLAM Multicast All channels distributed to all DSLAMs • IGMP Filtering • IGMP Snooping Multicast distribution not aligned with the topology • IGMP Message Throttling Not optimal Multicast distribution in case of topology failure • IGMP Message Suppression

Portal Monitoring Billing Subscriber Identity Address Policy Database Mgmt Definition

Policy Plane (Per Subscriber)

Access Edge Aggregation Node Multicast Aggregation Node • Multicast VPN • IGMP registration IGMP Snooping • PIM-SSM, SSM Mapping

Si Distribution Node Residential Service VLAN IP Unicast and Si 802.1q/PVC 802.1q Multicast Routing IP Unicast Routing or Si RFC 2547 bis MPLS VPN STB Aggregation Si DSL Node Access Node Distribution Si Node

Aggregation DSLAM Multicast Node • IGMP Filtering • IGMP Snooping • IGMP Message Throttling • IGMP Message Suppression

Presentation_ID © 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 36 Benefits of Distributed L3 Edge for Multicast Efficient Multicast Distribution L3 enables shortest path to source L2 needs IGMP snoop, replicates every m’cast packet around ring Wasted bandwidth, less room for HSI and other services Extra processing stress on L2 forwarding engines

Local Local Ad Server or Drop Ad Server or Drop Multilingual Drop Multilingual Server Drop Server L3 L3 L3 L3 DR 1 4 1 4 L2 H-VPLS PIM-SM L2 L3 L3 L2 L2 L3 L3

2 2 3 3

Ring-VPLS IP Multicast Implementation Implementation

Presentation_ID © 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential 37 Benefits of Distributed L3 Edge for Multicast Anycast Support L3 PIM Querier (not active) L3 192.1.1.1 192.1.1.1 L2 DSLAM DSLAM L2 L3 L3 H-VPLS PIM

L2 L2 L3 192.1.1.1 DSLAM 192.1.1.1 L3 DSLAM N-PE L3 PIM L3 PE-AGG DR

Ring-VPLS IP Multicast Implementation Implementation

No anycast support in PIM-SSM inherently supports anycast aggregation network Shortest path election through IGP No shortest-path election to the source Load sharing between multicast source because mcast traffic only enter through the designated router Instantiate multiple replication tree for same multicast destination All of above results in inefficient distribution multicast pkts

Security Threats IGMP V2 to v3 Map 192.1.1.1 192.1.1.1 L3 L2 DSLAM L3 DSLAM L2 L3 PIM-SM L3 H-VPLS Anycast + SSM L3 DSLAM L3 DR DSLAM L3 L2 192.1.1.1 192.1.1.1 Multi-Gig Attack L3 L2 Floods rng Spoof 192.1.1.1 Spoofing not possible Launch 1G to 10G Attack Ring-VPLS IP Multicast Implementation Implementation

SSM-Map not supported at the PE-AGG SSM enabled at the PE-AGG Attack launched from spoofed source SSM-Map enforces source aware mapping address can create outage on ring Minimizes ability to spoof service H-VPLS aggregation network is vulnerable Anycast works with SSM to spoofed sources

U-PE3 DSLAM R2 L3 L3 Logical View L3

R1 L3 DSLAM L3

? L3 R2

Ring-VPLS L3 Implementation Implementation L2 network segmented due to No impact to unicast traffic. IGP node failure converges around failed node Asymmetric routing can blackhole unicast traffic

Common misconception is that IP multicast causes slow channel change Only when mcast request has to go to the regional headend is it >100ms Typical PE-agg router will serve thousands of subscribers and probability is that the next channel is already multicast to the PE-agg node The main culprit is waiting for the I-Frame—this can be solved by a frame cache approaches

Channel Change Latency Factor Typical Latency

Multicast Leave for old Channel 50 msec Delay for Multicast Stream to Stop 150 msec Multicast Join for New Channel 50 msec Jitter Buffer Fill 150-200 msec Conditional access delay 0 msec – 2 sec I-Frame Delay 500 msec

MPLS/IP aggregation: MPLS TE/FRR Fast IGP convergence/BFD Multicast fast convergence IP services have sub-second convergence MPLS services have 50 ms convergence with MPLS FRR Ethernet/IP aggregation: RSTP Fast IGP convergence/BFD Multicast fast convergence IP services have sub-second convergence Ethernet services have sub-second convergence for hub and spoke and 2–3 seconds convergence for rings