PACKET-OPTICAL THE INFINERA WAY

INFINERA 1 OPTICAL FIBER PROVIDES almost lossless telephone service) to cover a wide range of We hope you find Packet-Optical the PREFACE transmission of signals at an ultra-wide solutions and networks with varying degrees Infinera Way informative and useful, whether range of frequencies. Packet switching, of capabilities and functionality. you use it to research a particular subject or implemented using the Ethernet family read the complete volume from beginning Packet-optical integration has some great of protocols and interfaces, offers one of to end. advantages in terms of cost and service the most efficient ways to sort and direct differentiation. Infinera´s technologies take The descriptions are kept independent streams of digital data. Packet-optical this one step further, with benefits including of product releases as much as possible. networking combines these two outstanding reduced equipment, lower operational costs Current details of the Infinera product technologies, positioning them to dominate and key capabilities such as low latency portfolio are available at www.infinera.com. the next generation of transport networks. and excellent synchronization, outlined Packet-Optical the Infinera Way was written in Chapter 2. Chapter 3 describes how Features of Infinera’s packet-optical solutions to help Infinera’s customers, prospects packet-optical networks are best managed that we believe to be unique are highlighted and partners, and anyone else who needs and how to take advantage of current and with this marker throughout the text. to have a better understanding of the future software-defined networking (SDN) packet-optical world. This book focuses on developments. Chapter 4 takes the reader The information included is subject to change the integration of higher Open Systems further into how the described technologies without further notice. All statements, information and recommendations are believed to be Interconnection (OSI) layer functionality into are leveraged by various applications, such accurate but are presented without warranty of optical systems to create packet-optical as business Ethernet, mobile fronthaul/ any kind. transport systems (P-OTS), i.e. packet- backhaul and cable TV backhaul. optical networks. The term P-OTS is widely For those looking for a better understanding used within the industry and has been of Layer 2 Ethernet technologies, Chapter 5 recycled from an earlier definition (plain old includes a more detailed description of how these technologies work and how they are leveraged in wide area networks.

2 INFINERA Footnote citations on page 111 INFINERA 3 CONTENTS

1 INTRODUCTION...... 6 2.3 MPLS-TP for Traffic Engineering 2.7.1 Benefits of the Packet-Optical 3.3.3 Applications of Intelligent Packet 4.4.2 A Backhaul Network Optimized 5.5.1 Carrier Ethernet: Ethernet as a and Service Scalability...... 26 Approach...... 38 Transport with SDN...... 57 for 4G/LTE and 5G...... 68 Transport Service...... 88 1.1 The Cloud-driven Transformation 2.4 Traffic Management...... 30 2.7.2 Advantages when Using Infinera’s 4.5 Switched Video Transport...... 69 5.5.2 The Carrier Ethernet of the Network...... 8 4 APPLICATIONS OF PACKET- 2.4.1 Traffic Shaping and Policing...... 30 Portfolio for Packet-Optical 4.5.1 Streaming 3D and HD Video to Architecture and Terminology...... 91 1.2 Packet-Optical Networking...... 9 Transport...... 40 OPTICAL NETWORKING...... 60 2.4.2 Implementing Traffic Shaping...... 31 the Home...... 69 5.5.3 Carrier Ethernet 2.0 Services...... 92 1.3 The Infinera Intelligent 4.1 Chapter Summary...... 62 4.5.2 Infinera’s Solution for Switched 5.5.4 Carrier Ethernet Service Transport Network Portfolio...... 10 2.5 Migrating Legacy TDM Services 3 OPERATING THE NETWORK...... 42 to Ethernet...... 32 4.2 Ethernet Services for Enterprises – Video Transport...... 70 Attributes...... 94 2 PACKET-OPTICAL 2.5.1 One Common Infrastructure 3.1 Chapter Summary...... 44 Business Ethernet...... 62 5.6 Carrier Ethernet Traffic 5 ETHERNET AND LAYER 2 Management...... 95 NETWORKING...... 12 for Ethernet and Legacy 3.2 Multi-layer Network 4.2.1 Serving Enterprise Customers...... 62 TECHNOLOGIES...... 72 TDM Services...... 32 Management...... 44 4.2.2 A Metro Network for Business 5.6.1 Bandwidth Profiles...... 95 2.1 Chapter Summary...... 14 2.5.2 Using the iSFP to Convert TDM 3.2.1 Management Framework...... 45 Ethernet...... 63 5.1 Chapter Summary...... 74 5.6.2 Class of Service and 2.2 The Principles of Packet-Optical Services for Ethernet Transport...... 33 3.2.2 Infinera’s Management 4.2.3 A Long-haul Network for 5.2 Ethernet Basics...... 74 Service Level Agreements...... 96 Integration...... 14 2.6 The Main Elements of an Infinera Software Suite...... 47 Business Ethernet...... 64 5.2.1 Ethernet Mode of Operation...... 74 5.7 Carrier Ethernet Operations, 2.2.1 Why Aggregate Traffic at Packet-Optical Network...... 34 Administration and Maintenance 3.2.3 Infinera Digital Network 4.3 Aggregation of IP Traffic – IP 5.2.2 Virtual LANs...... 76 Layer 2?...... 14 2.6.1 Ethernet Demarcation Units Administrator...... 48 Backhaul...... 64 (Ethernet OAM)...... 97 5.2.3 Ethernet Physical Media (PHY)...... 78 2.2.2 Ethernet Transport at Layer 2 (EDU) and Network Interface 3.2.4 A Unified Information Model for 4.3.1 IP-based Services over a 5.7.1 The Management Framework...... 97 Versus at Layer 1...... 15 Devices (NID)...... 35 Multi-layer Management...... 49 Common Infrastructure...... 64 5.3 Synchronization and Circuit 5.7.2 Standards for Ethernet OAM...... 98 Emulation Services over 2.2.3 Native Ethernet and OTN 2.6.2 Packet-Optical Transport Switches 3.2.5 Layer 2 Service Fulfillment...... 50 4.3.2 A Lean and Transport-centric 5.7.3 The Service Lifecycle...... 98 Framing...... 19 (EMXP) and the PT-Fabric...... 35 Ethernet...... 80 3.2.6 Layer 2 Service Assurance...... 50 Aggregation Network...... 65 5.7.4 Ethernet Service OAM – 2.2.4 Packet-Optical Transport 2.6.3 Optical Add-Drop Multiplexers 5.3.1 Synchronous and Asynchronous 3.2.7 Network Planning and 4.3.3 The Flexible Optical Network Transport...... 80 Performance and Fault with the XTM Series...... 20 (OADM, ROADM) and Other Brings Scalability...... 67 Management...... 100 Optical Elements...... 36 Optimization...... 51 5.3.2 Synchronization Standards...... 83 2.2.5 Packet-Optical Transport 4.4 Mobile Backhaul...... 67 5.8 Metro Ethernet Forum with the PXM...... 22 2.6.4 The DTN-X Packet-Switching 3.3 Software-defined Networking 5.4 Ethernet Protection...... 85 and Network Virtualization...... 51 4.4.1 4G/LTE and 5G Place New “Third Network” Vision...... 102 2.2.6 Leveraging the XTM Series Module (PXM)...... 36 3.3.1 Basic Concepts...... 51 Requirements on Mobile 5.4.1 Link Aggregation...... 86 and PXM in the Same Network...... 24 2.6.5 The Multi-layer Service Backhaul...... 67 5.4.2 Ethernet Ring Protection INDEX...... 104 Management System...... 37 3.3.2 Infinera’s Multi-layer SDN Vision.....56 Switching...... 87 2.7 Advantages of Packet-Optical 5.5 Carrier Ethernet Architecture Transport...... 38 and Services...... 88

4 INFINERA INFINERA 5 1 INTRODUCTION

6 INFINERA Footnote citations on page 111 INFINERA 7 INTRODUCTION

infrastructure, platform and software as a compute, storage and network functions established protocols, such as Generalized label-switched paths (LSPs) and Ethernet Client Traffic 1.1 THE CLOUD-DRIVEN Voice Transport TRANSFORMATION OF THE service (IaaS, PaaS and SaaS) offerings, is once provided from a single data center are Multiprotocol Label Switching (GMPLS), virtual connections (Figure 2). To optimize Video Technologies having a profound impact on both IT and rapidly being expanded to multiple centers but a common vision is to employ SDN the cost-to-performance ratio of Layer T, it Packet5$ NETWORK DC Interconnect Match network architectures among technology at geographically separate locations. As technology that can abstract the functions is of the utmost importance to match the Digital Over the past decade, IT infrastructure has Email Optical service providers. a result, operators need to scale, simplify of all Layer T technologies and present transport and switching technology to the Enterprise App seen a dramatic shift from local resources and increase flexibility in their transport standardized application programming traffic flow characteristics, and ultimately As a result, the traditional layered network to those located in third-party facilities. networks. Network functions virtualization interfaces (APIs) such that control can be to the end-user services. Also, these model of a hierarchical, functionally heavy Multiplex Whether we receive the services we use, (NFV) enables operators to migrate services driven by Layer C. An integrated packet- technologies are not necessarily exclusive. data network comprising several layers of including storage of our personal files, like security, voice and broadband remote optical approach to Layer T is manifested For example, Ethernet/MPLS packets may devices is evolving to a simplified model Switch streaming of personal and commercial video access server (BRAS) from dedicated in Infinera’s Intelligent Transport Network be carried in digital OTN circuits, which in of Layer C, cloud services, and Layer T, and, of course, photo albums and email, devices to software services on standard x86 architecture, which is a mix of packet, digital turn are carried by coherent optical super- intelligent transport (Figure 1). at home or on our mobile phones, they hardware within cloud data centers. NFV and optical functions that virtualize control channels as Layer T services are aggregated. all reside somewhere “on the network,” The adoption of cloud services, consisting allows service providers to drive down the at Layer C via open APIs, using both GMPLS Meanwhile, switching capabilities are or more fashionably, “in the cloud.” The of the upper OSI layers of a network, cost of network services, while delivering and SDN. essential at each of the layers in order FIGURE 2: Matching Network Technologies to acceleration of cloud services, such as is growing at a phenomenal rate. The them in a more granular and responsive to maintain the efficient use of capacity Traffic Flows manner. and provide alternative routing for traffic and OTN, but must also offer excellent 1.2 PACKET-OPTICAL engineering or protection events. Meanwhile, transport functions at the NETWORKING support for Layer 2 Ethernet functionality. Old Model New Model lower layers are converging as operators The choice of technologies to employ This functionality includes the advanced encourage vendors to develop the most Layer T traffic flows vary in many different within Layer T also has a geographical provisioning, management, verification and Firewall cost-effective network elements, giving ways, such as duration, size, time of day, aspect, splitting the industry into distinct protection of Layer 2 Ethernet services using SBC Layer C: rise to an intelligent transport layer, Layer burstiness and deterministic nature. As a sub-segments: metro/edge packet-optical L3–L7 Cloud eices a defined group of standardized protocols B-RAS T. Layer T combines Multiprotocol Label consequence, flows can be handled by systems and long-haul/core packet-optical and procedures. Scale MPLS PE Switching (MPLS) and Ethernet packet several technologies across the multi- systems. SDN Systems at the core of a network are dealing Virtualization services with the digital functionality of layered Layer T network, including optical L2/3 Packet Metro/edge packet-optical systems are with traffic from many applications, and Optical Transport Network (OTN) switching, wavelengths, digital OTN circuits, MPLS Layer T: aimed at applications toward the edge of an as such require less service awareness. L0–L3 Layer 1 OTN nteigent Transport the capacity scaling of wavelength-division optical network. These systems typically take However, they do require the capacity to Layer 0 WDM multiplexing (WDM) and the optical path the form of WDM-based optical networking handle higher-capacity links, as well as a flexibility of reconfigurable optical add- platforms with integrated Ethernet drop multiplexer (ROADM) functionality. FIGURE 1: A Simplified Network Model Comprising Cloud Services switching. They have varying degrees of The control plane of Layer T can use (Layer C) and Intelligent Transport (Layer T) support for Layer 1 technologies such as Synchronous Digital Hierarchy (SDH)/ Synchronous Optical Networking (SONET)

8 INFINERA INFINERA 9 INTRODUCTION

METRO REGIONAL/CORE 1.3 THE INFINERA INTELLIGENT INFINERA EE FARE IE AN NA MPLS-TP MPLS TRANSPORT NETWORK PORTFOLIO ER NA EA PBB-TE CE 2.0 MPLS-TP Infinera’s end-to-end portfolio for packet- Mobile

Layer 2 ACCESS AGGREGATION REGIONAL CORE optical networks consists of the platforms Front/Backhaul OTN SWITCHING (ODU-2) shown in Figure 4. OTN TRANSPORT, ODU-FLEX SWITCHING In the metro/edge space, Infinera is a Triple Play

Layer 1 pioneer of metro WDM through the XTM E Series, which today includes leading packet- XTG Series II 10G + Metro 100G 100G + Multicarrier Coherent optical metro access, aggregation and core Colorless, Directionless Colorless, Directionless, Contentionless Cable II NI E Gridless ROADM Overlay Gridless ROADM platforms. This platform delivers services Broadband II

Layer 0 Mixed Fiber Environment DCM Free Fiber designed for applications such as triple play broadband, cable broadband, business XTM Series DTN-X Family: XTC Series Ethernet and mobile transport. Integration of OTN + Layer 2 from edge to core without In the long-haul/core market, the DTN-X Business Ethernet technical or business compromise is the ultimate goal Family provides the world’s only commercial multi-terabit super-channel system based on Cloud Xpress Family DTN-X Family: XT Series large-scale photonic integrated circuit (PIC) Matching Network Technologies to Network Topology. FIGURE 3. technology. It includes the Packet Switching Source: IHS Markit Data Center INERNNE Module (PXM), which enables traffic over virtual local area networks (VLANs) or MPLS INFINERA EN FEIE RI INE E higher degree of OTN Layer 1 transport also switching across the whole chassis or LSPs with assured quality of service (QoS). In long-haul packet-optical networks, the capabilities versus Layer 2 Ethernet. This node, from any port to any other port, rather FIGURE 4. Infinera’s Intelligent Transport Network Portfolio geographical balance between Layer 1 and than just within ports on a single plug-in DTN-X family is a powerful companion to Layer 2 functionality is shown in Figure 3. unit as is common in metro/edge platforms. the XTM Series. Long-haul/core platforms typically are Long-haul/core packet-optical systems must capable of packet aggregation, OTN therefore support not only high capacity but switching, terabit-scale WDM transport and advanced ROADM functionality, all with a single control plane.

10 INFINERA INFINERA 11 2 PACKET-OPTICAL NETWORKING

12 INFINERA Footnote citations on page 111 INFINERA 13 PACKET-OPTICAL NETWORKING

2.1 CHAPTER SUMMARY The scalability and cost-effectiveness of an extra toolkit to complement Ethernet base stations and enterprise networks to Layer 1 aggregation (time or frequency Subscriber Aggregation Core Ethernet has made it the unifying service and enhance the transport capabilities more central locations. The traffic from the multiplexing), where the number of users Optical fiber provides almost lossless protocol for modern local and wide area and scalability of the network. Supervised connected equipment was transported and their data rates are fixed. Statistical L2 transmission of signals at an ultra-wide networking. Increasingly, consolidation of by a multi-layer management system that transparently to an IP core network at OSI multiplexing makes use of the fact that 1GbE range of frequencies. Packet switching, L2 the optical and Ethernet/Internet Protocol integrates the handling of OSI3 Layer 1 Layer 1, via wavelengths on fibers in ring or the information rate from each source 1GbE implemented according to the Ethernet (IP) transport infrastructures within the optical channels and Layer 2 Ethernet star topologies. As the number of endpoints varies over time and that the optical path’s 1GbE Switch family of protocols and interfaces, offers same network elements has become services, a flexible, cost-efficient and future- grew, the central IP core network had to be bandwidth only needs to be consumed 10GbE one of the most efficient ways ever to sort Switch the means to drive down both network proof telecommunications infrastructure is extended, requiring more IP routers at more when there is real information to send. 1GbE and direct streams of digital data. Packet- investment costs and the associated here and ready to be deployed. sites and with more ports, as indicated on • Statistical multiplexing in a Layer 2 optical networking fundamentally leverages operational costs. The additional support the left side of Figure 5. 1GbE This chapter deals with the principles of network also allows for the creation of new the outstanding characteristics of these for OTN1 and label-switching mechanisms transporting Ethernet traffic over WDM In this type of network, there are several types of transport services, for example 33% 100% 100% two technologies to implement a new (such as Multiprotocol Label Switching - networks and describes how these two reasons to introduce Layer 2 aggregation: services with more granular data rates and generation of telecommunication networks. Transport Profile, or MPLS-TP2) provides key technologies can be integrated in a with committed information rates that • The IP core network can be reduced in unifying architecture. It also highlights the may be temporarily exceeded. Hence the FIGURE 6: The Statistical Multiplexing of Ethernet Is size to just a few central routers instead advantages that arise from combining the operator may offer its customer a much Leveraged to “Fill the Pipes,” an Especially Useful of being spread out throughout a full Feature at the Edge of the Network Where Traffic Is two technologies, including how traffic is more differentiated service portfolio than IP Core Network IP Core Network metro network area. Layer 2 aggregation Often More Variable transported with minimal delay and without with only traditional Layer 1 transport equipment is generally less costly,4 loss of synchronization. services, as shown in Figure 6. consumes less power, has lower latency and requires less expertise to configure • Since data traffic is concentrated at Layer connectivity provides more rational traffic than IP routers. Thus, centralization of the 2 in the aggregation network, it can be Layer 2 Aggregation 2.2 THE PRINCIPLES OF PACKET- handling and reduces forwarding delay IP core network reduces the necessary handed over to IP core routers via a Layer 1 Aggregation OPTICAL INTEGRATION compared to using IP routers. (Star, Ring) investment and operational costs. few high-speed interfaces rather than 2.2.1 Why Aggregate Traffic at Layer 2? over many lower-speed interfaces. This • Layer 2 aggregation can perform The introduction of aggregation at simplifies administration and contributes statistical multiplexing of data traffic, and 2.2.2 Ethernet Transport at Layer 2 Layer 2, i.e. the Ethernet layer, brings to a lower cost per handled bit. the WDM channels of the underlying Versus at Layer 1 several benefits to network operators. optical network can be used much more • As an additional benefit, the aggregation Given the benefits of a Layer 2 aggregation Traditionally, metro aggregation networks efficiently than if only Layer 1 aggregation network itself can be used to offer network, it is important to understand how were implemented using dedicated WDM is used. Statistical multiplexing allows the services within the metro/regional area. this type of network differs from a traditional equipment that provided wavelengths bandwidth to be divided arbitrarily among For example, point-to-point Ethernet network performing Layer 1 aggregation of connecting, for example, BRAS, radio a variable number of users, in contrast to connections can be provided between Ethernet traffic over WDM. offices in a city center without loading FIGURE 5: Migrating from Layer 1 to Layer 2 Aggregation of Traffic any central router nodes. Such direct

14 INFINERA Footnote citations on page 111 INFINERA 15 PACKET-OPTICAL NETWORKING

Transporting Ethernet traffic between two sending subscriber Ethernet network and it is also unaware of any Ethernet service a very low extent will still carry them at 1 native Ethernet signal that can be handed such as 25 megabits per second (Mb/s), 200 remote sites with WDM as the underlying puts it into a digital wrapper adapted for information and can only manipulate the gigabit per second (Gb/s) as if they were over without any need for demultiplexing. Mb/s or 400 Mb/s transport services over a bearer technology can be done in two transmission over the WDM channel. At traffic at Layer 1 (Figure 7). 100% loaded. This may lead to unnecessary A Layer 2 solution will therefore normally physical 1 Gb/s port. fundamentally different ways: the receiving end, the wrapper is removed investments in Layer 1 equipment to add require a lower number of ports on the In a Layer 2 transport network (Figure Due to statistical multiplexing, a Layer 2 and the original frame is handed over to wavelengths in the WDM network. receiving switch/router, which may have • Transporting the Ethernet frames “as they 8), subscriber Ethernet networks are transport network solution is intrinsically the receiving Ethernet network. In this way, a beneficial impact on the cost of that are” (transparently) over a WDM channel, interconnected via an intermediate Ethernet In a Layer 2 transport solution, the incoming less deterministic than a Layer 1 solution. every single frame is forwarded without equipment. i.e. Layer 1 (optical) transport. switching function called the Carrier subscriber Ethernet frames are analyzed and The throughput of a Layer 2 network may modification between the two subscriber Ethernet network. A service VLAN (SVLAN) acted upon by the equipment located at the Since a Layer 2 network can use policing to suddenly change because a new service • Using an intermediate Carrier Ethernet networks. or an MPLS-TP pseudowire in the Carrier ingress point of the Carrier Ethernet network separate the effective data rate offered to a has been introduced or because the traffic network5 that in turn has its frames The Layer 1 transport solution provides Ethernet network is used to keep traffic from before being forwarded. It is possible to subscriber from the actual line rate available situation has changed. However, the Layer transported over one or more WDM a transparent path between the two each set of subscriber Ethernet networks concentrate the incoming flow of Ethernet on the access line, a Layer 2 network can 2 network can be made to behave in a channels, i.e. using Layer 2 (Carrier subscriber Ethernet networks, giving the fully separated, i.e. the SVLAN/pseudowire frames by statistical multiplexing and to also offer a more flexible and granular set deterministic way by the use of pre-defined Ethernet) transport. This is the technology highest possible QoS in terms of latency, establishes connectivity between the apply shaping and policing to the Ethernet of transport services than a Layer 1 network. capacity reservations, i.e. by the use of traffic used in a Layer 2 aggregation network. latency variation and packet loss. A Layer subscriber Ethernet networks belonging traffic. The Layer 2 solution can be made While a Layer 1 network typically only engineering. Both alternatives have advantages and 1 network is also fully deterministic and to the same subscriber. The frames of the fully “Ethernet service-aware” and can provides services at standard Ethernet line The table on the following page summarizes disadvantages. provides 100% throughput regardless of Carrier Ethernet SVLAN/pseudowire are analyze and act upon the Layer 2 traffic. rates, such as 1 Gb/s or 10 Gb/s, a Layer some of the main differences between Layer what services are carried by the Ethernet transported over channels of the WDM 2 network may offer much more flexibility, A basic Layer 1 Ethernet transport solution Multiple Ethernet signals are multiplexed 1 and Layer 2 wide area Ethernet transport. traffic. However, since a Layer 1 network network, just as described previously. takes every incoming frame from the using time-division multiplexing (TDM) is totally transparent to Ethernet traffic, A plain Layer 1 transport solution cannot in a Layer 1 solution. The aggregated concentrate the Ethernet traffic being signal is not a standard Ethernet signal, aggregated, which may result in low and consequently, demultiplexing must

Transparent transport of Ethernet frames utilization of WDM wavelengths. For be done before the original signals are Carrier Ethernet Network example, a Layer 1 network collecting handed over to a switch/router. A Layer 2 Gigabit Ethernet signals that is utilized to solution performs aggregation to a new Service VLAN

Subscriber Subscriber Subscriber Subscriber Ethernet 1 MXP Ethernet 2 Ethernet 1 Ethernet 2 MXP WDM channel (λ) EMXP EMXP WDM channel (λ) Packet-Optical Switch Packet-Optical Switch Transponder/ Transponder/ Muxponder Muxponder FIGURE 8: Layer 2 Ethernet Transport: Ethernet Traffic Is Transported via a Service VLAN in a Carrier Ethernet Network Extending Between Operator Sites FIGURE 7: Layer 1 Ethernet Transport: Ethernet Traffic Is Carried Transparently at Layer 1 over a WDM Wavelength

16 INFINERA Footnote citations on page 111 INFINERA 17 PACKET-OPTICAL NETWORKING

Layer 1 Ethernet Transport Layer 2 Ethernet Transport 2 erice acetptica Sytem • Full Layer 2 transport according to the • Plain Layer 1 transport, i.e. transponders TDM multiplexing: Collects and delivers traffic Statistical multiplexing: Collects traffic at one data without changing its format and data rate. rate and can deliver input from many sources over Metro Ethernet Forum’s (MEF) Carrier and muxponders that provide 100% one single physical interface at a higher rate. Ethernet 2.0 (CE 2.0) specifications, i.e. transparent Ethernet transport at OSI equipment that provides aggregation and Layer 1. Carrier thernet Fixed data rates: typically 100 Mb/s, 1 Gb/s, 10 Flexible selection of data rates. Allocation of concentration of Ethernet and other traffic Switch Gb/s and 100 Gb/s. bandwidth by policing and shaping of traffic. • Layer 2 transport. Using the PXM, a DTN-X with a selected set of Layer 2 functions that network supports Ethernet aggregation, Deterministic. “What goes in comes out.” Relies on traffic engineering to achieve deterministic support the transport task. Such functions behavior. switching and CE 2.0 services over an OTN include, for example, the Institute of TP core, as described in section 2.2.5. ranponer “Unaware” of Ethernet service information and can “Ethernet service-aware.” May analyze and act on Electrical and Electronics Engineers (IEEE) TP only manipulate traffic at Layer 1. Layer 2 control information, i.e. can be used to 802.3ad Ethernet Link Aggregation, traffic TP create Ethernet transport services with different characteristics and quality of service. shaping, policing and bandwidth profiles with guaranteed bandwidth allocation. The Lowest delay, jitter and packet loss. Statistical multiplexing implies a risk for delay, jitter XTM Series’ Layer 2 network elements – the and lost packets. 2.2.3 Native Ethernet and OTN Framing ine Sytem rop EMXP packet-optical transport switches – The Carrier Ethernet network of a Layer 2 mpifier Layer 1 performance management based on bit Layer 2 performance management based on VLANs are also Layer 1-aware, meaning that they transport network is built on WDM optical errors (cyclic redundancy check or CRC). and Ethernet Virtual Connections, latency, jitter, can be connected directly to a WDM link frame loss, etc. and support features such as forward error channels, i.e. the frames of the Carrier Embedded management channels via overhead Embedded management channels via separate correction (FEC) at the optical layer. Ethernet network are transported by the bytes in line signal wrapper. management VLAN at Layer 2. underlying WDM optical system. The FIGURE 10: The Primary Entities Forming a Node in an XTM Series Packet-Optical Network All the traffic units mentioned above are of Ethernet payload data can be packaged the plug-in type and can be installed side FIGURE 9. Characteristics of Layer 1 and Layer 2 Wide Area Ethernet Transport in the transport containers of the optical by side with optical units such as ROADMs in the XTM Series platform’s different system in several different ways. Optical OTN is a more recent addition to the It supports FEC and management functions The XTM Series provides the operator with Gigabit Ethernet connection is utilized. This chassis options. As an example, Layer transport standards/systems such as SDH, standards for public telecommunications for monitoring a connection end-to-end three principal alternatives for transport of enables the network operator to analyze the 2-capable EMXPs can be used at the edge SONET and OTN therefore incorporate networks and is sometimes referred to over multiple optical transport segments. Ethernet traffic: of the network to collect and aggregate various adaptation methods, e.g. Generic by its International Telecommunication wavelength utilization and avoid unnecessary OTN wraps any client signal in overhead investment in transponders/muxponders Ethernet traffic and hand it over to Layer 1 Framing Procedure (GFP), to allow for Union Telecommunication Standardization • Plain Layer 1 transport, i.e. transponders and information for operations, administration to launch additional wavelengths. Another transponders or muxponders that provide packet data transport. These methods Sector (ITU-T) name, G.709. The standard is muxponders that provide 100% transparent and management. The client signal to differentiating XTM Series feature is the continued transport with cost-efficiency and a allow the mapping of variable-length, designed as a “digital wrapper technology” Ethernet transport at OSI Layer 1. be transported is mapped into an optical ability to inject/extract management VLANs high quality of service (Figure 10). higher-layer client signals, such as a flow to transport both packet-mode traffic, • Ethernet-aware Layer 1 transport, i.e. on a Gigabit Ethernet client port. This enables channel payload unit (OPU). The OPU is The DTN-X Family with the PXM provides of Ethernet frames, to the channels of the such as IP and Ethernet, and legacy SDH/ muxponders that provide 100% transparent direct remote management of Layer 2 devices then encapsulated into the basic unit of a similar set of choices for the transport of WDM network. SONET traffic over fiber optics with WDM. transport but have Layer 2 features, such as via the same data communications network Ethernet traffic: information transport by the protocol, the providing information on to what extent a (DCN6) solution as for the optical transport equipment.

18 INFINERA Footnote citations on page 111 INFINERA 19 PACKET-OPTICAL NETWORKING

Header SVLAN User data categories: transport systems and switches. Signal Data Rate Application Examples Native Ethernet means that no additional ID Carrier Ethernet Network frame OTN transport systems follow OTN framing is applied to the data payload at ODU0 1.24416 Gb/s Transport of a timing-transparent transcoded (compressed) standards when framing transported signals 1000BASE-X signal or a stream of packets (such as Ethernet, MPLS the edge of the network. By treating the for a specific route. The transport systems or IP) using Generic Framing Procedure. Ethernet packets natively, it is possible OPU Carrier Ethernet frame are therefore compliant with the sections of to inspect them within the intermediate Overhead Optical Channel Payload Unit (OPU) ODUflex Any configurable rate Transport of a stream of packets (such as Ethernet, MPLS or IP) the OTN standards relating to transmission, (GFP) using Generic Framing Procedure. network nodes and to act upon the Ethernet such as framing, FEC, performance ODU2e 10.3995253164557 Gb/s Transport of a 10 Gigabit Ethernet signal or a timing-transparent headers so that the combined benefits of monitoring, etc. Transport systems use transcoded (compressed) 10 Gigabit Fibre Channel (10GFC) signal. Layer 2 intelligence and efficient Layer 1 ODU Payload transponders or muxponders to terminate transport can be realized. This becomes Overhead Optical Channel Data Unit (ODU) the signal. OTN switches take this one stage FIGURE 12: Characteristics of Some of the Optical Channel Data Units especially important at the edge of the further and add an OTN switching fabric that is capable of demultiplexing incoming OTU Payload Native Ethernet Overhead Optical Channel Transport Unit (OTU) OTN signals and switching the lower-speed control information, i.e. to perform • Data is sent as ordinary signal components to other high-speed statistical multiplexing and differentiation Ethernet frames. • Access to Ethernet interfaces. of services at the Ethernet level. control information at intermediate nodes. FIGURE 11: OTN Signal Structure and Terminology. The Carrier Ethernet Frame is Carried as the Payload • Framing according to the OTN of an Optical Channel Payload Unit (OPU) OTN framing standard: The frames of the Carrier • Data is packed into a 2.2.4 Packet-Optical Transport Ethernet network are carried over a WDM wrapper for transport via an OTN network. with the XTM Series wavelength by ODUs according to the OTN • Ethernet control The XTM Series supports two types of standard. This encapsulation is especially information is optical channel data unit (ODU), which is that optical channels may be underutilized if encapsulated. encapsulation of Carrier Ethernet traffic for favorable when the Carrier Ethernet carried within an optical channel transport an efficient multiplexing scheme is not used WDM transport: network extends over longer distances, unit (OTU), defining the line rate of the at the ODU level. Three ODU multiplexing or the data is to be transported via an connection (Figure 11). schemes important in a packet-optical intermediate core OTN network. Using Carrier Ethernet Network network are ODU0, ODUflex (GFP) and ODU2e framing between EMXPs allows OTN includes OTU standards for a fixed • Native Ethernet: The frames of the Carrier the use of OTN’s inherent FEC mechanisms ODU2e. ODU0 units are optimized to carry Ethernet network are transported as is, i.e. number of line rates covering the range and optical path monitoring bits, which 1 GbE signals within an ODU1 unit, which using the same preamble and end-of-frame Subscriber Subscriber from 2.6 Gb/s up to 112 Gb/s. These are important for long-reach links. Since Ethernet 1 Ethernet 2 in turn is carried by an OTU1 transport unit. sequence as on an ordinary LAN7 when bitrates have been selected based on the the encapsulation is in ODU format, these EMXP EMXP ODUflex units can be multiplexed into any forwarded over the WDM wavelength. This WDM channel (λ) overall requirements of the optical transport data units can also easily traverse any of the higher-rate ODUs and carried at the is the standard encapsulation recommended Packet-Optical Switch Packet-Optical Switch intermediate OTN switches transparently system, but are not necessarily multiples of by Infinera for metro/aggregation networks, appropriate OTU channel line rate (Figure 12). before reaching their final destination the bitrates of the signals to transport, e.g. since it allows each intermediate node (Figure 13). FIGURE 13: Native Ethernet and OTN Framing with XTM Series EMXPs. The Two-colored Bars Symbolize Gigabit Ethernet (GbE). The consequence is OTN systems are often divided into two to examine and manipulate the Ethernet Ethernet Frames, with the Control Information in Red

20 INFINERA Footnote citations on page 111 INFINERA 21 PACKET-OPTICAL NETWORKING

High priority Ethernet frame to be mapped into an ODU2e Native Ethernet OTN Framing and can range from 1.25 Gb/s to 100 Gb/s, Ethernet services and Layer 2 QoS. Instead of Business apps data unit ready for transport into the OTN allowing for right-sized transport and core, including OTN switches. This is very Ethernet frames are Ethernet frames are requiring a dedicated port for every service, maximum bandwidth efficiency (Figure 16). transported natively transported in OPUs the PXM supports port consolidation, which Low priority useful once the traffic has been aggregated over WDM channels within ODU and OTU can provide multiple services or flows on one Having the packet streams mapped to Signaling Interactive as much as possible to ensure the best with minimal extra containers according to voice, video overhead. the OTN standard. port, overcoming the requirement of one standard ODUflex containers allows for utilization of the 10 Gb/s circuit. Figure 15 physical port and fiber connection per service. multiplexing with other types of ODU contrasts the differing characteristics of native The service information The Ethernet frame Loss intolerant contained in the is wrapped inside containers and for switching the flows as data Ethernet and OTN framing. Ethernet frame can the OPU/ODU and be accessed at every cannot be read and appropriate at the optical layer using the An OTN core network can also provide a node of the network. acted upon without The PXM maps incoming streams of data DTN-X OTN Switch Module (OXM). Hence, Loss tolerant Operator unified transport layer where core nodes com- This allows for Layer 2 demultiplexing. packets, such as Ethernet VLANs, into it becomes possible to apply standard Layer data service differentiation flows that can be co-routed through the bine traffic from OTN muxponder-based Lay- at intermediate network 1 traffic engineering and protection to flows er 1 services with EMXP-based, ODU-framed nodes. network with other flows that have the of Ethernet packets as well. same destination. The packet flows with the Ethernet services. End-to-end performance No FEC without an Includes FEC and same destination are groomed into a single FIGURE 14: Service-aware Transport Enables a Differentiated Service Offering with Multiple Classes of Services monitoring is achievable even over multi- additional Layer 1 optical path monitoring transponder. mechanisms, which ODUflex container for efficient routing. with Different Characteristics ple operators’ networks through OTN with are important for long- tandem connection monitoring and Carrier distance links. ODUflex containers are built as necessary Ethernet’s inherent operations, administration Traffic can be carried Provides direct and maintenance (OAM) capabilities. “transparently” over an compatibility with N x 10 GbE network where decisions about traffic For example, frames containing data for OTN network. intermediate core prioritization are made and where traffic high-value and high-quality IP services OTN networks – ODU containers can be is aggregated to “fill the pipes.” The such as IP/MPLS, IP-virtual private network 2.2.5 Packet-Optical Transport with the PXM multiplexed and Consolidate 10G ports wrapping of traffic into full OTN can then be (IP-VPN) or virtual private LAN service (VPLS) switched like any other into 100G ports flow. done at the handover to the core network, can be switched to paths for transport to the Packet-aware The DTN-X Family of platforms performs ex OTN Core after aggregated pipes of traffic that are desired IP devices. By contrast, frames that Suitable primarily for Applicable in most U integrated WDM transport and OTN OD correctly shaped have been created, are destined for transport services (Ethernet, metro and regional situations but especially ODU ex switching, grooming and service protection at networks where traffic in core and long- 100 GbE N x 10 GbE avoiding wasted bandwidth. MPLS-TP or OTN) can be kept within the Layer 1, based on large-scale PIC technology. is aggregated and distance networks OD U optical transport network, with minimal Using the PXM, the DTN-X Family can map service differentiation is where traffic has been ex Native Ethernet uses the VLAN tag or MPLS applied. aggregated into larger use of expensive Ethernet switching and IP Ethernet VLANs to ODUflex subnetwork label to switch the frames to ports associated streams. routing resources (Figure 14). connections, which enables ODUflex-based with either IP services or transport services. FIGURE 16: The DTN-X Family with traffic engineering in long-haul networks. Some Characteristics of Native Ethernet Each of these service domains is optimized The EMXPs have OTN framing capabilities for FIGURE 15: Integrated PXM Consolidates Framing and OTN Framing Client Ports and Routes Packet N x 10 GbE and simplified for particular service types. ODU2e, including optional G.709 FEC on all With the introduction of the PXM, the DTN-X Family supports statistical multiplexing for Flows over ODUflex Subnetwork 10 Gb/s ports. The optional ODU2e framing Connections on 10 Gb/s ports allows the native 10 Gb/s

22 INFINERA Footnote citations on page 111 INFINERA 23 PACKET-OPTICAL NETWORKING

The flexible architecture of the packet PXM OXM DWDM-LM transport network allows for cost-efficient Core OTN

OTN aggregation of all types of packet-based Packet XTC-4/10 Processing OTN traffic. Service routers can be consolidated OTN Client to a few central sites and equipped with OTN (e.g: 100GbE) ODUj ODUj ODUj N x ODUj Super-channel high-speed interfaces, thereby reducing (e.g. ODUFlex (e.g. ODU2) (e.g. ODU2) (e.g. ODU2) (e.g. 5x or ODU2e) ODU4) overall network investment and operational PXM costs. The packet services offered adhere to PXM recognized Metro Ethernet Forum standards Client Service Support OTN Network Interfaces Network Service Support FIGURE 17: Mapping of Ethernet Traffic into ODUflex Containers Allows for Multiplexing and Switching at Layer 1 and can be managed according to well- Using •Standard Untagged OTN Ethernet Principles • ODU exi SNCs • MPLS-TP (Ethernet PWE) Metro 1 • Priority Tagged Ethernet • ODU2e SNCs* • 802.1Q (C-VLAN) Encapsulation*established OAM procedures defined PXM • 802.1Q (C-VLAN) Encapsulation • 802.1ad (C & S-VLAN) Encapsulation* SGW for Carrier Ethernet. Layer 1 protection CATV & Mapping• 802.1ad the (C Ethernet & S-VLAN) traffic Encapsulation to the ODU • MPLS-TP LSP* schemes can be applied in the core network Internet • MPLS-TP LSP* services, such as those defined by Metro Aggregation can be done in various ways, for example segment, while additional Layer 2 protection Ethernet Forum for CE 2.0, can be offered to Metro 2 Metro .N by carrying the Ethernet VLANs in MPLS- users in one or more geographical regions. schemes, such as MPLS-TP switching CCAP TP pseudowires (PWs), creating additional As shown by the dotted lines in Figure 19, and G.8032v2 Ethernet Ring Protection Business MSAN XTC-2 options for traffic engineering and packet services may span one or more metro Switching (ERPS), can be applied when Internet protection (Figures 17 and 18).8 networks, have their endpoints in an EMXP or services are delivered via EMXPs. NID be delivered directly to, for example, a service NID router from a node in the core network via the PXM in the DTN-X. 2.2.6 Leveraging the XTM Series and PXM in Mobile the Same Network Backhaul Packet Processing Steps: Hub • Incoming VLAN examined for The XTM Series, with its advanced handling Cell site destination router of Ethernet services, can seamlessly interwork VLAN 1 to A MPLS-TP ODU2e to A • VLANs mapped to with the PXM and its super-channel-based pseudowires (PWs) and then to transport of Ethernet traffic mapped to ODU Ethernet ODU ex or ODU2e* OTN VLAN 2 to B PWE X MPLS-TP FIGURE 19: Multiple Metro Networks Interconnected by a High-capacity OTN Core. Each Dotted Line Represents an Ethernet containers. Deploying a high-capacity OTN ODU ex to B containers • Multiple VLANs per Service Carried over the Packet-Optical Network VLAN 3 to B core based on the DTN-X Family with PXM PWE Y MPLS-TP ODU2e/ODU ex allows for the interconnection of multiple ODU ex is resized on demand metro networks into one intelligent packet ODU ex (hitless resizing) transport infrastructure, as depicted in Figure Hitless resizing FIGURE 18: Adaptation of 19. In this infrastructure, end-to-end Ethernet Ethernet Services into ODU Containers (Example)

24 INFINERA Footnote citations on page 111 INFINERA 25 PACKET-OPTICAL NETWORKING

In summary: In the access and aggregation MPLS-TP label-switched path (LSP) the MPLS-TP network. Using MPLS-TP Both the tunnel and the LSP can be Ethernet frame from user part of the transport network, where service terminology, the entry and exit nodes are envisaged as predefined circuits for Ethernet Payload granularity is required, a service-aware referred to as MPLS-TP provider edge information to follow through the network, header packet-optical mechanism is beneficial Carrier Ethernet Network (PE) nodes, and any intermediate nodes and consequently tunnels and LSPs are to support different QoS. Also, access being passed by the LSP are referred to configured in advance from the network to Ethernet OAM bytes and service tags as MPLS-TP provider (P) nodes. Often the management system. A key feature of MPLS labels enables end-to-end management of physical node performing the PE function MPLS-TP that distinguishes it from classic used for Subscriber Subscriber 9 switching Ethernet services. Using native Ethernet Ethernet 1 Ethernet 2 is called a label edge router (LER) and the IP/MPLS is that management framing offers benefits from revenue- EMXP EMXP intermediate transit node is called a label and protection are designed Ethernet MPLS PW Pseudowire payload generation, investment and operational WDM channel (λ) switching router (LSR). EMXPs can act as to operate without a dynamic header label label Packet-Optical Switch Packet-Optical Switch perspectives in the access and aggregation both an LER and an LSR, and may also control plane, i.e. similar to parts of the network. On the other hand, combine these roles (Figure 21). a traditional SDH/SONET Local Ethernet header Forwarded Ethernet frame OTN has all the benefits of a long-haul FIGURE 20: Using MPLS-TP to Define Label-switched Paths Within the Carrier Ethernet Network network, where circuits are set An MPLS-TP tunnel is a pre-defined optical transport network once the traffic has up by the management system. MPLS-TP transport path from the source FIGURE 24: Ethernet over MPLS Encapsulation been sufficiently aggregated and traverses LER to the destination LER. The MPLS-TP The actual data traffic is carried by a the core part of the network. Combining These can be addressed by the use of are switched based on their SVLAN tags and tunnel always has an active LSP that defines pseudowire inside the LSP/tunnel. One these two framing schemes within one MPLS-TP, which is the transport profile of media access control (MAC) addresses. the primary and working paths. It may MPLS-TP LSP may carry one or more A pseudowire is an emulation of a Layer 2, intelligent packet-optical transport multi-protocol label switching. also have a protection LSP that defines a pseudowires, i.e. the pseudowires offer a point-to-point, connection-oriented service architecture makes it possible to exploit the A label-switched path is defined between recovery path (Figure 22). means for multiplexing of traffic (Figure 23). over a packet-switching network (PSN), benefits of both technologies in a common MPLS-TP is a way to simplify Carrier nodes where traffic enters and leaves Ethernet services by pre-defining from one attachment circuit (AC) to another. infrastructure for service creation. Label Switching Router The pseudowire used in MPLS-TP is a connection-oriented services over packet- (LSR) LSR LSR Pseudowire MPLS-TP Tunnel/LSP based networking technologies in a way Active LSP MPLS-TP Tunnel connection established between two MPLS- that gives support for traditional transport TP LERs across the MPLS-TP tunnel/LSP with Label Edge Router LER LER 2.3 MPLS-TP FOR TRAFFIC LSP 1 the attachment circuit frames encapsulated operational models. It takes the advantages (LER) B B ENGINEERING AND SERVICE B as MPLS data (Figure 25, next page). SCALABILITY of MPLS concepts by adding more flexibility and network manageability than the basic A C A C A C While Carrier Ethernet networks and the use Ethernet SVLAN architecture (Figure 20). of service VLANs bring great advantages to LER LER LER Attachment Attachment Circuit (AC) the packet-optical network, there are some LSP 3 D Label Switched Path 2 D Protect LSP Circuit (AC) D (LSP 2) limitations in terms of protection options, Principles of MPLS-TP FIGURE 21: MPLS- FIGURE 22: Data LER + LSR LSR FIGURE 23: LSR traffic engineering and service scalability. MPLS is a technique that forwards packets TP Framework MPLS-TP Is Carried by a based on labels, as opposed to a standard Tunnel Pseudowire Defined Carrier Ethernet network where the frames Within the MPLS-TP Tunnel and LSP

26 INFINERA Footnote citations on page 111 INFINERA 27 PACKET-OPTICAL NETWORKING

which bring a mesh-based structure Applications layer - Tunnel PW MPLS-TP is fully supported by the EMXP to the wavelength routing and paths Applications Layer 7 range. Any physical port on an EMXP can Tunnel ID available through the physical network in/out support both native Ethernet and MPLS- AC pw-label AC TDM Ethernet Client service layer TP, allowing operators to deploy MPLS-TP for any services. One previous drawback Edge out label in label Transit out label in label Edge with Ethernet was that it was not well- Active in label out label in label out label Active where and when it makes sense for them. It is Pseudowire possible to run MPLS selectively per port or to suited for protection and restoration over LSP I/F I/F I/F I/F MPLS-TP layers LSP ID separate MPLS traffic based on MAC address mesh-based networks, as the available Label Switched (LSP)/Tunnel and VLANs within the same port. This allows protection schemes were largely based seamless migration and coexistence with the on point-to-point or ring architectures. Ethernet Link layer LER LSR LER two protocols running independently side by MPLS-TP is highly suited for a mesh-based side. environment and allows network operators ODU framing FIGURE 25: Relation Between Pseudowire, Tunnel and LSP to design network resilience strategies that Physical layers - WDM Layer 0/1 are closely aligned to the physical structure WDM Services can be extended over DTN-X- of the network, ensuring the best possible based networks through the PXM, which resilience and service uptime. A Layer 2 transport service is established Both Ethernet SVLAN and MPLS-TP supports MPLS-TP and VPLS over ODUflex in an OTN core network. FIGURE 26: MPLS-TP OSI Network Layers. The Two Variants of the Physical Layer Correspond to Native between two attachment circuits and the forwarding techniques have their own Ethernet Framing and OTN Framing Respectively service is carried by a pseudowire. The benefits, and it is often advantageous to In order to deploy MPLS-TP in a production MPLS-TP—Easy Service Creation pseudowire travels through the network in be able to offer services based on both network, it can be introduced either as Another advantage of MPLS-TP is that it an MPLS-TP tunnel. The MPLS-TP tunnel technologies. For example, multicast an overlay to the existing Ethernet or breaks service creation into two steps. First, to the network. Second, it brings a very network. Since each subscriber network is in turn mapped to at least one LSP: the services can often be deployed directly over incrementally as an evolutionary build-out in tunnels are created between endpoints familiar look and feel to service creation may include an extensive number of active LSP. SVLANs on Ethernet, whereas point-to-point parallel on the same networking hardware. within the network for service and protection as it is similar to the processes involved in devices and MAC addresses, this results trunks requiring protection can benefit Figure 26 on the next page summarizes how Infinera believes in a smooth evolution, paths for MPLS-TP-based services. Then traditional transport networks. This helps in a need for large MAC address tables more from the MPLS-TP features. However, transport in an MPLS-TP network relates to provided by a “ships in the night” capability, the network administrator simply creates operators migrate from traditional transport in each network node, creating various MPLS-TP pseudowires can also be used for the OSI model. Note that Ethernet framing with Ethernet and MPLS-TP acting in parallel new services by adding them to the tunnel networks to packet-optical networks. problems and extra equipment cost. Using multi-point communication using VPLS,10 is present at both the link layer and the as independent non-interfering protocols in endpoints as pseudowires, safe in the MPLS-TP, the subscriber MAC addresses which allows geographically dispersed sites client service layer. the same system. knowledge that all routing aspects of the are encapsulated within the pseudowire to share an Ethernet broadcast domain by service have already been handled. This payload and not seen by the intermediate connecting sites through pseudowires. MPLS-TP Resolves the brings two distinct advantages. First, it MAC Scalability Problem switches of the Carrier Ethernet network. MPLS-TP in ROADM-based Optical Networks makes the solution more scalable, as it is In an SVLAN-based Carrier Ethernet Packet-optical networks are often deployed simpler to add a large number of services network, all MAC addresses in the attached today over ROADM-based optical networks, subscriber Ethernet networks are visible to every switch within the Carrier Ethernet

28 INFINERA Footnote citations on page 111 INFINERA 29 PACKET-OPTICAL NETWORKING

2.4 TRAFFIC MANAGEMENT shaping is typically done per port, interface or other physical entity, while policing • The EMXP has a QoS Engine with 8 Queues Layer 2 networks rely on packet switching High ∙ Requires buffering + scheduling can be applied more granularly and per Transmit Outgoing where different CoS are mapped against ∙ Delays some packets and statistical multiplexing when forwarding queue packets Ethernet service, VLAN, etc. Incoming packets • Queues are emptied by a scheduler that information. Statistical multiplexing Medium Classify can operate in different modes Infrastructure level (per port) time time implies that the ports of the network can As shown in Figure 27, policing plays an Normal Before shaping After shaping ance ieacica is se o • Strict priority be allocated more “bandwidth” than is important role in traffic management of aance an ai oessciption o • Round Robin seices ote totally available, so a Layer 2 packet-optical Carrier Ethernet services and their individual Low SLAs, as described in section 5.6. • Weighted RR network must have an inherent mechanism Length de ned • While a frame is waiting to be scheduled out ∙ Requires no buffering to control the amount of information by queue limit ∙ Bandwidth is metered Traffic shaping is done by imposing an on the line it is buffered entering the network and passing additional delay on some packets such that over network links. These procedures • If there’s no buffer left the frame is dropped time time Service Level the traffic conforms to a given bandwidth Before policing After policing are collectively referred to as traffic • This is done in wirespeed profile. Traffic shaping provides a means Classification by: Queueing * In real implementations traf c is measured with leaky-bucket to allow controlled bursts management, and the two most important Interface hardware to control the volume of traffic being sent • Protocol buffer resources • Ethernet such tools are traffic shaping and policing. out on an interface in a specified period • Source interface (bandwidth throttling), and the maximum • VLAN FIGURE 27: The Difference Between Traffic Shaping and Traffic Policing • Prio-bits rate at which the traffic is sent (rate limiting). 2.4.1 Traffic Shaping and Policing Traffic shaping is a traffic management A drawback with traffic shaping is increased FIGURE 28: Traffic Shaping: Quality of Service Is Assured by Buffered Queues and Scheduled Forwarding of Packets latency and jitter for the Ethernet Virtual These switches do not have to be designed network, there is no such upper limit for the technique that delays some frames in a Connection (EVC), but the gain can be with the number of subscriber Ethernet number of subscribers that can be handled packet-optical network node in order to better throughout since the overall flow MAC addresses in mind. by a network using MPLS-TP. bring them into compliance with a desired 2.4.2 Implementing Traffic Shaping When a network element is congested, it traffic profile. Traffic shaping is a form of of frames may be improved: instead of starts to drop frames if there is no buffer Of course, as the MPLS-TP services rate limiting, as opposed to the policing of dropping traffic in a policer, it may be better Traffic shaping is typically done by assigning left to allocate. Weighted random early supported by Infinera’s XTM Series to shape the traffic to make sure no frames incoming packets of different priority to MPLS-TP Allows for a Virtually Unlimited bandwidth profiles, where excess frames are detection (WRED) is a way for a network are delivered over the same hardware are lost (or at least as few as possible), separate queues in the network element. Number of Customers simply dropped. Normally, traffic shaping element to ask clients to require fewer platform as native Ethernet services, they avoiding retransmissions at higher protocol For example, the EMXP uses eight queues The IEEE 802.1Q standard allows for a is not part of any subscriber service level resources from the network. However, also benefit from the same transport- layers. for different classes of services (CoS). The maximum of 4094 SVLANs in a Carrier agreement (SLA), but rather an internal WRED only acts on traffic that is transported like performance with extremely low queues are emptied by a scheduler that can Ethernet network, with one SVLAN typically network mechanism used by the operator to operate in different modes, e.g. strict priority by the TCP/IP protocol that ensures that latency and almost zero jitter, and can be required per subscribing customer. Since “even out” traffic flows and create fairness or round robin. While a frame is waiting to be data is re-sent. With WRED, different discard combined with synchronization schemes MPLS-TP uses tunnels and label-switched between users of network resources. Traffic scheduled for transmission out on the line, it profiles can be set for different CoS, and such as Synchronous Ethernet (SyncE) when paths to define the connectivity within the is buffered in the element, and if there is no TCP traffic can be dropped randomly so required, in mobile backhaul networks buffer left the frame is dropped (Figure 28). that the overall load on the network is for example.

30 INFINERA Footnote citations on page 111 INFINERA 31 • Provides TDM services at the edge of an Ethernet network Transparent transport of: • Transparent TDM over packet (TSOP and SaTOP) aoa stctes are outlines in open IETF drafts/standards* eea tes an potection potocos nconiation • TDM streams are “chopped up” and carried in ine ates 1 1 an 1 Ethernet packets PACKET-OPTICAL NETWORKING

decreased. Dropping traffic at random bandwidth perspective, Ethernet traffic has approach where the new packet-mode Steady stream of bits prevents a “global TCP sync,” in which all already surpassed the amount of legacy network also provides legacy TDM services • Layer 1 services such as E1, STM-1/OC3, TCP sessions try to increase their bandwidth TDM traffic. Operators are faced with the is an attractive alternative (Figure 29). channelized STM-1/OC3, STM-4/OC12 and TDM Network at the same time, creating a “perfect challenging task of maintaining existing STM-16/OC48 can easily be adapted for Many legacy TDM services can be replaced transport over Ethernet through the use of storm.” TDM services while upgrading networks by Ethernet equivalents and are well- Infinera’s Intelligent SFP (iSFP) pluggable for new Ethernet services, all in the most SDH: 810 bytes Encapsulation Ethernet suited for the packet-optical infrastructure optics that provide circuit emulation over headers headers Ethernet Network cost-efficient way. More capacity is required E1: 256 bytes described in the previous sections. However, SyncE-capable Ethernet. The iSFP modules 2.5 MIGRATING LEGACY TDM to cater to the growing amount of packet- fit into any EMXP Gigabit Ethernet port, there is a significant portion of traffic that Steady stream of Ethernet frames SERVICES TO ETHERNET mode traffic generated by Internet access, allowing a very flexible and tactical service cannot simply be migrated to Ethernet. video on demand and cloud computing, migration. 2.5.1 One Common Infrastructure for This fact creates a dilemma for network FIGURE 30: Transparent SDH/SONET and E1 over Packet while the shrinking amount of TDM traffic * RFC 4553 and draft-manhoudt-pwe3-tsop Ethernet and Legacy TDM Services operators as SDH/SONET systems start to must be taken care of to sustain revenues Transmission systems based on Layer reach end of life, or if a large SDH/SONET from existing services. Building a separate 2.5.2 Using the iSFP to Convert TDM Services 1 TDM technology, such as SDH and or even E11 network must be supported for new infrastructure for Ethernet traffic while for Ethernet Transport The TDM traffic is mapped into an Ethernet SONET, have been the basis for many a small number of services or subscribers. maintaining a shrinking SDH/SONET Virtual Connection that can be transported In Infinera’s iSFP solution, synchronization telecommunication services offered during network is one option, but an integrated The iSFP is used to deliver TDM services over either as an Ethernet service VLAN or via an transport is provided by the differential clock the last few decades. However, from a recovery (DCR) mechanism, transferring a network built for Carrier Ethernet services. MPLS-TP service. The EMXPs can also be used Infinera’s packet-optical platforms offer the SDH/SONET clock to the transparent It provides circuit emulation for STM-1/OC3, to perform Layer 2 aggregation to ensure full several alternatives for the handling of legacy STM-4/OC12 or STM-16/OC48 SDH/SONET emulated service with SyncE as a reference at utilization of higher-speed 10 Gb/s Ethernet Layer 1 CE 2.0 Services TDM-based services in a way that greatly services as well as E1 services or channelized both ends. DCR transfers the synchronization Layer 1 Point-to-Point Services ine facilitates a gradual shift toward one common, connections. The adaptation of TDM services clock to the emulated service and extracts it Point-to-Point Services STM-1 signals over a SyncE-capable Carrier 1 to Ethernet follows open Internet Engineering again at the far end of the service. (See also 1 1 ee packet-mode, Ethernet infrastructure for all Ethernet network. 1 Existing Layer 2 Services 1 ccess traffic. Task Force (IETF) standards (Transparent SDH/ section 5.3.2.) 1 ine 1 ansit The iSFP module performs packetizing of SONET over Packet and Structure-Agnostic 1 est eot tanspot • Existing Layer 1 services carried over the TDM service, converting for example an Time Division Multiplexing over Packet) for SDH/SONET or E1 networks can be STM1/OC-3 service into a 170 Mb/s Ethernet transported separately from Ethernet stream or an STM-4/OC-12 service into a TDM-over-packet transport (Figure 30). The two directions of the service can operate as two independent timing services using different wavelengths, while 680 Mb/s stream, creating a transparent bit Ethernet/MPLS-TP Transport still using the same WDM platform and domains or one direction can be frequency- SDH/SONET Ethernet CE 2.0 pipe between two locations in the packet- SyncE Capable optical transmission network. Infinera’s optical network. The existing TDM service, Synchronization locked to the other direction. platforms include powerful ROADMs for including payload structures, overhead bytes Network synchronization is the cornerstone In cooperation with an ecosystem partner, flexible handling of the optical paths and and protection protocols, is transparently of an SDH/SONET network. Here the quality Infinera also supports synchronization muxponders/transponders that adapt SDH/ migrated to the Carrier Ethernet network. of the underlying Ethernet network is critical. SONET traffic to optical transport. FIGURE 29: Migration Toward a Single, Packet-oriented Transport Infrastructure for All Services Service adaptation is done at the edge of the The Infinera XTM Series provides excellent network, and standard Ethernet frames are Synchronous Ethernet performance due to used between the ingress and egress points. patented innovations in circuit design.

32 INFINERA Footnote citations on page 111 INFINERA 33 PACKET-OPTICAL NETWORKING

Primary Reference Clock how these functions are distributed in a A third central objective when developing 2.6.2 Packet-Optical Transport Switches complete network and between products. the Infinera packet-optical portfolio has Infinera’s Ethernet Demarcation Unit (EMXP) and the PT-Fabric been to enable multi-layer traffic and service is an independent unit that supports A key objective in the development of the management. Depending on traffic load or service level agreement management In the metro aggregation network, traffic Infinera packet-optical portfolio has been capabilities, including sophisticated traffic link degradation, the packet-optical network enters via an EMXP in the first node. The same to expand the number of services that management and hierarchical quality of must be able to switch between different physical node may also include muxponders/ can be provided by and over an optical service mechanisms; standard end-to-end transponders using additional WDM channels 1 transport alternatives, in order to deliver operations, administration and maintenance 1 infrastructure. More services over the same for fully transparent Layer 1 transport, 1 1 the highest possible quality of service for and performance monitoring and extensive 1 with Layer 2 transport used only where 1 network mean more revenue and less cost 1 subscribers. The tight integration between fault management and diagnostics, all to 1 for the operator. By integrating a selected inspection and OAM information is required at Layer 2 and Layer 1 functionality in the reduce service provider operating costs and set of Layer 2 functions with the optical layer, intermediate points, and where traffic needs oon tiing eeence o nconos tenet architecture also enables software-defined capital expenses. A Z the network becomes much more potent to be aggregated by statistical multiplexing. ieentia coc ecoe networks and advanced traffic management. in terms of service offerings and can be The same set of demarcation functionality is The EMXPs are available in various made more scalable. The tight integration also available via Infinera’s Network Interface configurations and mounting options, such FIGURE 31: Using Synchronous Ethernet (SyncE) as a Reference for SDH/SONET Differential Clock Recovery Device, which is a port device supported by between Layer 1 and Layer 2 also makes it as one- or two-slot chassis-mount units and 2.6.1 Ethernet Demarcation Units and Network its parent EMXP. The NID performs service possible to increase resilience and improve a temperature-hardened access unit for Interface Devices OAM but leaves the service policing and traffic management in ways not possible deployment in non-controlled environments – tagging to be done by the EMXP, reducing all with full wire-speed forwarding on all ports management and monitoring. Probes can schemes are replaced by MPLS-TP and with less integrated approaches. Demarcation of a provided service is a key the cost and complexity of the customer- for all traffic types. be deployed in the network to monitor Ethernet protection options. Protection is located equipment. Another main objective of the chosen function of the packet-optical access network, SyncE quality, which is used as a reference supported by MPLS-TP for topologies such packet-optical architecture has been to as it enables the service provider to extend for both ends of the transparent TDM as ring, mesh and partial mesh. In addition, make it agnostic to the underlying transport its control over the entire service path, service. The probes can also be used to protection of the employed Ethernet Virtual starting from the customer handoff points. network technology. Traffic flows over a monitor the SDH/SONET sync quality of the Connections can be accomplished by the The customer’s equipment is connected to the ROADM-based, photonic network that connected systems to provide confirmation Ethernet Ring Protection Switching (ERPS) Carrier Ethernet network via a provider-owned provides connectivity and can be used for FIGURE 33. Two Examples of of the sync in the TDM service that is carried and Link Aggregation Group (LAG) options, demarcation device (Ethernet demarcation Infinera’s Range of Packet- transparent Layer 1 services. In addition, over the Ethernet network (Figure 31). as described in section 5.4. unit [EDU] or network interface device [NID]) FIGURE 32: Infinera’s Network Optical Transport Switches an Infinera packet-optical network can deployed at the customer location (Figure 32). Interface Device (NID) (EMXP) in the XTM Series, seamlessly interoperate with its transport These units enable a clear separation between a Chassis-based Unit and a network over MPLS-TP tunnels, Ethernet the user and provider Ethernet networks. 1RU Access Unit Protection 2.6 THE MAIN ELEMENTS OF AN SVLANs or OTN switches and any To maintain SDH/SONET protection, INFINERA PACKET-OPTICAL combination of these. the existing Automatically Switched NETWORK Optical Network (ASON) and subnetwork Having looked at the principles of packet- connection protection (SNCP) protection optical networking, it is time to discuss

34 INFINERA INFINERA 35 PACKET-OPTICAL NETWORKING

capabilities across the metro aggregation include Layer 1 transponders and management. The real benefits of packet- A further development of the EMXP and core networks. muxponders, EMXPs, ROADMs and other optical networking can only be realized concept is the Packet Transport Fabric optical network elements. with a truly integrated Layer 1 and Layer 2 (PT-Fabric), where a high-capacity EMXP The metro core network usually connects to transport platform and a unified Layer 1 and is interconnected to external 10 Gb/s and data centers, and also to IP core and mobile Tributary Modules Switch Modules Line Modules Layer 2 management system. 100 Gb/s input/output (I/O) client units OTN core networks, typically implemented by OTN 12 2.6.4 The DTN-X Packet-Switching via a separate frontplane bus. The optical a set of IP routers using full IP/MPLS for OTN A network for Carrier Ethernet services frontplane takes vertical-cavity surface- Module (PXM) OTN traffic engineering purposes. Normally OTN may be implemented as a separate emitting laser (VCSEL) technology used in Client ODUk ODUk <-> ODUJ ODUj ODUj N x ODUj Super-Channel operators prefer to have a clear demarcation (e.g. ODU0) (e.g. 5x ODU4) Layer 1 optical network with Layer 2 supercomputing, and brings it to optical The metro/regional packet-optical network (e.g. 10GbE) (e.g. ODU2) (e.g. ODU2 <-> ODU0) (e.g. ODU0) (e.g. ODU0) transport equipment for the first time. point toward the metro network at the described above can be further augmented, Ethernet switches attached externally edge of the IP/mobile core network, often both in the metro core and regarding over standard interfaces, as depicted in This frontplane enables the PT-Fabric to be referred to as a provider edge router. The integration with long-haul optical networks, OTN OTN OTN WDM Figure 35. Although conceptually simple, deployed in any combination of available slots Mapping Multiplexing Switching Super-Channels routers of IP/mobile core networks have by Infinera’s DTN-X Family of converged such a configuration results in a complex in single or multiple XTM Series chassis and broad functionality and very high capacity; digital photonic platforms. The DTN-X Family hierarchy of management systems. These without any backplane capacity limitations. is a terabit-capable super-channel system Block Diagram Showing the Main Functions of the Infinera DTN-X Family Multiple PT-Fabrics can be deployed in a they are therefore normally interconnected FIGURE 34: systems must be carefully integrated in leveraging large-scale photonic integrated single chassis, giving a total of 4 terabits of via a strict Layer 1 WDM network, since the circuit technology, integrating OTN switching switching within one XTM Series chassis. main objective is to provide “fat pipes” and WDM transport. With the inclusion of without any need for Ethernet aggregation. the PXM, the DTN-X family is transformed for long-haul purposes or in a densely 2.6.5 The Multi-layer Service FIGURE 35: Ethernet Services into a fully packet-aware platform where populated area, and where the packet- Management System Provided by Separate Ethernet Switches Attached to an Optical The metro aggregation and metro core packet flows can be aggregated and switched mode traffic now can be transported by The chosen network architecture has 2.6.3 Optical Add-Drop Multiplexers Network Result in a Complex networks are interconnected via the XTM on individual ODUs to facilitate traffic individual flows fully managed according a profound influence on the degree of Hierarchy of Management Systems Series packet-optical platform, which can and Other Optical Elements engineering (Figure 34). to OTN standards. Furthermore, the PXM operational simplicity that it is possible The Layer 2-specific elements of metro provide switching at OSI Layers 1 and is certified for and can terminate CE 2.0 to achieve when it comes to network Element Management aggregation and metro core networks System for the 2. In the metro core, Ethernet SVLANs services such as E-Line and E-LAN, making packet layer and MPLS-TP enable detailed handling use Layer 1 optical WDM channels for The PXM maps the Ethernet VLANs and it possible to set up Ethernet services Service Management of Layer 2 services, while WDM keeps the transport of Ethernet frames between MPLS services of the packet-optical network starting in an NID in the access network and System legacy transport services at Layer 1. A network nodes, as described in section to the Layer 1 OTN transport services. This terminating via a DTN-X PXM in the metro 2.2.3. All the above Layer 2-specific network mapping is especially cost-saving when Layer 2 (Packet) ROADM-enabled Layer 1 provides flexible core or long-haul network. More about how Integration wavelength switching capabilities, while a elements interwork seamlessly with the an OTN system is already in place, e.g. the PXM maps Ethernet traffic to OTN can Layer 1 (Optical) unified control plane provides management flexible optical networking elements at be found in section 2.2.5. Layer 1 when present in a truly integrated

packet-optical platform, such as Infinera’s Element Management System for the XTM Series. An XTM Series node may optical layer

36 INFINERA Footnote citations on page 111 INFINERA 37 PACKET-OPTICAL NETWORKING

nifie acetptica This multi-layer management approach Reduced Investment and Operational Costs Potential for Additional Service Revenues basis. Support for scaling Ethernet services Serice anaement an brings further benefits in terms of lower cost The network will have fewer physical units Service awareness is critical if the is the most important requirement, since ement anaement for management hardware, less training, through the integration of Layer 1 and differentiated QoS requirements of new Ethernet traffic will constitute the great less integration and simpler administration Layer 2 transport functions in the same multimedia applications and cloud services majority of future growth in bandwidth and maintenance of the entire network. hardware platform. This integration reduces are to be met end-to-end throughout the requirements. With an integrated packet- Especially for network operators that do packaging and cabling costs, as well as all network. To do that, it is important to retain optical platform, it is easy to upgrade not already have an extensive operations types of inventory. This integrated approach service transparency where necessary, i.e. transport capacity as subscriber demand support system (OSS) in place, the unified is designed to reduce network complexity to leverage all existing information present grows. packet-optical management system offers and lower operational expense (OpEx) in Ethernet VLAN tags and MPLS labels, for End-to-end OAM is important to ensure significant advantages over the integration significantly. example. This should be done until such the reliability and resiliency of the transport of multiple separate management systems. a point in the network where traffic with Efficient aggregation of Ethernet traffic can underlying the services carried over the the same QoS requirements has been fully solve the problem of underutilized WDM network. It is important to ensure that aggregated. 2.7 ADVANTAGES OF PACKET- channels at the edge of the network. By both the SLAs and service objectives that OPTICAL TRANSPORT filling the pipes, packet-optical transport Full service awareness enables the operator are internal to the service operator and increases the utilization of available to differentiate and market higher-value those that might explicitly be offered to Now that we have a better understanding bandwidth and facilitates an efficient services with specific SLAs, increasing a subscriber are met. Efficient tools for FIGURE 36: Ethernet Services Provided by an Integrated Packet-Optical Platform and Managed from a of the fundamental principles of packet- handover to the core network at a lowered service revenues. operations and maintenance are vital to Multi-layer Management System such as DNA-M to Simplify Operations and Reduce Cost optical networking, we can summarize the cost. achieve customer satisfaction. advantages, both in general terms and more If the network can distinguish between specifically as they are implemented with Better Customer Satisfaction high-value IP services and legacy traffic, it is the Infinera product portfolio. and Thereby Reduced Churn order to provide a useful Ethernet service status information and can manage both possible to offload legacy traffic to the most Massive scalability is easily achieved through provisioning and assurance environment. optical- and packet-mode equipment. Since cost-effective transport at Layer 1, avoiding Layer 1 and Layer 2 functions are handled the WDM functionality in an optical/ Using a true packet-optical network and an the consumption of equipment ports with by a single system, provisioning of Ethernet 2.7.1 Benefits of the Packet-Optical Approach Ethernet platform, where multiple services integrated multi-layer management system, higher complexity and cost, except where services affecting both Layer 2 and Layer Packet-optical networking in general helps can be assigned to the same wavelength such as the Infinera DNA-M Digital Network needed. 1 can be done by simple point-and-click the operator attain several types of valuable or to their own specific wavelengths. New Administrator for XTM Series, improves this commands from the management system. advantages: wavelengths can be added on an as-needed situation. Furthermore, Layer 2 services are monitored A multi-layer management system has from end to end and adequate Layer 1 access to both Layer 1 and Layer 2 network resources can be allocated directly, should optical paths be broken or changes in the traffic pattern occur.

38 INFINERA Footnote citations on page 111 INFINERA 39 PACKET-OPTICAL NETWORKING

2.7.2 Advantages to Using Infinera’s Portfolio level of functionality for the transport task. IEEE 1588v213 is a common standard Low Power Consumption Multi-layer Management for Packet-Optical Transport Including superfluous functions adds to both to deliver both phase and frequency Energy costs can be a significant item in the Infinera’s multi-layer DNA-M management equipment cost and operational complexity, information over a packet network, but it OpEx of any telecommunications network. system enables unified Layer 1 and Layer Packet-optical transport provides many especially in the aggregation and metro/ is sensitive to packet delay variation (jitter). The hardware elements of Infinera’s intelligent 2 OAM of the packet-optical network. general advantages, as described above. But regional segments of the network, which have IEEE 1588v2 can be deployed over any packet transport architecture have all been The management system supports the full other characteristics more related to the actual the majority of the network nodes. On the legacy Ethernet network, but it may rapidly designed with this in mind, with typical power range of operations, administration and implementation of the packet-optical nodes other hand, the node must include sufficient lead to quality problems if jitter becomes consumption as low as less than 7 watts (W) maintenance functions defined by MEF used in the network are also of significant features to make service differentiation, OAM too high. With Infinera’s transport-centric, per 10 Gigabit Ethernet, for example. for Carrier Ethernet 2.0. Furthermore, and necessary resilience possible. almost-zero-jitter implementation of network the system fulfills the requirements of importance. Infinera’s portfolio for packet- Not only does low power consumption save nodes, IEEE 1588v2 will converge fast and ITU-T Recommendation Y.1731, which also optical networking combines Infinera’s long The EMXPs and other Layer 2 equipment have on direct energy costs, but for every kilowatt can deliver the required time stamps. addresses performance management. and recognized optical networking experience all been designed with this balance in mind. saved in equipment power, an additional 0.5 with outstanding Ethernet/Layer 2 capabilities, Functionality of value in a metro aggregation kilowatt in reduced need for air conditioning including MEF Carrier Ethernet 2.0 services, or metro core network, such as the capability Low Latency is saved too. MPLS-TP and OTN-compatible transport. to handle MPLS-TP, is included, while most of The EMXPs and the PT-Fabric are designed Furthermore, the elements in this portfolio the complex IP handling has been omitted. for transport and capacity; they have no are truly optimized for the simplest and most central switching fabric, no input queues and High-density Implementation cost-efficient implementation of packet-optical no network processors limiting performance. The EMXPs and PT-Fabric have all been transport networks. Efficient Transport of Synchronization This results in minimal delay and no jitter designed to provide as high port density per Accurate synchronization is essential, within the EMXPs and the network. The unit as possible while still maintaining high especially in mobile networks. As an example, ultra-low latency has significant value overall, capacity. This translates to smaller footprint A Lean and Transport-centric Long Term Evolution (LTE)/4G and 5G and is crucial in certain applications, e.g. and reduced volume requirements, leading Implementation of Ethernet/Layer 2 networks need an accurate time stamp as between data centers and in algorithmic to less costly installations than with legacy Functions well as frequency synchronization. The EMXPs trading applications for the financial world. equipment. The packet-mode functionality of a packet- and PT-Fabric fully support ITU-T SyncE Additionally, the stability and low latency of optical node may include anything from a standards for the distribution of timing signals the EMXPs add minimal jitter and delay in simple Ethernet bridge to a full-fledged throughout the packet-optical network. mobile backhaul applications, enabling more IP router and more. When implementing a and longer radio hops when a combined transport-focused packet-optical node, it is of wireless and wired backhaul network is the utmost importance to select an optimal being deployed.

40 INFINERA Footnote citations on page 111 INFINERA 41 3 OPERATING THE NETWORK OPERATING THE NETWORK

3.1 CHAPTER SUMMARY In Chapter 3, we therefore take a closer multi-layer network management continues Network Management Software-defined Networking among operators and their vendors in order look at the network management principles to support the full range of infrastructure Traf c engineering Programmable services to facilitate business and network operations. The cost-effectiveness of a real network Device and network management (FCAPS) Dynamic multi-layer service provisioning applicable to packet-optical networks, management, from configuration and Traditional service management and optimization TM Forum (TMF, formerly TeleManagement is highly dependent on how easy it is and at how SDN complements network element management to monitoring, fault Forum), an industry association for service to operate and how well it can adapt to management for dynamic service delivery isolation and service assurance. Traffic OSS SDN Applications providers and their suppliers in the shifting service demands. While a well- and network optimization. engineering and service provisioning can App 1 App 2 App N telecommunications and entertainment designed multi-layer network management be shared between traditional network Network Multi-Layer industries, has played a central role in this The key point to emphasize is that multi- Manager SDN Platform system is critical to efficient operations, management and SDN platforms, with work. layer network management and multi-layer network operators are also increasingly hybrid operation that protects existing SDN are complementary. While SDN Frameworx, which was developed by TM looking to software-defined networking services while introducing new SDN-driven generally focuses on the automation and Forum, is a suite of best practices and technologies to enable more agile, dynamic services on the same physical network. With programmability of end user services, standards that enable a service-oriented, service delivery and multi-layer optimization. this approach, SDN can be introduced step automated and efficient approach to business by step, for example, in only one domain of operations. Frameworx provides hundreds the network or applied only to a particular of standardized business metrics that allow OSI layer. for benchmarking, as well as a suite of thernet erice no. 1 Physical Transport Network interfaces and APIs that enable integration 3.2 MULTI-LAYER NETWORK FIGURE 38: Service Provisioning via Network Management and SDN on the Same Physical Network across systems and platforms. Frameworx also includes adoption best practices and thernet erice no. 2 MANAGEMENT examples. Many of the advantages of packet-optical The central business process model used networking originate from the ability to he thernet irtua Connection Ethernet service and determine if a fault in mind. Infinera’s suite of management in Frameworx is the Business Process manage both the Layer 1 optical channels originated in the optical or packet switching systems, including the Infinera Digital Framework (eTOM),14 which also is maintained and Layer 2 Ethernet services of the network elements of the network. Network Administrator (DNA), provides by TM Forum (Figure 39, next page). This in a coordinated way (Figure 37). Layer network operators with full control of model describes the required business he optica channe λ 1 optical channels can be established at Coordinated handling of the optical and their integrated packet-optical network processes, including network management the same time as the Ethernet service Ethernet layers of the network calls for a and supports planning, deployment and and operations, of service providers, and attributes are assigned to the Ethernet multi-layer management system and, when operation of the network infrastructure in a defines their key elements and how they Virtual Connections that are using the Layer using SDN, a multi-layer SDN platform he optica fier an it ampifier cost-efficient and rational way. should interact. eTOM is a guidebook that 1 optical channels. This allows for an easy (Figure 38). These systems must be defines the most widely used and accepted procedure for end-to-end service creation. designed from the start to enable multi- FIGURE 37: An Ethernet Service in a Packet-Optical Network Is Created over Multiple Layers of Underlying standard for business processes in the Connections that Need to Be Managed in a Coordinated Way layer visibility and management, while also Importantly, it is only then possible to 3.2.1 Management Framework keeping ease of use for network operators telecommunications industry today. efficiently monitor the performance of an A great deal of the standardization of management procedures has taken place

44 INFINERA Footnote citations on page 111 INFINERA 45 OPERATING THE NETWORK

a predefined service quality level for an • Configuration management (C) refers to Stratey nratructure rouct peration ufiment urance iin optimal subscriber experience. functions for making orderly and planned Stratey nratructure rouct peration ufiment urance iin Commit iecyce iecyce Support eenue changes within the network. An important anaement anaement eaine anaement part of configuration management is aretin er anaement Cutomer eationhip anaement Infinera’s network management philosophy fully keeping an inventory of equipment, adheres to the relevant TM Forum standards. software releases, etc. in the nodes. For example, for integration with back-office Cutomer oSS na Serice eeopment anaement Serice anaement peration support systems, DNA comprises Frameworx- • Accounting (or administration) Cutomer anaement compliant web-services interfaces based on management (A) deals with functions that eource eeopment anaement eource anaement peration the TMF608 model. These interfaces hide the make it possible to charge users for the Serice Confiuration Serice roem na ppication Computin an etwor ppication Computin an etwor complexity of the underlying optical network network resources they use. ctiation anaement and are designed to reduce the time, risk and Suppy Chain eeopment anaement Suppierartner eationhip anaement costs associated with systems integration. • Performance management (P) comprises functions for monitoring and fine-tuning Serice the various parameters that measure the Infinera’s DNA multi-layer management performance of the network and forms nterprie anaement eource roiionin eource roue na functions (DNA-M) provide the following full the basis for service level agreements with anaement Strateic nterprie nterprie i nterprie ectiene nowee eearch annin anaement anaement anaement suite of MTOSI 2.015 -compliant interfaces: network users. eource erormance

inancia et Staehoer terna uman eource • Inventory • Security management (S) refers to eource anaement anaement eation anaement anaement • Alarms administrative functions for authenticating • Activation/provisioning users and setting access rights and other FIGURE 40: The Main eTOM Business Processes Supported by Infinera’s Management Software Suite • Performance statistics permissions on a per-user basis. FIGURE 39: The eTOM Business Process Framework. Source: TM Forum Another method often used to classify the many diverse actions involved in the The components included in Infinera’s suite management of communications networks 3.2.2 Infinera’s Management Software Suite network and its services. The suite has been of management software can broadly be For the management of packet-optical activities create an infrastructure that is the FCAPS suite: Infinera offers a full suite of tools for the created based on the Frameworx standards grouped into five categories, as depicted in networks, two areas (columns in the eTOM matches the supply of services with management of its optical and packet- and eTOM framework described above, but Figure 41 (see next page). model) are of particular interest: subscriber demand in an economical way, • Fault management (F) encompasses optical transport networks. These tools help given the role of Infinera as an infrastructure and with consistently high levels of quality functions for detecting failures and operators with tasks throughout the entire vendor, Infinera’s management suite focuses At the most fundamental level, each • Fulfillment, which involves supply chain and reliability. isolating the failed equipment, including service and network life cycle when planning, on the subset of activities in eTOM shown in network element of an Infinera packet- activities for assembling and making the restoration of connectivity. deploying and operating the packet-optical Figure 40. optical network supports local configuration services available to subscribers. These • Service assurance, which is the application of policies and processes to ensure that services offered over the network meet

46 INFINERA Footnote citations on page 111 INFINERA 47 OPERATING THE NETWORK

network subscribers direct access to certain 3.2.4 A Unified Information Model Inventory Provisioning NMS Performance Network Planning System network management capabilities through for Multi-layer Management a customer portal. Such a portal allows At the center of DNA’s multi-layer MTOSI 2.0 subscribers to monitor the performance of management architecture for packet-optical their services and adapt service parameters networks is the unified information model.

Optical & service planning without a need for manual operator There are several standards for modeling intervention. Finally, the suite includes multi-layer networks: the ITU-T G.805 Digital Network Administrator OSS Integration Toolkit Customer Portals planning and design tools, of great value recommendation provides a description

Netcool Open View OSS SDN • SNMP when preparing network deployment and of circuit-switched network connections OA OA • TL1 MDU MDU expansion. through a multi-layer network, while the • XML EMXP EMXP ROADM G.809 recommendation provides the same Intagrate into OSS environment End-user service management but for connectionless networks. G.805 3.2.3 Infinera Digital Network Administrator and G.809 provide generic methods for Element Managers Network Managers A central component in Infinera’s modeling and describing networks from management suite is the Digital Network a functional and structural architecture DTN-X Metro Core and Local Craft Digital Administrator, a cost-effective and perspective. XTM Series Long-haul XTM Series Terminal Network scalable carrier-class service and network Metro Access/Core Manager The G.805/G.809 model can be mapped management system. DNA provides into management information models using Local management of nodes Management of In nera networks a centralized system for operations, the equipment management functions administration and management of specified in the TMF608 (MTOSI 2.0) model. FIGURE 41: An Overview of Infinera’s Management Software Suite the entire packet-optical network and Management information models are FIGURE 42: Infinera Digital Network Administrator (DNA) hides the complexity of the underlying specifically important because they formally equipment to higher-order business and define and describe the reference points that operations support systems. Its integrated the operator’s OSS must interact with in order including all the OSI layers in between. At The fact that there is a single, unified and control by an internal element manager The network operator often has extensive management capabilities also provide the to manage a piece of transport equipment. the bottom of the multi-layer information information model from the fiber up to accessible through a local I/O device. other operations and support systems foundation for service management of model used by Infinera is the TMF608 model the Ethernet service is the enabler of Centralized, multi-layer management of for administration, billing, etc. of the individual end-to-end connections. DNA The unified information model allows the for the Layer 1 network. The higher-order unified management as well as all the all the elements in a network is done from services provided. Hence, the network increases the visibility of the network and entire network to be modeled, all the way models for connection-oriented Ethernet operational benefits from using a truly a network manager – Digital Network manager must interoperate smoothly simplifies many repetitive tasks, with the from the optical fibers run in the ducts up (802.1Q tunneling, or Q-in-Q), MPLS-TP and integrated packet-optical network. In a Administrator – that provides an end-to-end with the operator’s OSS environment, a goal of increasing the performance of to the services created on top of them, multi-point Ethernet models are added onto network with separate Layer 1 and Layer view of the connections and services. task accomplished by an OSS integration the network while lowering operational the Layer 1 model, as indicated in Figure 43 toolkit. It is also often desirable to give expenses (Figure 42). (on the next page).

48 INFINERA Footnote citations on page 111 INFINERA 49 OPERATING THE NETWORK

L1/L2 Planning tool L1/L2 Service L1/L2 Service MTOSI 2.0 provisioning Assurance compliant interfaces configuration process and reduces the time management processes from Layer 1 into 3.2.7 Network Planning and Optimization and cost to provision resources and services. Layer 2. The app is plug-and-play on top of Intelligent planning of network build-out XML- le the Layer 1 assurance web app, and not only is essential, both when new networks are To be able to deliver and provision a Layer OA OA adds Layer 2 management capability but established and when expanding an existing MDU MDU 2 service, the optical channels must also be MD EMXP EMXP also provides integrated Layer 1 and Layer network. Infinera’s management suite configured. One advantage with DNA multi- ROADM TMPC-MACA 2 management from one unified graphical therefore includes the Infinera Network layer management is that the operator can interface. Planning System (NPS), which facilitates provision an optical channel with the same planning tasks such as traffic engineering of management system before configuring a Performance management of a Layer 2 network optical links, planning of service routing and Point-to-point EVC Multipoint EVC Layer 2 service, effectively speeding up the comprises the measuring of throughput, delay, network optimization. The system supports (E-Line, E-Access) (E-LAN, E-Tree) entire provisioning process. jitter, and packet loss for a particular service. extensive reporting of results in formats such To define the endpoints between which such Provisioning services via a multi-layer SDN as deployment packages, equipment bills of measurements are taken, Maintenance Entity control platform (as outlined in Section 3.3), material, circuit traces and more. Groups and Maintenance End Points are TMF608 SVLAN such as the Infinera Xceed SDN controller, FIGURE 44: DNA-M Web App with a TMF608 MPLS-TP TMF608 based Ethernet connectionless defined, as described in section 5.7.4. information model multi-point information model Multi-layer View of an Ethernet Virtual information model automates and simplifies this process even further, because the platform includes multi- The assurance web app provides a Connection layer path computation and constraint-based complement to established Ethernet standards 3.3 SOFTWARE-DEFINED Layer 1 TMF608 based information model provisioning logic to enable the provisioning for fault management, such as IEEE 802.1ag NETWORKING AND NETWORK of all layers based on Layer 2 service (Connectivity Fault Management or CFM). The web app discovers and tracks the VIRTUALIZATION requirements. The goal of CFM is to monitor an Ethernet operational state of the Layer 2 services FIGURE 43: The DNA Multi-layer Management Architecture is Based on a Unified Layer 1 and Layer 2 network and pinpoint where a problem occurs, and the underlying Layer 1 paths that 3.3.1 Basic Concepts Information Model 3.2.6 Layer 2 Service Assurance while DNA-M provides a graphical multi-layer support them. Fault information is Software-defined networking is an approach A Carrier Ethernet network must provide user interface to help in quickly finding the indicated not only on the map but also to computer networking that allows the the ability to monitor, diagnose and centrally exact root cause of a networking problem, graphically for individual paths, links and management of network services through manage the network, using standards-based independent of whether it occurs on Layer 1 ports. If the origin of a networking problem an abstraction of the managed network 3.2.5 Layer 2 Service Fulfillment vendor-independent implementations. 2 service delivery, a unified view must be or Layer 2. resides in Layer 1, the user can turn to the and its functions. This network abstraction achieved in a higher-order OSS through For networks with Layer 2 equipment, DNA-M Infinera’s DNA takes this one step further by features in the Layer 1 assurance app to offer a provisioning web application that offering management across the layers to enables a physical decoupling of the system the integration of multiple systems. The quickly resolve the problem without having provides point-and-click provisioning for both further simplify management, and increase that makes decisions about where traffic is unified information model, by contrast, to change system or interface. The app also Ethernet SVLANs and MPLS-TP paths. The visibility of the network, with the goal of presents G.826 performance statistics for sent (the control plane) from the underlying enables all administrative and management provisioning app provides point-and-click reducing operational costs. Layer 1 (Figure 44). system that forwards traffic to the selected actions, such as planning, provisioning and creation of tunnels, label-switched paths The packet-optical assurance web app destination (the data plane). Thus, instead operations, to be performed across Layer 1 and pseudowires, which automates the of being executed in each network element, and Layer 2. in DNA-M extends the management system’s operational model and well-known

50 INFINERA INFINERA 51 OPERATING THE NETWORK

open platform, the Xceed solution includes about packet routing and forwarding, and Related to the centralization of routing Bandwidth Network Virtual Traditional SDN on Demand Planning Networking open application programming interfaces thus enable more flexibility in how services control is the problem of finding the best Application Plane

both “northbound” and “southbound” to are controlled. One useful application of paths for traffic flows in the SDN-controlled Virtualized Network Controller and from the controller and a published SDN control is to create packet-oriented infrastructure. In a dynamic control plane open data model. This allows Infinera services with some of the characteristics architecture for connection-oriented Control Plane customers or third-party developers to of connection-oriented16 services. networks, this calculation is done by some integrate additional functionality beyond Centralized routing control with SDN type of path computation function, which the range of Infinera-developed SDN enables more efficient traffic engineering of uses its knowledge of network status and applications, and to include non-Infinera connectionless data flows, of great value, topology, available resources and applicable network elements in the solution. for example, within data centers with a constraints and policies to identify a route large number of nodes and where network for a given request. As networks grow in Control An SDN controller may control how optical topology seldom changes. Centralized complexity with several interdependent Data Plane paths are established, how Ethernet frames control also allows for the programmability OSI layers and network domains, the task are switched and how IP packets are routed of routing decisions, thereby enabling the of path computation becomes increasingly in a packet-optical network, or in the case Layer 2 Infrastructure Layer 1 Infrastructure Layer 3 Infrastructure automation of tasks otherwise performed challenging and resource-demanding. Network equipment of a multi-layer SDN controller, all of the Network equipment manually from the management system. above. It may even direct the handling A path computation element (PCE) is a FIGURE 46: Network Virtualization Is a Central Feature of Software-defined Networks FIGURE 45: Software-defined Networking. of specific flows of application traffic, The concept of SDN is applicable not only software entity capable of computing a Control of the Traffic Flows Within the Network for example, a video stream that should to Layer 2 and Layer 3 packet networks, network path or route based on a network Is Separated from the Network Equipment and receive the same forwarding treatment by but also much more generally to all types graph and applying computational Placed in an SDN Controller all network elements. Thus, SDN allows of communications networks, including constraints, i.e. capable of performing the benefits of a PCE are only attainable if it is correlate path computations between all the network administrators and external the photonic layer (Layer 0) and optical path computation function. The main idea capable of multi-layer and multi-domain relevant layers. As an example, provisioning applications to have programmable and switching layer (Layer 1) of wide area behind a separate PCE is to isolate the path computations. a Layer 2 Ethernet Virtual Connection could the functions of the control plane can be centralized control of all types of traffic flows transport networks. When SDN is applied path computation function and place it involve coordinated path computations at In theory, the PCE can be located in centralized to one or more SDN controllers. within a network (Figure 45). to multiple layers in a wide area network, into a dedicated high-performance system Layer 2, Layer 1 and Layer 0. it is often referred to as multi-layer SDN or accessible via an open and well-defined the network management system, in The Infinera Xceed Software Suite is a Since packet-mode services such as IP, Transport SDN.17 interface and protocol. Path computation an SDN controller or in one or more Closely related to SDN is the concept of portfolio of software solutions based on the Ethernet. etc. are inherently connectionless can be restricted to a single OSI layer or network elements. However, given the network functions virtualization (Figure OpenDaylight platform. In addition to this (i.e. routing decisions are done “on the fly” to a single network domain, but the real computational resources required for multi- 46). The idea of NFV is to move network packet by packet by the individual network layer path computation, centralizing the functionality such as intrusion detection, elements), software-defined networking PCE to the SDN controller is generally the load balancing and network demarcation was originally used to centralize decisions most cost-efficient solution. When using a multi-layer SDN controller, the PCE must also be multi-layer, meaning that it can

52 INFINERA Footnote citations on page 111 INFINERA 53 OPERATING THE NETWORK

from specialized devices to software run over the same physical resources. Hence, OpenFlow enables SDN controllers to on common server platforms. Advances different service providers using the determine and set the paths of traffic flows APPLICATION LAYER

in technology have now made it possible transport network may “see” different across a network of switches. As an open Business Applications to run applications that once required Virtual Network 1 virtual networks, depending on the standard, OpenFlow also allows switches specialized central processing unit (CPU) resources they have been allocated (Figure from different vendors to be controlled from API API API and interface hardware on industry-standard 47). Furthermore, network virtualization the same SDN controller. Generally, the CONTROL LAYER server blades. Arguments in favor of the Enterprise A offers the owner of the physical transport OpenFlow protocol is carried within ordinary Network Services transition to NFV are both the lower cost of network the possibility to delegate more Transmission Control Protocol (TCP)/IP Network Services servers compared to specialized appliances network control to the service providers or packets sent between the controller and the and the rationalization of spares handling enterprises using its transport network. network elements. – instead of numerous specialized devices, Software-defined networking requires The OpenFlow standard was developed only server blades need to be kept in stock, some method for the control plane by ONF, but ONF does not provide INFRASTRUCTURE and the servers are given their functions LAYER to communicate with the data plane any implementation of either switch or by the installed software. In short, NFV Virtual Network 2 equipment making up the physical network controller software. There are, however, allows for more dynamic administration of infrastructure. OpenFlow, developed by quite a few open-source components network resources, making it possible for Physical Packet-Optical Network the Open Networking Foundation (ONF),18 available, including the source code for the network operator to respond to new Enterprise B was the first such standard communications OpenFlow itself, which will speed up both demands in a more agile way. interface defined between the control and the standardization and rate of adaptation Software-defined networking increases forwarding layers of an SDN architecture. of SDN. the flexibility of how the physical network FIGURE 47: Network Virtualization Allows Two Enterprises to “See” Different Virtual Networks, Although OpenFlow allows direct access to and FIGURE 48. OpenFlow Defines the Communication As SDN has matured and SDN principles can be used, since network services are They Use the Same Physical Infrastructure manipulation of the forwarding plane of Interface and Protocol Between the Control Plane and have been applied more broadly to a wide created dynamically by the controller and network devices such as switches and the Data Plane. Source: Open Network Foundation range of networks, additional protocols for the image it has of the network, rather routers, both physical and virtual (Figure 48). control plane-to-data plane communication than by direct manipulation of the physical resources and network functionality into a the virtual network matches the dynamic have been developed to complement switches, routers, etc. The SDN controller single, software-based administrative entity, reconfiguration capabilities of computing OpenFlow. Some examples of other thus enables network virtualization, i.e. it a virtualized network. and storage in cloud computing. “southbound” protocols from the control maps the status of the physical network Network virtualization brings a higher Network virtualization also enables the layer to the infrastructure are the Open degree of abstraction to the transport creation of logically isolated virtual networks vSwitch Database (OVSDB) management network, which is favorable when it comes over a shared physical network—multiple protocol, Representational State Transfer to mobility of resources, service creation virtual networks can coexist simultaneously (REST) and NETCONF. and management, especially in the context of cloud computing. The flexibility of

54 INFINERA Footnote citations on page 111 INFINERA 55 OPERATING THE NETWORK

Multi-layer SDN enables several important instances that interoperate with the IP 3.3.2 Infinera’s Multi-layer SDN Vision characteristics of a modern, flexible and control plane, for example IP routers, in Infinera envisions that future network cost-effective transport network, such as: legacy networks. SDN-enabled Intelligent architectures will be simplified into just two L0-3 Service Transport Networks simplify the creation of Mediation main layers: Layer C, cloud services, and Layer • Multi-layer virtual networks including Layer C software-defined virtual networks, improve T, intelligent transport. Layer C consists of all OSI Layers 0 to 2 (photonic/WDM, OTN, SDN Control geographical service scalability and can virtualized network functions, services and Ethernet and MPLS-TP) applications that run as software within a cloud reduce routing costs. environment, including virtualized routing Virtual networks can be defined in a flexible Multi-layer Control control. Layer T consists of integrated packet- way over physical network resources in optical networks. SDN plays a central role in one or more network domains. These Infinera’s multi-layer SDN architecture has E-Line this architecture as the bridge between virtual networks may span multiple network standardized, open software interfaces, Layer C and Layer T (Figure 49). CE A CE B Layer T layers and multiple network types, and can which make it possible for other higher-level be used by one or more customers with management and orchestration systems, as well as other external SDN-compatible web EMXP EMXP White Box complete isolation as needed between Video Video Merchant Si SDN principles are applicable to all OSI apps, to interact with the multi-layer SDN Source Source customers, protocols employed and OSI levels of an Infinera Intelligent Transport control platform. The Xceed multi-layer SDN layers. control platform is logically centralized, and Routers/ Network; hence, Infinera’s SDN vision for Switches the Xceed solution is also a multi-layer SDN • Automatic provisioning of MEF Carrier may be physically located in a single central Packets/Optical Optical node or distributed to multiple sites in the EMXP Transport Transport vision. This integrated, multi-layer approach Ethernet 2.0 services network. Thus, Xceed enables seamless to SDN offers several benefits: All Carrier Ethernet services as defined integration with other software-defined FIGURE 50: Bandwidth on Demand • Simplified service provisioning across by MEF CE 2.0 can be provisioned networks and is an excellent foundation for Multi-layer Packet-Optical Network multiple network layers automatically with SDN. Bandwidth can the virtualization of higher-order networks and services. be allocated to services dynamically, the network operator. The operator may, for traffic patterns. Dynamic services will also FIGURE 49: The Infinera SDN Vision: • A high level of automation when setting with restoration and protection functions example, adjust the bandwidth of a service automatically detect changes in network Multi-layer SDN Control of a Multi-layer up end-to-end paths in transport networks 19 centralized. dynamically based on customer requests topology and optimize services accordingly. Packet-Optical Network 3.3.3 Applications of Intelligent Packet • Improved network resource utilization as a (bandwidth on demand), or optimize the • Virtual routing – distributed routing Transport with SDN An SDN application of specific interest result of better topology information routing of an existing service as the network functionality With multi-layer SDN as an element of from a cost and efficiency standpoint is the • The programmability of new functions intelligent transport, new and attractive evolves without dropping a single packet As described in section 3.3.3, an Intelligent introduction of virtual routing, also referred options for service differentiation and (Figure 50). • Open standards and vendor neutrality Transport Network can employ SDN to to as distributed routing functionality (DRF), network operations become available to implement virtualized IP control plane Dynamic Layer 2 services with SDN provide a way for the network to reconfigure itself to meet bandwidth peaks and unpredictable

56 INFINERA Footnote citations on page 111 INFINERA 57 OPERATING THE NETWORK

within the transport network. The objective engine in the SDN controller. The routing Logical of DRF is to let virtualized routing engines, engine makes decisions and sends updated implemented as software instances residing routing control packets back toward the in the multi-layer SDN control platform, external peer routers. The routing engine for participate in Layer 3 routing decisions and the DRF also creates its own routing tables, control the physical network elements to which are turned into forwarding rules, make segments of the transport network controlling the packet flows in the physical behave as if they were IP routers when seen transport network. ASR ASR from the outside world of other IP routers. DRF enables significant simplification and Note that a DRF solution is not intended cost reduction, especially in aggregation EMXP as a direct replacement for conventional vRouter networks, which are common in metro areas. core routers, but can offer significant As shown in Figure 52, without a DRF, all Link for opportunities for capital expense (CapEx) protection IP traffic is forwarded to the central core EMXP EMXP reduction and a more seamless integration router, even traffic that could be sent directly of Layer 3 out to the network edge, and between the metro nodes. Adding more can extend the operational life of existing base stations with site routers typically also CSR CSR Physical routers in the metro environment. involves manual configuration changes to Figure 51 shows how a DRF function might the transport network. If DRF is enabled, FIGURE 51: A Physical Packet-Optical Network work. The DRF (shown in both logical and local IP traffic between sites is no longer Acting as a Virtual Router physical views in Figure 51) “listens” to Layer routed via the central core router. Since the 3 control plane traffic (e.g. Open Shortest DRF updates its routing tables dynamically, Path First [OSPF], Intermediate System to configuration changes at the IP level are Intermediate System [IS-IS] and Border detected automatically, without any manual FIGURE 52: Virtual Routing in a Mobile Backhaul Network Gateway Protocol [BGP] packets) from its intervention needed in the transport “real” router neighbors (peer routers) and network. passes the control packets to a routing

58 INFINERA Footnote citations on page 111 INFINERA 59 4 APPLICATIONS OF PACKET-OPTICAL NETWORKING

60 INFINERA Footnote citations on page 111 INFINERA 61 APPLICATIONS

4.1 CHAPTER SUMMARY of Ethernet services has already been networks use WDM as the underlying discovered by industries such as finance, transport technology and are enhanced with integrated Layer 2 functions, which are Packet-optical networks that provide healthcare, education, government, IT, retail, with Ethernet switching functions to carry located in the network nodes. All units are fully MEF-certified for Carrier Ethernet 2.0 Carrier Ethernet services are rapidly Metro Core real estate, legal, media and more. Ethernet traffic between the subscribers’ services and form an integral toolbox for becoming a primary infrastructure for Enlighten local area networks (LANs) and other data telecommunications network operators. The An increasing number of network operators the creation of a state-of-the-art packet- EMXP EMXP networks as needed. versatility of packet-mode Ethernet services are offering MEF-compliant Carrier Ethernet optical network. Furthermore, the EMXPs are seamlessly integrated with other traffic units and the capacity and scalability of optical services. In many cases, these services are The integrated packet-optical network and optical units in the Infinera XTM Series, ROADM ROADM replacing some of the operators’ legacy enables both transparent Layer 1 services transport have proven to be a winning forming a true packet-optical network. combination in a variety of applications. In EDU technologies (such as FR and ATM), while and more advanced Layer 2 (Carrier in other cases they coexist alongside Ethernet) services. Having the optical The Ethernet Demarcation Unit is designed this chapter, we take a brief look at some of EMXP EMXP the areas in which packet-optical technology other established wide area networking network as the base makes it possible to for minimal delay and jitter, providing unprecedented QoS and SLA fulfillment. The has already proven its superiority. EMXP technologies, such as Layer 3 virtual private support a broad range of traditional Layer EDU includes highly accurate and precise network services (also referred to as IP-VPN 1 transport services. Also, thanks to the Aggregation OAM and performance monitoring through services). integrated Layer 2 functions, it is possible to microsecond resolution and per-service offer the enterprise a wide range of Ethernet 4.2 ETHERNET SERVICES FOR For the network operator, Ethernet services visibility for all key OAM and SLA parameters, services with different bit rates, QoS and ENTERPRISES – BUSINESS stoe eises ipent nits seaess integate it 1 nits offer an additional business opportunity, enabling individual SLA monitoring and service tenet eacation nits pones s etc oing one SLAs, using bandwidth profiles and various differentiation. ETHERNET especially if Carrier Ethernet can be eto nteace eices integate pacetoptica neto protection and monitoring functions. The implemented in a cost-efficient way on the The EMXPs are scalable up to 440G and 4.2.1 Serving Enterprise Customers packet-optical network is ideally suited acetptica anspot itces tiae anageent same equipment already used to provide also designed to have ultra-low latency and The enterprise information and to support a multitude of enterprise communications technology (ICT) landscape Layer 1 transport services. Infinera and jitter. They can be equipped with long-reach subscribers, each with their own individual interfaces prepared for integration with the is constantly changing. New IP-based voice FIGURE 53: A Typical Network Configuration when Providing Carrier Ethernet Services to Enterprises in a Metro Area other packet-optical equipment vendors are requirements for the characteristics of the OTN transport core, have fast and flexible actively enabling this important transition. and video services, cloud computing, and wide area service they want to buy. OAM processing and are fully integrated with distributed data centers are adding new the XTM Series optical transport platform. requirements to the enterprise wide area are gradually being phased out by many attractive alternative to provide businesses data network (WAN), especially in terms network operators. Wide area connectivity with a best-of-breed, cost-effective wide 4.2.2 A Metro Network for All XTM Series equipment is managed by Infinera’s portfolio for metro Ethernet includes DNA-M, a multi-layer management system of increased bandwidth and enhanced based on TDM leased lines does not area networking solution for enterprise ICT Business Ethernet both demarcation units (EDUs and NIDs), that includes advanced functions for the Figure 53 shows the principal elements of performance. Legacy WAN connectivity provide the operational flexibility expected applications. Ethernet standards from the which can be placed at the customer site (as seamless provisioning and management of technologies such as frame relay (FR) by the modern enterprise (Figure 53). MEF, ITU and IEEE have now added features a typical packet-optical network deployed customer premise equipment or CPE), and Carrier Ethernet 2.0-compliant services. and asynchronous transfer mode (ATM) and functionalities that make Ethernet a by a network operator providing Carrier packet-optical transport switches (EMXP) As ICT managers look for solutions, Ethernet WAN-capable technology. The potential Ethernet services to enterprises in a metro services such as MEF-defined Carrier area. Its aggregation and metro core Ethernet services have emerged as an

62 INFINERA Footnote citations on page 111 INFINERA 63 APPLICATIONS

ODU ex-based Long-haul The infrastructure delivering the operator’s Mobile network Region 1 of Ethernet VLANs Region 2 4.3 AGGREGATION OF IP TRAFFIC – Aggregation IP BACKHAUL services is typically divided into an access

EMXP network, an aggregation network and a core Mobile 4.3.1 IP-based Services over a Common services EDU network. EMXP EMXP Infrastructure EMXP EMXP Fixed and mobile networks provide • The access network must serve millions of EMXP EMXP telecommunications services to end users – endpoints, each requiring a finite amount Mobile users PXM Common transport The internet, EMXP of capacity, while leveraging the available EMXP services that are generated by the operators infrastucture cloud services EMXP Hub themselves and by external entities. The access medium in an optimal way. and more... services can be Internet access, telephony • The aggregation network collects traffic with access to the public switched Access Fixed from a large number of end users via node services telephone network and consumer access to End-to-end Ethernet Virtual Connection hundreds or thousands of access nodes TV and media streams. Increasingly, all these in the access network, smoothing the Fixed network services are built on the IP suite of protocols, FIGURE 54: Providing Long-haul Carrier Ethernet Services Using the Infinera XTM Series and DTN-X Series individual peaks and troughs of traffic into with PXM with service provisioning to an end user a smoother average traffic flow. Packet- becoming equivalent to enabling the flow of optical technologies can bring significant IP traffic to and from the subscriber. benefits to this part of the FMC common Home and business users Much of telecommunications operators’ infrastructure. 4.2.3 A Long-haul Network enables ODUflex-based traffic engineering efforts today are centered on creating • The core network routes and distributes FIGURE 55: Fixed-Mobile Convergence Uses a Common Transport Infrastructure for IP Traffic from Mobile Users for Business Ethernet in the long-haul network (Figure 54). and Users in the Home and Enterprise. seamless services working over both the traffic between the aggregation For operators wanting to offer integrated Combining Infinera’s CE 2.0-certified XTM fixed and mobile networks – fixed-mobile networks and a finite number of entities long-haul and high-capacity Carrier Ethernet Series for metro Ethernet networks and convergence (FMC). One important services, the Infinera DTN-X Family of super- implementing the higher layers of the Infinera’s ODUflex-capable PXM allows element of an operator’s FMC strategy is to services, i.e. data centers, servers and channel systems presents a very attractive for both high-capacity, long-haul Ethernet use a common transport infrastructure of other networks. of access nodes to many fewer core fills the available bandwidth pipes more solution. Infinera’s DTN-X platforms perform transport and the termination of Ethernet integrated WDM and OTN switching, packet-optical equipment for all IP services, nodes. Traditionally, this task has been efficiently. That is why Infinera recommends services not only at regional metro irrespective of whether access is fixed or grooming and service protection at Layer networks, but directly at intermediate performed by SDH/SONET links and WDM the use of native Ethernet and MPLS-TP in mobile. This simplifies the implementation 1, based on large-scale photonic integrated DTN-X nodes. The broad Infinera portfolio 4.3.2 A Lean and Transport-centric wavelengths, but a more capacity-efficient the aggregation network. circuit technology designed for long-haul of common higher-layer service entities and thus creates a strong basis for providing Aggregation Network aggregation network is accomplished when data transmission. Using the PXM, the truly global Carrier Ethernet services with reduces both CapEx and OpEx (Figure 55). The role of the aggregation network is to Ethernet services are used to aggregate DTN-X Family can map Ethernet VLANs to Layer 1 protection in both regional and core multiple traffic streams. Ethernet leverages ODUflex subnetwork connections, which transport IP traffic from a large number network segments. the inherent possibilities for the statistical multiplexing of much of the data traffic and

64 INFINERA Footnote citations on page 111 INFINERA 65 APPLICATIONS

Millions of locations Thousands of locations Tens of locations network reduces delay and jitter, while 4.3.3 The Flexible Optical • A dedicated wavelength for the additional a need to transport legacy TDM traffic over making all traffic flows more predictable Network Brings Scalability traffic means minimal latency. yet another wavelength toward the core than in an IP network. node, for example. • The express wavelength can be equipped An aggregation network implemented Given the rapid demand for more bandwidth with error correction (OTN-FEC) to cater to Access network Aggregation network with the Infinera XTM Series is optimized from consumers and enterprises, the the longer distance. aggregation network requires flexibility 4.4 MOBILE BACKHAUL Core IP/MPLS for transport and does not introduce Access network to accommodate future growth. Infinera’s • Protection schemes can be implemented node unnecessary complexity or extra IP 4.4.1 4G/LTE and 5G Place New solution seamlessly integrates the capacity in the optical layer to improve resilience. EMXP functions. It is designed, in particular, for Requirements on Mobile Backhaul of an optical WDM network with the Layer Ethernet transport (Layer 2) according to the 2 switching functions of a Carrier Ethernet Thanks to the integrated packet-optical Mobile networks have been undergoing a MEF CE 2.0 specification, providing: network. It is easy to upgrade the capacity of functionality of the XTM Series, it is also rapid evolution over the last few decades possible to complement the EMXP with that shows no sign of abating anytime • High bandwidth utilization the transmission links between the nodes and TCP/P TCP/P the switching capacity as needed according to other types of muxponders, should there be soon. Modern LTE/4G and LTE-Advanced • The low latency necessary for IPTV, IP a pay-as-you-grow model. Ethernet Ethernet Ethernet telephony, video on demand and mobile Optical layer Optical layer Optical layer backhaul For example, assume that the demand for FIGURE 57: An Express Wavelength Is Added to the Aggregation Networks to Cater to Increased • Support for both point-to-point and bandwidth has been growing especially Bandwidth Demand from a Remote Node FIGURE 56: Aggregation of IP Traffic over a Packet-Optical Infrastructure. The Aggregation Network Can Be multicast applications fast at one particular remote node in the Made Less Complex and Given Better Performance if the IP Functionality Is Restricted to the End Nodes, i.e. aggregation network. Thanks to the packet- the Access Node Routers of the Core Network • Layer 2 performance management (utilization, latency, jitter, packet loss) optical integration of the XTM Series, an 10 GbE over one additional express wavelength can easily be • Efficient synchronization (full SyncE opened up for the high-traffic node, leading There is no need for advanced routing wires carrying the traffic. The handling of support) its Ethernet traffic directly to the core node and filtering functions in the aggregation the IP protocol, including IP/MPLS, should without loading any intermediate switches EMXP network (Figure 56). Instead, a carefully be restricted to as few nodes as possible, • Excellent protection and resilience LAG, (Figure 57). This has several benefits: chosen combination of Ethernet switching i.e. to the core nodes and as needed in the Ethernet Ring Protection Switching Version IP Router and optical functions should make the access nodes. Such a consolidation of IP/ 2 (ERPSv2), MPLS-TP linear protection • Only the high-traffic node and the core EMXP aggregation network act as a bundle of MPLS reduces both cost and operational • Predictable performance, in order to be node need to be upgraded. There is no EMXP complexity. Furthermore, the simpler easy to maintain and troubleshoot requirement for a forklift upgrade of the processing of Ethernet frames compared to whole aggregation network—pay as you Mobile SGW IP packets in the nodes of the aggregation • Low power consumption grow. 10 GbE EMXP over another

66 INFINERA INFINERA 67 APPLICATIONS

networks place tough demands on their The growth in data rates and higher copper wires or fiber. A primary challenge backhaul offerings and make maximum use 4.5 SWITCHED VIDEO TRANSPORT mobile transport networks, and the step number of cells place an ever-increasing for the mobile operator is to dimension the of installed fiber. network is ensured through technologies change in network performance that 5G demand for more capacity and reach on backhaul network to cope with upcoming such as SyncE, IEEE 1588v2 and a low 4.5.1 Streaming 3D and HD Video to the Home Implementing Ethernet services, rather brings will add to the demands on the mobile backhaul networks, whether they traffic demand, avoiding a bottleneck latency design philosophy. Cable television (CATV) operators/multiple- than IP/MPLS-based services with similar system operators(MSO)20 offer a wide range underlying transport network. are implemented by microwave links, for mobile services as well as end-user • Resilience. The network must include characteristics, can make the necessary of services over the transport infrastructure frustration over long response times and protection mechanisms that guarantee investments significantly lower, especially if unpredictable performance. carrier-class resilience against outages. An once built for TV distribution. These services the Ethernet services can be implemented XTM Series network supports protection encompass both entertainment and 4G/LTE introduced new requirements on on a previously deployed WDM platform, mechanisms such as link aggregation, communications for residential users as well synchronization and maintenance in the such as the XTM Series. The Ethernet ERPSv2 and MPLS-TP linear protection to as connectivity and value-added services Metro / Aggregation network, and mobile operators require products in Infinera’s XTM Series are fully ensure a high degree of reliability. to enterprises. Some services, like LAN Metro / Aggregation transport services with more stringent MEF CE 2.0-certified and have functionality • Efficient fronthaul integration. The flexible interconnect, may be created fully in-house service level agreements. For a regional that makes it possible to implement all the structure of the XTM Series allows for by the CATV operator/MSO, while others utility carrier or local operator sitting CE 2.0 service types shown in Figure 58. efficient integration with the Layer 1 WDM rely on external networks, data centers and on large fiber assets, this constitutes an Furthermore, a backhaul network elements necessary for a cloud-radio media hubs. important business opportunity if a network access network (C-RAN) architecture and implemented with the XTM Series has that allows the introduction of such services supporting mobile fronthaul configurations. A major trend in CATV networks is the rapid the following characteristics of immediate NNI NNI UNI can be built in a cost-effective manner. growth of traffic per end user and in the importance for mobile operators: • Efficient tools to manage operational overall network. The advent of streaming complexity. To be successful in the long run, high-definition and 3D video services it is also necessary to control and operate and cloud computing has made IP traffic E-Access E-LAN / E-Tree E-Line 4.4.2 A Backhaul Network the network in an efficient way. Infinera’s • Very little latency and jitter. Latency and DNA multi-layer management suite has grow some 30 to 50% per year. Network Optimized for 4G/LTE and 5G jitter have a cumulative effect as traffic UNI service-aware and integrated Layer 1 and operators also feel the need to radically The most attractive offering for a mobile passes through consecutive nodes in access UNI Layer 2 management functions that make lower the cost of their transport networks to operator running 4G/LTE and ultimately 5G rings, aggregation networks and the core. the network easy to operate. Service enable them to be profitable in the future, UNI UNI UNI services builds on established standards Latency must be low, predictable and stable, creation can otherwise be difficult for and is optimized for packet-mode transport not varying with load, throughput or packet and that requires reducing costs without complex networks, but with DNA, setting replication. compromising on scalability, efficiency, (Figure 58). Using Ethernet as the packet up integrated Layer 1 services, MEF-based switching technology makes it possible manageability or the ability to offer • Efficient mechanisms for synchronization. Layer 2 services and MPLS-TP services is differentiated services. eNB to benefit from the cost rationalization of Synchronization is critical in mobile backhaul easy. The DNA-M Portal even allows mobile Service GW eNB eNB Ethernet equipment, create tailor-made networks as users move from cell site to operators to monitor the performance of To meet these demands, network providers / MME cell site and expect uninterrupted service. subscriber services in real time. are going through a conversion from Proper performance of the packet-optical FIGURE 58: Carrier Ethernet 2.0 Services for Mobile Backhaul

68 INFINERA Footnote citations on page 111 INFINERA 69 APPLICATIONS

Converged Cable Aggregation/distribution Core power- and space-saving access node Access Aggregation/distribution Core Access Platform architecture. The new architecture, referred packet-optical technology for the best to as the Converged Cable Access Platform transport network economics. (CCAP), combines edge quadrature Infinera’s Switched Video Transport solution Enlighten amplitude modulation (QAM) and cable implements Internet Group Management CPE Thousands of business users aggregated to 10 modem termination system (CMTS) into Protocol, Version 3 (IGMPv3) features in the 1 Gb/s Multi-layer management Business CPE ports at 10 Gb/s PE router for one node. Such upcoming super nodes will EXMP to listen to the IGMP network traffic system business services require significant amounts of bandwidth between edge QAMs and multicast routers. Layer 2 By participating in the IGMP conversation in the aggregation/distribution network; PE router for CCAP Carrier Ethernet and switching the individual channels, the whereas traditional CMTS and edge QAM business services EMXP EMXP Infinera solution is designed to reduce the Layer 2 (EQAM) nodes could be served with 1 Gb/s, cost of the network significantly. The IGMPv3 capacities of 10 to 100 Gb/s per CCAP will Carrier Ethernet features, IGMP snooping and source-specific N x 1 Gb/s be required. multicast, allow network resources to be 10 Gb/s EMXP EMXP utilized in an efficient way, as a destination CMTS Hundreds of sites PE router for only receives the traffic intended for it as residential services aggregated to 10 4.5.2 Infinera’s Solution for opposed to a broadcast approach where 1 Gb/s ports at 10 Gb/s the traffic is distributed to everyone in the DOCSIS: 64 Gb/s Switched Video Transport PE router for Video on Demand: 16 Gb/s Infinera’s Switched Video Transport solution network (Figure 60). residential services Broadcast Digital TV: 2.6 Gb/s integrates selected Layer 2 functionality into the Layer 1 WDM network, which enables cost-efficient capacity increases and the FIGURE 59: New Access Nodes (CCAP) in CATV Networks Require Robust, High-capacity RF Gateway Aggregation Networks service differentiation capabilities typically and EQAM found in Ethernet networks.

The EMXP range allows operators the use FIGURE 60: An Overview of the Infinera Switched Video Transport Solution, Based on a Packet-Optical of unique features for the demarcation, Aggregation Network analog to digital distribution technologies convergence of legacy and next-generation aggregation and transport of Ethernet and packet-optical networks. The services onto Ethernet (Figure 59). services. Enabling the cost-efficient increase industry consensus is that packet-optical of metro network capacity, the EMXPs use Essential for the transformation of CATV networks will meet future needs, with the networks today is the introduction of DOCSIS 3.0- compatible21 equipment and the development of a new, super-dense,

70 INFINERA Footnote citations on page 111 INFINERA 71 5 ETHERNET AND LAYER 2 TECHNOLOGIES

72 INFINERA Footnote citations on page 111 INFINERA 73 ETHERNET AND LAYER 2 TECHNOLOGIES

topologies and distances. Standardized by 5.2.1 Ethernet Mode of Operation In shared Ethernet, the earliest mode 5.1 CHAPTER SUMMARY VPN3 IEEE in the IEEE 802.n family of standards, Systems communicating over Ethernet of Ethernet operation, frames were Packet switching may come in many Ethernet has largely replaced competing divide a stream of data into individual broadcasted to every possible receiver in a VPN1 VPN2 shapes, but in the context of packet-optical wired LAN technologies and is today the packets called Ethernet frames. Each broadcast domain, and a mechanism called networks for access and metro/regional dominating link layer protocol in data frame contains a source and destination carrier sense multiple access with collision applications, packet switching is almost networks. address and an error-checking code so detection (CSMA/CD) was used to avoid synonymous with Ethernet switching and the that damaged data can be detected and collisions on the transmission medium.24 Ethernet can be used in bus, star and mesh use of the Ethernet family of protocols. This retransmitted (Figure 61). topologies and over a variety of physical In the 1990s, the IEEE 802.3x standards Distribution chapter focuses on the characteristics of network Ethernet, and especially on how the original media, including coaxial cable, twisted pair A basic principle of the original Ethernet for Ethernet defined full-duplex operation Core network connectionless LAN protocols for Ethernet copper cable, wireless media, and optical standard is that destination and source between a pair of Ethernet stations, i.e. have been augmented to make Ethernet a fiber. Typical Ethernet data rates today addresses refer to unique physical ports simultaneous transmission and reception connection-oriented technology suitable range from 100 Mb/s (Fast Ethernet), 1 Gb/s attached to the transmission medium, the of frames over a twisted copper pair or for use in wide area transport networks, i.e. (Gigabit Ethernet or GbE), 10 Gb/s (10 GbE) MAC addresses, which are permanently fiber pair. At the same time, a flow control to make it into a Carrier Ethernet network. and on up to 100 Gb/s, with standards for written into the hardware of every Ethernet mechanism called the MAC control protocol Attention is also paid to how to handle Terabit Ethernet being developed. The network interface card (NIC). The Ethernet was introduced. If traffic gets too heavy, synchronization and how to create resilience term Terabit Ethernet is used to collectively protocol is concerned with the addressing, the control protocol can pause the flow of VPN1 in Ethernet networks. describe all rates above 100 Gb/s Ethernet. error checking and transport of frames frames for a brief time period. VPN2 To date, this includes 200 Gb/s and 400 across a physical transmission link and Today, practically all Ethernet LANs VPN3 This chapter is primarily intended as a Gb/s standardization, with future plans for a is referred to as a link layer or Layer 2 tutorial and source reference for those 23 and every WAN are based on switched full 1 Tb/s Ethernet capability. protocol. FIGURE 62: A Meshed and Switched Ethernet interested in the general Ethernet and Ethernet. In switched Ethernet, all Ethernet Layer 2 technologies used for wide area stations have their own individual full- networking. duplex connection to a central switch (sometimes called a multiport bridge). The contention problem within the switch, Basic Ethernet switches do not modify the 6 Bytes 6 Bytes 2 Bytes 46–1,500 Bytes 4 Bytes switch has a forwarding table that matches which needs to buffer frames from multiple Ethernet frames as they pass through and Destination Source Length/ Data FCS Ethernet stations’ MAC addresses with a 5.2 Ethernet Basics users trying to access the same destination are generally much simpler than Layer 3 Ethernet is a family of protocols and Type corresponding switch port and sends the simultaneously. Another advantage of IP routers because they operate at the networking technologies that was originally frame to the correct destination. (Jumbo frame 1,500 Bytes – 10 kBytes) switched Ethernet is that switches can link layer and do not run complex routing designed for local area networks in the Switched Ethernet removes the media be made non-blocking, allowing for protocols. Switches may also be both 1980s but is now also widely used for other 22 FIGURE 61: The Basic Ethernet Frame (the Frame Check Sequence Is Used for Error Control) contention and capacity problem inherent simultaneous traffic between several ports. cheaper and faster than IP routers because to shared Ethernet and reduces it to a Furthermore, each switch port now provides the full bandwidth of the Ethernet medium to the connected station.

74 INFINERA Footnote citations on page 111 INFINERA 75 ETHERNET AND LAYER 2 TECHNOLOGIES

the switching function can be implemented 6 bytes 6 bytes 4 bytes 2 bytes 4 bytes address. If the address is unknown, the 6 bytes 6 bytes 4 bytes 4 bytes 2 bytes 4 bytes entirely in hardware, rather than in software frame is flooded to all the switch ports Destination Source VLAN Length/ running on an expensive high-performance Data FCS except the incoming one. This creates one Destination Source SVLAN CVLAN Length address address tag type address address tag type type Data FCS processor. Finally, switches are simpler to single broadcast domain per switched manage than IP routers, since configuration network, which is a potential problem in does not involve the same complexity as FIGURE 64: IEEE 802.1Q/p Encapsulation of the VLAN Tag larger networks since broadcast frames FIGURE 66: IEEE 802.1ad Encapsulation of the SVLAN Tag with routers. are propagated and replicated throughout the entire network. The problem of large In a meshed Ethernet, shown in Figure 62 broadcast domains as well as the security A VLAN is a logical group of Ethernet station on a particular VLAN will only hear on the previous page, there are several be forwarded in infinite loops within the between each node of the network, and all problem of having all traffic available at stations that appear to one another as broadcast traffic from the other members of paths between nodes, and frames can network if no countermeasures are taken. other interconnecting ports on the switches every Ethernet station is overcome by the if they were on the same physical LAN the same VLAN. Using MAC address-based There must be one and only one open route must be blocked. Such a “one-route” introduction of virtual LANs. segment, even though they may be spread VLANs makes it possible to let the VLAN network topology is called a spanning across a large network. Each Ethernet span multiple switches. Interconnection tree. The Spanning Tree Protocol (STP) and between VLANs may then be provided by the Rapid Spanning Tree Protocol (RSTP) Layer 3 devices such as IP routers. in the Ethernet standards are distributed Service Provider algorithms that can be run by the switches All Ethernet frames in a VLAN have a distinct to form a spanning tree (Figure 63). identifier, called the VLAN identifier (VID), located in a designated VLAN tag field In wide area applications of Ethernet, e.g. SVID 1 (Figure 64), specified by the IEEE 802.1Q/p Carrier Ethernet, other mechanisms, such Multipoint standard and inserted in the frame by the as Ethernet Ring Protection Switching and Ethernet switch. The full VLAN tag field is manual configuration of virtual connections, Point to point 4 bytes long and contains a tag protocol are used to ensure that there is only one SVID 2 identifier (TPID) and a priority code point path between the ingress and egress nodes (PCP), which indicate the frame priority of the network. More information about this level. 12 bits of the VLAN tag are available can be found in section 5.4. SVID 4094 for VLAN identification, but two values are Multipoint reserved, making a maximum of 4094 VLANs possible in a single switched network using 5.2.2 Virtual LANs the basic standard. The task of the Ethernet switch is to move the frame from one LAN segment to FIGURE 65: Customers Running Their Own VLANs Inside the Service VLANs of a Service Provider VLANs can be used to implement virtual private networks, and VLAN frames include FIGURE 63: Three Virtual LANs (Green, Blue and Red), Each with Its another based on the destination MAC Own Individual Broadcast Domains, Interconnected by a Router

76 INFINERA Footnote citations on page 111 INFINERA 77 ETHERNET AND LAYER 2 TECHNOLOGIES

priority fields that may be used to create tunneling or Per-VLAN Spanning Tree. A over optical fiber. 100BASE-FX uses 1300 uses 850 nm light with a reach of 220 to 250 Name Medium Specific Distance services with different priorities, i.e. qualities. further refinement of the concept of service nanometer (nm) light transmitted via two meters (m) on multi-mode fiber. The long- This feature is of particular interest in wide VLAN is standardized in IEEE 802.1ah strands of optical fiber, one for receive and range PMD uses 1310 nm light with a reach 1000BASE-CX Twinaxial cabling 25 meters area applications where VLANs can be used specifying Provider Backbone Bridges the other for transmit. The maximum length of 550 m on multi-mode fiber and 5 km on 1000BASE-SX Multi-mode fiber 220 to 550 meters dependent on fiber to separate and classify traffic from different (PBB). The PBB standard supports complete is 2 kilometers (km) over multimode optical single-mode fiber. The PMD for shielded diameter and bandwidth sources and users, and to direct it along isolation of customer VLAN addressing fiber. copper reaches only 25 m. For unshielded 1000BASE-LX Multi-mode fiber 550 meters different paths of the wide area network. fields and service VLAN addressing fields. copper, which is common in many office The 100 in the media type designation refers 1000BASE-LX Single-mode fiber 5 km installations, multiple twisted pairs are A further development along the same to the transmission speed of 100 Mb/s. The 1000BASE-LX10 Single-mode fiber using 1310 nm wavelength 10 km used to send multi-level signals in a way theme is the use of VLANs as “virtual “BASE” refers to baseband signaling, which 5.2.3 Ethernet Physical Media (PHY) that extends the reach to 100 m. Ethernet 1000BASE-ZX Single-mode fiber at 1,550 nm wavelength ~70 km trunks” in service provider networks, means that only Ethernet signals are carried The Ethernet standard comprises a data First Mile (EFM), an initiative for Ethernet in allowing customers to run their own on the medium. The TX and FX refer to the 1000BASE-BX10 Single-mode fiber, over single-strand fiber: 1,490 nm 10 km link layer and an Ethernet physical media broadband access networks, later added downstream/1,310 nm upstream customer VLANs inside the service VLANs physical medium that carries the signal. (PHY) part, the latter being specific for the 1000BASE-LX10 and -BX10 (Figure 67). 1000BASE-T Twisted-pair cabling (Cat-5, Cat-5e, Cat-6 or Cat-7) 100 meters of the service provider (Figure 65, previous transmission media and data rate employed. page). 1000BASE-TX Twisted-pair cabling (Cat-6 or Cat-7) 100 meters When Ethernet is transported over a WDM XTM Series traffic units support both The XTM Series traffic units support both To allow for service VLANs, a 4-byte SVLAN wide area network, the Gigabit Ethernet electrical 100BASE-TX and optical 100BASE- FIGURE 67: Gigabit Ethernet Physical Media the electrical (1000BASE-T) and optical tag is added to the header of the Ethernet and 10 Gigabit Ethernet PHY standards FX client interfaces. frame in the IEEE 802.1ad standard. In are of the most interest since these are the (1000BASE-L) variants for single- and multi- the service provider domain, switching types of Ethernet deployed in metropolitan mode fiber in the GbE physical layer when interfacing to client systems. can then be based on the SVLAN tag and and other wide area networks. The Infinera 5.2.3.2 Gigabit Ethernet Physical Layer Both LAN PHY and WAN PHY operate over 5.2.3.4 40 and 100 Gigabit Ethernet destination MAC address, and the customer product portfolio includes transponders GbE can be transmitted over shielded fiber a short-range, long-range, extended-range Physical Layer VLAN (CVLAN) tag is used to create virtual and muxponders that can be equipped or long-reach PMD. Short-range PMD uses The 40 and 100 Gigabit Ethernet standards cables or shielded copper cables. It can 5.2.3.3 10 Gigabit Ethernet Physical Layer LANs within the customer domain. This with transceivers supporting Fast Ethernet, 850 nm light over multi-mode fibers up to encompass a number of different Ethernet also be transmitted over unshielded twisted 10GbE can be transmitted over fiber optic technology is also known as stacked VLANs, Gigabit Ethernet, 10 Gigabit Ethernet and 300 m; long-range uses 1310 nm light and physical layer specifications. A networking pairs of copper. This transmission is set up and copper cables, but copper cables are Q-in-Q25 or provider bridges (Figure 66, 100 Gigabit Ethernet. reaches 260 m on multi-mode and 10 km on device may support different PHY types to operate in full-duplex (most common) or only used over short distances such as previous page). single-mode fiber. The extended-reach PMD by means of pluggable optical modules. half-duplex mode. The standard defines a interconnections within a chassis. uses 1550 nm light and has a maximum The modules are not standardized by any Q-in-Q is primarily a way to overcome the physical media dependent (PMD) sublayer 5.2.3.1 Fast Ethernet (FE) Physical Layer For fiber optic cables, the physical layer can reach of 40 km on single-mode fiber. 10GbE official standards body but are defined by limitations on the VLAN identifier space. It that specifies the transceiver for the physical Fast Ethernet, i.e. Ethernet at 100 Mb/s, be implemented in two main variants: LAN is supported across the Infinera packet- multi-source agreements (MSAs). One such can also help in separating the customer medium in use. There are three types of has two predominant physical formats: PHY and WAN PHY, optimized for use in optical portfolio. and provider control domains when used PMDs for GbE: short-range, long-range 100BASE-TX, which runs over two local and wide area networks respectively. with other features like control protocol and shielded copper. The short-range PMD unshielded twisted pairs (UTP) made of copper wires, and 100BASE-FX, which runs

78 INFINERA Footnote citations on page 111 INFINERA 79 ETHERNET AND LAYER 2 TECHNOLOGIES

Physical Layer 40 Gigabit Ethernet 100 Gigabit Ethernet rate (one rate of bits per second), buffered Somewhere in every TDM network there is Primary Reference Clock (PRC) and multiplexed with other frames over an extremely accurate frequency source, a Stratum 1: International gateway ~ Backplane 100GBASE-KP4 intermediate links with higher data rates, primary reference clock (PRC) from which all Improved Backplane 40GBASE-KP4 100GBASE-KP4 and delivered at yet another rate to the other TDM clocks in the network directly or receiver. There is no fixed relationship indirectly derive their timing, i.e. frequency. 7 m over twinax copper cable 40GBASE-CR4 100GBASE-CR10 Stratum 2: Stratum 2 Stratum 2 between the timing, phase or frequency of Clocks derived in this manner are said to Transit exchanges 30 m over “Cat-8” twisted pair 40GBASE-T 100GBASE-CR4 the incoming bit stream and the outgoing be traceable to a PRC. The primary clock 100 m over OM3 MMF 100GBASE-SR10 bit stream from the network. signal is distributed “downward” through the network in order to synchronize all other 125 m over OM4 MMF 10GBASE-SR4 100GBASE-SR4 This is quite different from the principles devices, which are normally grouped into Stratum 3 Stratum 3 Stratum 3 2 km over SMF, serial 40GBASE-FR 100GBASE-CWDM4 of time-division multiplexed transmission separate stratum clock levels depending on technologies, such as PDH, SDH and 10 km over SMF 40GBASE-LR4 100GBASE-LR4 how “far” from the original PRC the device SONET, traditionally used in wide area 40 km over SMF 40GBASE-ER4 100GBASE-ER4 is located (Figure 69). Stratum 3: Stratum 3 Stratum 3 transport networks. In TDM, each stream Local exchanges FIGURE 68: 40 and 100 GbE Physical Media of information to be transferred over the The migration from such a synchronous network is allocated a specific timeslot TDM network to Ethernet-based in the transmission system, a procedure asynchronous transport introduces new that requires careful frequency and phase challenges. When a packet network is Stratum 4: agreement that supports 40 and 100GbE is 5.3 SYNCHRONIZATION AND synchronization of all intermediate network to support TDM-based services, it must PABXs 4 4 the C form-factor pluggable (CFP) agreement, CIRCUIT EMULATION SERVICES nodes handling the flow of passing bits. provide correct timing at the traffic which was adopted for distances of 100 meters interfaces. The transport of TDM signals OVER ETHERNET Today, services such as circuit-switched FIGURE 69: Stratum Clock Levels in a TDM Network for Circuit-switched Telephony. or more. Quad small form-factor pluggable through Ethernet requires that the signals telephony and storage area networks in data Clock Signals May Be Distributed over Several Paths to Ensure Redundancy (QSFP) and CXP connector modules support 5.3.1 Synchronous and at the output of the packet network comply centers are still based on TDM technologies, shorter distances. 100GbE is supported across Asynchronous Transport with the TDM timing requirements in but increasingly this TDM traffic needs to be the Infinera packet-optical portfolio. Packet switching technologies, including order for the attached TDM equipment to transported over packet-optical networks. It Ethernet, are inherently asynchronous, i.e. interwork correctly. Such an adaptation of The 40/100GbE standard supports only full- becomes necessary to emulate a traditional incoming frames are received at one data TDM signals to be transported by a packet Circuit emulation is closely related to the will be perfect. The service being carried by duplex operation (Figure 68). wireline circuit over an Ethernet network network is called a circuit emulation, and the concept of pseudowires. A pseudowire is the pseudowire may be SDH, SONET, ATM and to maintain synchronization between entity performing the adaptation is referred an emulation of a point-to-point connection or frame relay, while the underlying packet the ingress and egress ports of the Ethernet to as an interworking function (IWF) (Figure over a packet-switching network. The network may be a Layer 2 network such as wide area network. 70, next page). pseudowire emulates the operation of a Ethernet, an IP network or an MPLS network. “transparent wire” carrying the service, but it must be realized that this emulation rarely

80 INFINERA INFINERA 81 ETHERNET AND LAYER 2 TECHNOLOGIES

for the indirect generation of frequency 5.3.2 Synchronization Standards architecture at the physical level is used Primary Secondary Clock clock TDM circuit TDM circuit (differential timing). Two main principles and related standards to provide timing distribution from the are available for providing synchronization backbone to the access nodes. A reference The most advanced requirements on EMXP Circuit emulation service across an Ethernet network, as shown in timing signal traceable to a PRC is injected synchronization in today’s metro networks Figure 71: into a backbone Ethernet switch, and this EMXP EMXP EMXP Interworking Function Interworking Function are generated by mobile backhaul traffic, (IWF) (IWF) signal is then distributed to the next node, EDU which is dependent on both frequency, • Synchronous Ethernet. An ITU-T standard EMXP EMXP Ethernet which extracts timing from the incoming bit ESMC phase and time synchronization. For using Ethernet PHY media to distribute stream and synchronizes the outgoing bit example, for 3GPP2 base stations, including timing (frequency). SyncE uses PHY clock FIGURE 70: Emulation of a TDM Circuit over a Packet-mode Network. The Interworking Functions Provide stream to this rate. Timing for TDM circuits Traffic Interfaces for the TDM Circuits those for LTE, the following requirements transmissions and regenerates the clock Secondary transported over the packet network can be clock are typical: signal from the incoming bit stream in a recovered at the appropriate interworking way similar to traditional TDM systems, by • A frequency accuracy of 0.05 parts per functions for the circuit emulation service. FIGURE 72: In the Case of a Primary Clock Failure, Messages in use of phase-locked loop (PLL) circuitry. million (ppm) at the air interface the ESMC Ensure that a Secondary Clock Can Take Over A pseudowire encapsulates incoming cells, from what is used to distribute frequency, This approach requires SyncE support bit streams and protocol data units (PDUs) and they normally rely on time stamps that • 2.5 microseconds (µs) time accuracy in every network node traversed by and transports them through tunnels set up are sent between nodes. Time stamps may between neighboring base stations, i.e. ± the Ethernet signal and only provides The EMXPs used as nodes in an Infinera the primary synchronization source. If this in the packet network. also in some network applications be used 1.25 µs difference to coordinated universal frequency and phase synchronization, not packet-optical network fully support the ITU-T primary clock fails, the messages can identify a In addition to the synchronization of time (UTC) time of day synchronization. Synchronous Ethernet recommendations secondary clock that then becomes the primary. frequencies required by TDM networks, • Precision Time Protocol (PTPv2) – IEEE (G.8262/Y.1362 and others) for jitter and The SSM messages are sent over the Ethernet wander tolerances, supported frequencies, Synchronization Message Channel (ESMC), as other types of network are dependent on FIGURE 71: Standards for Primary Reference Clock (PRC) 1588v2. An IEEE standard using Layer 2 Traceable reference frequency and clock specifications for Synchronous described in ITU-T Recommendation G.8264. having the exact same time in every node. Synchronization over Ethernet. SyncE embedded OAM packets with the highest Uses Ethernet PHY to Transfer a ~ Ethernet equipment clocks (EEC). The Frequency is a relative entity, measured priority to ship clock/phase and time of In an Infinera packet-optical network, the Reference Frequency to Every Node. SyncE EMXPs are also compatible with ITU-T relative to a frequency standard, e.g. the Synchronous Ethernet function is entirely based IEEE 1588v2 Uses Ethernet Frames day information across a packet network. Recommendation G.823 regarding clock PRC, with respect to jitter, wander and slip. on the Ethernet PHY media and its circuitry in Carrying Time Stamps to Send Current Special hardware is often used to process selection logic, possible clock quality levels, the EMXPs. Using either of the Layer 2 traffic Time represents an absolute, monotonically “Time of Day” to the Nodes these packets for higher accuracy, but the noise tolerances, noise generation and Packet network forwarding mechanisms (see Chapter 2), MPLS- increasing value, generally traceable back to packets remain in standard Ethernet frames. transfer limits, holdover performance, etc. the rotation of the earth (day, hour, minute, e.g. Ethernet TP label-switched paths or Ethernet service second). Mechanisms to distribute absolute The EMXPs can do automatic selection VLANs, has no effect on timing or the ESMC. of the synchronization source to improve time in a network are significantly different A physical Ethernet interface using MPLS-TP Time stamp Slave clock 5.3.2.1 Synchronous Ethernet synchronization resilience. Network-level will forward Synchronous Ethernet in the same Time stamp With Synchronous Ethernet, a master/slave Synchronization Status Messages (SSM) way as a physical Ethernet interface using IEEE 1588V2 complying with ITU-T Recommendation an Ethernet user network interface (UNI) or Slave clock G.781 are sent between nodes, identifying network to network interface (NNI) (Figure 72).

Master clock

82 INFINERA INFINERA 83 ETHERNET AND LAYER 2 TECHNOLOGIES

5.3.2.2 Precision Time Protocol PTP is based on IP multicasting and can (TC). The BC mode of operation relies such a way that the incoming service clock MAC Client MAC Client (PTP) – IEEE 1588v2 be used on any network that supports primarily on the master/slave relationship frequency is replicated as the outgoing IEEE 1588 (PTP) enables sub-microsecond multicasting. Precision is typically in the between network elements. In the TC service clock frequency. In network- synchronization of clocks by having a master range of 100 ns to 100 µs, depending on the mode of operation, the network elements synchronous operation, the packet network Distributor Collector Distributor Collector clock send multicast synchronization packets real-time capabilities of the end systems. add an arrival and departure time stamp to operates in a fully synchronized manner containing time stamps. All IEEE 1588 The PTP standard can distribute time/phase, outgoing messages, indicating the delay using a PRC-traceable clock, but this does receivers correct their local time based on frequency or both. It is resilient because a that has been added by internal processing not necessarily preserve the timing of the the received time stamp and an estimate failed network node can be routed around (Figure 73). external TDM service. Using differential MAC MAC MAC MAC of the one-way delay from transmitter to and the time can be taken from more than timing, the difference between the external The TC mode of operation, which EMXPs receiver. one master clock. IEEE 1588v2 packets TDM service clock and the network support, has several advantages since it fully comply with Ethernet and IP standards reference clock is encoded and transmitted does not require any preconfiguration, and are backward compatible with all across the packet network. Differential is easily interoperable between network existing Ethernet switching and IP routing timing makes it possible to recover the elements and handles various types of equipment. There is no requirement for external TDM service clock at the far end of asymmetries in the network. IEEE 1588 TC intermediate Ethernet switches traversed the packet network. can be used for Ethernet VLANs, Ethernet by the emulated circuit to be IEEE 1588v2- FIGURE 74: Link Aggregation Functional Diagram service VLANs and MPLS-TP pseudowires. More information on timing recovery and aware, as they see the timing frames as Output buffer transport of legacy TDM services such as normal data. The BC mode of operation allows SDH and SONET over an Infinera packet- Corr: + 1231.662356 the network to realign the clock and The convergence time of the PTP protocol, optical network can be found in section 2.5 compensate for the wander that builds up in supposed to be noticed by the subscriber. with the Ethernet/Layer 2 protection i.e. the time it takes for the protocol to of this book. the network. The main purpose of an automatic mechanisms. achieve the desired level of synchronization, protection switching (APS) mechanism is Ethernet Spanning Tree Protocol and is dependent on the quality of the to guarantee the availability of backup Corr: + 861.452344 Rapid Spanning Tree Protocol can prevent underlying packet network. Although 5.4 ETHERNET PROTECTION resources and ensure that switchover is 5.3.2.3 Differential Timing Output buffer the EMXPs are not directly involved in loops and assure backup paths. However, Timing recovery for a TDM circuit emulation Wide area services, such as telephony, achieved within milliseconds. PTP/1588v2 protocol handling, the very low both protocols are too slow to respond service requires that the timing of a signal is internet access and video on demand, wander, jitter and delay within an EMXP- Protection switching can be implemented to network failures and are not used in similar on both ends of the packet network, require a high level of availability; 26 based packet-optical network makes the at various OSI network layers. An optical packet-optical networks. Instead, other i.e. at the “outside” of the IWFs. The clock unavailability is typically only tolerated for PTP protocol converge extremely rapidly. transmission network may include mechanisms, such as Link Aggregation of the TDM service must be preserved in a few minutes per operating year. When alternative fiber routes and protection Group (LAG) and Ethernet Ring Protection IEEE 1588 specifies two different failures occur in the network, they are not switching at Layer 1. Ethernet/Layer 2 can Switching are more suitable. For Layer FIGURE 73: IEEE 1588 Transparent Clock (TC) Mode of Operation mechanisms to maintain synchronization: perform protection switching as well, and 2 networks employing MPLS-TP label- boundary clock (BC) and transparent clock there may also exist protection mechanisms at higher OSI layers. This section deals

84 INFINERA Footnote citations on page 111 INFINERA 85 ETHERNET AND LAYER 2 TECHNOLOGIES

MAC client can treat the group as if it were sent between two endpoints (hosts). The a single link. Link aggregation is capable LACP protocol will therefore assign each State and protection are coordinated between of increasing both the capacity and the “conversation” between two hosts to one the EMXPs involved, using a special signaling protocol (Inter-Control Center Protocol or availability of the communication channel specific physical link and send all frames ICCP). Protection LAG 1+1 and N+N are EMXP Carrier Peer between devices interconnected by belonging to that conversation over this link. supported by the Infinera packet-optical EMXP EMXP ICCP Ethernet System 2 Ethernet. Link aggregation can also provide Dedicated hash algorithms, analyzing the network. Peer load balancing, in which traffic is spread fields of the frame header (VLAN ID, MAC System 1 across several physical links in order to avoid address, MPLS-TP label, etc.), are used to The XTM Series’ multi-chassis LAG brings a number of benefits, such as: the overload of any single link (Figure 74, spread the conversations evenly across the EMXP • Increased network availability LAG FIGURE 75: LAG Assigns Different Conversations previous page). available physical links. Between Hosts to Separate Physical Links • Fast protection switching (< 50 ms) that allows for demanding customer SLAs FIGURE 76: Multi-chassis LAG Most implementations of LAG now conform to 5.4.1.2 Multi-chassis LAG (compare to IP-level protection < 5 s) what used to be clause 43 of the IEEE 802.3- Multi-chassis LAG (MC-LAG) refers to the • Network redundancy without complex 2008 Ethernet standard, informally referred ability to set up LAGs in which one or both switched paths (see section 2.3), even more signaling between nodes to as “802.3ad.” This includes Infinera’s endpoints have physical links that end in However, Ethernet as such does not allow for by the use of ERPS, as specified in ITU-T advanced protection schemes such as EMXPs, which have been interoperability- MPLS-TP linear protection are available. two or more different chassis, or in the • Few interoperability issues, since each loops since frames would circulate forever, Recommendation G.8032 (Figure 77). tested against several other vendor solutions. side can see the other just as in standard so the loop has to be blocked at some point With the EMXPs, it is possible to define a link case of the XTM Series, EMXPs. MC-LAG is An Ethernet ring is made up of two or more 5.4.1 Link Aggregation protection LAG in the network. This can be accomplished aggregate group consisting of up to eight supported by many vendors, but is not fully 5.4.1.1 Principles Link aggregation, as nodes interconnected via transmission links. ports. All ports in the group must have the standardized, and its implementation varies • Supported by almost all client equipment defined by IEEE 802.3ad, is a method for same port speed. by vendor. Multi-chassis LAG uses normal aggregating two or more parallel physical LACP signaling toward the connected peer transmission links to a LAG, such that a and is “seen” by the connected peer as an 5.4.2 Ethernet Ring Protection Switching Normal operation A failure is detected Traf c ow is restored A dedicated protocol, the Link Aggregation ordinary LAG (Figure 76). Ring-based networks are often attractive Control Protocol (LACP), is used to since they can offer a simple form of A B authenticate and coordinate the network redundancy in a network consisting of many A B A B Failure Failure elements participating in the LAG (Figure 75). nodes. A redundant path for each node is F C F C provided by just one additional link that F C It is important to understand that a Layer 2 Blocked closes two adjacent branches to form a loop. Blocked Signaling E D E D Open network may not change the order of frames E D

FIGURE 77: How Ethernet Ring Protection Switching Works. The Solid Line Between Nodes C and D Represents a Ring Protection Link (RPL)

86 INFINERA INFINERA 87 ETHERNET AND LAYER 2 TECHNOLOGIES

Each node has two links connected to its • Mixing protected and unprotected VLANs Packet-switched wide area networks, adjacent nodes. To avoid loops, one of The Ethernet Ring Protection Switching however, especially those based on available with EMXP-based nodes provides ERPSv2 supports both revertive and non- the links in the full ring is always blocked. Ethernet, offer other advantages to network sub-50 ms protection for Ethernet traffic using revertive operation after the condition that This link, which is blocked under normal operators than those typically found in ring topology, and ensures that there are no caused the switch has been cleared to conditions, is called the Ring Protection legacy carrier networks: Carrier Ethernet Network loops formed at the Ethernet layer. minimize unplanned traffic hits. ERPSv2 also Link (RPL). One of the nodes connected to includes “manual switch” and “force switch” • Most potential users of a WAN or the RPL is designated the RPL owner and is EMXP-based nodes can also support multiple rings, so that several Ethernet protection operator administrative commands. metropolitan area network (MAN) service Ethernet Virtual Connection responsible for controlling the status of the switching rings can be joined at one physical have an Ethernet-based LAN and want link. In the example above, the link between The XTM Series includes support for both location. This makes the Infinera packet- to extend it to multiple sites. It makes node C and node D is the RPL and node C ERPSv1 and ERPSv2. optical solution extremely useful for the sense for a carrier to offer Ethernet-type is the RPL owner. aggregation of traffic from multiple rings transport services, since customers are Subscriber Ethernet Subscriber Ethernet covering many sites, for example, in a mobile When a failure is detected, the RPL owner familiar with the protocol, and their backhaul network. 5.5 CARRIER ETHERNET is responsible for unblocking the RPL and ARCHITECTURE AND SERVICES equipment already has Ethernet interfaces. opening it for traffic. A link failure can be Subscriber Ethernet • Ethernet is the all-dominant Layer 2 data detected by link-down events or OAM 5.5.1 Carrier Ethernet: Ethernet as Ethernet Ring Protection Switching networking protocol, which means that frames, for example, a loss of continuity. a Transport Service Version 2 adds some very useful features. economies of scale have made Ethernet When a failure has been detected, the ring Legacy carrier networks provide transport Most important is that it introduces more switching equipment very attractive from FIGURE 78: Carrier Ethernet: A Connection-oriented Version of Ethernet Provided is said to be in a “signal failure” state. The services with very high availability and advanced Ethernet ring interconnection a cost perspective. Naturally, carriers as a Service to Subscribing Customers node detecting the failure condition will predictable performance, using, for architectures (multi-ring/ladder network), want to take advantage of the continued block the traffic on the failed link and inform example, SDH or SONET multiplexing with the concept of sub-rings. This performance and cost evolution of the the other nodes in the ring. schemes. Packet-switched transport creates the ability to have different rings Ethernet technology. networks are different, as they offer When the nodes are informed about a ring interconnected at two or more points, new services based on features such • Many non-Ethernet wide area To transport Ethernet traffic efficiently in Ethernet for LAN use, into a more failure, the learned MAC addresses are avoiding a single point of failure. as asynchronous transport, statistical technologies such as TDM, ATM, frame metro and regional networks, the carrier/ predictable and “circuit-like” channel flushed. In the example above, the nodes in ERPSv2 can support multiple ERP instances multiplexing and full connectivity between relay, etc. can be replaced or emulated by network operator/service provider must suitable for wide area networking. the ring will relearn the destination MACs of on a single ring. In addition to the possibility multiple endpoints. The classical “carrier- Ethernet alternatives, making a transport establish an Ethernet of its own – a the stations participating in the blue traffic, 27 The need for such a Carrier Ethernet has to allocate a specific set of VLANs to a ring grade” characteristics, such as quality of network based on Ethernet more uniform Carrier Ethernet network – that forwards which will now flow over the unblocked RPL. been recognized for some time. The Metro instance, there are also a number of new service, security and high availability, must and less complex to operate than a traffic between the customer LANs “in the Ethernet way.” Using Ethernet as Ethernet Forum was formed in 2001 by deployment possibilities: be realized by new mechanisms in a packet network using multiple other technologies. vendors and service providers to develop network. the transport mechanism requires the • MPLS-TP backbone VLANs addition of functions that transform the standards for ubiquitous LAN interconnect • Common spans with other rings connectionless and broadcast-oriented

88 INFINERA Footnote citations on page 111 INFINERA 89 ETHERNET AND LAYER 2 TECHNOLOGIES

services over metropolitan optical networks. • Ethernet regarded as a service. This is the E-Transit) that enable transparent, private recovery from failures is comparable to accepts and delivers Ethernet frames, i.e. The principal concept was to bring the Metro Ethernet Forum’s scope and view. line, virtual private line and multipoint- that of SDH/SONET networks or better. a dedicated, physical demarcation point Standardized simplicity and cost model of Ethernet to Carrier Ethernet services concern the user- to-multipoint connectivity over the wide between the responsibility of the service • Quality of service. Carrier Ethernet Services wide area networks, while adding stability, to-user transfer of Ethernet 802.3 frames area network. The services are provided provider and that of the subscriber. The supports the delivery of critical predictability and manageability. Since over any available physical transport layer. independent of the underlying transport attached customer equipment (CE) can be applications that are expected to meet then, MEF has issued a wide range of protocols and media used, with a a router, switch or computer system, and A Carrier Ethernet network is by definition high performance levels. The performance Quality technical specifications for Carrier Ethernet high level of choice and granularity of the physical medium of the UNI can be Scalability a two-layer structure, consisting of a parameters of Carrier Ethernet are of equipment, specifications that are adhered bandwidth as well as quality of service copper, coax or fiber operating at 10 Mb/s, physical transport layer, which can be quantifiable and measurable so that they Service to by Infinera (Figure 78, previous page). options. The services require no changes 100 Mb/s, 1 Gb/s, 10 Gb/s or 100 Gb/s, CARRIER WDM, SDH/SONET, Ethernet PHY or any can be included in an SLA for voice, video to customer LAN equipment or networks according to the IEEE 802.3 Ethernet PHY/ ETHERNET Metro Ethernet Forum defines a Carrier other physical transport technology, and and data over converged business and and accommodate existing network MAC protocol. Ethernet network as a set of certified a pure Ethernet frame handling layer, the residential networks. connectivity, such as time-sensitive TDM network elements that are interconnected Ethernet MAC (ETH) layer.28 The Ethernet The UNI functions are divided between the traffic and signaling, while being delivered • Service management. Carrier Ethernet and provide Carrier Ethernet services, locally services offered are created “on top of” CE and provider edge equipment as the over a single Ethernet connection service providers are expected to Service Reliability and worldwide. transmission technologies such as WDM function sets UNI-C and UNI-N, respectively. between network and customer. manage large numbers of customers and Management optical networks. The following discussion Sometimes the CE does not support all the In this context, it is important to remember their multiple services, spanning wide focuses on Carrier Ethernet services and • Scalability. The scale of a customer UNI-C functions; in such cases a network that the term “Ethernet” is ambiguous and geographical areas. Carrier Ethernet Source: Metro the Ethernet MAC layer. Details of how the LAN and that of a service provider interface device or an Ethernet demarcation Ethernet Forum is standardized in various ways by multiple includes advanced capabilities for Ethernet MAC layer is carried by the optical network are fundamentally different in unit belonging to the Carrier Ethernet interest groups. provisioning, maintaining and upgrading WDM layer of the Infinera XTM Series are terms of geographical reach, number network is located at the customer site and FIGURE 79: Carrier Ethernet Attributes as Ethernet services (Figure 79). • Ethernet regarded as a “point-to- described in Chapter 2.2.4, “Packet-Optical of users (endpoints) and bandwidth. acts as the actual physical demarcation Defined by Metro Ethernet Forum point” transmission link, i.e. the physical Transport with the XTM Series.” Carrier Ethernet is scalable in all those point of the service. Carrier Ethernet characteristics of Ethernet framing and dimensions, while allowing the service demarcation is a key element in Carrier Metro Ethernet Forum has defined five main 5.5.2 Carrier Ethernet Architecture transmission. This is the scope and view of provider to use various underlying Ethernet networks as it enables service attributes of Carrier Ethernet that distinguish and Terminology IEEE 802.3. transport technologies to achieve the best providers to extend their control over the it from the familiar LAN-oriented Ethernet, A service provided by a Carrier Ethernet the equivalent of a “circuit,” and it is the total economy. entire service path, starting and ending at • Ethernet regarded as a packet-switched and make it suitable as a transport service network starts at one user network interface Ethernet Virtual Connection that is assigned the customer handoff points. network infrastructure. This is the 802.1 offered by carriers. The five attributes are: • Reliability. Carrier Ethernet is resilient (UNI) and ends at another UNI. The UNI the various characteristics – attributes – to (bridging) view as well as the managed and reliable. Protection mechanisms is the point where the service provider The association between two or more UNIs which a customer subscribes. • Standardized services. Carrier Ethernet Ethernet network view by ITU-T SG15/ are available to provide end-to-end and via the Carrier Ethernet network is referred provides five standardized service types SG13. individual link protection. The speed of to as an Ethernet Virtual Connection. In the (E-Line, E-LAN, E-Tree, E-Access and Carrier Ethernet world, this association is

90 INFINERA Footnote citations on page 111 INFINERA 91 ETHERNET AND LAYER 2 TECHNOLOGIES

Ethernet Virtual Connection (EVC) relay or ATM permanent virtual circuits classes of service. A service also defines the Leaf (PVC). transparency of Layer 2 control protocols and how they should be handled. The Carrier Ethernet specifications allow for Root Customer Customer three types of Ethernet Virtual Connections: For Ethernet services based on EVCs, two Equipment Equipment (CE) (CE) point-to-point EVC, multipoint-to-multipoint variants are defined, differentiated by the EVC and rooted multipoint (point-to- method of service identification used at Service Provider 2 Service Provider 1 multipoint) EVC, which are used to create the UNIs. Services using port-based UNIs, Carrier Ethernet Network 1 Carrier Ethernet Network 2 Carrier Ethernet services (Figure 81). i.e. where there is only one UNI and EVC Leaf per physical port of the provider edge Point-to-point EVC: Each Multipoint-to-multipoint EVC: Rooted Multipoint EVC: Each device, are referred to as “private,” while EVC associates exactly two Each EVC associates ≥ two UNIs. In EVC associates ≥ two UNIs with one User Network Interface External Network to User Network Interface 5.5.3 Carrier Ethernet 2.0 Services services using UNIs that are VLAN-based UNIs with each other. this diagram, three sites joint-share a or more UNIs as roots—the roots can (UNI) Network Interface (UNI) In the Carrier Ethernet network, data and multiplexed over the same physical multipoint EVC and can forward forward to the leaves, while each Ethernet frames to each other. leaf can only forward to the roots. (ENNI) is transported across point-to-point, interface are referred to as “virtual private.” point-to-multipoint and multipoint-to- For example, an E-Line service that is port- FIGURE 80: Carrier Ethernet Basic Concepts and Terminology multipoint EVCs and OVCs according to based is referred to as Ethernet Private Line. FIGURE 81: Ethernet Virtual Connection Types the attributes and definitions of a set of E-Access and E-Transit services, which are well-defined Ethernet service types. Each based on Operator Virtual Connections and service type provides transparent data primarily defined for use within a service service types are used in more specialized Implementing CE 2.0 services gives the Sometimes an Ethernet Virtual Connection association of UNIs and ENNIs within a transport between the UNIs and/or ENNIs provider’s network, have been restructured applications, such as mobile backhaul network operator a whole new range of passes through the networks of more than single CE network in which at least one of of subscribers and operators. Three of the to eliminate the port/VLAN distinction and applications in which several Carrier services to offer “on top” of the legacy one service provider. The interface between these external interfaces is an ENNI. service types are based on EVCs (i.e. are for (Figure 82, next page). Ethernet service providers cooperate. The Layer 1 transport service, thereby adding two such service providers is referred to as services between UNIs) and two are based The EVC, i.e. the association between two E-Transit service addresses the multi- new revenue streams to an existing network an external network to network interface on OVCs (i.e. at least one endpoint is an The five Carrier Ethernet 2.0 service types or more UNIs, performs two basic functions: operator wholesale of Carrier Ethernet investment. Furthermore, implementing (ENNI) (Figure 80). ENNI). EVC-based service types are E-Line, target different applications. E-Line services connectivity. Ethernet services, rather than IP/MPLS- • Connects two or more subscriber sites E-LAN and E-Tree, while E-Access and are typically used to replace private When an Ethernet Virtual Connection spans based services with similar characteristics, (UNIs), enabling the transfer of Ethernet E-Transit are based on OVCs. TDM lines and frame relay VPNs, as well The Infinera packet-optical portfolio is MEF multiple Carrier Ethernet networks, it is service frames between them. as for internet access. The other three CE 2.0-certified, including up to 100 Gb/s composed of segments in each CE network A complete MEF Carrier Ethernet service where relevant, and has the functionality that are concatenated together to form the • Prevents data transfer between subscriber consists of an Ethernet service type needed to implement all the service types EVC. These segments are called Operator sites that are not part of the same EVC. associated with one or more bandwidth above in a packet-optical network. Virtual Connections (OVCs). An OVC is the This capability enables an EVC to provide profiles and supporting one or more data privacy and security similar to frame

92 INFINERA INFINERA 93 ETHERNET AND LAYER 2 TECHNOLOGIES

or VLAN-based UNI, and with more detailed when caused by protection switching, make Service Type Port-based Service VLAN-aware Service characteristics as specified by its Ethernet Ethernet QoS mechanisms necessary in virtually all E-Line Ethernet Private Line Ethernet Virtual Private Line service attributes, which are listed in MEF Service networks. specifications 6.1 and 10.2. Type Ethernet Ethernet Carrier Ethernet defines several traffic Ethernet Private LAN E-LAN Ethernet Virtual Private LAN The service attribute represents a service Service Service management mechanisms, which are characteristic, which in turn is further Attribute Attribute described below. defined by a set of Ethernet service attribute Parameters

EVC Services E-Tree Ethernet Private Tree Ethernet Virtual Private Tree parameters. The attributes and parameters ee ee customize the overall performance and 5.6.1 Bandwidth Profiles quality of service for each individual service FIGURE 83: Each Subscribed-to Ethernet Service Type Has an Associated Set of Attributes and A bandwidth profile is a set of traffic Access EPL Access EVPL and subscriber (Figure 83). Parameters Defining Its Behavior in Detail parameters that defines the maximum Access E-Line average bandwidth available for the The Ethernet service attributes are of three it 1 an 1 customer’s traffic. An ingress bandwidth E-Access main types: Access E-LAN profile limits traffic transmitted into it s an 1 1 Per-EVC service attributes defining the 3 Per-UNI service attributes, such as traffic in this way is referred to as traffic the network and an egress bandwidth characteristics of the Ethernet Virtual • UNI ID and physical interface management, and involves mechanisms profile can be applied anywhere to Transit E-Line Connection as such, such as: capabilities: data rate, frame format such as queuing, scheduling and policing of control overload problems from multiple

OVC Services it s E-Transit • EVC ID and type; point-to-point or • Ingress and egress bandwidth profiles Ethernet frames. With traffic management UNIs sending data to an egress UNI Transit E-LAN multipoint • Service multiplexing capability in place, it is possible to guarantee a certain simultaneously. Frames that meet the profile it s • List of connected UNIs • Layer 2 control protocol processing quality of service for a given service with are forwarded; frames that do not meet the • Customer VLAN ID and class of service respect to, for example, data rate, delay, profile are dropped. preservation jitter and packet dropping probability. FIGURE 82: Ethernet Service Types Defined in MEF Carrier Ethernet 2.0 and Their Relation Bandwidth profiles allow service providers to Ethernet Virtual Connections and Operator Virtual Connections. Source: Metro Ethernet • EVC performance: frame delay (latency), 5.6 CARRIER ETHERNET TRAFFIC Quality of service guarantees are important to offer services to users in increments lower Forum inter-frame delay variation, frame loss MANAGEMENT if network capacity is insufficient, especially than what is set by the physical interface ratio and availability The different applications, users and data for real-time streaming multimedia speed. Also, they provide the possibility 2 EVC-per-UNI service attributes, such as: flows in a Carrier Ethernet network require applications such as voice over IP and to engineer the network and make sure • UNI/EVC ID that certain parts of the network are not can make the necessary investments 5.5.4 Carrier Ethernet Service Attributes different priorities and performance IPTV, since these often require a fixed bit significantly lower, especially if the Ethernet A user of Carrier Ethernet subscribes to an • Customer VLAN ID/EVC mapping guarantees. The process of differentiating rate and are delay- and loss-sensitive. In overloaded. services can be implemented on the Ethernet service type (E-Line, E-LAN, E-Tree, • Ingress/egress bandwidth profiles per the absence of network congestion, QoS previously deployed WDM platform. E-Access, E-Transit) with either a port-based class of service mechanisms are in principle not required. However, temporary changes in traffic patterns and reconfigurations, for example

94 INFINERA INFINERA 95 ETHERNET AND LAYER 2 TECHNOLOGIES

Total Bandwidth at UNI • Yellow: the service frame is not subject Ingress bandwidth profiles can be applied PORT-BASED PORT/VLAN-BASED 5.7 CARRIER ETHERNET to SLA performance guarantees, but will per UNI (to all traffic regardless of VLAN tag OPERATIONS, ADMINISTRATION Ingress Bandwidth be forwarded on a “best-effort” basis. or EVC ID), more granularly on an EVC basis, EVC1 EVC1 Pro le Per EVC1 AND MAINTENANCE These frames have lower priority and are or even based on a class of service marking Ingress Bandwidth Ingress Bandwidth (ETHERNET OAM) UNI EVC UNI EVC2 discard-eligible in the event of network such as a customer-applied VLAN priority tag 2 Pro le Per Ingress UNI Pro le Per EVC2 EVC1 EVC2 Using Ethernet as an end-to-end wide congestion. (Figure 85). Ingress Bandwidth EVC3 EVC3 Pro le Per EVC area network service rather than as a link CIR 3 EIR CIR EIR • Red: the service frame is to be discarded at Compliance to the bandwidth profile is layer protocol creates a need for a new the UNI by the traffic policer. determined through two “leaky-bucket” set of operations, administration and algorithms, using a principle referred to as Bandwidth profiles can be defined per maintenance mechanisms and protocols. two-rate three-color marker (TrTCM). Ethernet Virtual Connection and per class PORT/VLAN/CoS-BASED Service providers must be able to provision EIR traf c is of service and are governed by a set of and maintain large volumes of Ethernet marked CE-VLAN CoS 6 Ingress Bandwidth Pro le Per CoS ID 6 CIR parameters, the most important being: services and subscribers in a rational and orange—will 5.6.2 Class of Service and EVC1 CE-VLAN CoS 4 Ingress Bandwidth Pro le Per CoS ID 4 cost-efficient way. EIR be dropped • Committed information rate (CIR), which Service Level Agreements UNI EVC3 at congestion defines the assured bandwidth, expressed The integration of real-time and non- CE-VLAN CoS 2 Ingress Bandwidth Pro le Per CoS ID 2 Furthermore, an end-to-end wide area points as bits per second. real-time traffic over Ethernet requires Ethernet service, i.e. an Ethernet Virtual EVC differentiating packets from different 2 Connection, often involves one or more • Excess information rate (EIR), which carriers/network operators providing the Conceptual Example with Three EVCs applications and providing differentiated FIGURE 84: defines “extra” bandwidth that may be Sharing the Same UNI. The Three CIRs Can Always performance according to the needs of each underlying transmission capacity in addition temporarily used, expressed as bits per Be Met; the Three EIRs Cannot Always Be Met application. When a network experiences to the Ethernet service provider. Carrier Simultaneously second. FIGURE 85: Three Types of Bandwidth Profiles Are Defined in MEF 10.1 congestion and delay, some packets must Ethernet OAM requires the coordination • Committed burst size (CBS) and excess burst be dropped or delayed. This differentiation of OAM performed by a number of size (EBS), which define temporary bursts is referred to in Carrier Ethernet as class of administrative entities and by different of information that can be handled. service. technical systems. MEF 10.2 specifies three levels of bandwidth in the data, such as customer VLAN IEEE provider and its customers. In addition to profile compliance for each individual CoS can be applied at the EVC level 802.3 “q” or “p” tag markings. the various parameters of the bandwidth service frame:29 (with the same CoS applied to all frames profiles (CIR, EIR, etc.), the SLA typically also 5.7.1 The Management Framework The EMXP also supports a simpler bandwidth The class of service settings, together with transmitted over the EVC), or applied within specifies maximum values for various types Ethernet OAM builds on an established • Green: the service frame is subject to profile that only uses the traffic parameters: bandwidth profiles, are used to make service the EVC by customer-defined priority values of frame delay and frame delay variation management framework and terminology service level agreement performance • Rate, expressed as bits per second. level agreements between the service • Burst size, expressed as bytes (Figure 84). (jitter), and values for the availability of the guarantees. subscribed-to service (Figure 86, next page).

96 INFINERA Footnote citations on page 111 INFINERA 97 ETHERNET AND LAYER 2 TECHNOLOGIES

Carrier Ethernet Service Provider such as Simple Network Management Network management and fault management. Application EVPL Profiles, Sample CoS Obectives Protocol Version2 (SNMPv2), defines a Management The IEEE, ITU-T and MEF OAM standards COMPANY MIB System Q standard network management interface provide the means to monitor and execute Committed Excess Metro Fiber Ethernet Virtual Private Line Services Information Information Frame Frame for the administration and maintenance of a the required actions on the Carrier Ethernet. Priority Rate Rate Delay Loss Ratio Customer Customer particular network element (Figure 88). SNMPV2 o cas 1 s s 1 Equipment Equipment Ethernet service provisioning comprises (CE) (CE) nteactie UNI ENNI UNI the processes of setting up the required siness an 1 1 s s 1 EVCs for a customer and assigning them conse ieo 5.7.2 Standards for Ethernet OAM Network 1 Network 2 pogaing attributes such as bandwidth profiles and From an OAM perspective, there are several eepesence s s 1 Demarcation Demarcation class of service. The provisioning process standards that work together in a layered unit unit teae also includes procedures for testing ie content s 1 fashion to provide Carrier Ethernet OAM. Ethernet Virtual Connection the service once it is set up, but before ontent istite IEEE 802.3ah defines OAM at the link level. eeopent an it is turned over to the customer. The With more of an end-to-end focus, 802.1ag nonea tie s 1 FIGURE 88: The Ethernet OAM Framework and Terminology configuration of the service is checked for eie defines connectivity fault management for correctness and verified against its service identifying network-level faults, while ITU acceptance criteria (SAC). Y.1731 adds performance management, FIGURE 86: Examples of Service Level Agreements for Different Applications. Source: Metro Ethernet Forum which enables SLAs to be monitored. operators conduct initial turn-up testing to The described life cycle of an Ethernet The Carrier Ethernet provisioning process The functions of these OAM layers are verify that the EVC is operational and fulfills service is depicted in Figure 90, next page. specified by MEF is based on the ITU-T implemented either in a standalone network the subscribed-to characteristics. While The involved processes fall into three main Specification Y.1564. using the concept of a data model, a A management information base is a demarcation device (i.e. the NID or EDU) the EVC is in use, all parties involved—the categories: provisioning, performance management information base (MIB), database representation of the managed or integrated into the node equipment (i.e. subscriber, the network operators/carriers, describing the status of the individual objects in a telecommunications network. into the EMXP, in Infinera’s case) (Figure 89). and the service provider—want to monitor FIGURE 89: thernet Serice elements in the managed network This database, normally located in the the same EVC to ensure that it adheres to Specification 2.1a an .1731 the specified SLA regarding delay, jitter, Standards for (Figure 87). central network management system, Carrier Ethernet keeps an updated view of network element 5.7.3 The Service Lifecycle loss, throughput, availability, etc. Finally, OAM etwor anaement The life of an EVC starts with a service order when the EVC is not needed any longer, the etwor an connectiity Sytem status by sending queries to the elements, 2.1a an .1731 etwor eement 1 Confiuration comman initiated by a customer. The order contains assigned network resources should be freed and is also used for configuration and Cutomer Cutomer various types of EVC information, such as up and made available to other EVCs. uipment uipment arm tatu meae etc. provisioning activities. An MIB, together etwor etwor UNI locations, bandwidth profiles, class of C interace interace C with an associated management protocol eice eice service, etc. After provisioning the EVC, The Network Management System Carrier thernet etwor FIGURE 87: the service provider and involved network Uses a Data Model – an MIB – to Keep Track of or the Status of the Individual Network Elements 1 in in 2.3ah 2.3ah

98 INFINERA INFINERA 99 ETHERNET AND LAYER 2 TECHNOLOGIES

MEF specifications ITU-T Y.154 • The more sophisticated end-to-end In this model, an entity that requires Customer Customer service level performance and fault management is called a maintenance entity Equipment Equipment New (CE) Demarcation Demarcation (CE) customer 1. Provisioning of the circuit management of the Ethernet Virtual (ME). An ME is essentially an association order • Pre-test (turn-up) Operator 1 Operator 2 • Verify compliance with SLA Pass Connection is often referred to as Ethernet between two maintenance endpoints within service OAM (SOAM) and is addressed by an OAM domain, where each endpoint MEF Specifications 17, 30 and 31, and the corresponds to a provisioned reference Ethernet Virtual Connection Fail ITU-T Y.1731 and IEEE 802.1ag standards. point. For example, in the previous figure, the green arrow between the two CEs MEF specifications ITU-T Y.154 In a real wide area network, Ethernet Virtual represents a subscriber ME. Connections may span several networks, Customer Domain 3. Fault management 2. Performance management each with their own management needs. A maintenance entity group (MEG) consists • Sectionalize the problem • Does the EVC meet the SLA Since managing functionality “end-to-end” of the MEs that belong to the same service • Identify and correct • Is the customer happy Provider means different things to the end customer, inside a common OAM domain. The Domain the network operator and the Carrier MEs exist within the same administrative MEF specifications Yes No IEEE 02.1ag and End of life Ethernet service provider, Ethernet SOAM boundary and belong to the same point- IEEE 02.3ah must handle and interact with performance to-point or multipoint Ethernet Virtual Operator 1 Operator 2 Source: Metro Ethernet Forum and fault management over several Ethernet Connection. For a point-to-point EVC, the Domain Domain Tear down circuit OAM domains. An OAM domain is simply MEG contains a single ME. Return to inventory a network or sub-network of elements An MEG endpoint (MEP) is a provisioned FIGURE 90: The Life Cycle of an Ethernet Service According belonging to the same administrative entity FIGURE 91: Hierarchical OAM Domains Define the OAM Responsibilities and Flows of OAM Data to Metro Ethernet Forum. Some of the Involved Standards OAM reference point that can initiate and managing them (Figure 91). Have Been Indicated terminate proactive OAM frames. It can also Recognizing the fact that Ethernet networks initiate and react to diagnostic OAM frames. often encompass multiple administrative The MEPs are indicated by triangles in the domains, IEEE 802.1, ITU-T SG 13 and MEF figures. action on the transit Ethernet traffic flows. a wide set of performance and fault have adopted a common, multi-domain The MIPs are indicated by circles in the management activities for the EVC and its 5.7.4 Ethernet Service OAM – Performance • Link level performance and fault An MEG intermediate point (MIP) is any SOAM reference model. The Carrier figures. sub-components, such as: and Fault Management management as defined by IEEE 802.3ah intermediate point in an MEG that can react Ethernet is portioned into customer, service Performance management and fault provides mechanisms to monitor link to some OAM frames. An MIP does not The concepts of the SOAM reference model provider, and operator maintenance levels. management comprise the processes of operation and health, and for elementary initiate OAM frames; neither does it take are summarized by Figure 92 (on the next Service providers have end-to-end service Performance Management monitoring EVCs for proper operations, fault isolation. page), indicating the six default MEG levels responsibility; operators provide service • Frame delay. Measurement of one-way and discovering any problems and correcting considered by MEF. faults that have occurred. transport across a sub-network. two-way (round-trip) delay from MEP to Given the above SOAM reference model, MEP. MEF specifications 30.1 and 35 define

100 INFINERA INFINERA 101 ETHERNET AND LAYER 2 TECHNOLOGIES

Self-service Subscriber Subscriber • Linktrace is an on-demand OAM function functional requirements and APIs for LSO Web Portal Equipment Operator A NEs Service Provider Operator B NEs Equipment initiated in an MEP to track the path to a that support capabilities for fulfillment, Default 1 2 3 4 5 6 7 8 Service Provider OSS/BSS destination MEP. It allows the transmitting control, performance, assurance, usage, Business Applications MEG node to discover connectivity data about security, analytics, and policy across Level APIs APIs Subscriber ME the path. multi-operator networks. This approach 6 overcomes some of the existing complexity Test ME • Loopback is an on-demand OAM function End-to-end 5 by defining service abstractions that hide Lifecycle Service Orchestration EVC ME used to verify the connectivity of an MEP 4 the complexity of underlying technologies SP ME with its peers. APIs APIs and network layers from the applications User Operator 3 Operator A ME Operator B ME Service Lifecycle Service Service Lifecycle Service 2 and users of the services, thus facilitating, Endpoint Orchestration Endpoint Orchestration 1 E-NNI ME for example, customer self-service of 5.8 METRO ETHERNET FORUM APIs network capacity. APIs “THIRD NETWORK” VISION VM or VNF MEP (Up orientation) MIP Network Operator Network Operator MEP (down orientation) Logical path of SOAM PDUs Current trends in data communications Subscriber (WAN Service Provider) (Data Center) imply a scenario in which network operators SUMMARY FIGURE 92: Example of Ethernet SOAM Maintenance Entities. Source: Metro Ethernet Forum can deliver self-service, on-demand Operator Virtual Connection Operator Virtual Connection Optical fiber provides almost lossless communication services over multiple, End-to-End Network as a Service transmission of signals at an ultra-wide interconnected networks. Metro Ethernet range of frequencies. Packet switching, Forum refers to this evolution as the FIGURE 93: The Metro Ethernet Forum’s “Third Network” Vision. Source: MEF 2015. • Inter-frame delay variation. Differences used to proactively detect the loss of implemented according to the Ethernet emergence of a “Third Network” (Figure between consecutive frame delay connection between endpoints. Continuity family of protocols, offers one of the 93). A central element of MEF’s Third measurements. check is also used to detect unintended most efficient ways ever to sort and direct Network vision is the ability to enable connectivity between MEGs, as well as to streams of digital data. With packet- platform, represents a powerful toolbox for designed suite of multi-layer management • Frame loss ratio. The number of frames services not only between physical service verify basic service connectivity and health. optical networking, these two outstanding the deployment of these new networks. The and control tools. delivered at an egress UNI compared to endpoints such as Ethernet ports (UNIs), technologies are positioned to dominate architecture encompasses elements that the number of transmitted frames over a • Remote defect indication signal. When a but also between virtual service endpoints As the adoption of cloud services the next generation of transport networks. are fully certified for Carrier Ethernet 2.0 specified time, e.g. a month. downstream MEP detects a fault, it will running on a blade server in the cloud, accelerates, it is imperative that the In addition, the continuing evolution driven services, capable of both native Ethernet signal the condition to its upstream MEP(s). connecting to virtual machines (VMs) or transport network becomes more intelligent • Availability. Downtime is measured over by industry groups ensures dependable and and OTN-adapted transport and with This behavior is similar to the remote defect virtual network functions (VNFs). for the emerging Layer C to thrive. The a period of time, e.g. a year, and used to open standards for future needs. additional scalability and resilience enabled indicator function in SDH/SONET networks. simplified Layer T network is best built using calculate the availability of the service. MEF aims to achieve this vision by defining by MPLS-TP. Special attention has been Infinera’s packet-optical technology, realized a combination of scalable technologies like • Alarm indication signal. An MEP can send Lifecycle Service Orchestration (LSO) given to efficient synchronization, ultra-low Fault Management by the XTM Series platform, the DNA multi- super-channels and intelligent capabilities an alarm signal to its higher-level MEs, capabilities and supporting application latency and minimal jitter in small as well as • Continuity check. “Heartbeat” messages layer management suite and the Xceed SDN delivered using packet-optical technologies thereby informing them of the disruption, program interfaces. MEF will define large networks. Configuring and managing are issued periodically by the MEPs and combined with rich software management immediately following the detection an Infinera packet-optical network is simple and control applications. of a fault. and straightforward thanks to the carefully 102 INFINERA INFINERA 103 6 INDEX

104 INFINERA Footnote citations on page 111 INFINERA 105 1 circuit emulation...... 80 distributed routing functionality (DRF)...... 56 service attribute...... 94 express wavelength...... 67 I 10 GbE...... 79 class of service (CoS)...... 96 DOCSIS...... 70 service attribute parameter...... 94 external network to network interface IEEE 1588v2...... 83 (ENNI)...... 92 A cloud services...... 8 DTN-X...... 36, 64 service provisioning...... 99 IEEE 802.1ag...... 98 committed service type...... 94 active LSP...... 27 IEEE 802.1Q/p...... 77 burst size (CBS)...... 96 shared...... 75 F aggregation network...... 14, 65 E IEEE 802.3ah...... 98 information rate (CIR)...... 96 station...... 75 Fast Ethernet...... 74, 78 ATM...... 62 E- IEEE 802.n...... 74 connection-oriented service...... 26 switch...... 75 FCAPS suite...... 46 attachment circuit (AC)...... 27 Access...... 90 IGMPv3...... 71 connectionless service...... 52 switched...... 75 fixed mobile convergence (FMC)...... 64 automatic protection switching...... 85 E-LAN...... 90 Infinera portfolio...... 10, 40 forward error correction (FEC)...... 18 control plane...... 51 E-Line...... 90 Synchronization Message Channel...... 83 Infinera’s management software suite...... 47 frame check sequence (FCS)...... 74 Converged Cable Access Platform E-Transit...... 90 synchronization over...... 82 Intelligent B (CCAP)...... 70 frame relay (FR)...... 62 E-Tree...... 90 traffic management...... 95 bandwidth profile...... 95 SFP (iSFP)...... 33 convergence time...... 84 Frameworx...... 45 element manager...... 48 Virtual Connection (EVC)...... 91 boundary clock...... 84 transport...... 8 CSMA/CD...... 75 frontplane...... 36 EMXP...... 21, 35 Ethernet demarcation unit (EDU)...... 35, 63 Transport Network architecture...... 9 broadcast domain...... 75 customer enterprise network...... 62 Ethernet Ring Protection Switching interworking function (IWF)...... 81 burst size...... 96 equipment (CE)...... 91 (ERPS)...... 87 G ERPSv2...... 88 IP backhaul...... 64 Business Ethernet...... 62 portal...... 48 Ethernet-aware Layer 1 transport...... 15 G.709...... 19 Ethernet...... 74 IP-VPN...... 22, 62 VLAN...... 78 eTOM Business Process Framework...... 45 G.781...... 83 frame...... 74 C excess G.8032...... 87 meshed...... 76 L Carrier Ethernet 2.0...... 88 D burst size (EBS)...... 96 G.805/G809...... 49 OAM...... 97 G.8264...... 83 label network...... 16, 88 data plane...... 55 information rate (EIR)...... 96 physical media...... 78 Generic Framing Procedure (GFP)...... 19 edge router (LER)...... 27 service...... 62, 90 differential clock recovery (DCR)...... 33 Gigabit Ethernet...... 74 switched path (LSP)...... 26 CFP...... 80 differential timing...... 82, 84 switching router (LSR)...... 24 Digital Network Administrator (DNA)...... 48

106 INFINERA INFINERA 107 Layer management Multiprotocol Label Switching (MPLS)...... 8 OpenFlow...... 55 Packet-Optical Q C...... 8, 56 framework...... 45, 97 muxponder...... 18, 36 operations, administration Transport Switch (EMXP)...... 35 Q-in-Q...... 49, 78 and maintenance...... 97 T...... 8, 56 information base (MIB)...... 98 transport system...... 3 quality of service (QoS)...... 35, 88 operations and support Layer 1 MEF “Third Network”...... 102 N system (OSS)...... 48 networking, applications...... 62 aggregation...... 15 MEG native Ethernet...... 19, 20 operator virtual connection (OVC)...... 92 transport, advantages of...... 38, 44 R Ethernet transport...... 16 endpoint...... 101 path computation network optical channel Rapid Spanning Tree Protocol (RSTP)...... 76 Layer 2 intermediate point...... 101 element (PCE)...... 53 functions virtualization (NFV)...... 53 data unit (ODU)...... 20 ring protection link...... 87 aggregation...... 14 level...... 101 function...... 53 interface card (NIC)...... 74 payload unit (OPU)...... 19 ROADM...... 28, 36 Ethernet transport...... 17 Metro Ethernet Forum (MEF)...... 90 interface device (NID)...... 35, 62 transport unit (OTU)...... 20 point-to-point EVC...... 92 leased line...... 62 mobile manager...... 48 Optical Transport Network (OTN)...... 8 policing...... 30 S Lifecycle Service Orchestration (LSO)...... 102 backhaul...... 67 planning...... 51 port device...... 35 OSS integration toolkit...... 48 scalability...... 26, 39, 67, 90 Link Aggregation...... 86 fronthaul...... 69 segments...... 64 power consumption...... 41 OTN SDH...... 9, 19, 32 Control Protocol (LACP)...... 86 MPLS-TP...... 26 virtualization...... 51 Precision Time Protocol (PTPv2)...... 82, 84 framing...... 19 SDN...... 51 Group (LAG)...... 85 tunnel...... 27 non-blocking switches...... 75 primary reference clock (PRC)...... 81 switch...... 20 applications...... 57 link layer...... 74 MTOSI 2.0...... 46 Provider transport system...... 20 controller...... 52 multi-chassis LAG...... 86 O Backbone Bridges...... 78 M multi-layer OAM domain...... 100 P bridges...... 78 management...... 38 edge (PE)...... 27 MAC address...... 26, 74 ODU2e...... 20 Packet Switching Module (PXM)..... 22, 36, 64 management system...... 45 pseudowire...... 16, 24, 80 maintenance entity group (MEG)...... 101 ODUflex (GFP)...... 20, 64 Packet Transport Fabric (PT-Fabric)...... 36 network management...... 44 PT-Fabric...... 36 SDN...... 50 PXM...... 22, 36, 64 multipoint-to-multipoint EVC...... 92

108 INFINERA INFINERA 109 service T virtualized network...... 54 FOOTNOTES assurance...... 46, 50 TDM dervices, migrating to Ethernet...... 32 VLAN awareness...... 39 identifier (VID)...... 77 1 Optical Transport Network is a standard for 13 The IEEE 1588 standards describe a hierarchical where frames can have a data payload of up to time-division multiplexing (TDM)...... 80 multiplexing and switching signals at OSI Layer 1. master-slave architecture for clock distribution 10,000 bytes. in computer networks, originally known as the frame...... 96 TM Forum...... 45 tag...... 77 2 Multiprotocol Label Switching—Transport Profile 23 The number 2 refers to the second layer in the is a mechanism to create virtual connections. Precision Time Protocol (PTP). standardized OSI reference model for data fulfillment...... 50 TMF608...... 46, 49 3 The Open Systems Interconnection (OSI) model 14 Enhanced Telecom Operations Map. communications. lifecycle...... 98 is a conceptual model that characterizes and 15 Multi-Technology Operation Systems 24 In CSMA/CD, the devices (called stations) can traffic W standardizes the communication functions of a Interface (MTOSI) is a TM Forum standard for broadcast data over the medium whenever it telecommunication or computing system. implementing interfaces between operations and is idle. If more than one station transmits at the VLAN (SVLAN)...... 26, 78 engineering...... 22, 26 weighted random early 4 Analysts estimate that Layer 2 equipment is only support systems (OSS). same time and signals collide, the transmission detection (WRED)...... 31 is stopped by the involved stations, which will acceptance criteria (SAC)...... 99 management...... 30 30 to 50% of the cost of Layer 3 equipment. 16 Connection-oriented service: A semi-permanent connection is established before any useful then wait for some random time and then restart 5 See section 5.5 for more information about transmission. level agreement (SLA)...... 96 shaping...... 30 Carrier Ethernet as defined by Metro Ethernet data can be transferred, and a stream of data is Forum. delivered in the same order as it was sent. 25 The “Q” refers to the name of the IEEE 802.1Q SNMPv2...... 98 standard for VLANs. transparent clock...... 84 6 A data communications network is used for the 17 Transport SDN is sometimes used to specifically X management and control of network equipment. designate integrated Layer 0/Layer 1 networks 26 Service restoration times of 30 seconds or more software-defined networking (SDN)...... 51 Transport SDN...... 52 operating under the SDN scheme. are typical for STP. 7 Known as e.g. 10G BASE-X. SONET...... 9, 19, 32 XTM Series...... 18, 20, 24, 36 18 ONF is a trade organization standardizing the 27 In many Metro Ethernet Forum technical 8 See section 2.3 for more on MPLS-TP. OpenFlow protocol and related technologies. specifications and some other literature, the Spanning Tree Protocol (STP)...... 76 U 9 Note that the use of the term “router” is historic 19 ”White box” is a generic term for minimalistic Carrier Ethernet network is referred to as a metro and neither requires nor precludes the ability packet switches built from merchant silicon Ethernet network (MEN), which is the same thing. statistical multiplexing...... 15 Y unified information model...... 49 to perform IP forwarding. It is sometimes used switching chips and commodity CPU and 28 The ETH layer is sometimes referred to as the instead of “node” in an MPLS context. memory. path layer. switched video transport...... 69 Y.1731...... 41, 98 user network interface (UNI)...... 83, 91 10 VPLS is defined in IETF RFC 4761 and RFC 4762. 20 North American operators of cable and direct- 29 A service frame is a subscriber Ethernet frame to SyncE...... 30, 83 11 E1 is a 2 Mb/s signal in the older Plesiochronous broadcast television systems are often referred be forwarded by a service in a Carrier Ethernet. Digital Hierarchy (PDH) standard. to as MSOs. In other parts of the world, the abbreviation CATV operator is used. synchronization...... 33, 80 V 12 MPLS used in conjunction with IP and its routing protocols. 21 Data Over Cable Service Interface Specification. probe...... 34 DOCSIS 3.0 has been ratified as ITU-T video services...... 69 Recommendation J.222. Synchronization Status virtual 22 For Gigabit Ethernet, some vendors provide Message (SSM)...... 83 equipment that supports a jumbo frame option, Synchronous Ethernet (SyncE)...... 30, 83 LAN (VLAN)...... 76 private LAN service (VPLS)...... 22, 28 private network (VPN)...... 77 routing...... 56

110 INFINERA INFINERA 111 www.infinera.com

112 INFINERA Footnote citations on page 111