Editor’s Note

Editor’s Note service provider transport networks worldwide today and morphing into SDN-WAN, ready to serve future Packet transport networks requirements. Continuous expansion and re- are amid a quiet yet invention of data forwarding, signaling and routing successful evolution cycle. techniques has secured its long and continuing After years of software- lifetime. defined networks (SDN) By the way – the number of vendors participating in revolution noise without our event keeps growing continually due to another many actual deployments, aspect of the secret sauce: It is not trivial but the industry converges on rewarding to implement MPLS/SDN, and there are evolutionary solutions for many different viable approaches. Microwave SDN. Today’s SDN-WAN Carsten Rossenhövel vendors showed full MPLS data plane and control Managing Director, EANTC solutions are interoperable, plane integration this time; a white-box router vendor integrate pre-existent MPLS implementations, and together with a company producing protocol stacks adopt the diversity and complexity of service for white boxes demonstrated SDN integration; and provider network routing and services. Admittedly, a first-time participant showed how to jump-start into the MPLS/SDN use case scenarios we tested this SRv6 successfully. All these are great success stories, year were some of the most complex so far; yet, it expanding service providers’ choices and solution seems this is the most viable evolutionary path to diversity. fulfill future service requirements. Fast forward to the future: 2018 is the year that we 21 vendors participated in the EANTC interopera- will remember as “the year before 5G deployments bility event this time – one of the largest numbers took off.” Things need to get real now: Operators ever. The level of interoperability for Ethernet VPN discover that there is a lot to do which cannot be (EVPN) services and Segment Routing (SR) delayed any further – backbones need to scale in implementations over MPLS (SR-MPLS) has been very anticipation of the 5G traffic growth and much reassuring. We have seen tangible progress in the higher number of base station sites; software-defined number of successful multi-vendor combinations, the network controllers need to calculate optimal paths maturity of implementations as expressed by more for each of the new slices (service classes); mobile complex test cases, and the efficiency of edge computing and service virtualization will create configuration and troubleshooting. There is a major much more East-West traffic as well. Multiple evolution going on quietly: Legacy signaling network parts and elements need to be integrated to protocols LDP and RSVP-TE will no longer be needed form a consistent end-to-end 5G transport network. in the future, greatly improving It has been great to witness the scalability and efficiency of how the participating vendors core and aggregation net- Table of Content are getting ready to fulfill these works. 5G transport network Editor’s Note ...... 3 Path-Computation Element Pro- requirements. One of the 5G- tocol (PCEP) tests were much Introduction ...... 4 relevant test areas is network more promising than last year Participants and Devices ...... 4 clock synchronization. It was as well; router (“PCC”) and Interoperability Test Results ...... 5 supported by a large group of controller (“PCE”) implementa- vendors again – participants Segment Routing ...... 5 tions are increasingly aware of who have continually improved multi-vendor scenarios and Ethernet Virtual Private Network ..... 13 multi-vendor network clocking ready to interoperate with each Topology...... 20 at our events for about ten other in a collaborative way. In Software Defined Networking...... 23 years now. The solutions are other areas such as Segment rock-solid meanwhile, and this Microwave ...... 30 Routing over IPv6 (“SRv6”), a test area is always one with few vendors are doing ground Clock Synchronization ...... 32 most reliable results in our breaking work; it might take a Summary...... 38 interop events. little more time until SRv6 Next year, we will increase becomes widely deployable. clock synchronization precision What is the secret sauce of these successful requirements further for some of the 5G Release 15 developments? It is about continuity and consistent requirements. In addition, we plan to expand the standardization along the lines of industry SDN tests, revisiting areas that showed only limited requirements. At the 101st IETF meeting in London a participation or success this year. In the coming few weeks ago, George Swallow retired. As one of years we will focus much more on domain and the main inventors of MPLS standards, he and many potentially even service orchestration: Auto-mated other key contributors have spent two decades service provisioning, fault management, defining, expanding, and maturing today’s MPLS- performance monitoring, and other management based service provider packet transport networks. aspects become increasingly important. It seems this As a result, MPLS is still around, running virtually all has always been the successful motto of MPLS:

3 MPLS + SDN + NFV World Congress 2018 Multi-Vendor Interoperability Test

Never rest – always aspire to improve further. There As always, packet clock synchronization area is a lot to do to get 5G services off the ground; let’s showed consistent results with many vendors get started! supporting the latest PTP profiles for time/phase synchronization. In the microwave section we saw a great deal of Introduction effort to integrate the wireless transport into IP/MPLS transport networks. We find this to be a critical For the last couple of years, we have been focusing requirement to support next-generation mobile on testing Ethernet VPNs, Segment Routing (SR) and networks, where end-to-end network slicing will play the use of Path Computation Elements (PCE) to a key role for the diverse 5G use-cases. influence traffic forwarding and routing policies. This year it was the time to probe the maturity of these technologies by encouraging the participation Participants and Devices of more vendors than ever in our interoperability hot- staging. Our stretch goal for this year’s event was Participants Devices integration. We had the pleasure of performing tests with a total Adva FSP150 ProVMe of 21 vendors, with test scenarios of up to 10 vendors in the same topology, performing any-to-any Arista Networks 7050SX2-72Q interoperability. 7280SR-48C6 It was definitely a year of consolidation for Segment BISDN GmbH Basebox controller (external) Routing as the new standard for MPLS-enabled Switch AG7648 networks. All our test scenarios involving MPLS in the (Delta Electronics) Segment Routing, Ethernet VPNs and Software Defined Networking sections were carried out using Calnex Calnex Paragon-T Segment Routing. Therefore, it showed us how Calnex Paragon-X mature vendor implementations are and a clear view, whereto the industry is moving forward. Cisco ASR 9000 We additionally tested for the first time SR implemen- CSR1kv tations using IPv6 in the data plane (SRv6) and we IOS XRv9000 verified vendor interoperability of some new NCS 5500 proposed standards: Network Services Orchestrator • “BGP Signaling of IPv6-Segment-Routing-based (NSO) VPN Networks” Nexus 7702 Nexus 93180-FX • “Segment Routing Prefix SID extensions for BGP” • “IPv6 Segment Routing Header (SRH)” Delta Electronics AGC7648A • “SRv6 Network Programming” ECI Telecom NPT-1800 • “Topology Independent Fast Reroute using Segment Routing” Ericsson Baseband 6620 In the EVPN section, we saw most of the new vendor Baseband 6630 faces, with broad support for EVPN bridging and MINI-LINK 6651 EVPN Integrated Routing and Bridging (IRB). EVPN is MINI-LINK 6352 already a well established technology for Data MINI-LINK 6366 Center use-cases but we are seeing it more and more MINI-LINK 6691 present as a unified control-plane technology to MINI-LINK 6693 provide L2VPN/L3VPN services across WAN and Router 6371 Core network deployments. Router 6471 Implementations of Path Computation Element Router 6672 Protocol (PCEP) also showed a good level of Router 6675 maturity. This year we could test a total of 31 Huawei CX6608 combinations of different vendor/products inter- CX600-X2-M8A oping as PCE and Path Computation Clients (PCC). NE40E-X2-M8A This was also the first year where we could test NE40E-M2K disaggregated hardware/software, with some white- NE9000-8 box and Network Operating System (NOS) vendors Network Cloud Engine (NCE) participating. Furthermore, in the NETCONF/YANG section we IP Infusion OcNOS-AS7712-32X reached a notable milestone by provisioning an end- OcNOS-Virtual Control Machine to-end L3VPN service in a multi-vendor environment by using standardized IETF YANG models.

4 Interoperability Test Results

Terminology Participants Devices We use the term tested when reporting on multi- Ixia IxNetwork vendor interoperability tests. The term demonstrated Novus One refers to scenarios where a service or protocol was evaluated with equipment from a single vendor only. Juniper MX80-P Networks MX104 Test Equipment MX240 QFX10002-72Q With the help of participating test equipment QFX5110-48S vendors, we generated and measured traffic, emulated and analyzed control and management Meinberg LANTIME M1000S protocols and performed clock synchronization LANTIME M4000 analysis. We thank Calnex, Ixia and Spirent Communications for their test equipment and support Metaswitch Metaswitch CNRouter throughout the hot staging. Networks

Microsemi TimeProvider 2300 TimeProvider 2700 Segment Routing TimeProvider 4100 Segment Routing is becoming the de-facto SDN TimeProvider 5000 architecture. Leveraging the source routing TimeProvider 5000 Expansion 10 paradigm, SR brings scalability, simplicity and end- NEC iPASOLINK VR to-end traffic engineering to MPLS and native IPv6 networks Nokia 7750 SR-7 Its architecture allows the use of different control- Network Services Platform (NSP) planes models. In a distributed model, Network Elements (NEs) use a dynamic routing protocol to Oscilloquartz OSA5421 HQ++ allocate and distribute Segment Identifiers (SIDs). The routing protocol used for this purpose could be an Spirent Attero-100G IGP, such as IS-IS or OSPF with SR extensions, or Communications TestCenter (STC) BGP with SR extensions. TestCenter Virtual (STCv) In a centralized model, external controllers can be UTStarcom SOO Station leveraged for computation of paths that are then UAR500 encoded in a SID list. A variety of methods including PCEP, BGP, NETCONF could be used to signal these ZTE ZENIC WAN Controller SR policies to the NEs. For the former, results are Corporation ZXR10 M6000-3S covered in the PCEP section of this white paper. ZXR10 M6000-5S Additionally, the SR architecture can be instantiated ZXR10 M6000-8S PLUS over various data planes: SR over MPLS (SR-MPLS) ZXR10 M6000-18S and SR over IPv6 (SRv6). ZXR10 T8000-18 Throughout the SR section we will present several Table 1: Participants and Devices executed test cases using different combinations of control plane protocols and data plane encap- sulations. Interoperability Test Results Segment Routing over IPv6 (SRv6) As usual, this white paper documents only positive results (passed test combinations) individually with vendor and device names. Failed test combinations IPv6 Routing over SRv6 are not mentioned in diagrams; they are referenced Segment Routing uses network programming function anonymously to describe the state of the industry. concepts to enable packet forwarding through a Our experience shows that participating vendors specific path, different from the default IGP shortest quickly proceed to solve interoperability issues after path. A number of standard SRv6 functions are our test so there is no point in punishing them for specified in the SRv6 Network Programming IETF their willingness to learn by testing. Confidentiality is draft (filsfils-spring-srv6-network-programming). vital to encourage manufacturers to participate with In this scenario we tested both the END and END.X their latest - beta - solutions and enables a safe functions. environment in which to test and to learn. The END function is the most basic function. It is used to steer traffic along the shortest-path to the advertising node.

5 MPLS + SDN + NFV World Congress 2018 Multi-Vendor Interoperability Test

On the other hand, the END.X function is used to IPv4 VPN over SRv6 steer traffic along the shortest path to the advertising The draft “dawra-idr-srv6-vpn” defines procedures node and then cross-connect it to a particular and messages for BGP SRv6-based EVPNs and L3 neighbor. VPNs in order to provide a migration path from During our test, we verified the expected END and MPLS-based VPNs to SRv6 based VPNs. END.X data plane forwarding behavior and IPv6 SR In order to provide an SRv6 based VPN service, the Header (SRH) handling by Cisco and UTStarcom egress PE signals an SRv6-VPN SID with the VPN devices for SRv6 encapsulated traffic generated by route via MP-BGP. SRv6-VPN SID refers to an SRv6 Spirent TestCenter (STC) and Ixia IxNetwork. SID that may be associated with one of the END.DT In our first scenario, UTStarcom’s UAR500 per- or END.DX functions defined in the IETF draft “filsfils- formed the END function and Cisco’s NCS 5500 spring-srv6-network-programming”. performed the END.X function. In the second In our test, vendors configured an IPv4 L3VPN and scenario, the roles were inverted accomplishing the the egress node performed the END.DT4 function, same results. which performs the Endpoint (END) function with decapsulation and IPv4 table lookup. Cisco The ingress PE encapsulates the VPN packets in an NCS 5500 ::4 outer IPv6 header where the destination address is Metric=10 the SRv6-VPN SID provided by the egress PE. Metric=100 Additionally, the ingress PE inserts the Segment Routing Header (SRH) which allows traffic enginee- Spirent ::1 Spirent ring based on the SIDs listed in the Segment-list field TestCenter TestCenter (SID-list). Metric=10 Metric=10 Additionally, during this test case execution, the UTStarcom transit node (P) performed the END function, UTStarcom UAR500 updating the packet’s IPv6 destination address with UAR500 ::4 the next SID. This behavior was possible due to Metric=10 inclusion of Node P to the SID-list of the afore- Metric=100 mentioned SRH. During the test, unidirectional traffic from TG1 to Spirent ::1 Spirent TG2 was sent by the test generator. TestCenter TestCenter Metric=10 Metric=10 Cisco NCS 5500 Cisco Ixia Ixia NCS 5500 IxNetwork P IxNetwork ::4 UTStarcom Metric=10 UAR500 Metric=100 UTStarcom UAR500 Ixia ::1 Ixia IxNetwork IxNetwork Metric=10 Metric=10 UTStarcom Spirent Spirent UAR500 TestCenter P TestCenter UTStarcom UTStarcom UAR500 ::4 UAR500 Metric=10 UTStarcom UAR500 Metric=100

Ixia ::1 Ixia SRv6 Domain Network Node IxNetwork IxNetwork Metric=10 Metric=10 Physical Link Access Network

Cisco Traffic Generator Emulator/ NCS 5500 Traffic Generator Figure 2: IPv4 VPNs with SRv6 Core Service Traffic 1 Service Traffic 2 SRv6/IS-IS Domain END Function In this test we additionally verified that BGP can be used to advertise the reachability of prefixes in a Physical Link END.X Function particular VPN from an egress Provider Edge Emulator/ Network Node (egress-PE) to ingress Provider Edge (ingress-PE) Traffic Generator nodes. Figure 1: SRv6 IPv6 Routing

6 Segment Routing

Segment Routing Then, we verified that the SR nodes (mapping clients) and LDP Interworking process the advertised mapping and program their MPLS forwarding accordingly. We tested that an SR mapping server can be used to Lastly, we verified that edge network nodes do a provide interworking between SR and LDP networks. proper encoding of the data path and that services The mapping server advertises a remote-binding function appropriately between SR and LDP-only segment id for prefixes attached to non-SR capable nodes. LDP nodes. The following vendors participated in setup 1 as: Additionally, we verified that an end-to-end LSP can • LDP Node: Arista 7280SR-48C6, Cisco NCS be built when one part of the network is Segment 5500, Delta AGC7648A, Spirent TestCenter Routing enabled and the other part relies on LDP • SR Node: Arista 7280SR-48C6, IP Infusion exclusively for label allocation and distribution. OcNOS (AS7712-32X), Juniper MX80-P First, we verified that the mapping server advertises the range of prefixes corresponding to non-SR • SRMS Node: Arista 7280SR-48C6, Cisco NCS capable nodes and their associated SIDs/Labels. 5500, Nokia 7750 SR-7 The following vendors participated in setup 2 as: Setup 1 • LDP Node: Ericsson MINI-LINK 6693 Spirent Spirent • SR Node: Ixia IxNetwork SRMS TestCenter TestCenter • SRMS Nodes: IP Infusion OcNOS (AS7712-32X), Nokia 7750 SR-7 Cisco Arista Juniper NCS 5500 7280SR-48C6 MX80-P LDP Domain SR Domain Segment Routing Anycast Segment — Disjointness in Dual Plane Spirent Spirent Networks TestCenter SRMS TestCenter In this section we verified that Segment Routing Arista Cisco Juniper Anycast Segment could be used to disjoint traffic 7280SR-48C6 NCS 5500 MX80-P forwarding paths within dual plane networks. LDP Domain SR Domain Segment Routing provides a new solution for disjoint Spirent Spirent paths within dual plane networks. Disjointness allows TestCenter SRMS TestCenter to transport different traffic services across disjoint paths. This can be achieved by using SR Anycast Cisco Arista segment in SR routers. NCS 5500 7280SR-48C6 Anycast Segment Identifier (Anycast SID) is specified LDP Domain SR Domain for a set of routers within the same data plane. Each SR router in the Anycast set advertises the same Spirent Spirent Anycast-SID, which represents ECMP-aware, shortest- SRMS TestCenter TestCenter path IGP route to the closest node of that Anycast set. In this test, we tested that the service traffic can be Delta Nokia IP Infusion diverted to specific data plane based on a AGC7648A 7750 SR-7 OcNOS configured policy in the ingress PE. LDP Domain SR Domain The following vendors participated as: • Ingress PE node: Ixia IxNetwork, Juniper MX104 Setup 2 • P Node: Ericsson Router 6675, Juniper MX80-P

Ixia • Anycast Nodes: Arista 7280SR-48C6, Cisco ASR 9000, ECI NPT-1800, Ericsson Router 6675, IxNetwork Nokia Huawei CX6608, Nokia 7750 SR-7 7750 SR-7 Ericsson Ixia • Egress PE Node: Ixia IxNetwork, Juniper MX80-P MINI-LINK 6693 IxNetwork IP Infusion Segment Routing Per-CoS Steering OcNOS LDP Domain SR Domain into Multi-Plane Network

IS-IS/SR Domain MP-BGP With the same test bed as the previous test, depicted IS-IS/LDP Domain Network Node in Figure 4, we verified that a policy in the ingress PE can divert traffic into different data planes by Access Network Physical Link leveraging DSCP marking to differentiate traffic flows Emulator/ Traffic Generator and mapping them two different data planes. Traffic Generator Figure 5 shows the participants who tested this Figure 3: SR/LDP Interworking scenario.

7 MPLS + SDN + NFV World Congress 2018 Multi-Vendor Interoperability Test

Anycast SID: Anycast SID: index 101 index 101 Juniper Ericsson Juniper Ericsson MX80-P Router 6675 MX80-P Router 6675

Juniper Juniper MX80-P Juniper MX104 Juniper X80-P MX80-P Huawei ECI Huawei ECI CX6608 NPT-1800 CX6608 PT-1800 Anycast SID: index 102 Anycast SID: index 102

Anycast SID: Anycast SID: index 101 index 101 Juniper Ericsson Juniper Spirent MX80-P Router 6675 MX80-P TestCenter

Juniper Arista MX104 Juniper 7280SR- Spirent MX80-P 48C6 TestCenter Huawei Arista Huawei Arista CX6608 7280SR-48C6 CX6608 7280SR-48C6 Anycast SID: index 102 Anycast SID: index 102

Anycast SID: index 101 Data Plane 1 Ericsson Cisco CoS Marking 1 Router 6675 ASR 9000 Data Plane 2 CoS Marking 2

Physical Link Network Node

Emulator/Traffic Generator Ixia Ixia IxNetwork IxNetwork Figure 5: Segment Routing Per-CoS Traffic Steering Cisco Nokia ASR 9000 7750 SR-7 BGP Segment Routing Anycast SID: index 102 Segment Routing can be used in large scale Data Data Plane 1 Service Traffic 1 Centers as a simple solution to provide traffic Data Plane 2 Service Traffic 2 engineering and fast re-route capabilities in the DC fabrics. BGP is a popular choice as routing protocol Physical Link Network Node in Clos topologies due to its scalable intrinsec Emulator/Traffic Generator properties. For these reasons, we arranged two topologies using Figure 4: Segment Routing — Anycast Segment BGP-LU. In the first one, we tested 5 Leaves and 1 Disjointness in Dual Plane Networks Spine exchanging IPv4 prefixes and their associated labels using BGP Labeled Unicast (BGP-LU) NLRI. The following vendors participated as: Then we tested a similar topology with one Spine • Ingress PE node: Arista 7280SR-48C6, Juniper and three leaves but this time vendors enabled the MX104 BGP Prefix-SID attribute in the BGP-LU NLRI. • P Node: Juniper MX80-P In the latter test bed, we additionally validated the • Anycast Nodes: Arista 7280SR-48C6, ECI NPT- correct SR forwarding and SID advertisement. 1800, Ericsson Router 6675, Huawei CX6608 We tested full-mesh traffic forwarding between all Leaf nodes, and between the Leaf nodes and the • Egress PE Node: Juniper MX80-P, Spirent node emulators—Ixia and Spirent (were applicable). TestCenter The following vendor equipment participated in the BGP-LU scenario: Arista (7280SR-48C6), Cisco (NCS 5500), Ixia (IxNetwork), Juniper (MX104) and Spirent (TestCenter).

8 Segment Routing

Ixia IxNetwork and Spirent TestCenter acted as failures. The LFA approach is applicable when the traffic generators and emulated Leaves. protected node or Point of Local Repair (PLR) has a For the Prefix-SID Label in Labeled Unicast NLRI direct neighbor that can reach the destination (BGP-LU + SID) variant of the scenario; Arista, Cisco without looping back traffic to the PLR. When the and Ixia participated with the same equipment as in protected path fails, the traffic was sent to the the BGP-LU setup. neighboring node which in turn forwards the traffic In this case, Ixia IxNetwork was used as a traffic to the destination. generator and emulated Leaf. When above conditions are not met, Topology Independent Loop-free Alternate (TI-LFA) can be used instead. It relies on segment routing to build a Spine Cisco protection mechanism based on proven IP-FRR Layer NCS 5500 concepts. TI-LFA does not require any additional signalling between the Point of Local Repair (PLR) and the repair node — typically called PQ node. In both cases, the destination is protected against the Leaf failure of a link. Additionally, the SRLG protection Layer describes the situation, in which the destination is protected assuming that a configured set of links Cisco Arista Juniper Ixia Spirent share fate, with the primary link which has failed. NCS 5500 7280SR- MX104 IxNetwork TestCenter 48C6 During the test, initially we performed baseline measurement for packet loss using bidirectional traffic from a traffic generator. Cisco Spine Afterwards, vendors configured LFA/TI-LFA on the NCS 5500 Layer network nodes and verified that the network nodes installed backup forwarding entry in FIB. While the traffic was running via the primary path we disconnected the link and measured the service interruption time based on the packet loss. We saw Leaf that the traffic was taking the backup path. Layer Finally, vendors configured a new link (link 2) and Cisco Arista Ixia configured it with the same SRLG as the failed link. NCS 5500 7280SR-48C6 IxNetwork We tested TI-LFA with SRLGs in this case. We tested the following three scenarios: FRR / LFA Physical Link Topology Layers link protection, TI-LFA link protection and TI-LFA local SRLG protection for both SR-MPLS and SRv6 data eBGP-LU eBGP-LU + SID plane options. DC Fabric Leaf Node Vendors participating in the IP FRR/LFA link protection tests with MPLS data plane (Figure 7) Access Network Traffic Generator were: • Emulator/ Spine Node/ PQ Node: ECI NPT-1800, Ericsson Router 6675, Traffic Generator Route Server Huawei CX6608 • PLR Node: ECI NPT-1800, Ericsson Router 6675, Figure 6: BGP Segment Routing Huawei CX6608, Juniper MX80-P • P Node: ECI NPT-1800, Huawei CX6608, During this test we found out that one of the vendors Juniper MX80-P acting as Route Server was not able to process/ propagate BGP updates with Prefix-SID TLVs, • Egress PE: ECI NPT-1800, Ericsson Router 6675, occasioning BGP session flaps upon reception. Due Huawei CX6608, Juniper MX80-P to this we removed the vendor from the Route Server Vendors participating in the TI-LFA link protection function. tests with MPLS data plane (Figure 8) were: • PQ Node: Cisco ASR 9000, ECI NPT-1800, Resiliency and Monitoring Huawei CX6608, Juniper MX-80-P • PLR Node: Cisco ASR 9000, ECI NPT-1800, Segment Routing FRR/TI-LFA Ericsson Router 6675, Juniper MX80-P • P Node: ECI NPT-1800, Ericsson Router 6675, Segment Routing aims to be a transport technology Huawei CX6608, Juniper MX80-P that support services with tight Service Level Agreement (SLA) guarantees. Therefore, SR must • Egress PE: Ericsson Router 6675, Huawei provide a local repair mechanism capable of CX6608, Juniper MX80-P, Nokia 7750 SR-7 restoring end-to-end connectivity in case of link

9 MPLS + SDN + NFV World Congress 2018 Multi-Vendor Interoperability Test

Spirent Ixia TestCenter IxNetwork Huawei Nokia CX6608 7750 SR-7 Spirent ECI Ixia Cisco Metric=10 Metric=20 TestCenter NPT-1800 IxNetwork ASR 9000 Metric=10 Metric=10

Juniper Cisco MX80-P ASR 9000 Metric=100 Metric=10 Metric=10 Metric=10 Ericsson Ericsson Router 6675 Router 6675 Spirent Spirent TestCenter TestCenter Juniper Huawei MX80-P CX6608 ECI Spirent Huawei Spirent Metric=10 Metric=20 NPT-1800 TestCenter CX6608 TestCenter Metric=10 Metric=10

Ericsson Juniper Router 6675 MX80-P Metric=100 Metric=10 Metric=10 Metric=10 ECI Ericsson NPT-1800 Router 6675

Spirent Spirent TestCenter TestCenter Ericsson Juniper Router 6675 MX80-P Huawei Spirent Juniper Spirent Metric=10 Metric=20 CX6608 TestCenter MX80-P TestCenter Metric=10 Metric=10

ECI Ericsson NPT-1800 Router 6675 Metric=100 Metric=10 Metric=10 Metric=10 Huawei ECI CX6608 NPT-1800

Spirent Spirent TestCenter TestCenter ECI Ericsson NPT-1800 Router 6675

Spirent Juniper Spirent Juniper Metric=10 Metric=20 TestCenter MX80-P TestCenter MX80-P Metric=10 Metric=10

Huawei ECI CX6608 NPT-1800 Metric=100 Metric=10 Metric=10 Metric=10 Ericsson Huawei Router 6675 CX6608

PQ Node Traffic Generator PQ Node Traffic Generator

SR Domain Protected Path SR Domain Protected Path Post-convergence Link Failure Post-convergence Link Failure Path Path Physical Link Network Node Physical Link Network Node

Figure 7: SR FRR/LFA Link Protection Figure 8: SR-MPLS TI-LFA Link Protection

10 Segment Routing

Vendors participating in the TI-LFA local SRLG For SRv6, the TI-LFA implementation of UTStarcom is protection tests and MPLS data plane were: Cisco based on centralized controller SOO Station that ASR 9000 as PLR Node, Ericsson Router 6675 as P runs PQ algorithm and calculates post-convergence node, Cisco ASR 9000 as PQ node and Nokia path. 7750 SR-7 as egress PE node. SRv6 TI-LFA

Spirent Spirent TestCenter TestCenter Metric=10 Nokia Spirent Metric=20 7750 SR-7 TestCenter Metric=10 Spirent Metric=20 TestCenter UTStarcom UTStarcom Metric=10 UAR500 UAR500 Link 2 Metric=10 Metric=10 Cisco Cisco ASR 9000 ASR 9000 UTStarcom Metric=10 Metric=10 UAR500 SRv6 TI-LFA with SRLG Ericsson Router 6675 Spirent TestCenter Traffic Generator Metric=10 PQ Node Metric=20 Spirent SR Domain Protected Path TestCenter Link Failure Metric=10 Post-convergence Link 2 Path SRLG UTStarcom UTStarcom UAR500 Physical Link Network Node UAR500 Metric=10 Metric=10

Figure 9: SR-MPLS TI-LFA Local SRLG Protection UTStarcom UAR500 During the SR-MPLS tests we observed that many vendors could Fast Reroute but not all of them were PQ Node Traffic Generator able to test TI-LFA, and even fewer were able to test SR Domain Emulator/ TI-LFA with SRLG groups. We encourage vendors to Traffic Generator work on these features so we can test more combi- Post-convergence Access Network nations next year. Path Protected path Link Failure SRv6 FRR SRLG Network Node Ixia Physical Link IxNetwork Figure 11: SRv6 TI-LFA Protection Metric=10 Metric=10 Ixia Vendors participating in the SRv6 tests for FRR, TI-LFA IxNetwork and TI-LFA with SRLGS were: • PQ Node: UTStarcom UAR500 UTStarcom UTStarcom • PLR Node: UTStarcom UAR500 UAR500 UAR500 Metric=100 Metric=10 • Egress PE: Ixia IxNetwork, Spirent TestCenter UTStarcom We were glad to test the same set of features for SR- UAR500 MPLS and SRv6, it shows that the gap between the two implementations is closing. PQ Node Traffic Generator SR Domain Emulator/ Traffic Generator Label Switched Path (LSP) Ping/Trace Post-convergence The IETF standard RFC 8287 defines the LSP ping Path Access Network and traceroute method for Segment Routing with Protected path Link Failure MPLS data plane. Similar to conventional LSP ping/ traceroute, the SR fault detection and isolation tools Physical Link Network Node are also based on MPLS echo request and echo Figure 10: SRv6 FRR/LFA Link Protection reply. But SR LSP ping/traceroute include a new TLV type, the Segment ID sub-TLV.

11 MPLS + SDN + NFV World Congress 2018 Multi-Vendor Interoperability Test

On receipt of the sub-TLV carried in an MPLS echo During the test, we observed that vendors have request sent by the sender LSR, the LSR responder different interpretations regarding the sub-TLV type/ needs to check the segment ID obtained from the sub- length in FEC stack TLV for SR LSPs. Some vendors TLV with the local advertised segment ID, to claimed that the standard (RFC 8287) does not say determine if the MPLS echo request has been clearly whether to consider the reserved octets to be forwarded from the correct path. The LSP ping/ part of the Length or not. After the interop event, a traceroute response is carried in a MPLS echo reply. technical errata request was raised at IETF by one of First, we verified that a SR sender can initiate an LSP the participating vendors to clarify the length of the ping/traceroute request to a target SR responder, Sub-TLVs. which responded with an LSP ping/traceroute reply to the SR sender.

Setup 1: LSP Ping/Traceroute

Nokia 7750 SR-7 Cisco Huawei ASR 9000 N40E-M2K

Nokia 7750 SR-7

Huawei N40E-M2K Nokia Cisco 7750 SR-7 ASR 9000

Nokia 7750 SR-7

Setup 2: Ping

Nokia 7750 SR-7 Huawei Nokia N40E-M2K 7750 SR-7

Cisco ASR 9000

Ping/Traceroute SR/IGP Domain Physical Link Network Node Link Failure

Figure 12: LSP Ping/Traceroute

Vendors participating in the LSP ping/traceroute tests were: • Ingress Node: Cisco ASR 9000, Huawei N40E- M2K, Nokia 7750 SR • P Node: Cisco ASR 9000, Huawei N40E-M2K, Nokia 7750 SR • Egress PE: Cisco ASR 9000, Huawei N40E-M2K, Nokia 7750 SR

12 Ethernet Virtual Private Network

Ethernet Virtual Private This test verifies the EVPN functionality over MPLS data plane which is based on Segment Routing. Network In the SR domain, IS-IS with SR extensions were used as the IGP protocol. Multi-Vendor A/A Site for an EVPN In this scenario, we run 4 different vendor combi- MPLS VLAN-Based Service nations. In all of them, Cisco IOS XRv9000 was acting as Route Reflector. Ethernet Virtual Private Network (EVPN) provides After checking the IGP and the Segment Routing separation between the data plane and control information, we verified that the Routes Type 1, 2, 3, plane, which allows the use of different encapsu- and 4 were correctly imported. lation mechanisms in the data plane such as MPLS Finally, we sent unicast end-to-end traffic using Ixia and Virtual Extensible LAN (VXLAN). IxNetwork and Spirent TestCenter and observed no packet loss. For multi-homed sites an additional CE device was used to setup a Link Aggregation Group (LAG) to the Cisco PE nodes. For this role either Nokia’s virtual switch, Nokia IOS XRv9000 within 7750 SR-7 chassis, or Arista’s 7280SR-48C6 7750 SR-7 was used. Ixia We verified the aliasing functionality in Active-Active Nokia IxNetwork multi-homed sites and that the Non-Designated 7750 SR-7 Cisco Forwarder (NDF) was blocking remote’s site BUM NCS 5500 traffic. For the PE roles we tested the following devices: Arista 7280SR-48C6, Cisco NCS 5500, Cisco ASR Cisco 9000, Juniper MX104 and Nokia 7750 SR-7 in Arista IOS XRv9000 Arista multi-homed site configuration; and Ixia IxNetwork 7280SR- 7280SR- performing PE emulation in a single-homed 48C6 48C6 Arista configuration. The detailed vendor combinations are Arista 7280SR- depicted in Figure 13. 7280SR- Juniper Arista 48C6 48C6 MX104 7280SR-48C6 Ethernet Line (E-Line) The MEF defines a point-to-point Ethernet service as Ethernet Line (E-Line). The IETF now proposes a Cisco solution framework in order to support this service in Arista IOS XRv9000 Arista MPLS networks using EVPN. The IETF discussed the 7280SR- 7280SR- features in the “VPWS support of the EVPN” draft 48C6 48C6 Arista and requires the use of VPWS to meet the E-Line Arista 7280SR- requirements. In addition, EVPN also supports 7280SR- Cisco Arista 48C6 inherited functions to make the VPWS implemen- 48C6 ASR 9000 7280SR-48C6 tation more effective. In the test, we setup a point-to-point connection between two given PEs as depicted in Figure 14. We configured an EVPN instance between each pair Cisco and enabled VPWS inside EVPN instances. Arista IOS XRv9000 Arista 7280SR- 7280SR- Due to time constraint issues, some participating 48C6 48C6 Arista vendors only tested in single-homing mode. During Arista 7280SR- such test, we verified that the ESI field was set to zero and that the Ethernet Tag field mapped to the 7280SR- Nokia Arista 48C6 VPWS identifier, both of which were carried in the 48C6 7750 SR-7 7280SR-48C6 EVPN AD per EVI route. Ethernet Link LAG The following vendors participated in this scenario: Cisco NCS 5500 (dual-homed), Huawei NE9000-8 Access Network CE Node (single-homed), Juniper MX104 (single-homed) and MPLS/SR Network Nokia 7750 SR-7 (dual-homed) PE routers. Additio- Leaf Node MP-BGP nally, Ixia IxNetwork and Spirent TestCenter acted as emulated PEs and traffic generators. Emulator/ Route Reflector Traffic Generator Cisco IOS XRv9000 was used as route reflector, and a virtual-switch running in Nokia’s 7750 SR-7 router Figure 13: Active-Active EVPN with was used as CE device. MPLS/Segment Routing Transport

13 MPLS + SDN + NFV World Congress 2018 Multi-Vendor Interoperability Test

We also verified that the two PEs exchanged the ESI Leaf label, used to identify BUM traffic generated Spirent from a leaf, as per the RFC8317. TestCenter The following vendors joined this test scenario: Huawei Juniper MX104 and Nokia 7750 SR-7. NE9000-8 Ixia IxNetwork EVPN Enhancements

Cisco ARP Proxy Nokia IOS XRv9000 Within EVPN, PEs advertise MAC/IP addresses, 7750 SR-7 along with an MPLS label, to other PEs using Multi- Juniper Protocol BGP (MP-BGP). Nokia MX104 ARP proxy functionality of EVPN eliminates ARP 7750 SR-7 Cisco flooding within the transport network by advertising NCS 5500 MAC addresses along with their corresponding IP addresses in the MAC/IP advertisement route, type- Ethernet Link E-Line Service 1 2. When a PE receives ARP request from its directly attached hosts, it intercepts the ARP Request and Access Network E-Line Service 2 performs an IP/MAC lookup for the requested IPs. If MPLS/SR Network MP-BGP the lookup is successful, the PE will send an ARP Reply in behalf of the requested IP endpoint. The ARP Leaf Node Route Reflector Request will not be flooded through the EVPN Emulator/ network or any other attachment circuit. If the lookup Traffic Generator CE Node is not successful, the PE will flood the ARP Request in the EVPN network and the other local CEs. Figure 14: E-Line Services with SR-MPLS Transport Setup 1 Ethernet Tree (E-Tree) The MEF defines a rooted-multipoint Ethernet service as Ethernet Line (E-Line). Again, the IETF proposes a Arista Juniper solution for supporting this service in MPLS networks 7280SR-48C6 MX104 by using EVPN. Arista In the setup we verified that the EVPN technology 7050SX2- Arista can support E-Tree functional requirements. Based on 72Q Juniper 7050SX MX240 the current IETF standard (RFC 8317), we tested a root/leaf per AC (attached circuit) scenario. Cisco Juniper Metaswitch Nexus 93180-FX QFX5110- CNRouter Root 48S Leaf IP Infusion Juniper Nokia Leaf Spirent OcNOS TestCenter MX104 7750 SR-7 Spine Root Leaf Ixia Layer ZTE IxNetwork Nokia Juniper Leaf M6000-8S 7750 SR-7 MX104 PLUS Setup 2

EVPN Instance Router IP/MPLS Network Ethernet Link IP Infusion Arista BISDN OcNOS 7050SX2-72Q Basebox Aggregation Network Access Network

Figure 15: E-Tree Service - DC Domain Leaf Node Leaf or Root Sites per AC MP-eBGP Route Server We verified that the MAC addresses learned on a leaf Attachment Circuit where advertised with the Emulator/Traffic Generator expected leaf flag and installed in the remote PE as leaf MAC addresses. Figure 16: EVPN with ARP Proxy

14 Ethernet Virtual Private Network

We performed the ARP proxy tests with two different setups. In the first setup, only symmetric IRB forwar- ding was used. In setup 2 on the other hand, pure L2 Em_Host traffic was forwarded, and routing was not involved Cisco Nokia in the setup. Nexus 7750 SR-7 Thanks to the broad vendor support for ARP Proxy, 93180-FX this year we could setup a typical CLOS topology, Cisco with the spine layer serving as Route-Server for eBGP Nexus 93180-FX sessions for both the underlay routing (IPv4), and the (BGW) overlay (EVPN). Arista 7050SX2-72Q and Juniper QFX5110-48S acted as Route Servers for setup 1. IGMP MP-BGP In the Leaf role for setup 1, as depicted in Figure 16 the following vendors participated: Arista 7050SX2- DC Network Aggregation Network 72Q, Arista 7280SR-48C6, Cisco Nexus 93180-FX, Dual-homed Leaf/BGW Node IP Infusion OcNOS (AS7712-32X), Juniper MX104, Juniper MX240, Metaswitch CNRouter, ZTE ZXR10 Figure 17: IGMP Proxy with EVPN M6000-8S PLUS. For setup 2, the leaf role was fulfilled by BISDN Basebox and IP Infusion OcNOS (AS7712-32X). EVPN Routing Arista 7050SX2-72Q acted as Route Server. Unfortunately there was not enough vendor support EVPN - Integrated Routing and Bridging to test ND Proxy this year. Ethernet VPN Integrated Routing and Bridging (IRB) status is currently “work in progress” at the IETF and IGMP Proxy provides solution for inter-subnet forwarding in data center environments with EVPN. MP-BGP EVPN The goal of IGMP proxy mechanism is to reduce the enables communication between hosts in different flood of IGMP messages (both Queries and Reports) VXLAN overlay networks by distributing Layer 3 in EVPN instances among PE Routers, just like ARP/ reachability information in the form of either a host IP ND suppression mechanism in EVPN reduces the address route(route type-2) or an IP prefix (route flooding of ARP messages over EVPN. type-5). Depending on the required lookup at the Hosts in a VXLAN domain express their interests in ingress or/and egress Network Virtualization Edge multicast groups on a given subnet/VLAN by (NVE), the draft defines two different semantics for sending IGMP membership reports (Joins) for their IRB: Asymmetric and symmetric IRB model. interested multicast group(s). Furthermore, an IGMP While the asymmetric IRB semantic requires both IP router (e.g., IGMPv1) periodically sends membership and MAC lookups at the ingress NVE with only queries to find out if there are hosts on that subnet MAC lookup at the egress NVE, in the symmetric IRB still interested in receiving multicast traffic for that semantic, both IP and MAC lookup are required at group. both ingress and egress NVEs. Furthermore, if there is no physical/virtual multicast We tested using a three-stage Clos topology for all router attached to the EVPN network for a given profiles, also referred to as a “Leaf and Spine” multicast group (*,G), or multicast sender (S,G), it is network (as discussed in RFC 7938). The route desired for the EVPN network to act as a distributed servers acted as spine switches and aggregated a anycast multicast router for all the hosts attached to set of horizontal EVPN PE devices as leaves. A fixed that subnet. number of IPv4 subnets was connected to every leaf In this test Cisco Nexus 93180-FX and Nokia device. Then we configured eBGP in the physical 7750 SR-7 participated as PE routers. We emulated network between spine and leaf devices (as a host behind Cisco’s single-homed PE sending underlay). External-BGP was used between the spine IGMP reports and we observed how this were and leaf to advertise the overlay EVPN routes with a propagated as Route-Type-6 (SMET route) over the VXLAN forwarding plane. EVPN overlay. In this setup, the EVPN-VXLAN was accessed by both The Nexus 9000 Leaf originated the EVPN Route- IPv4 subnets connected to the leaf device whereas Type6 based on the Join received from the Emulated the subnets were emulated by the traffic generators. Host. The Multi-Site Border Gateway (BGW) relayed In all test combinations we verified full-mesh connec- this message and forwarded the SMET-Route to the tivity between all leaves with traffic generators. external EVPN Speaker (Nokia) seamlessly. On the other hand, the current behavior in one vendor implementation is to send encapsulated IGMP messages to feed IGMP Proxy to avoid unnecessary unknown multicast flooding.

15 MPLS + SDN + NFV World Congress 2018 Multi-Vendor Interoperability Test

EVPN - Symmetric Integrated Routing and EVPN - Asymmetric Integrated Routing and Bridging Bridging Setup 1 The following vendors successfully participated in the test as EVPN PE: • Setup 1: Arista 7050SX2-72Q (dual-homed), Spine Arista 7280SR-48C6 (dual-homed), Metaswitch Layer Juniper Arista CNRouter, Spirent TestCenter (STC), QFX5110-48S 7050SX2-72Q • Setup 2: Arista 7050SX2-72Q (dual-homed), Arista 7280SR-48C6 (dual-homed), Ixia IxNetwork, Nokia 7750 SR-7 Nokia Arista 7050SX2-72Q was placed as Spine node 7750 SR-7 acting as Route Server. The setup is depicted in Ixia Figure 19. Arista IxNetwork 7050SX2-72Q Setup 1

Cisco Spine Arista Nexus 93180-FX Layer Arista 7280SR-48C6 7050SX2-72Q

Setup 2

Spine Metaswitch Spirent Arista Arista CNRouter TestCenter Layer Juniper Arista 7050SX2- 7280SR- QFX5110-48S 7050SX2-72Q 72Q 48C6

Setup 2

Juniper Spine Arista MX104 - MX240 Spirent Layer Arista Leaf 7050SX2- TestCenter Layer 72Q 7050SX2-72Q

Ixia IxNetwork Arista 7280SR-48C6 Nokia Ixia Arista Arista 7750 IxNetwork 7050SX2- 7280SR- SR-7 Dual-homed L3 VNI 72Q 48C6 DC Network MP-BGP

Route Server Leaf Node Dual-homed MP-BGP

DC Network Figure 18: EVPN Symmetric IRB Leaf Node Route Server As depicted in Figure 18, the following vendors successfully participated in the test as EVPN PE routers: Figure 19: EVPN Asymmetric IRB • Setup 1: Arista 7050SX2-72Q (dual-homed), Arista 7280SR-48C6 (dual-homed), Cisco Nexus 93180-FX, Ixia IxNetwork, Nokia 7750 SR-7 • Setup 2: Arista 7050SX2-72Q (dual-homed), Arista 7280SR-48C6 (dual-homed), Ixia IxNetwork, Juniper MX104, Juniper MX240 (dual-homed), Spirent TestCenter (STC) Juniper QFX5110-48S and Arista 7050SX-72Q acted as Spine switches and BGP route servers.

16 Ethernet Virtual Private Network

EVPN IP-VRF-to-IP-VRF Interface-full with SBD IRB Model Ethernet VPN Prefix Advertisement draft status is We observed some issues in this test case, as some “work in progress” at the IETF and provides a vendors do not handle the data plane properly when solution for efficient handling of inter-subnet forwar- different VNI value used in Type 2 and Type 5 ding in a data center environment with EVPN. In an Routes, though the VNI of type 5 route is irrelevant in EVPN network environment, there is a requirement this case. for IP prefix advertisement for subnets and IPs For this reason, IxNetwork and Spirent TestCenter in residing behind an IRB interface. This scenario is setup 1 and 2 respectively could only successfully referred to as EVPN IP-VRF-to-IP-VRF. generate traffic to Nokia 7750 SR-7 node. The EVPN prefix advertisement draft provides diffe- rent implementation options for the IP-VRF-to-IP-VRF Setup 1 model: • Interface-less model, where no Supplementary Broadcast Domain (SBD) and overlay index are Ixia Nokia required IxNetwork 7750 SR-7 • Interface-full with unnumbered SBD IRB model, Setup 2 where SBD is required as well as MAC addresses as overlay indexes • Interface-full with SBD IRB model, where SBD is Spirent Nokia required as well as Gateway IP addresses as TestCenter 7750 SR-7 overlay indexes In the test we focused on EVPN-VXLAN for VXLAN Setup 3 data plane provisioned within the data center fabric. External BGP (eBGP) was used for both underlay and overlay NLRI exchanges. The eBGP configura- Juniper Arista tion on the Spine nodes was modified so that the QFX5110- 7050SX2- next-hop attribute was not changed during BGP 48S 72Q update propagation between Leaf nodes. Juniper ZTE ZXR10 M6000-18S For all tests, we verified that the VXLAN virtual MX104 - MX240 network identifier (VNI) was directly mapped to the EVPN EVI. We confirmed that the RT-5 (IP Prefix Juniper Nokia ZTE ZXR10 advertisement route) carried the correct IP Prefix and QFX10002- 7750 SR-7 M6000-8S PLUS length, as well as the corresponding Gateway IP 72Q address (zero in case of the Interface-less model). The route-tables were verified via CLI. Additionally, a Leaf Node L3 VNI RT-2 was used in interface-full mode. It carried MAC address length and MAC address. The IP length was DC Network MP-BGP set to 0. Following, we sent IPv4 test traffic from all Dual-homed Device Pair IPv4 subnets to any other IPv4 subnets, and expected to receive traffic on all IPv4 subnets without any packet loss. Figure 21: Interface-full with SBD IRB Model This test is depicted in Figure 21 and the following Interface-full with Unnumbered SBD IRB vendors participated as PE routers: Model • Setup 1: Ixia IxNetwork, Nokia 7750 SR-7 This test is depicted in Figure 20 and the following • Setup 2: Nokia 7750 SR-7, Spirent TestCenter vendors participated as PE routers: • Setup 3: Juniper MX104-MX240 (multi-homed), • Cisco Nexus 93180-FX, Nokia 7750 SR-7 Juniper QFX10002-72Q, Nokia 7750 SR-7, ZTE ZXR10 M6000-18S, ZTE ZXR10 M6000-8S PLUS Additionally in setup 3, Arista 7050SX2-72Q and Juniper QFX5110-48S acted as Spines/BGP Route Cisco Nokia servers in this scenario. Nexus 93180-FX 7750 SR-7

Leaf Node L3 VNI

DC Network iBGP/eBGP

Figure 20: Interface-full with Unnumbered SBD IRB Model

17 MPLS + SDN + NFV World Congress 2018 Multi-Vendor Interoperability Test

Spine Layer Juniper Arista QFX5110-48S 7050SX2-72Q

Leaf Layer Arista Cisco Juniper Nokia ZTE ZXR10 7050SX2-72Q Nexus 93180-FX MX104 - MX240 7750 SR-7 M6000-18S

Ixia Arista Juniper Spirent ZTE ZXR10 7280SR-48C6 IxNetwork QFX10002-72Q TestCenter M6000-8S PLUS

Physical Link L3 VNI Leaf Node Route Server

DC Network MP-BGP Emulated PE Dual-homed Device Pair

Figure 22: Interface-less Model EVPN IP-VRF-to-IP-VRF

Interface-less Model EVPN Interworking Interface-less model was the most widely supported model to do IP routing with Prefix-advertisement EVPN and IP-VPN Interworking (route type-5) in EVPN overlays. This test is depicted in Figure 22 and the following vendors participated as PE routers: ZTE ZXR10 • Arista 7050SX2-72Q (dual-homed), Arista Nokia 7280SR-48C6 (dual-homed), Cisco Nexus M6000-8S PLUS 7750 SR-7 93180-FX, Ixia IxNetwork, Juniper MX104, (GW) Juniper MX240 (dual-homed), Juniper Ixia QFX10002-72Q, Nokia 7750 SR-7, Spirent Arista IxNetwork TestCenter, ZTE ZXR10 M6000-8S PLUS, ZTE 7050SX2- Arista Cisco ZXR10 M6000-18S 72Q 7280SR- ASR 9000 48C6 Arista 7050SX2-72Q and Juniper QFX5110-48S (GW) acted as route servers. Cisco We did not observe any issues which show mature IOS XRv9000 Arista vendor implementations and clear definition in the 7050SX2- standard procedures. 72Q

Access Network EVPN L3 VNI DC Network VXLAN IP-VPN MPLS Route Server/ Leaf/PE Node Route Reflecor

MPLS/SR Network Emulated PE

Dual-homed Device Pair

Figure 23: EVPN and IP-VPN Interworking - Setup 1

18 Ethernet Virtual Private Network

Nokia Spirent ZTE ZXR10 7750 SR-7 TestCenter T8000-18

Arista Cisco Cisco Huawei Huawei 7050SX2- Arista Nexus 93180-FX Nexus 7702 NE9000-8 CX6608 72Q 7050SX2-72Q (BGW) (EVPN-IP-VN GW) (EVPN-IP-VPN GW)

DC 1 DC 2 Arista Ixia Cisco 7280SR- Nexus 93180-FX 48C6 IxNetwork

Access Network EVPN L3 VNI MPLS/LDP Network Route Server

DC Network VXLAN IP-VPN MPLS Emulated PE Leaf/PE Node

Figure 24: EVPN and IP-VPN Interworking - Setup 2

One of the most important use cases of EVPN is Setup 2 was performed to allow vendors to interconnection of data centers across an IP/MPLS participated performing a different network function core. The goal of this test is to verify an interworking or demonstrate the same capabilities in different use case between a WAN network that is based on product lines. IP-VPN and data center networks that are based on For setup 2 (Figure 24), we tested the following EVPN. vendors: The system must provide control plane and data • EVPN-VXLAN PE Role: Arista 7050SX2-72Q, plane interworking between the EVPN network and Arista 7280SR-48C6, Cisco Nexus 93180-FX, the IPVPN technology supported. Ixia IxNetwork, Huawei CX6608, Nokia 7750 For setup 1 (Figure 23), we tested the following SR-7, Spirent TestCenter vendors: • IP-VPN/MPLS PE Role: ZTE ZXR10 T8000-18 • EVPN-VXLAN PE Role: Arista 7050SX2-72Q, • IP-VPN - EVPN Gateway Role: Cisco Nexus Arista 7280SR-48C6, ZTE ZXR10 M6000-8S 7702, Huawei NE9000-8 PLUS • Border Gateway Role: Cisco Nexus 93180-FX • IP-VPN/MPLS PE Role: Ixia IxNetwork • BGP Route Server Role: Arista 7050SX2-72Q • IP-VPN - EVPN Gateway Role: Cisco ASR 9000, Nokia 7750 SR-7 • BGP Route Reflector Role: Cisco IOS XRv9000 • BGP Router Server/Spine Role: Arista 7050SX2- 72Q

19 MPLS + SDN + NFV World Congress 2018 Multi-Vendor Interoperability Test

SDN Controller Orchestrator Controller Path Comp

Cisco Huawei Cisco Nokia NSO NCE(IP Domain) IOS XRv9000 NSP

Huawei Juniper Arista CX6608 QFX5110 7050SX2-72Q Cisco Nexus 7702 ZTE ZXR10 ZTE ZXR10 T8000-18 M6000-8S PLUS Juniper BISDN Nokia QFX10002 Spirent Basebox 7750 SR-7 Ixia TestCenter IxNetwork Nokia ZTE Cisco ZTE Arista 7750 SR-7 ZXR10 ASR 9000 ZXR10 IP Infusion 7050SX2- M6000-18S Ixia 72Q M6000-5S OcNOS Metaswitch Arista IxNetwork CNRouter 7280SR- ZTE ZXR10 Cisco Nokia Nexus 93180 Huawei 48C6 M6000-18S Arista Juniper 7750 MX240 NE9000-8 SR-7 7280SR-48C6 Huawei NE40E-M2K 12:50:00 Meinberg LANTIME M1000S ZTE Ixia Ericsson IxNetwork Router 6672 ZXR10 Arista Adva M6000-3S 7280SR- FSP150 NEC 48C6 Spirent ProVMe TestCenter Ericsson iPASOLINK VR Huawei Baseband 6620 CX6608 12:50:00 Meinberg LANTIME M4000 Ericsson Ericsson Router 6672 Baseband 6630

Calnex Huawei Calnex Paragon-X CX600-X2-M8A Paragon-T Data Center 1 - VXLAN

12:50:00 UTStarcom Microsemi UAR500 TimeProvider 2300

Segment Routing Clocking Core Router

Data Center Microwave Access Device Physical Link MPLS/LDP Route Reflector/ Microwave Link SDN Controller & Orchestration Route Server

20 Topology

& Orchestration mputation Element (PCE) NETCONF

ZTE Spirent Ixia Huawei ZENIC WAN TestCenter IxNetwork NCE(super) Controller

Spirent Huawei Delta NE9000- TestCenter AGC7648A 8 Ericsson Juniper MINI-LINK MX104 6693 Cisco Cisco NCS5500 Segment Routing NCS 5500 Core Nokia Ericsson MPLS 7750 SR-7 Router 6371 Cisco ASR 9000 ECI Metaswitch CNRouter Cisco Ixia IP Infusion NPT-1800 IOS XRv9000 IxNetwork OcNOS Huawei Arista CX6608 Ericsson 7280SR-48C6 SRv6 UTStarcom Router 6471 Ericsson UAR500 Arista Router 6675 7280SR-48C6 Ixia Spirent IxNetwork TestCenter Juniper 12:50:00 MX80-P Cisco NCS5000 Microsemi TimeProvider 2300

Ericsson 12:50:00 Microsemi MINI-LINK 12:50:00 TimeProvider 4100 6693 Meinberg LANTIME M4000 12:50:00 12:50:00 Ericsson Oscilloquartz Microsemi MINI-LINK OSA5421 HQ++ TimeProvider 2700 6691 Microwave Ericsson Ericsson 12:50:00 MINI-LINK 6366 MINI-LINK 6351 NEC Microsemi iPASOLINK TimeProvider 5000 VR Ericsson MINI-LINK 6352 Microwave Huawei NE40E-X2-M8A Data Center 2 - MPLS

Packet Clock Synchronization

Emulator Synchronous Node Controller Devices Managed by PCE

Phase/Frequency Analyzer 12:50:00 Clock Node Data Center Server Devices Managed by NETCONF

Synchronous Virtual Network Microwave Node Microwave Node Function Dual-homed Leaf

21 MPLS + SDN + NFV World Congress 2018 Multi-Vendor Interoperability Test

Cisco IP Infusion ZTE ZXR10 Spirent Nexus 93180-FX OcNOS M6000-18S Juniper TestCenter MX240 Huawei CX6608 Arista 7280SR- Arista 48C6 Arista Nokia 7750 SR-7 7280SR-48C6 RR 7050SX2-72Q Arista (GW - DF) (GW) 7280SR-48C6

DC 1 DC 2

Arista Metaswitch ZTE ZXR10 Ixia 7050SX2-72Q CNRouter M6000-8S PLUS IxNetwork

Access Network MP-BGP MPLS/SR Network Route Server/Route Reflector

DC Network VXLAN Leaf/PE Node Emulated PE Dual-homed

Figure 25: EVPN VXLAN - SR-MPLS Interworking

EVPN-VXLAN and EVPN-MPLS Interworking The MAC/IP advertisement routes in the VXLAN network segments were carrying the RD of common In data center networks, it is more common to run a EVI, the MAC addresses, the VNI associated with the pure IP network without MPLS support. In those MAC, and VXLAN encapsulation as extended cases, VXLAN is one of the data plane options for community. multi-tenancy. The goal of this test is to verify an interworking use case between a WAN network that In this scenario, we tested the following vendors: is based on EVPN-MPLS and data center networks • EVPN-VXLAN PE Role: Arista 7050SX2-72Q that are based on EVPN-VXLAN. (dual-homed), Arista 7280SR-48C6 (dual- This test focuses on “Integrated Interconnect solution” homed), Cisco Nexus 93180-FX, Huawei

which means the NVO Gateway (GW) and the CX6608, Ixia IxNetwork, Spirent TestCenter, WAN Edge functions are integrated into the same ZTE ZXR10 M6000-18S, ZTE ZXR10 M6000-8S system. This system must provide control plane and PLUS, Metaswitch CNRouter (dual-homed), IP data plane interworking between the EVPN-VXLAN Infusion OcNOS (AS7712-32X) (dual-homed), network and the EVPN-MPLS technology supported • VXLAN-MPLS Gateway Role: Arista 7050SX2- in the WAN. 72Q, Nokia 7750 SR-7 This year we had the opportunity to test the MPLS • Route-reflector/server Role: Arista 7050SX2- section exclusively with Segment Routing, using ISIS 72Q, Juniper MX240 SR extensions. As shown in Figure 25, the devices in the data center network provided EVPN-VXLAN connections. The IP/ MPLS edge routers provided the interconnection between the EVPN-VXLAN network and EVPN-MPLS for the control plane and data plane interoperability. Once we tested the BGP sessions in the underlay and overlay, we checked the BGP routing table on each of the IP/MPLS edge devices (PE). Each of the PE devices received route type-3 from each remote EVPN PE. We generated unicast Ethernet traffic between the sites and started to verify the MAC/IP advertisement (route type-2) routes in each of the network nodes. The MAC/IP advertisement routes in the IP/MPLS network segment were carrying the RD of common EVI, the MAC addresses, and the MPLS label associated with the MAC.

22 Software Defined Networking

Software Defined Networking with the PCE after the session’s restoration and that test traffic was not affected by the PCEP session Service providers aim to increase the agility of their interruption. networks. Adopting centralized network manage- Finally, the PCE deleted the LSP and we verified that ment protocols and application-specific service or- the LSP information on the PCC were deleted. chestration architecture can be instrumental to help Following the LSP’s deletion, test traffic followed the service providers achieve this goal. The following IGP shortest path as expected. Figure 26 depicts two sections describe our Path Computation Element successful participant combinations. Protocol (PCEP) and NETCONF/YANG interopera- bility test descriptions, results and overall inter- ZTE ZENIC operability findings. WAN Controller Path Computation Element Protocol PCEP is defined by the IETF in RFC 5440 as a ZTE mechanism to communicate between a Path Cisco ZXR10 M6000-5S Computation Client (PCC) and a Path Computation ASR9000 Cisco ZTE ASR9000 Element (PCE). PCEP sessions run over TCP and ZXR10 T8000-18 enable PCC and PCE nodes to exchange path Spirent computation requests, responses, session status and TestCenter reports. Traffic engineering paths are computed by the PCE and pushed towards the PCC. Both the PCE and PCC can trigger this computation and provide the path requirements. PCEP can also be used to re- Huawei Cisco NE9000-8 optimize and update existing paths. ASR9000 Nokia In order to demonstrate all interoperable PCEP 7750SR-7 combinations, we designed the tests with a uni- directional LSP per combination. This meant that a Cisco single PCC head-end was sufficient to showcase IOS XRv9000 PCC-to-PCE interoperability. This year, we noticed a growing interest in PCEP’s usage to govern SR-TE paths. Most participating Cisco vendors favored testing PCEP with SR-TE over RSVP- Huawei ASR9000 TE, which we tested thoroughly in 2017. NE9000-8 Ericsson Router6672

PCE-initiated SR-TE Paths in a Stateful PCE Spirent Model TestCenter In environments where the LSP placement needs to change in response to application demands, it is useful to support the dynamic creation and tear Ericsson down of LSPs. Huawei Router6672 In this test, we verified the creation of an SR-TE path NE9000-8 Huawei CX6608 within a single IGP domain when initiated by the PCE. We also verified the state synchronization and deletion of LSPs. Spirent TestCenter The test topology included three network nodes, one of which acted as PCC. Participating vendors chose IS-IS-TE to synchronize the TED information. The LSP was restricted to a suboptimal path to ensure that our Huawei test traffic did not follow the IGP shortest path. Ericsson CX6608 Router6672 Huawei We started the test by verifying the stateful PCEP NE9000-8 session state and TED information on the network nodes. After initiating the LSP from the PCE, we Ixia checked the LSP database on the PCE and ensured a IxNetwork single transport path entry on each PCC. In order to verify state synchronization, we asked the participating vendors to terminate the PCEP session Nokia (by the PCE or PCC), clear the LSP database on the Cisco 7750SR-7 PCE, then re-establish the PCEP session. We verified ASR9000 Cisco ASR9000 that existing LSPs on the PCC were synchronized

23 MPLS + SDN + NFV World Congress 2018 Multi-Vendor Interoperability Test

Spirent Ixia TestCenter IxNetwork

ZTE Huawei ZTE ZXR10 Nokia ZTE NE9000-8 ZXR10 M6000-5S 7750SR-7 Nokia T8000-18 ZXR10 M6000-3S 7750SR-7

Nokia Ixia NSP IxNetwork

Nokia Huawei Cisco 7750SR-7 Metaswitch NE40E-M2K ASR9000 Huawei CNRouter Huawei NE9000-8 NE9000-8

Cisco IOS XRv9000 PCEP Session Optimal LSP Ethernet Link Alternative LSP IS-IS Domain PCE Cisco Network Node Nokia ASR9000 7750SR-7 Huawei NE9000-8 Figure 26: PCE-initiated SR-TE Path Creation and Deletion Nokia NSP It is worth mentioning that current vendor support for bandwidth constraints was limited, hence we excluded that requirement from the test. Also one vendor implementation reported an abnormal LSP Ixia state synchronization process that triggered the Ixia IxNetwork IxNetwork Nokia deletion of the LSP after the PCEP session had been 7750SR-7 restored. Some test combinations were prevented by a Maximum SID Depth (MSD) mismatch. Spirent Finally, we observed that a single vendor implemen- TestCenter ted the TED state synchronization differently which created interoperability challenges and generated error reports during the LSP database re-synchro- nization process. Cisco Huawei ASR9000 NE40E-M2K Huawei PCC-initiated SR-TE Paths in a Stateful PCE NE9000-8 Model

Cisco According to definition, once a PCC has successfully IOS XRv9000 established a PCEP session with a selected PCE node, it can send a path computation request to that PCE (PCReq message) containing the path attributes and constraints. After receiving the path request, the ZTE PCE pushes the computed LSP down towards the ZTE ZXR10 Cisco ZXR10 M6000-5S requesting PCC. This can be helpful for applications T8000-18 ASR9000 that demand certain network conditions to operate. In this test, we verified the SR-TE LSP initiation and Ixia creation process by checking the TED and LSP IxNetwork information on the PCEP peers and sending test traffic over the created tunnel. The traffic was generated from the PCC side towards a secondary Nokia network node via a suboptimal path – defying the Huawei 7750SR-7 IGP shortest path route. Similar to the PCE-initiated NE9000-8 Nokia 7750SR-7 test, vendors chose IS-IS-TE to synchronize the TED information.

24 Software Defined Networking

After the creation of the PCC-initiated LSP, we Cisco verified the delegation of the LSP from the PCC to the IOS XRv9000 PCE by checking the LSP’s “Delegate” flag on the PCEP peers. Finally, we tested the LSP termination and confirmed that the TED and LSP database were cleared. After the termination, as expected our test Cisco Nokia ASR9000 traffic followed the IGP shortest path – as opposed to 7750SR-7 Huawei the suboptimal path. Figure 27 depicts successful NE9000-8 combinations. Nokia Spirent NSP TestCenter

Nokia Cisco 7750SR-7 Huawei ASR9000 Huawei Cisco NE9000-8 NE9000-8 ASR9000 Nokia 7750SR-7 Spirent Cisco TestCenter IOS XRv9000

Nokia Huawei 7750SR-7 Cisco NE40E-M2K Huawei Huawei ASR9000 CX6608 NE9000-8 Ericsson Router6672 PCEP Session Optimal LSP Spirent Ethernet Link Alternative LSP TestCenter IS-IS Domain Network Node PCE

Ericsson Figure 27: PCC-initiated SR-TE Huawei Router6672 Path Creation and Deletion NE9000-8 Huawei CX6608 We noticed a few differences in implementing PCC initiation between vendors. Some vendors used Path Nokia NSP Computation Report – with an empty Explicit Route Object (ERO) versus using PCReq to initiate the LSP by the PCC. Also some implementations delegated the LSP to the PCE by default or left no option for Ixia revoking the delegation. Ixia IxNetwork IxNetwork Nokia 7750SR-7 SR-TE Path Update and Re-optimization in a Ixia PCEP Network IxNetwork The SDN architecture centralizes network manage- ment decisions. The PCE nodes should be able to make localized and holistic service updates accor- Nokia ding to a variety of conditions. Cisco 7750SR-7 In this test we verified the ability of the PCEP peers to ASR9000 Cisco ASR9000 compute and install re-optimized paths when the state of the network changes. Given the wide range Cisco of possible triggers, and differences in supporting IOS XRv9000 those triggers by vendors, we asked participants to use one of three options to trigger path recom- putation: • Increase a link cost on the original path ZTE ZTE ZXR10 • Interrupt a link on the primary path Cisco ZXR10 M6000-5S • Manually trigger path updates on the PCE in case T8000-18 ASR9000 of PCE emulation solutions (i.e. Ixia IxNetwork, Spirent TestCenter) 25 MPLS + SDN + NFV World Congress 2018 Multi-Vendor Interoperability Test

Since this test shares most of the preliminary steps Inter-Domain Segment Routing Traffic with PCE-initiated SR-TE Paths in a Stateful PCE Engineering - BGP-LS and PCE Integration Model 23 and PCC-initiated SR-TE Paths in a Stateful PCE Model 24, we ran the LSP update process as an ZTE intermediate step within those tests whenever ZENIC WAN Controller possible. With the exception of a single test combi- nation, all vendor combinations that are listed in Figure 26 and Figure 27 also took part in this test. In those combinations, we reused the test topology Cisco ZTE where the updated LSPs took the direct path instead ASR9000 ZXR10 Cisco ZTE of suboptimal one as highlighted in the figures. ASR9000 M6000-5S ZXR10 Other successful vendor combinations that partici- M6000-18S pated in this test but were not previously listed are Cisco depicted in Figure 28. IOS XRv9000

ZTE ZENIC WAN Controller

Cisco ZTE ASR9000 ZXR10 ZTE Cisco ZTE T8000-18 ASR9000 ZXR10 ZXR10 Cisco ZTE M6000-3S ASR9000 M6000-5S ZXR10 T8000-18 Cisco Cisco IOS XRv9000 IOS XRv9000

ZTE Cisco Cisco ZTE ZXR10 ASR9000 ASR9000 Cisco Nokia Nokia ZXR10 M6000-5S T8000-18 ASR9000 7750SR-7 7750SR-7

Cisco Nokia IOS XRv9000 Network Services Platform

Cisco Nokia ASR9000 7750SR-7 Huawei Cisco Nokia NE9000-8 ASR9000 7750SR-7 Cisco Nokia Nokia ASR9000 7750SR-7 NSP PCEP Session BGP-LS Ethernet Link Optimal LSP Nokia IS-IS Domain 1 Network Node Huawei 7750SR-7 Cisco NE9000-8 IS-IS Domain 2 PCE ASR9000

Figure 29: BGP-LS and PCE Integration PCEP Session Optimal LSP Ethernet Link Alternative LSP The visibility of topology, Traffic Engineering IS-IS Domain Network Node Database (TED) and Link State Database (LSDB) in an inter-domain network remains local to each PCE domain. To create a TE LSP that transits through two Figure 28: Additional SR-TE or more network autonomous systems, a PCE can Path Update and Re-optimization utilize BGP Link State (BGP-LS). In this test, we asked participating vendors to set up an optimal inter-IGP-domain LSP. We verified PCEP sessions between the PCC and PCE, and that the PCE established BGP-LS with a multi-domain router.

26 Software Defined Networking

We also confirmed that the PCE had the full network NETCONF/YANG topology for both domains. Afterwards, we asked the PCE to create an end-to-end LSP across both Many service providers are investigating zero-touch network domains and verified the LSP on the PCC networks to reduce their service orchestration and nodes. The test topology and vendor roles are operations overhead. Part of the convergence depicted in Figure 29. process is to successfully define and implement a multi-vendor network service. Cisco IOS XRv9000, Nokia Network Services Platform and ZTE ZENIC WAN Controller took the The combination of network configuration proto- PCE role while Cisco-ASR 9000, Nokia 7750 SR-7 cols – such as NETCONF and RESTCONF – with and ZTE ZXR10 M6000-3S participated as PCC. YANG modeling language are potential instruments. The topology differed slightly due to the number of Standardization bodies attempt to define service involved vendors and physical ports availability on catalogues that can be used by different vendors to the network nodes. define the same network service, such as Layer 3 and Layer 2 VPNs. The following two tests cover both use cases. Inter-AS Segment Routing Traffic Engineering L3VPN Service Creation and Termination Another use case of PCEP is to program SR-TE paths across autonomous systems. This can be a valuable In this test, we defined the parameters for a L3VPN tool when application-specific traffic engineering can service. IETF RFC 8299 defines this service model in make use of the AS topology and signal the most YANG. The test expected that a controller will optimal path towards the PCC head-end. translate the service parameters from YANG to vendor-specific NETCONF/YANG configuration In this test, the PCE learned the multi-domain parameters on the network nodes. topology via BGP-LS. In addition, the inter-AS link was modeled using SR Egress Peering Engineering The service models specified the interface para- (EPE) SIDs. The PCE then used this information to meters, virtual routing and forwarding (VRF) instance compute the optimal path and push it towards the and CE-PE routes. The service was created and PCC head-end. After we verified the creation of the terminated using the orchestrator’s northbound inter-AS LSP we shut down one link on the path in interface. Vendors chose MPLS or SRv6 as their order to trigger the recomputation of the path. An preferred transport data plane. alternative path was computed and pushed to the To verify the service creation and deletion we PCC successfully. We verified the label stack on the checked the running configuration on the network PCC and asked the PCE vendor to terminate the LSP. nodes. Following the service creation, we sent traffic The termination was successful and the LSP infor- through the network expected no traffic loss. We also mation was cleared on the PCEP peers. ZTE ZXR10 confirmed that the service orchestration was non- T8000-18 participated as PCC head-end, while intrusive by comparing the configuration on network Cisco IOS XRv9000 took the role of PCE. The full nodes before and after the test. The device topology and the LSPs are displayed in Figure 30. configurations matched with the exception of one case where the device configuration had extra unreadable characters. The responsible vendor Cisco explained that the added characters do not interfere IOS XRv9000 with the device operations. Six vendors successfully participated in this test. Both, Cisco NSO and Huawei NCE acted as ZTE ZXR10 T8000-18 NETCONF/YANG orchestrators. ECI NPT-1800, Cisco Ericsson Router 6471, Metaswitch CNRouter and Cisco ASR9000 UTStarcom UAR500 acted as provider edge. Cisco ASR9000 ZTE NSO exposed its northbound interface using Nokia ZXR10 NETCONF/YANG, while Huawei NCE exposed its 7750SR-7 M6000-5S northbound API using RESTCONF/YANG. Juniper Successful combinations are depicted in Figure 31. MX240 Finally, it is worth mentioning that the YANG data model for L3VPN over SRv6 data plane has not yet PCEP Session BGP-LS been defined in the standard by IETF (indicated as Ethernet Link Optimal LSP work in progress in draft-raza-spring-srv6-yang-01), Alternative LSP so NETCONF behavior for setting up L3VPN service AS 65000 over SRv6 data plane had proprietary implemen- Network Node AS 66000 tation at this stage with provisioning performed by a PCE controller in several steps (transactions).

Figure 30: Inter-AS SR-TE

27 MPLS + SDN + NFV World Congress 2018 Multi-Vendor Interoperability Test

Cisco Cisco Application NSO NSO

Spirent ECI Ericsson Spirent Spirent UTStarcom UTStarcom Spirent TestCenter NPT-1800 Router 6471 TestCenter TestCenter UAR500 UAR500 TestCenter

Cisco Application NSO SRv6 NETCONG/YANG Access Network Test Traffic Network Node Data Link Traffic Generator Orchestrator Spirent Metaswitch ECI Spirent TestCenter CNRouter NPT-1800 TestCenter Figure 32: L2VPN Service Creation and Termination Huawei Application NCE(IP Domain) Multi-Vendor/Multi-Domain Controllers Orchestration The SDN architecture can span multiple network domains. Typically, each domain is controlled by a domain controller. Inter-domain services can be Spirent Huawei Spirent Cisco orchestrated by a multi-domain controller that has a TestCenter NE40E-M2K TestCenter IOS XRv9000 full view of its subdomains. Cisco In this test, we experimented with three management Application NSO protocols: NETCONF, RESTCONF and PCEP. The three combined managed two network domains. The domains were connected directly via two provider edges. We asked the vendors to create and delete an end-to-end L3VPN service modeled in RFC 8299. Spirent UTStarcom UTStarcom Spirent Four vendors took part in this test. Cisco NSO TestCenter UAR500 UAR500 TestCenter managed two Ericsson provider edges via NETCONF/YANG. The other domain contained two Huawei provider edges managed using PCEP by MPLS NETCONG/YANG Huawei’s NCE(IP domain). A single Huawei SRv6 RESTCONF/YANG NCE(Super) instance managed the domain Access Network Test Traffic controllers using NETCONF/YANG and RESTCONF/YANG. Network Node Data Link Figure 33 describes the test topology and Traffic Generator Orchestrator connectivity between the participating components. At the beginning of this test we checked manage- Figure 31: L3VPN Service Creation and Termination ment sessions and configuration information on the network nodes and controllers. We then asked the L2VPN Service Creation and Termination multi-domain controller to trigger the service creation and monitored the north-bound interface on both Another use case of NETCONF/YANG is L2VPN domain controllers. The multi-domain controller re- service modeling and deployment. This test setup modelled the service to two different services using was similar to the previous L3VPN one with a slight RFC 8299 YANG and pushed them into the two difference in the service parameters – such as the domain controllers. Each domain controller trans- Service VLAN. In this test, the service was directly lated the service into device configuration and modelled on the orchestrator due to the lack support pushed them towards the network nodes. To verify for draft YANG models such as the draft-ietf-l2sm- the service creation, we checked the device l2vpn-service-model by the IETF. configuration and sent traffic via the service path. The orchestrator initiated and terminated the L2VPN The multi-domain controller deleted the service service on the PEs successfully. We verified the successfully and the devices configuration was network service by sending traffic between the two cleared successfully. customer sites. Two vendors took part in this test: Test traffic was dropped after the service’s deletion Cisco NSO acted as the orchestrator while as expected. UTStarcom UAR500 acted as PE. SRv6 was chosen by UTStarcom as the transport data plane.

28 Software Defined Networking

Huawei NCE(Super)

Huawei Cisco NCE(IP Domain) NSO

Huawei Ericsson NE40E-M2K Router 6471 Huawei Ericsson CX6608 Router 6371

NETCONF/YANG Ethernet Link RESTCONF/YANG L3VPN PCEP Session Network Node Controller/Orchestrator IS-IS Domain

Figure 33: Multi-Vendor/Multi-Domain Controller Orchestration

Virtual CPE Integration with NETCONF NETCONF can be extended to manage virtualized resources such as the virtual customer premises equipment (vCPE). Adva took part in this test with two FSP150 ProVMe devices, connected back-to-back, to illustrate the WAN integration of vCPE components. On each of the two Adva units, a virtual router from a third party vendor was instantiated and configured manually.

Cisco NSO

Spirent Adva Adva Spirent TestCenter FSP150 FSP150 TestCenter ProVMe ProVMe

Customer/DC NETCONF White Box Virtual Router Virtualization Layer Data Link Test Traffic

Figure 34: Virtual CPE Integration with NETCONF

The successful configuration was verified by sending user data across the two hybrid CPEs and their virtual routers using Spirent TestCenter. Subsequently, NETCONF sessions were established between each vCPE virtual router and the Cisco NSO. Due to lack of time for test execution, further application of NETCONF/YANG was not covered.

29 MPLS + SDN + NFV World Congress 2018 Multi-Vendor Interoperability Test

Microwave Modulation Channel Participants Scheme Spacing As operators upgrade their networks with LTE-A in preparation for a 5G future, questions are being Ericsson 4096 QAM 112 MHz asked around the role Microwave transport will play. MINI-LINK 6691 We see a trend of integrating the more specialised Ericsson microwave devices into the standard IP/MPLS router 4096 QAM 112 MHz domain, and thus looked into two particular aspects MINI-LINK 6366 of this in the following tests. Ericsson 4096 QAM 112 MHz MINI-LINK 6651 Bandwidth Notification NEC 2048 QAM 56 MHz Today Mobile Backhaul networks are often built as iPASOLINK-VR an overlay with routers sitting on top of microwave devices. In the past there was limited communication Table 2: Modulation and Channel Spacing Used between these two domains, but with the bandwidth notification messages (ETH-BN) defined by ITU-T Layer 3 Microwave MPLS-based Y.1731, it is now possible for the microwave systems Services to signal a change in bandwidth to the routers. The aim of this test was to confirm the capability to This enables a router to apply service policies to the establish IP/MPLS service on a microwave platform traffic it sends on to the microwave system based on crossing or terminating on existing infrastructure. the bandwidth information within the ETH-BN We tested two different combinations relying on packets. different transport profiles, and verified that a L3VPN At the beginning of this test, the Microwave nodes service can be set up between IP/MPLS capable were using the maximum modulation possible, as microwave systems and IP/MPLS aggregation depicted in Table and sent end-to-end traffic. In the routers in multi-vendor scenario. next step we emulated severe weather conditions in the link between the microwave nodes by using a RF In the first scenario we used IS-IS as the IGP protocol attenuator and verified that the aggregation router and LDP for the MPLS label allocation/distribution. In could process the bandwidth notification messages the second we changed the IGP to OSPF with LDP. (ETH-BN) and accordingly apply service policies to We created both end-to-end services between two the traffic sent to the microwave system, based on different microwave vendors with standalone routers the bandwidth information within the ETH-BN. participating as aggregation router, as well as directly between two microwave vendors. In the tests, Ericsson MINI-LINK 6366, Ericsson MINI- LINK 6691 and NEC iPASOLINK VR microwave nodes acted as PE/P nodes. Juniper MX80 NEC Ericsson participated as P node in the combination which was iPASOLINK VR Router using L3VPN with ISIS and LDP. In another 6672 combination with only Ericsson MINI-LINK 6691 and NEC iPASOLINK VR, we tested the L3VPN with OSPF and LDP. Ericsson Ericsson Ericsson MINI-LINK MINI-LINK Router 6366 6691 6672 Ericsson Ericsson Juniper NEC Microwave/ Aggregation MINI-LINK MINI-LINK MX80 iPASOLINK VR P/PE/Node Router 6366 6691 Microwave Link Access Network Ethernet Link IP/MPLS Network Ericsson Ericsson NEC ETH-BN RF Attenuator MINI-LINK MINI-LINK iPASOLINK VR 6366 6691 Figure 35: Bandwidth Notification Microwave/ Aggregation We successfully tested the following combinations: PE/P/Node Outer Ericsson Router 6672 acted as the aggregation router in both combinations. The microwave link was Microwave Link Access Network established between two NEC iPASOLINK VR and Ethernet Link IP/MPLS Network an Ericsson MINI-LINK 6366 and MINI-LINK 6691. Figure 36: Layer 3 Microwave MPLS Based Services

30 Microwave

Layer 3 Microwave Transport Resiliency Bringing IP/MPLS to the microwave network provides additional resiliency options in the access network and increases the end-to-end service availability. The goal of this test was to show that a microwave node can react to degradation of the radio link by re-routing the traffic via a different path. We used Spirent TestCenter to act as CE and send bidirectional traffic across the network. We verified that the microwave nodes were using the main path with the maximum modulation scheme available as depicted in Table and that no packets were lost. We then emulated severe weather conditions by reducing the available bandwidth of the channel with a RF attenuator. In this test, Ericsson MINI-LINK 6691 and NEC iPASOLINK VR acted as P nodes, Ericsson MINI-LINK 6366 and NEC iPASOLINK VR acted as the microwave stations and PE nodes. Ericsson MINI- LINK 6651 and NEC iPASOLINK VR participated as resiliency nodes. Juniper MX80 acted as PE router.

Ericsson NEC MINI-LINK 6366 iPASOLINK VR

NEC Ericsson iPASOLINK MINI-LINK VR Ericsson 6691 NEC MINI-LINK iPASOLINK VR 6651

NEC NEC Juniper iPASOLINK iPASOLINK MX80 VR VR NEC iPASOLINK VR

Ericsson Ericsson Juniper MINI-LINK MINI-LINK MX80 6366 6691 Ericsson MINI-LINK 6651

Microwave Node Access Network PE/P Node Microwave Link IP/MPLS Network Ethernet Link Backup Traffic Path Main Traffic Path RF Attenuator

Figure 37: Layer 3 Microwave Transport Resiliency

31 MPLS + SDN + NFV World Congress 2018 Multi-Vendor Interoperability Test

Clock Synchronization The Calnex Paragon-X was used to provide the PTP impairments. The measurements for phase output of In this year’s event we focused on time/phase the devices under test was done either with the delivery, including a lot of resiliency scenarios, using Calnex Paragon-X or the Calnex Paragon-T. full and assisted partial timing support setups, as In one additional setup of this test case we planned well as PTP deployments at network edges for legacy to use a transparent clock instead of the boundary networks supporting only Sync-E. clock, but in this case the slave did not manage to We tested the behavior of the time signal delivery in lock to the grandmaster while using the impairment. optimal and suboptimal conditions: network delay asymmetry, hold-over performances, source failover Calnex between two grandmaster clocks. GPS We defined the accuracy level of ±1.5 μs (ITU-T Paragon-X recommendation G.8271 accuracy level 4) as our end-application goal, with 0.4 μs as the phase 12:50:00 budget for the air interface. Therefore, the require- Microsemi ment on the network limit, the last step before the Meinberg Calnex Oscilloquartz TimeProvider OSA5421 end-application, had to be ±1.1 μs. LANTIME Paragon-X 2300 M1000S The primary reference time clock (PRTC) was GPS using an L1 antenna located on the roof of our lab. Calnex The synchronization test team was really prolific this Paragon-T year, with above 40 successful combinations, willing to test brand new software versions, products and interface types, including PTP over 100 GbE. Microsemi Our tests helped to discover several small issues but TimeProvider the R&D departments of the vendors reacted quickly 4100 12:50:00 Ericsson Calnex Meinberg providing patches and troubleshooting support. Router 6675 Paragon-X LANTIME Oscilloquartz M4000 Phase/Time Partial Timing Support OSA5421 This test was performed using only the ITU-T Slave Boundary Grandmaster G.8275.2 profile (PTP telecom profile for Phase/ Clock Clock Clock Time-of-day synchronization with partial timing support from the network), without any physical 1 PPS Link PTP 8275.2 frequency reference – such as SyncE. Clock Link — ToD Synchronous In this setup the grandmaster clock was provided Node with GPS input, while the slave and boundary clock Frequency/ started from a free running condition. Phase Analyzer Reference Clock In the first step, we enabled PTP on the boundary Impairment Tool clocks while the Calnex Paragon-X was emulating a PSN PDV according to the profile defined in G.8261 test Microwave Node Grandmaster case 12. In the combination including Meinberg 12:50:00 Clock LANTIME M4000 as T-GM and Ericsson Router 6675 as T-BC (as well as all the other occurrences of Figure 38: Phase/Time Partial Timing Support this combination in the current document), the boundary clock did not manage to lock to the Phase/Time Assisted Partial Timing grandmaster clock using this attenuation profile. For this reason, we agreed to use an attenuated version Support of this impairment, where all parameters were This test was performed using the ITU-T G.8275.2 reduced by 50%. profile between the grandmaster and boundary After the boundary locked to the grandmaster clock, clock, with the participants having the choice of we let the slave clock also lock to the boundary clock running G.8275.1 or G.8275.2 between boundary via PTP and verified that the phase accuracy and and slave clocks. frequency of the ITU-T G.823 SEC mask require- Both the grandmaster and boundary clocks were ments were satisfied. connected to GPS and the slave clock locked via PTP We successfully tested the following combinations: to the boundary clock. We then started the impair- Meinberg LANTIME M4000 and Oscilloquartz ment which emulated a PDV according to the profile OSA5421 HQ++ participated as grandmaster defined in G.8261 test case 12. Some combinations clock, Ericsson-Router 6675 and Meinberg LANTIME used an attenuated version of this impairment, where M1000S as boundary clock and Microsemi all parameters were reduced by 50%. TimeProvider 2300, Microsemi TimeProvider 4100, Upon disconnecting the GPS from the T-BC-P we and Oscilloquartz OSA5421 HQ++ as slave clock. verified that it switched to PTP and confirmed that the

32 Clock Synchronization

output met the phase accuracy requirement of ±1.1 μs and frequency requirements. In the last step we Calnex GPS reconnected the GPS antenna to the T-BC-P and Paragon-X repeated the measurement. We successfully tested the following combinations: Ericsson Router 6471, Meinberg LANTIME M4000, 12:50:00 Calnex Microsemi TimeProvider 4100 and Oscilloquartz Microsemi Meinberg Oscilloquartz Paragon-X OSA5421 HQ++ participated as grandmaster TimeProvider LANTIME OSA5421 clock, Ericsson-Router 6675, Ericsson MINI-LINK 2300 M1000S 6651, Ericsson MINI-LINK 6366, Meinberg Calnex LANTIME M1000S and Oscilloquartz OSA5421 Paragon-X HQ++ participated as boundary clock, Oscillo- quartz OSA5421 HQ++, Microsemi Time Provider 2300, Microsemi TimeProvider 4100 Huawei NE40E-M2K, Huawei NE40E-X2-M8A and Ericsson Ericsson Baseband 12:50:00 Baseband 6630 participated as slave clock. 6630 Calnex Ericsson The Calnex Paragon-X was used to provide PTP Meinberg Paragon-X Router 6471 impairments. The measurements for phase output of LANTIME M1000S the devices under test was done either with the Huawei Calnex Paragon-X or the Calnex Paragon-T. NE40E-X2-M8A GPS GPS Calnex Calnex Paragon-X Paragon-T

Huawei Oscilloquartz 12:50:00 NE40E-M2K OSA5421 Ericsson Calnex Meinberg 12:50:00 Router 6675 Paragon-X LANTIME Ericsson Oscilloquartz Calnex Microsemi Microsemi M4000 Router 6371 OSA5421 Paragon-X TimeProvider TimeProvider 4100 4100 Meinberg Slave Boundary Grandmaster LANTIME Clock Clock Clock M1000S GPS Calnex Slave Boundary Grandmaster Paragon-X Clock Clock Clock

1 PPS Link PTP 8275.1 Clock Link — ToD PTP 8275.2 12:50:00 Frequency/Phase Sync-E Ericsson Ericsson Meinberg Calnex Oscilloquartz Analyzer Synchronous MINI-LINK MINI-LINK LANTIME Paragon-X OSA5421 Node 6651 6366 M1000S Impairment Tool Reference Microwave Node Slave Boundary Grandmaster Clock Clock Clock Clock Grandmaster PSN 12:50:00 Clock 1 PPS Link PTP 8275.1 Figure 40: Phase/Time Assisted Clock Link — ToD PTP 8275.2 Partial Timing Support Frequency/Phase Synchronous Analyzer Node In one combination we observed a problem with the Impairment Tool Reference boundary clock: the APTS backup path did not reach Clock the ready/locked state while using both ITU Microwave Node G.8275.1 and 8275.2 profiles at the same time PSN downstream towards different clients. In another Grandmaster 12:50:00 Clock combination we observed that the boundary clock was acting as T-GM and thus did not increment the Figure 39: Phase/Time Assisted steps. Removed value and did not show the T-GM Partial Timing Support parent ID to the slave clocks. The vendor explained that since the T-BC standardization process is still ongoing, this feature is not implemented yet.

33 MPLS + SDN + NFV World Congress 2018 Multi-Vendor Interoperability Test

Phase/Time Assisted Partial Timing Calnex Support: Delay Asymmetry Paragon-T GPS This test was performed using the ITU-T G.8275.2 profile between the grandmaster and boundary

clock, with the participants having the choice of 12:50:00 running G.8275.1 or G.8275.2 between boundary Oscilloquartz Ericsson Calnex Meinberg and slave clocks. OSA5421 Router 6675 Paragon-X LANTIME M4000 Calnex GPS Calnex Paragon-X Paragon-X

12:50:00 12:50:00 Ericsson Ericsson Meinberg Calnex Oscilloquartz Microsemi Meinberg Calnex Oscilloquartz MINI-LINK MINI-LINK LANTIME Paragon-X OSA5421 TimeProvider LANTIME Paragon-X OSA5421 6651 6366 M1000S 2300 M1000S

12:50:00 Huawei Calnex NE40E-X2-M8A Ericsson Ericsson Microsemi Meinberg 12:50:00 Paragon-X MINI-LINK MINI-LINK TimeProvider LANTIME Meinberg Calnex Ericsson 6651 6366 2700 M4000 LANTIME Paragon-X Router 6471 Ericsson M1000S Slave Boundary Grandmaster Baseband Clock Clock Clock 6630

1 PPS Link PTP 8275.1 Calnex Paragon-T Clock Link — ToD PTP 8275.2 Frequency/Phase PSN Analyzer Synchronous NEC Impairment Tool Node iPASOLINK VR 12:50:00 Oscilloquartz Calnex Microsemi Reference Microwave Node OSA5421 Paragon-X TimeProvider Clock 4100 Grandmaster Clock Huawei 12:50:00 NE40E-M2K Figure 41: Phase/Time Assisted Partial Timing Support: Delay Asymmetry Slave Boundary Grandmaster Clock Clock Clock Both the grandmaster and boundary clocks were connected to GPS. We started the impairment 1 PPS Link PTP 8275.1 emulating a PDV according to the profile defined in Clock Link — ToD PTP 8275.2 G.8261 test case 12. Some combinations used an Frequency/Phase Sync-E attenuated version of this impairment, where all Analyzer Synchronous parameters were reduced by 50%. Impairment Tool Node After disconnecting the GPS from the boundary Reference clock, we used the Calnex Paragon-X to introduce an Microwave Node Clock additional delay asymmetry of 125 μs and verified Grandmaster that the boundary could calculate and compensate PSN 12:50:00 Clock the asymmetry introduced. We successfully tested these combinations: Ericsson Figure 42: Phase/Time Assisted Partial Router 6471, Meinberg LANTIME M4000, Micro- Timing Support: Delay Asymmetry semi TimeProvider 4100 and Oscilloquartz OSA5421 HQ++ participated as grandmaster The Calnex Paragon-X was used to provide PTP clock, Ericsson Router 6675, Meinberg LANTIME impairments. The measurements for phase output of M1000S, Microsemi TimeProvider 2700 and the devices under test was done either with the Oscilloquartz OSA5421 HQ++ participated as Calnex Paragon-X or the Calnex Paragon-T. boundary clock, Ericsson MINI-LINK 6651, Ericsson In this test we observed different behaviors of the MINI-LINK 6366, Ericsson Baseband 6630, Huawei devices acting as boundary clock, depending on the NE40E-X2-M8A, Huawei NE40E-M2K, Microsemi internal oscillator quality. Upon inserting the delay TimeProvider 2300, NEC iPASOLINK VR and asymmetry, some T-BCs kept the lock to the Oscilloquartz OSA5421 HQ++ participated as grandmaster clock, while others went in holdover slave clock. mode (ClockClass 160) while measuring the 34 Clock Synchronization

asymmetry introduced, preferring the internal In this setup, both grandmasters were provided with oscillator time. As soon as the calibration was a GPS signal from a common GPS antenna. We performed, they reverted to ClockClass 6. allowed the boundary clock to lock to the primary grandmaster and then degraded the primary Phase/Time Synchronization: grandmaster’s quality by disconnecting its GPS Source Failover input. We verified that the boundary clock switched over to the secondary grandmaster and measured Calnex the slave clock’s transient response. We also tested if GPS Paragon-T the correct clockClass values are being signalled by the grandmasters according to the telecom profiles, which allows the alternate best master clock UTStarcom algorithm running on the boundary clock to correctly select the best grandmaster during each step of the UAR500 12:50:00 tests. Oscilloquartz We used the priority2 field as tie-break parameter. Microsemi OSA5421 We successfully tested the following combinations: TimeProvider Ericsson Baseband 6620, Meinberg LANTIME 2300 Ericsson Router 6675 12:50:00 M4000, Meinberg LANTIME M1000S, Microsemi Meinberg TimeProvider 5000 and Oscilloquartz OSA5421 Huawei LANTIME HQ++ participated as grandmaster, Ericsson Router NE40E-X2-M8A M4000 6675, Huawei NE40-M2K and NEC iPASOLINK VR participated as boundary clock, Huawei NE40E-X2- M8A, Meinberg M1000S, Microsemi TimeProvider NEC 2300, NEC iPASOLINK VR, Oscilloquartz iPASOLINK VR OSA5421 HQ++ and UTStarcom UAR500 participated as slave clock. Calnex Paragon-X Furthermore two combinations were using 100 GbE 12:50:00 links: in the connection between Ericsson Router Meinberg 6675 acting as boundary clock and Huawei NE40E- LANTIME X2-M8A and UTStarcom UAR500 acting as slave NEC M1000S clocks. iPASOLINK VR Huawei In one failed combination (not shown in this report) NE40E-M2K 12:50:00 we observed an issue related to the ptpTimeScale Oscilloquartz Ericsson flag set to TRUE, while the currentUTCOffset was set OSA5421 Baseband to 36s and the currentUTCOffsetValid was FALSE. 6620 The PTP standard section 8.2.4.2 a) states that in case of ptpTimescale TRUE the currentUTCOffset 12:50:00 shall be obtained from the primary reference (GPS in Meinberg Meinberg this case). Therefore we would have expected the LANTIME LANTIME UTC offset to be 37s and the valid flag set to TRUE. M1000S M4000 This issue led the T-BC not to lock to the grandmaster NEC clock. iPASOLINK VR 12:50:00 Huawei Microsemi CX600-X2-M8A TimeProvider 5000 Slave Boundary Grandmaster Clock Clock Clock

1 PPS Link PTP 8275.1 Clock Link — ToD Sync-E Frequency/Phase Synchronous Analyzer Node Impairment Tool Reference Clock Microwave Node PSN Grandmaster Clock 12:50:00

Figure 43: Phase/Time Synchronization: Source Failover

35 MPLS + SDN + NFV World Congress 2018 Multi-Vendor Interoperability Test

Phase/Time Synchronization with GPS full Timing Support: Microwave Calnex Transport Paragon-T We started the test with the slave clock in free running mode and generated a constant bit rate at Huawei the maximum line rate for the maximum modulation NE40E-X2-M8A scheme (10% of 576 byte packets, 30% of 64 byte 12:50:00 Meinberg packets, 60% of 1518 byte packets) and expected Ericsson LANTIME no traffic loss. MINI-LINK Meinberg M4000 After the slave clock locked, we performed baseline 6352 LANTIME measurements using the Calnex Paragon-T device. To M1000S emulate severe weather conditions, we reduced the bandwidth between the two nodes of the microwave network using an RF attenuator. As expected the Huawei 12:50:00 nodes reacted by changing the modulation used. CX600-X2-M8A Ericsson Meinberg We then verified that the PTP traffic was unaffected LANTIME by the change of modulation, as it was prioritized MINI-LINK 6651 & 6366 M4000 over other data traffic and the slave clock output Meinberg retains the required quality level. Since the band- LANTIME width decreased accordingly, we saw that data M1000S packets were dropped according to the available bandwidth. Ericsson In the first setup, the microwave stations acted as 12:50:00 Router 6371 boundary clocks, while in the second setup they NEC Microsemi were acting as transparent clocks. iPASOLINK VR TimeProvider We successfully tested the following combinations: 5000 Meinberg LANTIME M4000 and Microsemi Time- Microsemi Provider 5000 participated as grandmaster clock, TimeProvider Ericsson MINI-LINK 6352, Ericsson MINI-LINK 6651 4100 and NEC iPASOLINK VR participated as boundary Slave Boundary Grandmaster clock, NEC iPASOLINK VR participated as a Clock Clock Clock transparent clock, Ericsson Router 6371, Huawei CX600-X2-M8A, Huawei NE40E-X2-M8A, Calnex GPS Meinberg LANTIME M1000S and Microsemi Time- Paragon-X Provider 4100 participated as slave clock.

Ericsson MINI-LINK 6651 12:50:00 Meinberg NEC Huawei LANTIME iPASOLINK VR NE40E-M2K M4000 Slave Transparent Grandmaster Clock Clock Clock

1 PPS Link PTP 8275.1 Clock Link — ToD PTP 8275.2 Frequency/Phase Sync-E Analyzer Synchronous Impairment Tool Node Reference Microwave Node Clock Grandmaster PSN 12:50:00 Clock RF Attenuator

Figure 44: Phase/Time Synchronization with full Timing Support: Microwave Transport

36 Clock Synchronization

Phase/Time Synchronization: Calnex GPS Degradation of Primary Source Paragon-X According to the architecture defined in ITU-T G.8275 a boundary clock can become a grand- master and can also be slaved to another PTP clock. 12:50:00 The goal of this test was to check the capability to Meinberg Ericsson swap the role of a boundary clock’s port from master NEC LANTIME iPASOLINK VR Router 6675 to slave and vice-versa. M1000S This test was performed using the ITU-T G.8275.1 profile. Both the grandmaster and one of the boundary 12:50:00 clocks (BC-A) were provided with a GPS signal. We Meinberg Huawei Ericsson allowed the grandmaster and the boundary clock A LANTIME NE40E-X2-M8A Router 6675 to lock to GPS input. The boundary clock A acted as M1000S primary grandmaster for the upstream boundary clock (BC-B).

We then disconnected the antenna of the boundary 12:50:00 clock A to emulate a GPS failure and verified that Meinberg Ericsson Microsemi both boundary clocks locked via PTP to the central LANTIME Router 6371 TimeProvider grandmaster. In the last step, we recovered the GPS M1000S 4100 of the boundary clock A and verified that the boundary clock B locked again to the downstream boundary clock A. 12:50:00 We successfully tested the following combinations: Meinberg Oscilloquartz Microsemi Ericsson Router 6675 and Microsemi TimeProvider LANTIME OSA5421 TimeProvider 4100 participated as grandmaster clock, Ericsson M1000S 4100 Router 6371, Huawei CX600-X2-M8A and NEC iPASOLINK VR and Oscilloquartz OSA5421 HQ++ participated as boundary clock B, Meinberg 12:50:00 LANTIME M1000S participated as boundary clock Meinberg Ericsson Microsemi A. The link between Ericsson Router 6675 and LANTIME MINI-LINK TimeProvider 6651 & 6693 Huawei NE40E-X2-M8A was based on 100 GbE M1000S 4100 interface. In other failed combinations (not shown in the picture), the devices acting as BC-A were not able to 12:50:00 swap the operation mode of the port from Master to Meinberg Ericsson Microsemi Slave, as this function was not implemented. LANTIME MINI-LINK 6352 TimeProvider M1000S 5000

12:50:00 Meinberg Huawei Microsemi LANTIME NE40E-M2K TimeProvider M1000S 5000 Boundary Boundary Grandmaster Clock A Clock B Clock

1 PPS Link PTP 8275.1 Clock Link — ToD PTP 8275.2 Frequency/Phase Sync-E Analyzer Synchronous Impairment Tool Node Reference Microwave Node Clock Grandmaster PSN 12:50:00 Clock

Figure 45: Phase/Time Synchronization: Degradation of Primary Source

37 MPLS + SDN + NFV World Congress 2018 Multi-Vendor Interoperability Test

Time/Phase holdover with Sync-E In the upstream between the boundary and the support in the core grandmaster clock only Sync-E was available, while the ITU-T G.8275.1 was used in the downstream link GPS between boundary and slave clock. Calnex In the first step we allowed the slave clock to gain a Paragon-X stable lock, then we emulated a GPS failure of the antenna connected to the boundary clock. We then measured that the phase accuracy require- ment of ±1.1μs is fulfilled for at least 30 minutes Meinberg while in hold-over. LANTIME 12:50:00 We successfully tested the following combinations: M1000S Ericsson Oscilloquartz Meinberg LANTIME M4000 and Oscilloquartz Router 6471 OSA5421 OSA5421 HQ++ participated as grandmaster Huawei clock, Ericsson Router 6471, Meinberg LANTIME NE40E--M8A M1000S and Oscilloquartz OSA5421 HQ++ participated as boundary clock, Huawei NE40E-X2- M8A, Meinberg LANTIME M1000S and Microsemi 12:50:00 TimeProvider 4100 participated as slave clock. Microsemi Meinberg Oscilloquartz In one additional combination (not shown in the TimeProvider LANTIME OSA5421 picture) we observed that after GPS has been 4100 M1000S unplugged, the T-BC went into holdover-within-spec, clockClass 135 as expected, but after only 7 minutes changed to holdover-out-of-spec, clockClass 165. In Ericsson this situation the slave clock lost it PTP lock to the MINI-LINK boundary clock.

6651 12:50:00 Oscilloquartz Meinberg LANTIME Summary Huawei OSA5421 M4000 NE40E-M2K It was a pleasure for the whole EANTC team to have 21 dedicated vendors with us in the lab in Berlin. During two intensive weeks, they made great Slave Boundary Grandmaster progress and thus jointly advanced the industry. Clock Clock Clock Kudos to all of them and we are looking forward to see what 2019 will bring! 1 PPS Link PTP 8275.1 Clock Link — ToD PTP 8275.2 Frequency/Phase Sync-E Analyzer Synchronous Impairment Tool Node Reference Microwave Node Clock Grandmaster PSN 12:50:00 Clock Figure 46: Time/Phase Holdover with Sync-E Support in the Core

In scenarios where it is not possible to deploy PTP in the core network, it is common to have one or more “distributed grandmaster clocks” provided with GPS at the edges of the network. The goal of this test case was to verify the holdover performance of a boundary clock in relation to phase/time stability while GPS is unavailable and a Sync-E is used as backup to GPS. In this setup, both the grandmaster and the boundary clocks were provided with a GPS signal from a common GPS antenna.

38

EANTC AG Upperside Conferences European Advanced Networking Test Center

Salzufer 14 54 rue du Faubourg Saint Antoine 10587 Berlin, Germany 75012 Paris - France Tel: +49 30 3180595-0 Tel: +33 1 53 46 63 80 Fax: +49 30 3180595-10 Fax: + 33 1 53 46 63 85 [email protected] [email protected] http://www.eantc.com http://www.upperside.fr

This report is copyright © 2018 EANTC AG. While every reasonable effort has been made to ensure accuracy and completeness of this publication, the authors assume no responsibility for the use of any information contained herein. All brand names and logos mentioned here are registered trademarks of their respective companies in the United States and other countries. 2018 v5