Considerations, Best Practices and Requirements for a Virtualised Mobile Network CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
About the GSMA Network 2020 Contents The GSMA represents the interests of mobile operators The GSMA’s Future Networks Programme is designed to worldwide, uniting nearly 800 operators with almost 300 help operators and the wider mobile industry to deliver all-IP companies in the broader mobile ecosystem, including handset networks so that everyone benefits regardless of where their and device makers, software companies, equipment providers starting point might be on the journey. and internet companies, as well as organisations in adjacent industry sectors. The GSMA also produces industry-leading The programme has three key work-streams focused on: events such as Mobile World Congress, Mobile World Congress The development and deployment of IP services, The 5.1.4 NFV carrier-grade reliability scope 24 Shanghai, Mobile World Congress Americas and the Mobile 360 evolution of the 4G networks in widespread use today, 1 Introduction 3 Series of conferences. The 5G Journey developing the next generation of mobile 1.1 Scope 4 5.2 C arrier Grade Service – Requirements 26 technologies and service. 1.2 Definitions 4 for NFV Reliability For more information, please visit the GSMA corporate website at www.gsma.com. Follow the GSMA on Twitter: @GSMA. For more information, please visit the Network 2020 website 1.3 Abbreviations 5 5.3 NFV Reliability Assessment Standard 26 at: www.gsma.com/network2020 1.4 References 6 5.4 Best Practices 28 5.4.1 Use case scenarios 28 2 NFV reference architecture 7 5.5 Conclusion 29
3 Requirements for Network Orchestrators 9 6 Migration 31 3.1 Introduction 10 6.1 Introduction 32 3.1.1 Network Services 10 6.1.1 Migration from physical network to 32 3.1.2 Virtualised Network Functions 11 virtualised network 3.1.3 Network Functions Virtualisation 11 6.1.2 Software upgrades 32 With thanks to contributors: Infrastructure 6.1.3 Interoperability 32 AT&T 3.2 NFV Orchestrator functional requirements 11 6.2 Best practices 32 China Mobile China Telecom 3.3 Other NFV related architectures 12 6.2.1 Migration 32 China Unicom 3.4 Best Practices 14 6.2.2 Software upgrades 33 KDDI 3.4.1 Planning on NFV based network 14 KT 6.3 Requirements 34 LG Uplus management and orchestration 6.3.1 Migration 34 architecture NTT DoCoMo 6.3.2 Software upgrades 34 Orange 3.4.2 Example of 5G Telco-MANO architecture 15 Proximus America Movil 7 Performance benchmark for NFV 35 4 Virtualised network security 17 SK Telecom infrastructure Telecom Italia 4.1 Introduction 18 7.1 Performance benchmark scope 36 Telefónica 4.1.1 Maturity 18 Telekom Austria 7.2 State of the art in NFV performance 37 Telenor Group 4.2 Architectural Challenges 18 benchmarking TeliaSonera 4.3 Best practices 19 T-Mobile US 4.4 Conclusions 19 8 Vertical interoperability 41 United States Cellular 8.1 Introduction 42 Vodafone 4.5 Actions 19 Ascom Network Testing 8.2 Requirements for vertical interoperability 42 Cisco 5 Carrier grade NFV/ Reliability 21 8.3 Challenges and potential risks 42 Ericsson 5.1 Introduction 22 Gemalto 8.4 Best Practices 43 HP 5.1.1 Carrier-grade 22 Huawei 5.1.2 T he essential value of carrier-grade 22 Nokia reliability Oracle America Samsung Electronics 5.1.3 NFV carrier-grade reliability challenges 23 ZTE CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK 1 Introduction
2 3 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
1.1 Scope 1.2 Those are Definitions Resource orchestration Subset of Network Functions ETSI European Telecommunications Virtualisation Orchestrator Standards Institute This document outlines the key considerations in Term Description functions that are responsible ETSI GS ETSI Group Specification the deployment of network virtualisation in a mobile Lifecycle management Set of functions required to for global resource management network environment. The topics covered within manage the instantiation, governance FCAPS Fault, Configuration, Accounting, maintenance and termination of a Performance and Security represent solutions to the potential obstacles mobile Service lifecycle Set of phases for realizing a Virtualised Network Function or service from conception and GS-O Global Service Orchestration operators may face when wishing to capitalize on Network Service identification to instantiation and ICT Information and Communications network virtualisation (covering both Network Functions Network controller Functional block that centralizes retirement Technology Virtualisation and Software-Defined Networking). some or all of the control and Software-Defined Networking A set of techniques that enables management functionality of a IoT Internet of Things to directly program, orchestrate, network domain and may provide This document provides an overview of the steps control and manage network IPC Instructions Per Cycle an abstract view of its domain to resources, which facilitates the mobile operators should take to adopt this technology other functional blocks via well- iTLB/dTLB instruction Translation Lookaside design, delivery and operation and where appropriate, provide an indication of defined interfaces Buffer / data Translation of network services in a dynamic Lookaside Buffer what they will need to complete the work and which Network Function Functional block within a and scalable manner network infrastructure that has ISB Industry Standard Benchmarking external organisations are best placed to deliver it. SDN Orchestration A process that oversees and well-defined external interfaces directs a set of software-defined ISG Industry Specification Group and well-defined functional The document also outlines a number of examples networking activities and behaviour KPI Key Performance Indicator and approaches that have been taken by operators interactions with the objective of Network Functions Principle of separating Network carrying out certain work in an KVM Kernel-Based Virtual Machine to identify and address the gaps. These best practice automated manner. Virtualisation Functions from the hardware L1/L2/LLC Level 1/Level 2/ Last Level Cache examples are provided as guidance and do not they run on by using virtual Virtualised Infrastructure Functional block that is hardware abstraction Open-O Open Orchestrator Project represent the consensus of the GSMA members Manager responsible for controlling and participating to this activity. Network Functions Functional block that manages managing the Network Functions OPEX Operating Expense Virtualisation Infrastructure Virtualisation Orchestrator the Network Service lifecycle OPNFV Open Platform for NFV and coordinates the management compute, storage and network of Network Service lifecycle, resources, usually within one OS Operating System Virtualised Network Function operator’s Infrastructure Domain OSM Open Source MANO project lifecycle and NFV infrastructure Virtualised Network Function Implementation of an Network to ensure an optimized allocation OSS Operations Support Systems Function that can be deployed on of the necessary resources and a Network Function Virtualisation OVS Open vSwitch connectivity Infrastructure PGW Packet Data Network Gateway Network Functions Totality of all hardware and Virtualised Network Function Internal component of a Virtualisation Infrastructure software components that build PNF Physical Network Function Component Virtualised Network Function up the environment in which providing a defined sub-set RAN Radio Access Node Virtualised Network Functions of that Virtualised Network are deployed RFC Request For Comments Function’s functionality. Network service orchestration Subset of Network Functions MANO Management and Orchestration Virtualisation Orchestrator MME Mobility Management Entity functions that are responsible 1.3 Abbreviations for Network Service lifecycle MVNO Mobile Virtual Network Operator management Term Description NextGen Next Generation API Application Programming Network service descriptor Template that describes the NF Network Function deployment of a Network Service Interface NFV Network Functions Virtualisation including service topology ATIS The Alliance for as well as Network Service Telecommunication Industry NFV-O Network Functions Virtualisation characteristics such as Service Solutions Orchestrator Layer Agreements for the Network Service on-boarding BSS Business Support Systems NFVI Network Functions Virtualisation Infrastructure and lifecycle management of its CAPEX Capital Expenditure instances NOC Network Operations Centre CNI Critical Network Infrastructure Network Service Composition of Network NS Network Service Functions and defined by its CPU Central Processing Unit SDN Software Defined Networking functional and behavioural DPDK Data Plane Development Kit specification SDN-O Software Defined Networking DUT Device Under Test Orchestration Type of composition where one Orchestration ECOMP Enhanced Control, Orchestration, particular element is used by SHV Standard High Volume the composition to oversee and Management & Policy SGW Serving Gateway direct the other elements EOSL End of Service Life UMO Unified Management and Quota Upper limit on specific types of EPC Evolved Packet Core resources Orchestration EM Element Management vACL virtual Local Area Network Access Lists
4 5 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
vCG-NAPT virtual Concatenation Group [17] GS ETSI NFV ETSI GS NFV 001: “Network Function Network Address Port 001 Virtualisation (NFV); Use Cases” Translation [18] GS ETSI TST ETSI GS NFV-TST 001: “pre- vEPC virtual Evolved Packet Core 001 deployment testing, report on validation of NFV environments and vFW virtual Fire Wall services” vMME virtualised Mobility Management [19] GS ETSI TST ETSI GS NFV-TST 002: “Testing Entity 002 Methodology; Report on NFV vPE virtual Provider Edge Interoperability Testing Methodology” vSwitch virtual Switch [20] GS ETSI PER ETSI GS NFV-PER 001: “NFV 001 performance and portability best VIM Virtualised Infrastructure practices” 2 Manager [21] GS ETSI IFA ETSI GS NFV-IFA 003 V2.1.1: “Network VM Virtual Machine 003 Functions Virtualisation (NFV); VNF Virtualised Network Function Acceleration Technologies; vSwitch benchmarking and acceleration VNFC Virtualised Network Function specification” NFV reference Component [22] draft-ietf- https://datatracker.ietf.org/doc/draft- bmwg-virtual- ietf-bmwg-virtual-net/ 1.4 References net-04 [23] draft-ietf- IETF: draft-ietf-bmwg-ipsec-term-12. architecture Ref Doc Number Title bmwg-ipsec- txt, “Terminology for Benchmarking [1] GS ETSI NFV Network Functions Virtualization term-12 IPsec Devices”. 002 (NFV); Architectural Framework [24] draft-ietf- IETF: draft-ietf-bmwg-ipsec-meth-05. [2] GS ETSI NFV- Network Functions Virtualisation bmwg-ipsec- txt, “Methodology for Benchmarking MAN 001 (NFV); Management and Orchestration meth-05 IPsec Devices”. [3] GS ETSI NFV- Network Functions Virtualisation [25] draft-ietf- IETF: draft-vsperf-bmwg-vswitch- IFA 009 (NFV); Management and bmwg- opnfv-01.txt, “Benchmarking Virtual Orchestration; Report on Architectural vswitch- Switches in OPNFV”. Options opnfv-01 [4] GS ETSI NFV- Network Functions Virtualisation [26] draft-kim- IETF: draft-kim-bmwg-ha- IFA 010 (NFV); Management and bmwg-ha- nfvi-01.txt, “Considerations for Orchestration; Functional nfvi-01 Benchmarking High Availability of NFV requirements specification Infrastructure”. [5] ITU-T Y.3300 Framework of software-defined [27] GS ETSI NFV- ETSI GS NFV-REL001 “Network networking REL001 Functions Virtualisation (NFV); [6] ISO/IEC Reference Architecture for Service Resiliency Requirements” 18384-1 Oriented Architecture (SOA RA) Part 1: [28] ETSI NFV-SEC ETSI GS NFV-SEC 007 “Network Terminology and Concepts for SOA 007 Functions Virtualisation (NFV); Trust; [7] NGMN 5G white paper Report on Attestation Technologies and Practices for Secure Deployments” [8] AT&T Ecomp Enhanced Control, Orchestration, Management & Policy Architecture; [29] ETSI GS NFV- ETSI GS NFV-SEC 013 “Network http://about.att.com/content/dam/ SEC 013 Functions Virtualisation (NFV) Release snrdocs/ecomp.pdf 3; Security; Security Management and Monitoring specification” [9] Open-O Open-O architecture; www.open-o.org [30] ETSI GS NFV- ETSI GS NFV-SEC 012 “Network [10] OSM Open Source MANO; http://osm.etsis. SEC 012 Functions Virtualisation (NFV) Release org/ 3; Security; System architecture [11] ETSI GS NFV- Network Function Virtualisation (NFV); specification for execution of sensitive REL006 V0.0.3 Reliability; Specification on Software NFV components” (2016-07) Update Process [31] ETSI TS 103 ETSI TS 103 308 “CYBER; Security [12] Yardstick https://wiki.opnfv.org/display/ 308 baseline regarding LI and RD for NFV yardstick and related platforms” [13] Bottlenecks https://wiki.opnfv.org/display/ bottlenecks [14] StorPerf https://wiki.opnfv.org/display/storperf [15] VSPerf https://wiki.opnfv.org/display/vsperf/ VSperf+Home [16] CPerf https://wiki.opnfv.org/display/cperf
6 7 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
Figure 1 shows the NFV reference architectural and The NFV Orchestrator is responsible for the functional blocks [1]. The functional blocks are: management and orchestration of NFV infrastructure, • Operations Support Systems (OSS) and Business as well as the software resources and delivering the Support Systems (BSS). network services on the NFV infrastructure. The ISG ETSI NFV [3] identifies the following management and • Element Management (EM); orchestration data repositories: • Virtualised Network Function (VNF); • VNF Catalogue; • Service, VNF and Infrastructure Description; • NS Catalogue; • VNF Manager(s); • NFV Instances repository; • NFV Orchestrator; 3 • NFVI Resources repository • Virtualised Infrastructure Manager(s) (VIM); • NFV Infrastructure (NFVI), including hardware and The NFV Orchestrator has two main responsibilities: virtual compute, storage and network resources • Network Service Orchestration: managing the Requirements for lifecycle of network services. This includes the A VNF is an implementation of a Network Function management of the Network Services templates and that can be deployed in an NFV Infrastructure. VNF Packages. The Network Service Orchestration Examples of mobile network functions are Mobility provides the management of the instantiation of Network Orchestrator Management Entity (MME), Serving Gateway (SGW), VNFs, in coordination with VNF Managers; Radio Access Node (RAN) and Packet Data Network • Resource Orchestration: providing an overall view of Gateway (PGW). the resources to which it provides access and hides the interfaces of the VIMs present below it
Figure 1: NFV architecture Lifecycle management of Network Services
Management & Orchestration
NFV OSS/BSS Orchestrator Service, VNF and Infrastructure Description EM EM EM EM NFV Manager(s) VNF VNF VNF VNF VNF
Virtualised NVF Infrastructure (NVFI) Infrastructure Manager(s)
Management of the services Control management of Lifecycle management provided by the NF the virtualised resources of VNF instances
Source: GS ETSI NFV-002 Network Function Virtualization (NFV); Architecture Framework
8 9 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
3.1 Introduction 3.1.1 Network Services 3.1.2 Virtualised Network Functions 3.2 NFV Orchestrator functional requirements Network Functions Virtualisation requires a new set of The Network Service Orchestration is responsible for The management and orchestration elements of a VNF The Network Orchestrator shall fulfil all requirements management and orchestration functions to be added the Network Service lifecycle management including include fulfilment, assurance and security management, specified in clause 6 of ETSI NFV-IFA 010 [5] to the current model of operations, administration, operations such as: however, the focus in NFV is the decoupling of the VNF applicable to an NFVO. These requirements include maintenance and provisioning. The Network Functions • On-board and management of Network Service software from the hardware. The decoupling of Network functional requirements for VNF lifecycle, Network Virtualisation Management and Orchestration has the Descriptors; Functions from the physical infrastructure results in a Service lifecycle and virtual resource management new set of management functions that are focused on role of managing the NFVI and control the allocation of • Instantiate Network Service; summarized as follows: the creation and lifecycle management of virtualised resources needed by the NSs and VNFs. • Network Service: • Scale Network Service; resources for the VNF, referred to as VNF Management. The management and orchestration takes place at • Update Network Service; VNF Management functions are responsible for the VNF’s • lifecycle management (instantiation, scaling, updating, termination); three different levels as depicted in Figure 2. • Terminate Network Service lifecycle management including operations such as: • information management (NS descriptor); • Instantiate VNF (create a VNF instance using the • performance management; VNF on-boarding artefacts); • fault management • Scale VNF (increase or reduce the capacity of the Figure 2: Management & Orchestration at different abstraction levels VNF); VNF: • Update and/or Upgrade VNF (support VNF • lifecycle management (instantiation, scaling, software and/or configuration changes of various termination); complexity); • information (VNF package) management; NS • Terminate VNF (release VNF-associated NFVI VNF • configuration management (initial configuration and resources and return it to NFVI resource pool) NVF Orchestrator (NFVO) update); VNF 3.1.3 Network Functions Virtualisation Infrastructure • performance management; VNF Network Functions Virtualisation Infrastructure • Indicator management (NFVI) resources under consideration are both • fault management virtualised and non-virtualised resources, supporting virtualised network functions and partially virtualised Virtualised resource management: network functions. VNF • direct/indirect, reservation; VNFC Virtualised resources are offered for use through • capacity, fault, performance; VNFC VNF Manager (VNFM) abstracted services, for example: • network forwarding path; • Network, including: networks, subnets, ports, • information; VNFC addresses, links and forwarding rules, for • quota, allowance the purpose of ensuring intra- and inter-VNF connectivity; Other requirements are also considered for: • Compute including machines, and virtual machines, • Multi-Tenancy (Tenant management); as resources that comprise both CPU and memory; • Infrastructure resource management; • Storage, including: volumes of storage at either Virtualised Infrastructure • Software image management; block or file-system level Manager (VIM) • NFV acceleration management; • Security consideration; Virtual Switches SDN Storage machines & Routers controllers • Policy administration
Source: Orange
10 11 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
3.3 Other NFV related architectures In the case of the OSM project, as shown in Figure In NFV reference architecture, the VIM can play 4, the SDN controller is a well-defined part of the Figure 4: OSM architecture mapping onto ETSI NFV framework elements the role of an SDN application, sitting on top of the resources managed by the Resource Orchestrator, northbound interfaces of the SDN controller. The rather than accessing it exclusively through the VIM VIM delegates to the SDN controller the connectivity interface. This provides finer control of network management of virtualised network resources that configuration, decouples resource and cloud-native GUI & Design-Time Tools OSS/BSS are required by network services and their constituent lifecycle management and supports multi-VIM services network functions. This architecture does not propose at the Resource Orchestrator level. a direct interface between the NFV-O and SDN controllers. However, in some architecture (e.g. AT&T Network Service Orchestrator ECOMP architecture), a Master Orchestrator for NFV EMSs and SDN acts as an NFV-O and as a direct consumer of the APIs exposed by an SDN controller. Specific NFV Management and Orchestration VNFMs
Figure 3: ECOMP orchestration architecture VNFs Resource Orchestrator Ve-Vnfm VNF Configuration PNFs (Includes VIM/SDN & Abstraction Management & Orchestration Connectors)
OSS/BSS NFV Orchestrator NFVI
Application Nf-Vi Controller ODL EM EM EM EM OpenVIM OpenStack OpenVIM
Floodlight VNF VNF VNF VNF VNF Network Controller Main NFV reference points
Infrastructure Source: OSM Project NVF Infrastructure (NVFI) Cloud Controller
In another architectural approach, the SDN Orchestrator can be separated from the NFV Source: ECOMP www.openecomp.org Orchestrator (e.g. Open-O architecture) to manage the SDN service connectivity independently from NFV (e.g. for VPN on Demand use case). In this approach, a Global Service Orchestrator GS-O is placed on the top of the SDN-O and NFV-O orchestrators.
12 13 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
Figure 5: Open-O orchestration architecture Figure 6: possible NFV based network management and orchestration architecture GS-O
GUI Portal UMO (Unified Management Legacy OSS Abstract NBI and Orchestration)
Service Service Service Lifecycle Mgr. Decomposer Formation
EMS (Legacy) EMS (New) VNFM SDN-O NFV-O
GUI Portal GUI Portal
Abstract NBI Abstract NBI TAS CSCF SBC vTAS vCSCF vSBC SDN Lifecycle Mgr. SDN Monitor NVF Monitor PNFs VNFs VPN NS Lifecycle Mgr. SDN Res. Mgr. Traffic Optimise NVF Res. Mgr. VIM Source: China Mobile Abstract SBI Abstract SBI
SDN ACCESS/WAN SDN EMS/NMS VIM SDN NVF SDN VNFM VIM Driver Controller Drivers Driver Drivers Driver Controller Drivers Driver Drivers The UMO is a combination system that has the 3.4.2 Example of 5G Telco-MANO architecture capability of network orchestration, VNF and NS ETSI’s Network Functions Virtualisation (NFV) lifecycle management. It also covers policy design Industry Specification Group (ISG) is defining NFV- and management, resource management and VNF MANO architecture. This is a framework for the Source: Open-O www.open-o.org FCAPS management. management and orchestration of all resources in the cloud computing infrastructure as well as NS and VNF The benefit of this architecture is that the role and lifecycle management. 3.4 Best Practices responsibilities of legacy OSS and UMO are clear, 3.4.1 Planning an NFV based network management separate and defined. The UMO can be considered Some network functions are hard to transform to and orchestration architecture a unified system designed to manage the whole virtualised network functions (VNFs) and will remain Figure 6 shows an example of an architecture for a virtualised network. All the information needed for as physical network functions (PNFs). future NFV based network management strucuture a mobile network service, VNF and NS resource from an implementation based on China Mobile’s management will be handled by the UMO only, which These hybrid physical and virtual network network. In this architecture, the complete network has no impact on the legacy network. environments should be considered for a 5G Telco management system can be divided into two parts. management and orchestration architecture. 5G Telco- The right section is a sub-system managing VNFs MANO is required to provide end-to-end network under UMO (Unified Management and Orchestration). provisioning for 5G network slicing on demand and The UMO is China Mobile’s NFV based network one-view network monitoring instead of monitoring management and orchestration system. The left physical and virtual network(s) separately. 5G Telco- section shows the legacy OSS: in order to avoid having MANO should also be easy to implement. to reconstruct the existing sub-system, the legacy OSS will stay unchanged and will continue to manage existing PNFs. An interface between legacy OSS and UMO is not precluded and could be a proprietary interface, based on operators’ requirements.
14 15 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
Figure 7 shows an example of a 5G Telco-MANO In addition to NFV-MANO defined interfaces, Figure architecture. 6 shows interfaces between an E2E orchestrator and domain controllers as well as between domain In this architecture, NMS-Assurance is used for end- controllers and EMS that are necessary for the 5G to- end network assurance. NMS-Assurance is the Telco MANO architecture to work. most widely used network assurance system in PNF networks and can be extended as a virtual networks By following this approach, a 5G Telco-MANO, end- assurance system with the help of EMS and VNF. Thanks to-end on-demand network provisioning and one- to the reuse of NMS- Assurance, end-to-end network view network assurance in physical and virtual hybrid assurance can be easily implemented to avoid the networks can be achieved. complexity of reconstructing a system from scratch. 4 E2E orchestrator and domain controllers are new elements designed to support end-to-to-end network provisioning and configuration. In each network domain, Virtualised a network controller (for example, Transport-SDN, Cloud SDN) will provision and configure the network function with an open interface regardless of whether they are physical or virtualised network functions, with network security the interfacing of an E2E orchestrator. In this way, an E2E orchestrator can provision a 5G network slice using both PNFs and VNFs end-to-end.
Figure 7: Possible end-to-end management and orchestration architecture for 5G
Customer Portal/OSS/BSS
E2E Infra Manager
NMS E2E Orchestrator SON (E2E Assurance) (E2E Provisioning/Control
Repository NFV-O
Domain Controller EMS VNFM (Adaptation)
EMS
VNF PNF (vCore-C, vCore-U..) (AU, OLT...) VFVI VIM
Source: KT
16 17 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
4.1 Introduction 4.2 Architectural Challenges 4.3 Best practices 4.4 Conclusions NFV can greatly amplify existing security problems This section lists the security aspects that The GSMA Fraud and Security Group’s (FASG) • Security for NFV is still a work in progress in terms of impact. The vulnerabilities are similar operators deploying a virtualised network need preliminary discussions on NFV security highlighted 5 • Security involves all NFV architecture components to those of today, but instead iNFV concentrates to take into account. key security risks operators need to take into account: (VNF level, NFVI, MANO) them in one place and increases the likelihood of • Legal and Regulatory Compliance failure: Core risks • Integrity of infrastructure and VNF • LI community requires the strongest requirements a common mode failure. In many ways, it puts all are about geographic deployment and security of • LI Requirements / Security for LI Function will our “security eggs in one basket”. In traditional • Several Trust Domains: solutions, particularly in heterogeneous environments; telecommunications equipment, a number of factors depend on National Regulations / Countries –– A Trust Domain is a collection of entities that • Isolation Failure: Core risk is around functions helped to frustrate would be attackers such as physical • Still quite difficult to establish the right level of share security policies “escaping” from allocated resources, enabling security, proprietary software, hardware, installation requirements for RFI/RFP –– Even in the context of a single CSP: prejudicial data access to extensive areas of and configuration and reduced the ability to exploit –– a dedicated trust domain for LI in addition to information (at rest, in motion or in memory); vulnerabilities. 4.5 Actions an admin trust domain • Denial of Service: Risks include flooding of public 4.1.1 Maturity –– Regulation geographical constraints with location interface or resource exhaustion (e.g. on internal • All vendors must be able to credibly answer (with No vendor is at an advanced stage when it comes to attestation ensuring that LI can only take place network or memory); evidence) a set of basic questions around the security of their NFV products in accordance with compliance with national security regulations regarding on known POI/IAP • Topology Validation and Enforcement: Risk is ETSI TS 103 308 [30] NFV. A number of interested parties are working hard • Separation of LI function executions from other VNF around malicious configuration (e.g. VM layer with standardisation bodies to include and embed and Hypervisor is considered a requirement misconfiguration); • Operators should consider a delay in the use of security from the ground up. We do not anticipate NFV where sensitive functions are involved, unless –– Segregation of Lawful Interception information/ • Security Logging and Incident Management: seeing mature NFV products/ solutions implementing appropriate mitigations are available flows. Monitoring behaviour of infrastructure Risks arise from failures to log or manage issues the latest security standards (e.g. ETSI NFV SEC) should not decrease the confidentiality of a from complex interactions across domains • Government regulatory advice should provide in the near future. Therefore, any product currently Lawful Interception solution greater clarity, now, on the potential security risks available today is highly unlikely to have the required FASG also elaborated on mitigation strategies for the around NFV, and be able to legally and competently –– Encrypted targets DB in a separate/secured security built in. Indeed, many NFV vendors are new above risks highlighting the importance of a strong deal with this topic to telecommunications security concepts. The cloud context cooperation between operators, manufacturer and technologies being used to underpin NFV are not of –– Secure Encryption where appropriate regulators. Three recommendations the maturity required to underpin Critical Network –– a location to store “properly” keys ensuring that are made: Infrastructure (CNI) and iits related obligations. they can’t be compromised by e.g. a privileged • Vendors shall be required to adopt best-of-breed hypervisor manager or other process. The main threat vectors for a virtual network remain security features as specified by ETSI NFV SEC and fundamentally unchanged: –– Secure execution operators: • Loss of availability: Attacks that result in crashing –– Location attestation means a virtual network element or rendering it unusable –– Root of Trust (NFV SEC 007 [27]). through flooding/denial of service; –– Multiple Trust Domains (NFV SEC 013 [28]). • Loss of confidentiality: Either caused by –– Execution Enclaves for Sensitive Applications eavesdropping or leakage of sensitive data; (NFV SEC 012 [29]) • Loss of integrity: Resulting from a modification of • The open-source community needs to be motivated data during transit (man-in-the-middle-attack) to include basic security measures. A “fixing as we go or in the virtual network element, as well as from along” approach is not suitable for this environment; unauthorised access to a virtualised network function; • Everyone needs to develop a secure architecture • Loss of control: Loss of control can take place at with separate zones of trust the network level where the attacker controls the network exploiting a protocol or implementation flaw or at virtual function level Another threat that needs to be considered in virtualised networks is the possible absence of a single entity overseeing the whole network. This is because a virtual network (Figure 1) can be easily subdivided in to different administrative domains each with different elements requiring the establishment of secure interaction between themselves.
18 19 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK 5 Carrier grade NFV/ Reliability
20 21 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
5.1 Introduction 5.1.2 The essential value of carrier-grade reliability 5.1.3 NFV carrier-grade reliability challenges When service providers refine their plans to introduce 5.1.1 Carrier-grade Telecom networks must be always-on and guarantee With all the industry initiatives around NFV, network NFV into their networks, they are also reviewing In telecommunications, “carrier-grade” refers to a a level of service because society, business and reliability has become a hot topic, as shown in a survey the implications for high reliability on a network system, or hardware or software component that is industries increasingly depend on reliable connections published by Heavy Reading in Jan 2015, as well as infrastructure that incorporates virtualised functions. extremely reliable, well tested and proven. Carrier for both routine and critical communications. some challenges experienced by the industry when There are numerous initiatives underway currently grade systems are tested and engineered to meet or implementing NFV: Always-on reliability is mandatory and telecom service to specify, align and promote NFV carrier-grade exceed the “five-9s” (99.999%) availability standards, • Service recovery in a multi-layer or multi-vendor providers have built their networks, reputations and capabilities for achieving efficient carrier-grade that provide resiliency and fast fault recovery. environment revenue streams on a foundation of carrier-grade NFV solutions. These include ETSI NFV Proof of • Carrier-grade reliability when NFV has components Product or service development within the reliability. A carrier-grade network guarantees a ‘five- Concept, Open-Platform for NFV project, Carrier with a lower grade of reliability telecommunications industry has traditionally followed 9s’ availability standard, that allows for no more than Network Virtualisation Awards, ATIS (The Alliance for rigorous standards for stability, protocol adherence 5.27 minutes of downtime per year per service. If there • New skills and process requirements Telecommunication Industry Solutions) and various and quality, reflected by the use of the term ‘carrier- is just one failure in the system, the NOC (Network • Multi-component interoperability management supplier ecosystems. Operations Centre) will be notified and proceed with grade’ to designate equipment demonstrating this • Security concerns reliability. Over time, telecom service providers have remediation in less than 5 minutes so that the five-9s • Complexity in E2E problem demarcation engineered an extensive range of sophisticated target can still be met. Hence there is an overwhelming features into their networks, to the point where they need to automate the entire process, including quick can guarantee their high reliability. service recovery processes and to provide a seamless transfer from the failing element to healthy elements. However, it is difficult to achieve the same grade of high reliability in each component of the NFV network. What This level of service is typically required by high-value Figure 9: Challenges faced by the NFV Testing Market is important for service providers is to be able to offer enterprise customers who often will pay a premium for carrier grade services that meet the required availability, Service Level Agreements that specify high availability. reliability and performance requirements for end-to-end Carrier-grade scalability and robustness 27.2% voice, video, data or converged-services. Seamless control and provisioning of physical and virtual networking infrastraucture 23.7% Openness and interoperability 17.1%
Real-time and dynamic provisioning 13.5% Figure 8: The benefits of Carrier-grade NFV for Operators NFV global reach and cross administration 10.3%
Other 8.2%
0% 5% 10% 15% 20% 25% 30% Reduce Improve Cost of Poor Customer Source: Heavy Reading Mobile Networks Insider Quality Churn
Benefits Increase Protect O&M Efficiency Revenue
Enhance Brand Image
22 23 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
5.1.4 NFV carrier-grade reliability scope 1. Each product should be highly reliable and fault 3. Most of the challenges are in network maintenance. Practically, there are 3 key areas in achieving NFV-driven resilient. Different types of resiliency can be applied Even if a redundant network is deployed, failures carrier-grade reliability: product, network deployment such as “1+1” or “N+M” (simple 1+1 redundancy can happen. Therefore, it is critical to follow a (design and integration) and network maintenance. is deprecated as a recommended strategy proactive, predictive and pre-emptive approach for resilience). For stateful applications, data that identifies incidents before they impact users. synchronization and restoration mechanisms should This can be achieved using extensive telemetry be in place for state continuity in case of component to monitor the different components in the NFV Figure 10: End-to-End Carrier-grade Reliability Scope failure. Furthermore, to respond to a failure of a network. Moreover, this complexity and scale physical or virtual element within an NFV platform, requires fault conditions to be identified by network the management software must be able to detect probes with management tools in place that Network Deployment failed components, hosts, or virtualisation solutions should be responded to by pre-programmed event Product Network Maintenance (Design & Integration) (e.g. VMs) immediately and recover automatically, responders (autonomics – big data analytics. This if possible through an autonomous response allows the creation of algorithms to perform tasks mechanism (example: virtual machine migration). within seconds that may otherwise take operations Active/Stand-by Unified Monitoring Distributed Pooling engineers hours or days to resolve. Design 2. Networks should be deployed in a way so that Quick Service Recovery 1. When large scale failure does occur, emergency N+M Redundancy failure in a node will not impact service continuity. Application handling capabilities are needed to recover the This can be achieved with distributed design across services quickly and minimize impact. Contingency Plan multiple sites, deploying additional resources Resource Security 2. Organizations must ensure staff have the right HA1: VM N+M Redundancy and proper dimensioning. Also, it is important Clustering Hardening skills as well as the appropriate tools and right Demarcation capabilities to consider a feature deployment and system processes in place. They must also ensure Cloud OS HA2: VM Migration configuration that can protect a network from network management capabilities meet the Fault Detection different threats. It is desirable that the network is same dependable level of carrier-grade reliability able to identify events or drivers in the network as Dimensioning Resilient Design expected by customers. HA: Multiple NICs, SPs, RAID Automation well as external to the network that could degrade COTS or otherwise impair the quality or availability of the supplied services. For example, if a network Solid plan involving different components and solutions to build a bridge received a significant increase in traffic beyond the level it was engineered to expect.
NOTE: ETSI NFV ISG (Industry Specification Group) Carrier Grade has provided some guidelines and best practices on Enterprise (IT) Reliability Reliability 99.9% product reliability and resilient network design [27]. 99.999%
Source: Huawei
24 25 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
5.2 Carrier Grade Service – Requirements for 5.3 NFV Reliability Assessment Standard NFV Reliability In order to provide an ultra-reliable, highly secure Figure 12: Carrier-Grade NFV Reliability Assessment Framework As illustrated in the diagram below, there is a daunting and fully connected network, an in-depth and list of requirements for achieving carrier-grade comprehensive NFV network assessment framework Network Resource Network Network Network Operations & 6 Dimensions reliability and deliver the always-on connectivity has been developed. Health Capacity Protection Evolution Handling Maintenance expected by service providers and their customers. Such reliability is attained by complying with a very Hardware Software Overload Inter- Network Staff / stringent criteria of availability, security, performance, Health Licensing Protection operability Information Organisation serviceability and management requirements, which (Example: Skills Design and fall into six primary dimensions (6D) for a NFV-driven Virtualisation Zero Physical Autonomous Topology) Layer Health downtime Tools/ carrier-grade system. Resources Response Mechanism Software Platforms (Example: Upgrade Emergency (Example: Case 360° VNF Health Link Virtual Network Processes Assessment Utilisation Machine Deployment Telemetry) Elements/ Migration) of new Figure 11: Requirements for Carrier-Grade NFV Reliability Formulas Connectivity function, Disaster feature, Recovery Processes Scaling Security technology Capabilities Grey Failure Hardware Load Resiliency upgrade and Balancing expansion MANO Carrier-Grade Reliability Alarms/ Input Design Statistics Configuration Topology Processes Features Log-files
Source: Huawei Availability, Performance, Security, Serviceability, Management 1. Network Health 2. Resource Capacity NFV brings a lot of new challenges to operators due How efficiently and effectively resources are utilised to its architecture. With regards to network health, (e.g. software, hardware, links) is critical for NFV operators need to consider several new elements network reliability. Operators need to monitor and NFV 6D Assessment Standard when compared to PNF (physical network function) avoid overloading resources above the network’s Network Health, Resource Capacity, Network Protection, that should be in a healthy status, otherwise they designed thresholds and maintain load-balance Network Evolution, Emergency Handling, Operations & Maintenance may impact a network’s reliability. Also, if there is between the resources. NFV introduces scaling (Measurement, Results, Gap Analysis, Benchmarking, Improvement) deterioration in a basic service KPI then it should be capabilities in order to enhance a network’s agility treated quickly and automatically, to minimize the and reliability. downtime or eliminate the impact on users. Some of the critical elements which should be monitored 3. Network Protection Source: Huawei regularly and proactively include hardware and Networks should be protected from all possible software components, the virtualisation layer as threats which can impact services. NFV brings a well as connectivity and network performance. number of new challenges such as cloud computing, Predictive fault detection becomes more important component scalability and resiliency, information, in NFV, because applications use infrastructure cyber security and network access provisioning. A delivered by other vendors. For example, operators network should be protected from known threats, should identify grey failures in advance, such as a such as signalling storms (overload protection), memory leak or CPU overutilization (based on a denial of service attacks and hardware or software traffic model) that have not yet affected the service failures. A network should be designed properly for performance and end users. its resilience (no single point of failure) and should be able to recover automatically when possible (migration, reconstruction etc.). A robust network shouldn’t be vulnerable to any of these threats.
26 27 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
4. Network Evolution 6. Operations & Maintenance 5.5 Conclusion A network is always evolving with the introduction In order to maintain high reliability, operational Carrier-grade is demanding, but it is an essential of new features, functions, services, technologies, efficiency is critical. ICT operational transformation is feature of today’s telecom networks to maintain software upgrades and hardware equipment etc. taking place in operator organizations to adopt NFV strict reliability requirements as both the enterprise NFV increases the complexity further with a larger technology. Service providers need to build teams customers or consumers have been conditioned to number of components and vendors. A reliable, with skilled staff for new technologies and products, expect extreme reliability in our networks. However, high-performance network might be impacted after efficient tools for monitoring (e.g. telemetry), with careful planning and the assistance of telecom- a change in one of the layers. For example, when a fault (e.g. root cause analysis) and performance system engineering experts, NFV network and service vendor’s hardware is incompatible with some of the management, processes for routine maintenance designers (new versions of service software will components for a server capacity expansion which considering the dynamic environment of NFV be necessary to fully utilise the new opportunities creates interoperability issues impacting network networks. For example, an added complexity in NFV of virtualisation), service providers can build this reliability. As a result, openness is critical in driving is driven by the implementation on an as needed capability into their NFV deployments and then, with innovation and enhancing reliability. Open interfaces basis. Historically once an application was put into carrier-grade systems to confidently commercialize (APIs) play a vital role in building carrier-grade the network it stayed in the network. Going forward their NFV services, knowing that they will meet services. Additionally, it is important to introduce applications will be applied to the network as needed business and technology objectives while satisfying software upgrade procedures with zero downtime then all or a part of them will be removed. In this their customers. in order to minimize the impact to end users. situation, the configurations of the network will be changing on an ongoing basis. Service providers know that they need to continue to 5. Emergency Handling meet those expectations as they transition to NFV. Failures may happen any time in the network. Without this assurance for NFV, they run the risk of 5.4 Best Practices Quick recovery and service continuity are very losing their high-value customers and seeing increased important, so emergency handling capabilities, 5.4.1 Use case scenarios subscriber churn which could offset the many business including skills, processes, information (network All actors in the industry are invited to use the above benefits provided by NFV. No new technology is topology) and automation, are essential. That is framework to build use cases to get carrier-grade worth that risk, regardless of the potential saving in important for physical networks, but in NFV we reliability results, to maintain openness, and to avoid CAPEX and OPEX. So, in order to provide an ultra- need to consider recovery in this more complex, vendor or technology lock-in. At this juncture, China reliable, highly secure and full connected network, a multi-layer environment. Mobile and Huawei have announced partnerships to robust, 360 degree, in-depth analytical NFV network carry out the laboratory testing on top of live systems. assessment framework is required.
Figure 13: Carrier-Grade NFV Reliability Assessment Methodology
Input 6 Dimensions E2E Assessment Assessment Results
Statistics Network Health Resource Configuration Capacity O&M Resource Network Capability Capacity Alarms/Log-files Health Network Protection 6 Dimensions Design and their 0 O&M elements 20 Capability 40 Assessment Systems Assessment Topology Emergency 60 Handling Network 80 Network Network Evolution Protection Processes 100 Evolution Emergency Features Handling
Source: Huawei
28 29 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK 6 Migration
30 31 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
6.1 Introduction 3 – Vertical Interoperability Verification As an example the case of MME deployment is 6.2.2 Software upgrades This section describes migrations and software In NFV, there are multiple components provided discussed here. In this example, the operator who has The types of NFV software to be upgraded are upgrades from both a deployment and software usually by different vendors. Each layer’s software has launched commercial LTE services is going to migrate identified in [11]. These are as follows; upgrade aspect after the l aunch of a virtualised to be upgraded or updated periodically. Any change to its native MME to a virtualised one. In general, the • VNF domain software any layer may impact the interoperability and therefore operator has to keep the existing native MMEs operating network launch. Operators should consider • MANO domain software interoperability between entities comprised of PNF the service performance. Before implementing a so that it can continue to serve existing users, therefore • NFVI software and VNF. Not all network functions are well-suited to sotfware upgrade in the network, it is important to the operator would initially introduce the virtualised be virtualised and in this section, we examine three verify it in an operator’s test bed (or any other external MMEs as additional facilities. In this example, newly From those components, the most critical one is the categories from an operator perspective. LAB), mirroring the exact environment with the same added vMMEs are added to an existing MME pool. VNF, since it provides the services. The process should components end-to-end. Once it is verified, the ensure a zero-downtime software upgrade in order 6.1.1 Migration from physical network to virtualised After a certain period of time, the EOSL (End of software release could then be safely rolled-out in the to provide high availability (including planned and network Service Life) of the existing native MMEs is reached, live environment. unplanned downtime) and improve a user’s experience. This category considers several migration paths from the operator may want to introduce the Next The NFVI software upgrade process is not as complex physical to virtualised, e.g., single functionality level 4 – Software upgrade/update frequency Generation core network (NGCN) to access one of its as VNFs, but the process should protect service virtualisation, PNF/VNF-mixed operation in the unique Vertical layout may be composed by different vendors unique functions e.g. network slicing. In this example, continuity. Finally, the MANO software upgrade process functionality (e.g., the operation which includes vMME and each vendor may provide software releases the deployment approach is described as a combo- can not impact the service, but only operations (e.g. and existing MME in the same pool), operation without (patches or versions) in different time periods. Some node deployment of both EPC and NextGen core. commands, auto-scaling may not work etc.). utilizing NFV orchestrator etc. vendors may release software packages once a month, some quarterly or semi-annually etc. Each time, a There are several different processes to be followed for 6.1.2 Software upgrades multi-vendor verification process may be needed, software upgrades based on industry best practice. Several aspects should be considered in this category, which is time consuming and requires additional Figure 14: MME migration path For each component, the same or different strategy for example, VNF application software upgrade, NFVI resources. Operators should plan accordingly and may be executed and this depends on: (e.g., host OS/hypervisor) upgrade and VIM upgrade consider how to optimize the process per case. (e.g., OpenStack). There are also some other important 4G 5G • Software architecture differences in software upgrade procedures between 6.1.3 Interoperability • Availability of Additional resources NFV and traditional networks. This category considers the interoperability between Native • Traffic migration capabilities the physical entity and the virtualised entity or MME • Network Design 1 – Process network connection using virtualised networking i.e., The software architecture is different for each VNF, and SDN in the NFV configuration. Addition so consequently, the software procedure should be to an Virtualised MME adjusted accordingly. It is recommended that the VNF existing Figure 15: VNF Architecture 6.2 Best practices pool structure and number of VNFCs be considered, before upgrading the VNF. Another difference between NFV 6.2.1 Migration and a traditional network is that in the NFV it is possible Two possible approaches to introduce virtualised products in an operator network following the launch Each VNF may be to access a shared pool of resources (i.e. servers) that VNF1 composed of 1 or can be used when a process needs them. This provides of commercial LTE services are discussed below. EOSL multiple VNFCs flexibility to operators or vendors who could apply a 1. Introduction of a new system for newly developed different strategy based on their needs. services (e.g., IoT services, facilities for MVNOs and EPC + NextGen Core services for the enterprise customers); 2 – Zero downtime 2. Expansion to existing facilities VNFC1 VNFC2 Each VNF/VNFC may be composed of several instances and this could provide more flexibility As the introduction of new systems for newly VNFCI VNFCI VNFCI VNFCI Source: KDDI to operators in order to perform a zero-downtime developed services is not necessarily a migration, the 1 2 N 1 software upgrade. rest of this section will focus on area of expansion.
Each VNFC may have 1 or multiple instances (parallable)
Source: Huawei
32 33 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
1. Using VNF in Pool, so it is easier to migrate traffic 6.3 Requirements from one VNF to another one. Then upgrade the 6.3.1 Migration “empty” (not loaded) VNF without risk. The following aspects should be considered for the migration from physical to virtualised: 2. Rolling SW Upgrade (instance by instance), it is applicable when there are multiple instances. It is used • It should be possible to share PNF and VNF for Active-Active or Active Stand-By mode, to isolate an resources for the same network function instance, upgrade it and then put it back to the traffic. NOTE: The “resources” are not the resources which are provided by the NFVI, such as compute, storage 3. Scale-out/Scale-In process, when deployment of or network resources VNFC instances with the new SW (in the same VNF), 7 • In a mixed environment consisting of both physical migrate traffic to them and kill old instances (scale-in). and virtualised network functions, it should be Additional resources are needed. possible to maintain the same interfaces between 4. New VNF deployment, initially instantiate a new network functions Performance benchmark VNF, migrate traffic to that one and then terminate the 6.3.2 Software upgrades “old” VNF. Traffic migration may require configuration The following elements should be considered for from external nodes. software upgrades; for NFV infrastructure 5. NFVI SW Upgrade (i.e. host OS, KVM, Open V • The traffic should be switched seamlessly for all the switch or DPDK) uses live VM migration to minimize methods of SW upgrade described above the impact. VMs are migrated from NFVI (old SW) to • Migration of active VNFs (live migration) from the the upgraded NFVI area and continue upgrading all compute node in operation to the maintenance zone the resources. should be executed without interruption when VIM 6. MANO SW Upgrade process is not as critical as software or NFVI software will be upgraded. If the other components, since there is usually no service interruption occurs, it should be minimized impact. Any method highlighted above could be used • Required hardware acceleration functionality should based on the SW architecture. be available for the destination of the migration • Due to complexity, automated software upgrade / update process is preferred • Quick roll-back process in case of failure • Ensure zero downtime for the services during VNF upgrade
34 35 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
7.1 Performance benchmark scope In addition to the vendors and operators’ own 7.2 State of the art in NFV performance benchmarking In the OPNFV Colorado release, test cases covering When NFV was brought to the industry, it was claimed tests, many standard organizations and open In OPNFV, there are several test projects which have Performance/Speed and Capacity/Scale are available, to help operators shorten their service time to market source communities are very supportive of NFV been initialized to benchmark the performance of Reliability/Availability part will be finished in the and offer agility in system expansion, while simplifying performance benchmarking experiments. Usually a NFV system and its components. Yardstick and coming releases. network operation through the use of common this work will involve test framework and traffic Bottlenecks are the names given to projects used The scope of Yardstick is the development of a test servers. However, since NFV is built on common x86 generators, as benchmarking could not be performed to benchmark system level performance, while framework as well as test cases and test stimuli to servers, service level performance is often questioned. in live networks. Other common characteristics of StorPerf, VSPerf and CPerf define methodology enable NFVI verification. This methodology is aligned Given the circumstances, vendors and operators performance benchmarking work are as follows: in benchmarking storage, virtual switch and SDN with ETSI TST 001 [18]. should team up to provide some performance metrics • Include testing instructions that are sufficiently controller performance respectively. and carry out tests. specified (prerequisites, procedures, output) The Bottlenecks [13] project aims to find system Yardstick [12] is a project that aims to verify the • Results are repeatable bottlenecks by testing and verifying the OPNFV Each single portion of the NFV system can be infrastructure compliance from the perspective of a infrastructure in a staging environment before benchmarked, for example, VNF performance, NFVI • Test cases can be performed on different vendor’s VNF. Based on the uses cases that have been defined committing it to a production environment. It defines performance, storage performance, virtual switch device/system in ETSI GS NFV 001 [17], each use case implies specific an automatic method for executing benchmarks to performance etc. Each level of these performance • The produced results are useful and meaningful for requirements and complex configuration on the validate the deployment during the staging phase. benchmarks will help operators to decide which the users underlying infrastructure and test tools. In order to find The framework has four components: Workload product will fit into their NFV systems. From the end- a system level benchmark, in Yardstick, every single generator and VNFs (WV), Monitor and Analysis (MA), to-end service perspective, it is possible to benchmark In this chapter, state of the art NFV performance VNF work-load performance metric is broken down into a Deployment and Configuration (DC), Automated service deployment. For example, how quickly a NFVI benchmarking is outlined. Performance metrics, test number of characteristics/performance vectors and each Staging (AS). The architecture is shown as follows: could be set up, how long it would take to upload a methodologies and use cases that have been defined performance vector is then represented by a test case. VNF package and how long to instantiate a VNF or will be helpful for vendors and operators who want to a Network Service etc. Based on these benchmarks, carry out the tests in their own labs. operators can tell if it is feasible to deliver a low cost, high performance solution based on NFV.
Figure 17: YardStick performance metrics and test suite
Figure 16: NFV performance benchmark scope Performance/Speed Capacity/Scale Reliability/Availability Compute • Latency for random memory access • Number of cores and threads • Processor availability (error free • Latency for cache read/write • Available memory size processing time) operations • Cache size • Memory availability (error free Projects Working Groups Test Framework SUT/DUT/FUT for NFV • Processing speed (instructions per • Processor utilisation (max, average, memory time) second) standard deviation) • Processor mean-time-to-failure • Throughput for random memory • Memory utilisation (max, average, • Memory mean-time-to-failure access (bytes per second) standard deviation) • Number of processing faults per ETSI Sufficiently Defined VSwitch • Cache utilisation (max, average, second • TST-001 Instructions standard deviation) • TST-002 Test Framework Controllers • PER-001 Network • Throughput per NFVI mode (frames/ • Number of connections • NIC availability (error free + byte per second) • Number of frames sent/received connection time) • IFA-003 Repeatable Results Storage Traffic Generator • Throughput provided to a VM • Maximum throughput between VMs • Link availability (error free (frames/byte per second) (frames/byte per second) transmission time) OPNFV NFV1 • Latency per traffic flow • Maximum throughput between NFVI • NIC mean-time-to-failure • Yardstick Applicable to Different • Latency between VMs nodes (frames/byte per second) • Network timeout duration due to link • Bottleneck Vendors Management • Latency between NFVI nodes • Network utilisation (max, average, failure • StorPerf • Packet delay variation (jitter) standard deviation) • Frame loss rate • VPerf Useful and Meaningful between VMs • Number of traffic flows Benchmarking • CPerf Reliability Results for Users • Packet delay variation (jitter) between NFVI nodes IETF (BMWG) Servers Test Traffic • Virtual-Net-04 Storage • Sequential read/write IOPS • Storage / Disk Size • Disk availability (error free disk Framework Generator Unambiguous Output • HA-NFVI-01 VNFs • Random read/write IOPS • Capacity allocation (block-based, access time) • Latency for storage read/write object-based) • Disk mean-time-to-failure operations • Block size • Number of failed storage read/write • Throughput for storage read/write • Maximum sequential read/write IOPS operations per second Source: Huawei operations • Maximum random read/write IOPS • Disk utilisation (max, average, standard deviation)
36 37 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
TST 001 provides an informative report on methods for IFA 003 specifies performance benchmarking metrics Figure 18: Bottlenecks test framework pre-deployment testing of the functional components for virtual switching, with the goal that the metrics of an NFV environment, including VNFs, NFVI and will adequately quantify performance gains achieved MANO. The recommendations focus on lab testing and through virtual switch acceleration conforming to the Automated the following aspects of pre-deployment testing: associated requirements specified herein. It defines the Staging (AS) • Assessing the performance of the NFVI and its critical aspects of vSwitch performance by treating the Workload VNF VNF Monitor and ability to fulfil the performance and reliability vSwitch as a Device Under Test (DUT), with specific generator Analysis (MA) requirements of the VNFs executing on the NFVI configurations that are consistent across instantiations WV of a vSwitch on a computing platform. It also uses • Data and control plane testing of VNFs and their the existing testing and benchmarks specifications interactions with the NFV Infrastructure and the (see [22], [23], [24] and [26]) to measure the NFV MANO performance of the DUT under specific configurations • Validating the performance, reliability and scaling Hypervisor ODL DPACC OpenStack and conditions, such as vSwitch physical to physical, Deployment and capabilities of Network Services Configuration vSwitch virtual to virtual, etc. Infrastructure (DC) TST 002 provides interoperability test methodology In IETF, the draft-ietf-bmwg-virtual-net-04 that is applied to NFV by analysing some of the core [22] document researched by Benchmarking Source: OPNFV Bottlenecks project NFV capabilities and the interactions between the Methodology Working Group (BMWG) has defined functional blocks defined within the NFV architectural the considerations for benchmarking virtual network framework required to enable them. It describes functions and their infrastructure. This document The workload generator generates workloads which The Network Service Benchmarking (NSB) is an open two types of testing: Conformance Testing and investigates additional methodological considerations go through VNFs, and then analysis units will monitor source framework that allows benchmarking and Interoperability Testing. necessary when benchmarking VNFs instantiated and the infrastructure and VNF status. Test results will characterization of VNFs by testing with real-life traffic PER 001 provides a list of minimal features which hosted in general-purpose hardware, using bare-metal then be analysed to find out the system bottleneck. scenarios on Bare Metal, Standalone Virtualised, and the VM Descriptor and Compute Host Descriptor hypervisors or other isolation environments such as Therefore, a wide range of different hardware Managed Virtualised environments. The NSB test should contain for the appropriate deployment of Linux containers. It lists several new benchmarks and resources and software configurations will be used to harness is inherited from the OPNFV Yardstick open VM Images over an NFVI, in order to guarantee related metrics, such as time to deploy VNFs, time to locate a system’s bottleneck. source solution (also upstream the patches back to high and predictable performance of data plane migrate VNFs, time to create a virtual network in the Yardstick community developed in test harness), workloads while assuring their portability. In addition, general-purpose infrastructure, etc. it also defines The Storage Performance Benchmarking for NFVI and incorporates a benchmarking methodology that the document provides a set of recommendations several new considerations which must be addressed (StorPerf) [14] is a project that aims to provide a tool facilitates deterministic and repeatable benchmarking on the minimum requirements which hardware and to benchmark VNF and their supporting infrastructure, to measure block and object storage performance in on Industry Standard High Volume (SHV) servers. It hypervisor should have for a NFVI suitable for different like hardware configuration parameters (shelf an NFVI. The project will define a test suite, including provides several VNFs: vCG-NAPT, vACL, vFW, vPE, workloads (data-plane, control-plane, etc.) present in occupation, CPUs, caches, memory), etc. test cases, metrics and test process to find the vEPC as the samples for demonstration, and it also can VNFs. According to the workload analysis in clause benchmarks, which can provide a good preview of integrate 3rd-party VNF as the benchmarking tool for Another document, “Considerations for Benchmarking 6, clause 7, it provides a recommendation on the expected storage performance behaviour for any type operators, which can help in characterizing VNFs by High Availability of NFV Infrastructure” [27] minimum requirements that a Compute Host should of VNF workload. measuring Network, VNF and NFVI KPIs: lists additional considerations and strategies for have for a NFVI suitable for data-plane workloads, and, benchmarking high availability of NFV infrastructure The vSwitch Performance (VSPerf) [15] project will • Network KPIs: e.g. Traffic Generator KPIs like clause 8 gathers the list of features which the Compute when network functions are virtualised and performed develop a generic and architecture agnostic vSwitch packets in, packets out, throughput, latency as per Node Descriptor and the VM Descriptor templates in NFV infrastructure. With VNFs and NFVI replacing testing framework and associated tests, which will RFC 2544 etc. should contain for the appropriate deployment of VM the legacy network devices, operators have several serve as the basis for validating the suitability of • VNF KPIs: e.g. packets in, packets out, packets Images over an NFVI, in order to guarantee high and issues to cope with such as availability, resiliency and different vSwitch implementations in a Telco NFV dropped, etc. predictable performance while preserving portability non-measurable failures. Above all, they want to deployment environment. • NFVI KPIs: e.g. CPU utilisation, IPC, L1/L2/LLC across different servers. ensure the availability of the VNF products and their The Controller Performance Testing (CPerf) cache hit/misses, iTLB/dTLB hit/misses, OVS stats, infrastructures. From the operator point of view, the [16] project will serve as a performance testing memory bandwidth & latency, etc. availability is the most important feature and the benchmarking tests for the high availability of NFV environment for the SDN controller portion of the In ETSI, ETSI GS NFV-TST 001 [18], TST 002 [19], PER infrastructure are also important. This document large, realistic, automated deployments required by 001 [20] and IFA 003 [21] have been produced by ETSI investigates considerations for high availability of NFV OPNFV. The initial focus of CPerf is OpenDaylight, but Industry Specification Group (ISG) to benchmark the Infrastructure benchmarking tests. later-on the project tends to leave the controller part NFV-related performance and test methods. of the test matrix open and will welcome collaboration with other controller communities.
38 39 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK 8 Vertical interoperability
40 41 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
8.1 Introduction 8.3 Challenges and potential risks 3. Service Assurance 8.4 Best Practices Physical mobile networks are made up by equipment When vendors use non-standardised non- Service performance and high availability are also Incompatible systems elements or software versions provided by multiple vendors in general. This is made standardised interfaces there could be potential risks dependent on interoperability. When there is a failure can cause further network faults, and require possible by the clear definition of open interfaces in the network. Some of the potential impacted areas in the network, the information should be propagated compatibility management. The key elements are to between the network nodes specified by 3GPP. are described below: quickly to the upper layers, based on the standard use standardised interfaces and proper compatibility Such a multi-vendor environment fosters innovation interfaces so that each component can use this validation processes. and competition, resulting in advantages both for 1. Vertical Integration information effectively. For example, a hardware operators and subscribers. With characteristics such as hardware-software failure should trigger an alarm to users allowing them 1. Standardised Interfaces decoupling and hardware generalization, NFV further to act immediately. NFV solutions should be based on interoperable multi- It will be beneficial for virtual networks to be made by opens up carriers’ network architecture. NFV is one vendor ecosystems with open source technologies equipment provided by multiple sources. huge complex ICT systems integration project, involving Automation also plays an important role in NFV or standardised protocols and interfaces. ETSI’s IFA multiple technologies, interfaces and multiple vendors. lifecycle management and different components have Workgroup focus on interface standardisation and in-built self-healing mechanisms (i.e. VM migration in 8.2 Requirements for vertical interoperability several open source projects have developed platforms It is important that vertical cloud platform integration case of host failure). These mechanisms are triggered based on this, such as OPNFV, OPEN-O, OSM etc. There Figure 18 highlights an example of a virtual network is performed smoothly and quickly. Interoperability under specific conditions, such as notifications that have also been several documents released by ETSI. where individual components are procured from a plays a key role in vertical integration, especially in a have been received from the network. If this kind of Figure 20 shows some of the important work done by number of different vendors. multi-vendor ecosystem. information is not received then the system will not the ETSI IFA and SOL Working Groups. recover automatically, and cause network outage. For networks to operate correctly it is important that 2. Network Service Deployment 2. ETSI Plugtests the interfaces between the individual blocks of the Interoperability can cause problems in Network Service 4. Software Upgrade ETSI identified the need for thorough industry architecture are tightly specified and implemented. deployment, since multiple components provided by Any change in the software of a component, in any interoperability testing and took on the initiative to different vendors are involved during this process. This layer, could cause problems to the other layers. Failing to define interoperable interfaces is expected start organizing NFV Plugtests. They will offer test includes from OSS/BSS and Portals up to NFVO, VNFM, Consequently, this could impact existing services and to result in a higher total cost of ownership as well as a sessions where vendors and Open Source projects will VIM, Cloud OS and COTS. Descriptors, such as NSD, their users. Vertical verification is recommended in slower time to market. Operators will need to ensure that be able to assess the level of interoperability of their VNFD should also follow the standardised format for a order to test interoperability. Horizontal verification the open source community developing the architecture implementations and verify the correct interpretation successful service instantiation. challenges remain the same as in traditional network, understand the importance of vertical interoperability. of the ETSI NFV specifications. so there is no real difference.
Figure 20: MANO to IFA Mapping – reference points Figure 19: example of multivendor virtual network Reference of the functional specification IFAxxx (stage 2) ETSI GS NFV IFA XXX Orchestrator IFA013 SOL005 OSS/BSS NFV Orchestrator vendor Reference of the solutions specification SOLxxx OSS NFVO (stage 3) ETSI GS NFV SOL XXX Os-Ma EM EM EM EM Management & Orchestration & Management REST APIs IFA014 NFV IFA007 SOL003 SOL001 Manager(s) IFA011 VNF VNF VNF VNF VNF Or-Vnfm NSD IFA008 SOL001 Service, SOL002 VNF and EM & NFV Descriptors Infrastructure VNFM Or-Vi (templates) VNF Virtual resources Description Ve-Vnfm VNF Compute Storage Network IFA011 Vi-Vnfm Package SOL001 IFA002 Virtualisation layer IFA018 VNFD
Execution IFA006 Hardware Virtualised Infrastructure IFA004 IFA005 IFA019 Compute Storage Network Manager(s) NFVI VIM Open Stack APIs TOSCA/YAML Profile Nf-Vi (extensions required) +CSAR archive format Infrastructure vendors Source: ETSI NFV
42 43 CONSIDERATIONS, BEST PRACTICES AND REQUIREMENTS FOR A VIRTUALISED MOBILE NETWORK
The component for the test sessions are Virtual Network Functions (VNFs), Management and Orchestration (MANO) solutions and NFV Infrastructure (NFVI) with pre-integrated Virtual Infrastructure Manager (VIM).
3. Integration and Verification Most of the interoperability issues appear during the integration phase. Multiple vendors are involved in this phase, so cooperation and coordination are important. Either operators or one of the assigned services suppliers should lead this complex project and act as a prime system integrator, having extensive IT and CT experience, in order to deliver an integrated E2E solution.
A proper verification plan is required for interface compatibility and service validation. For this purpose, an operator’s test bed could be used, using the same multi- vendor ecosystem. Alternatively, a vendor’s NFV Lab could be used for solution validation or pre-packaged, pre-integrated and pre-tested solutions. The same process applies for a software upgrade process.
44 Find out more at www.gsma.com/network2020
GSMA HEAD OFFICE Floor 2 The Walbrook Building 25 Walbrook London EC4N 8AF United Kingdom Tel: +44 (0)20 7356 0600 Fax: +44 (0)20 7356 0601