Network Architecture and Management

Torino, January 12th 2021

GRUPPO TIM The evolution of TIM's network architecture and management

Alessandro Capello Technology Innovation Agenda

#1 Network architecture

#2 Capacity planning and QoS management

#3 Network management

#4 Evolution of network architecture and management

2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 2 Requirements Network architecture For all services: • Connectivity matrix • Throughput • Latency • Availability Goals

Where? Satisfy the requirements at the minimum cost What? Physical Technologies topology Infrastructure Constraints

The buildings hosting network equipment The ducts carrying the optical fiber cables How?

Interface speed © TIM

Device fan-out and switching performance PUBLIC PUBLIC USE

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 3 The architecture of the (old) Telephone Network

Branch Office Distribution (SL) network

Class 4 (SGT) Branch Office Distribution (SL) network Designed to carry a single type of service: Transmission Class 5 Distribution network (SGU) network transmission of analog voice over the frequency band 300–3400 Hz

Class 4 Branch Office Distribution Hierarchical structure (SL) network (SGT) Circuit switching technology Very well-known mathematical models to control the quality offered to customers Cabinet (blocking probability) Small/very small SL SL SGU SGT ~30 ALL RIGHTS RESERVED

~500 2021,

~ 2000 © TIM ~ 10000

~ 150000 PUBLIC PUBLIC USE

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 4 The architecture of the Data Network

DSLAM OLT

Distribution network With the growth of the Internet, data services based on packet Point of Presence (PoP) switching technologies become predominant Backbone Router Data network carries both fixed and

Distribution mobile traffic for a variety of network Aggregation customers and services Router BNG Services: web browsing, voice, video PE Distribution Transmission network network streaming, business applications, etc. AAA L3 services VPN Customers: residential, small business, large corporates, Distribution network Backbone wholesale (other operators)

2021, © TIM © TIM

Backbone Aggregation Access PUBLIC PUBLIC USE

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 5

The backbone

2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 6 The optical network

Network diameter: 2400-3100 km (working-protection) 52 ROADMs

Optical links with 96 channels @ 100 Gbps and 200 Gbps

2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 7 The Points of Presence

BNG Backbone Aggregation Network CDN

Mobile Core 2021, © TIM © TIM

PoP PUBLIC PUBLIC USE

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 8 BNG – Broadband Network Gateway

• Residential and SOHO customers • Retail: Internet Access, Internet Access + VoIP • Wholesale • 20-100K customers per node • Point-to-Point VLAN to Access Nodes (PPPoE for customer separation) • RADIUS protocol for communication with AAA servers • Per session Authentication based on “loop ID” (and Username and Password)

BNG • Dynamic configuration of a Virtual Interface per customer based on service profile (including Access Control List, Rate limiting, QOS, VPN, IPv6, …) • Address assignment • Per session Accounting • Legal documentation requirements • Collection of statistics

2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 9 PE – Provider Edge router

Business services • Internet Access, L2/L3 VPN, VoIP 2-10k customers per node Strong SLA (i.e. maintenance operations only during the night)

Point-to-Point VLAN to Access Nodes (Q-in-Q for customer separation) E

P MPLS on interface backbone side • Solution for providing both Internet Access and L2/L3 VPNs (and for adding IPv6) BGP protocol used to monitor the status of the link between the PE and the CE • In case of failure backup connections are activated in less than 30s • New solution based on Bidirectional Forwarding Detection (BFD): less CPU intensive, more reactive

2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 10 Limitations of current architecture

Network upgrades are expensive considering The introduction of new network services is the current traffic growth slow

Many hops between source and destination Need to deploy physical devices Network functions are developed within Integration with legacy management systems specialized network equipment Legacy network devices are expensive and

many features are not used in practice

2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 11 Agenda

#1 Network architecture

#2 Capacity planning and QoS management

#3 Network management

#4 Evolution of network architecture and management

MARKET SENSITIVE

2021,

CONFIDENTIAL PUBLIC PUBLIC USE

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 12 Capacity planning and QoS management

Traffic matrix has completely changed in the last 20 years

Voice traffic Data traffic

International Internet Volume

Backbone Backbone Locality Area A Area B Area A Area B

Capacity planning and Capacity planning involves QoS management means Quality of Service upgrading the network differentiating traffic management are the infrastructure (interface behavior in case of ALL RIGHTS RESERVED main tools to manage speed, number of network congestion 2021, traffic evolution interfaces, ecc.) to carry © TIM

more traffic PUBLIC PUBLIC USE

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 13 Capacity planning

The network design avoids single points of failure All internal links are dimensioned following the 50% rule: each link can’t be loaded more than 50% of its capacity @ peak hour

Continuous monitoring of the link load 50% rule 50% Warning thresholds are defined to allow planning for upgrades

Peak and average traffic rates are collected for all interfaces Traffic matrix is estimated using gravity models

Algorithms for traffic prediction must consider both legacy and new services

prediction

2021,

Traffic Traffic PUBLIC PUBLIC USE

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 14 QoS Management

Protect critical traffic in case of multiple simultaneous faults Classification based on IP precedence, MPLS EXP or Ethernet CoS 3, 4 queues: a strict priority (typically for voice) queue and 2, 3 WFQ for different data

classes DiffServ Critical point is traffic marking: you cannot trust IP precedence of Internet IP packet… most

of the traffic in the backbone is in the Default class

2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 15 QoS for business customers

For a business access, 3 parameters are defined: • A peak bandwidth • A Minimum Guaranteed Bandwidth (MGB): the minimum bandwidth that will be available also in congestion conditions • The Real Time Bandwidth (RTB): the maximum traffic that is allowed in the Low Latency Queue The network applies a limitation of the peak bandwidth at the maximum value defined by the selected offer Peak bandwidth is usually equal to the bandwidth available on the local loop

Business customers Business It may happen that the local loop bandwidth is lower than the peak bandwidth: differentiated queuing at the access level is still required to avoid discarding of critical traffic

2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 16 QoS for business customers

Access #1 Access #N WFQ WFQ • 3 queues for each customer LLQ LLQ • LLQ + 2 queues with WFQ (Mission Critical and Default) • Example of weights is 30:70 • Hierarchical queuing

configuration • multiple access on a single interface • In case of congestion the bandwidth is C-VLAN

Queue Queue distributed through the different accesses

proportionally to the MGB parameter

PHY GE

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 17 QoS for residential customers

For a residential access only a peak bandwidth is defined The network applies a limitation of the peak bandwidth at the maximum value defined by the selected offer

customers For each customer, 1 or 2 queues (LLQ and default) are configured Hierarchical queuing In case of congestion the bandwidth is distributed through the different sessions

proportionally to the Peak bandwidth and the MGB parameters

Residential

2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 18 Agenda

#1 Network architecture

#2 Capacity planning and QoS management

#3 Network management

#4 Evolution of network architecture and management

MARKET SENSITIVE

2021,

CONFIDENTIAL PUBLIC PUBLIC USE

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 19 Network management: FCAPS

FCAPS is a network FCAPS categorizes the working objectives of network management framework management into five levels created by ISO

Fault Configuration Accounting Performance Security

2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 20 FCAPS: Fault management

• A fault is an event that has a negative significance. • The goal of fault management is to recognize, isolate, correct and log faults that occur in the network. • Trend analysis can be used to try to predict errors so that the network is always available. This can be established by monitoring different things for abnormal behavior. • When a fault or event occurs, a network component will often send a notification to the network operator using either a proprietary or open protocol such as SNMP to collect information about network devices • In turn, the management station can be configured to make a network operating center (NOC) aware of problems, allowing appropriate action to be taken. This notification is supposed to trigger manual or automatic activities. For example, the gathering of more data to identify the nature and

Fault management severity of the problem. • Fault logs are one input used to compile statistics to determine the provided service level of individual network elements, as well as sub-networks or the whole network. They are also used to

2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 21 FCAPS: Configuration management

• The goals of configuration management include: 1. to gather and store configurations from network devices (this can be done locally or remotely). 2. to simplify the configuration of the device 3. to track changes that are made to the configuration 4. to configure ('provision') circuits or paths through non-switched networks

management 5. to plan for future expansion and scaling • Configuration management is concerned with monitoring system configuration information, and any changes that take place. • This area is especially important, since many network issues arise as a direct result of changes made to configuration files, updated software versions, or changes to system hardware. • A proper configuration management strategy involves tracking all changes made to network

hardware and software. Examples include altering the running configuration of a device, updating Configuration

the OS version of a router or switch, or adding a new modular interface card.

2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 22 Configuration stages for network devices

Stage Actions 1. Initial configuration typically applied on power-on, boot-up. 2. Information entered on serial consoles, buttons/switches on device 3. Configurations that need knowledge of physical connectivity, typically entered by operators on the truck-roll Day-0 • ID of the Interface connected to the switch for example (assuming there is not automatic topology discovery method) 4. Configurations that are considered pre-requisite. 5. Configuration required to establish communication between device and service orchestration system (or network management system). Configuration pushed by service orchestration system that are common Day-1 to all Service instances. (example, NTP, Syslog, SNMP Trap destination

On-Going configurations pushed on the device for day-to-day operations. 2021, Day-2..N

Configurations pushed to realize one instance of a service © TIM PUBLIC PUBLIC USE

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 23 FCAPS: Accounting management

• The goal is to gather usage statistics for users. • Accounting management is concerned with tracking network utilization information, such that individual users, departments, or business units can be appropriately billed or charged for accounting purposes. • Accounting is often referred to as billing management. Using the statistics, the users can be billed

and usage quotas can be enforced. These can be disk usage, link utilization, CPU time, etc.

Accounting management Accounting

2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 24 FCAPS: Performance management

• Performance management is focused on ensuring that network performance remains at acceptable levels. • It enables the engineering department to prepare the network for the future, as well as to determine the efficiency of the current network, for example, in relation to the investments done to set it up. The network performance addresses the throughput, network response times, packet loss rates, link utilization, percentage utilization, error rates and so forth. • This information is usually gathered through the implementation of an SNMP management system, either actively monitored, or configured to alert administrators when performance move above or below predefined thresholds. • Actively monitoring current network performance is an important step in identifying problems before they occur, as part of a proactive network management strategy. • By collecting and analyzing performance data, the network health can be monitored. • Trends can indicate capacity or reliability issues before they affect services. Also, performance

management process . Alarms vary depending upon the severity of the problem.

2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 25 FCAPS: Security management

• The network is protected against hackers, unauthorized users, and physical or electronic sabotage. • The confidentiality of user information is maintained where necessary or warranted. • Security systems also allow network administrators to control what each individual authorized user can (and cannot) do with the system. • Security management is not only concerned with ensuring that a network environment is secure,

but also that gathered security-related information is analyzed regularly.

Security management Security

2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 26 FCAPS: Packet network vs Optical network

Packet network Optical network

Alarms collected by the Element Management Fault management Alarms collected using SNMP System

Configuration is done via a graphical interface provided by the EMS Manual configuration using Command Line Configuration Configuration between different domains relies Interface on the Network Management System (higher level system talking to multiple EMS)

Network statistics are collected using SNMP Statistics collected by the Element Management Accounting (pull mode) System

Performance data (mainly obtained through Statistics collected by the Element Management Performance active methods) are collected using SNMP (pull System

Mainly dedicated systems using network © TIM Security statistics (in addition to security mechanisms Mechanisms at data plane level

at data plane and control plane level) PUBLIC USE

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 27 Limitations of current network management

Manual configurations are error Fault management is ultimately Network upgrades are slow prone based on human interpretation

The collection of network NMS implies per-vendor statistics via SNMP «pull mode» integrations: high cost of is inefficient and limit the

retrieved from devices

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 28 Agenda

#1 Network architecture

#2 Capacity planning and QoS management

#3 Network management

#4 Evolution of network architecture and management

MARKET SENSITIVE

2021,

CONFIDENTIAL PUBLIC PUBLIC USE

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 29 First technology trend: Software Defined Networks Open interface to applications Application . . . Application (NorthBound API) Well-defined open APIs Controller Traditional Feature F F Network . . . Network network Service Service OS Abstract Network View Custom Hardware Network OS

Feature F F

OS Feature F F Open interface to Custom OS packet forwarding Hardware Hardware (SouthBound API) Custom Hardware

Hardware Hardware

• The Control Plane is logically centralized 2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 30 Second technology trend: Disaggregation of networking functions

Disaggregation replicates in networking technology what happened in IT technology with the migration from proprietary mainframe systems to microprocessor systems with operating systems

from third parties

Images from Nick McKeown, Stanford University PUBLIC PUBLIC USE

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 31 Third technology trend: Network Function Virtualization

CPU and chipset are optimized to handle new types of workloads efficiently. Functionalities that in past were feasible only with specialized hardware (e.g. ASIC) can be built using general purpose CPU

Virtual network functions Network implemented as a coherent Functions set of Virtual Resources

a bunch of sophisticated Virtualiz technologies capable of ation creating Virtual Resources out of the Infrastructure

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 32 ETSI NFV Architecture

NFV Management & Orchestration (Specified in ETSI GS NFV-MAN 001)

Os-Ma-nfvo Network Service Management OSS/BSS NFV Orchestrator (NFVO) → Manage combinations of connected VNFs

NS VNF NFV NFVI Catalog Catalog Instances Resources

Or-Vnfm Ve-Vnfm-em VNF Management EM VNF Manager (VNFM) → Manage individual VNFs Ve-Vnfm-vnf VNF Vi-Vnfm

Vn-Nf Virtualised

Manager 2021, Nf-Vi (VIM) → Manage the use of NFVI resources

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 33 Motivations for SDN and NFV

Lower risk of vendor lock-in; the availability of many suppliers can reduce costs Easier automation of operational processes thanks

to open APIs SDN & Global optimizations (by means of centralized

disaggregation control) Cost reduction Faster Time to Market HW resources can be dynamically allocated on the basis of real needs: better efficiency and scalability HW resources can be easily (i.e. automatically) re- assigned to different network applications: faster NFV deployment of new network applications

infrastructure 2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 34 Evolution of network architecture

Telco central offices will progressively look like small data

centers

2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 35

Central Office Re-architected as a Datacenter

https://opencord.org/ PUBLIC USE

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 36 Open issues

Technological aspects Organizational aspects

• Lack of cloud-native virtualized network • Automation of operational processes has applications huge impact on Service Providers’ internal • Performance using shared, general organization purpose HW resources are less predictable • New skills are required • Functions that use a lot of bandwidth (User Plane) require ad-hoc engineered

solutions

2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 37 Evolution of network management

Services can be described using standard description languages (e.g. YANG, YAML, etc.) Service descriptions can be passed to network controllers through programming interfaces Configuration (API) Automation The network controllers transform the service description into device configurations and automatically apply changes

Streaming telemetry provides a continuous flow of statistics from the devices to the NMS using a push model: • Source-timestamped Telemetry • Event-driven: push as soon as the data changes (low latency), push only when the data changes (low throughput)

Closed The control loop can be closed with code that is able to correlate statistics to service behavior 2021,

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 38 Closed loop

Analytics Intelligence

Provide specific decisions and Provide insights based on collected Decision recommendations data Analysis AI models, policies and intents

Data collection Orchestration & Control

Automate workflows to handle

provide live fault and performance Execution 2021, Collection Individually steer the state of

data © TIM

managed entties PUBLIC PUBLIC USE

The evolution of TIM's network architecture and management Alessandro Capello, Technology Innovation 39 Grazie