Time Division Multiplexing (TDM) to Internet Protocol (IP) White Paper

TDM to IP White Paper

FAA Telecommunications Infrastructure (FTI)-2 Time Division Multiplexing (TDM) to Internet Protocol (IP) White Paper

Networks and Telecommunications Community of Interest FTI-2 Working Group Technology, Performance, and Operations Subcommittee

Date Released: May 11, 2017

Synopsis Virtually all local exchange carriers in the U.S. will migrate their networks from the current TDM based architecture to an architecture that is based on Ethernet access to an IP fabric. Two primary drivers for this migration are (1) more efficient bandwidth in an IP environment and (2) equipment manufacturers and chip set providers will soon discontinue production of TDM based equipment. This white paper will attempt to address some of the fundamental issues relating to TDM to IP migration; it also addresses timelines, migration strategies and network adaptive technologies for those applications that continue to demand a TDM interface.

The American Council for Technology (ACT) – Industry Advisory Council (IAC) is a non-profit educational organization established to create a more effective and innovative government. ACT-IAC provides a unique, objective and trusted forum where government and industry executives are working together to improve public services and agency operations through the use of technology. ACT-IAC contributes to better communications between government and industry, collaborative and innovative problem solving and a more professional and qualified workforce. The information, conclusions and recommendations contained in this publication were produced by volunteers from industry and government advisors supporting the objective of more effective and innovative use of technology by federal agencies. ACT-IAC volunteers represent a wide diversity of organizations (public and private) and functions. These volunteers use the ACT-IAC collaborative process, refined over thirty years of experience, to produce outcomes that are consensus-based. The findings and recommendations contained in this report are based on consensus and do not represent the views of any particular individual or organization. To maintain the objectivity and integrity of its collaborative process, ACT-IAC does not accept government funding. ACT-IAC welcomes the participation of all public and private organizations committed to improving the delivery of public services through the effective and efficient use of IT. For additional information, visit the ACT-IAC website at www.actiac.org.

The ACT-IAC Networks & Telecommunications (N&T) Community of Interest (COI) The N&T COI mission is to provide clarity, impartial feedback, and points for consideration on networks and telecom issues identified in collaboration with the federal government and industry. The N&T COI provides a forum where government and industry executives are working together on key telecommunication issues such as interoperability, information sharing, communications architectures, wireless technologies, converged internet protocol based services, security, and continuity of service. The N&T COI established a working group to facilitate collaboration between government and industry on matters concerning the upcoming FTI-2 effort.

Disclaimer This document has been prepared to contribute to a more effective, efficient and innovative government. The information contained in this report is the result of a collaborative process American Council for Technology-Industry Advisory Council (ACT-IAC) 3040 Williams Drive, Suite 500, Fairfax, VA 22031 www.actiac.org ● (p) (703) 208.4800 (f) ● (703) 208.4805 Advancing Government Through Collaboration, Education and Action Page ii TDM to IP White Paper in which a number of individuals participated. This document does not – nor is it intended to – endorse or recommend any specific technology, product or vendor. Moreover, the views expressed in this document do not necessarily represent the official views of the individuals and organizations that participated in its development. Every effort has been made to present accurate and reliable information in this report. However, ACT-IAC assumes no responsibility for consequences resulting from the use of the information herein. This paper was prepared by ACT-IAC after consultation with the Federal Aviation Administration. The information and opinions contained herein are those of the ACT-IAC and are not reflection of any planned strategy or approach to FTI-2 by the FAA. Copyright © American Council for Technology, 2017. This document may not be quoted, reproduced and/or distributed unless credit is given to the American Council for Technology- Industry Advisory Council. For further information, contact the American Council for Technology-Industry Advisory Council at (703) 208-4800 or www.actiac.org.

7.1 PSEUDOWIRE ...... 6 7.1.1 How Pseudowire Emulates TDM ...... 7 7.1.2 Available Pseudowire types ...... 8 7.2 MICROWAVE AND SATCOM ...... 9 7.3 4G/LTE WIRELESS ...... 9 7.4 TDMA-VSAT WIRELESS ...... 9 8. NETWORK ADAPTATION ARCHITECTURE ...... 10 9. NETWORK ADAPTATION MIGRATION STRATEGY ...... 11 10. NETWORK TIMING CONSIDERATIONS FOR MIGRATING FTI FROM TDM TO ETHERNET TRANSPORT ...... 12

10.1 NETWORK TIMING USING TDM TRANSPORT NETWORK ...... 13 10.2 ASYNCHRONOUS T1 TRANSPARENCY IN THE TRANSPORT NETWORK ...... 14 10.3 ETHERNET TRANSPORT FOR TDM PSEUDOWIRE ...... 15 10.4 METRO ETHERNET FORUM MEF 3 ...... 16 10.5 METRO ETHERNET FORUM MEF 8 ...... 16 11. THE INTERNET SOCIETY NETWORK WORKING GROUP RFC 5087 ...... 17

11.1 TIMING RECOVERY ...... 17 11.2 CLOCK RECOVERY ...... 19 11.3 SYNCHRONOUS NETWORK SCENARIOS ...... 20 11.3.1 Provider Edge (PE) Synchronized Network ...... 20 11.3.2 Customer Edge (CE) Synchronized Network ...... 21 11.3.3 Relative Network Scenario ...... 21 11.3.4 Adaptive Network Scenario ...... 22 11.3.5 Network Synchronization ...... 22 12. SUMMARY/RECOMMENDATIONS ...... 23 13. AUTHORS AND AFFILIATIONS ...... 24

List of Tables Table 7-1. Pseudowire Types...... 8

American Council for Technology-Industry Advisory Council (ACT-IAC) 3040 Williams Drive, Suite 500, Fairfax, VA 22031 www.actiac.org ● (p) (703) 208.4800 (f) ● (703) 208.4805 Advancing Government Through Collaboration, Education and Action Page v TDM to IP White Paper 1. Introduction The topic of TDM to IP Migration addresses the plans of virtually all local exchange carriers in the U.S. to migrate or have already migrated their networks from the current TDM based architecture to an architecture that is based on Ethernet access to an IP fabric. Although there are a number of reasons that can be cited as drivers for this migration, two primary reasons are, (1) bandwidth utilization in an IP environment is significantly more efficient than the current TDM based architectures we see today, and (2) equipment manufacturers and chip set providers are now limiting and will soon discontinue production of TDM based equipment, forcing carriers to plan for and eventually migrate to an alternative technology platform. In addition, IP networks offer much greater opportunities for the carriers to provide new products such as software defined networks, network function virtualization and a range of network and cloud based products and services. This white paper will attempt to address some of the fundamental issues relating to TDM to IP migration; it also addresses timelines, migration strategies and network adaptive technologies for those applications that continue to demand a TDM interface. Carrier specific information regarding retirement of TDM services and plans for migration of some of those services, especially the timing of service migration, are not contained within this paper as information of that nature is considered sensitive from a competitive standpoint. Given that carrier plans in this area will have a direct and significant impact on any organizations planning the acquisition of network based services, one-on-one non-disclosure briefings with carriers may provide an avenue for those planning such a migration to brief users on the specifics of their plans and how it will effect end users. 2. Background  With the explosive growth in mobile data and revenue generating applications, the carriers envision the all-IP network as an important step to future growth and shareholder value, and most importantly meeting market requirements.  Some carriers have announced sun-setting dates for TDM services and analog technologies. Others have signaled their intent to do the same. The sun-setting procedures require significant planning and coordination with regulatory commissions governing tariff pricing and service delivery. All sun-setting is done with deliberate notification timelines to customers. Sun-setting is generally based on the economics of the services, considering usage volumes, revenue, and operational expenses.

American Council for Technology-Industry Advisory Council (ACT-IAC) 3040 Williams Drive, Suite 500, Fairfax, VA 22031 www.actiac.org ● (p) (703) 208.4800 (f) ● (703) 208.4805 Advancing Government Through Collaboration, Education and Action Page 1 TDM to IP White Paper  Government agencies are working to develop plans to migrate their systems to IP technologies.  Although bandwidth for IP-based applications is growing significantly within the Federal Aviation Administration (FAA), the operational communications systems used by the FAA are still 90% TDM based.  Given the size, age, mission criticality and budgets of these government systems, there is great concern that the government agency migration plans may not align with the carrier sun-setting plans. 3. Implementation Timeline  The carriers will transition services at different times.  There will be support for TDM for many years while the move to Ethernet access gains momentum. Even as momentum shifts to Ethernet access, there will still be pockets of continued TDM support.  The specific timelines for services retirement for each individual carrier is unknown and therefore the solution cannot be timeline-based. Additionally, new replacement technologies are also a factor in creating a timeline. The solution must work today and tomorrow, regardless of when the carrier chooses to discontinue support for TDM and analog services. 4. Fundamental Issues  Timing: TDM services have very strict synchronization requirements. The FAA will migrate to IP-based applications but during the migration time period, the existing TDM based systems must be supported. Any interim solution that provides TDM support over a packet based network must provide a highly accurate and highly available synchronization capability.  Equipment obsolescence: Aging infrastructure in carrier networks is reaching its end of life. Carriers have no incentive to replace technology that is losing its customer base. Carrier investments are based on growth services and operational excellence, which is a positive condition for market leading growth services like Ethernet and fiber based solutions. In addition, it should be noted that carriers do not manufacture the equipment they place in their infrastructures. Carriers rely on manufacturers that sell their equipment in a global market where, outside the U.S., demand is primarily for IP and Ethernet. As a result, manufacturers have or are making plans to discontinue production of TDM based equipment. Furthermore, manufacturers are finding that many of the chip sets

American Council for Technology-Industry Advisory Council (ACT-IAC) 3040 Williams Drive, Suite 500, Fairfax, VA 22031 www.actiac.org ● (p) (703) 208.4800 (f) ● (703) 208.4805 Advancing Government Through Collaboration, Education and Action Page 2 TDM to IP White Paper needed to produce TDM based equipment are no longer available. These circumstances are driving carriers to an IP fabric with Ethernet access.  Loss of competent TDM technicians: With the migration away from TDM technologies, technicians are employment incentivized to learn the latest and greatest technologies (i.e., IP and Ethernet) and have no desire to learn the technologies that are being sunset by the carriers (i.e., TDM, Analog). This has caused a noticeable increase in restoration times for these types of services, and decrease in employable market staffing.  Diversity and Avoidance: Any mission-critical system requires a high level of availability and to maintain a high level of availability, it is important to design redundant paths from source to destination. The paths must be diverse such that a failure on one path will not impact the end-to end FAA service. TDM-based circuits have a discrete path they are provisioned on and are therefore fairly easy to design redundant paths that are diverse. TDM provisioned services are at times re-groomed within the network. However, with packet based networks, it becomes more difficult to ensure that a primary path and a backup path do not share any common equipment or cables. Yet, the inherent design of packet networks is to have multiple sets of routing which can make packet networks better able to redirect data for delivery than a TDM network. It is difficult to monitor the diversity over time to ensure that the carrier has not re-groomed their network and potentially negated the diversity originally designed.  Security: TDM circuits are considered dedicated circuits, meaning once they are configured, the data traversing the circuit cannot be mixed with or monitored by another entity (layer 1 separation). With the advent of Ethernet circuits, customer traffic is now separated at the data link layer or layer 2. The data passes through layer 2 switching devices which passes customer traffic over shared ports but separates the traffic via Multiprotocol Label Switching (MPLS) or Virtual Local Area Network (VLAN) tags. This level of separation may not be considered as secure as the previous TDM layer 1 separation provided.  Wireless: Although the majority of IP based systems can successfully utilize wireless Fourth Generation (4G)/ (Long Term Evolution (LTE) as an alternative to wireline, TDM Pseudowire will not work in most cases over this wireless technology. With the carriers moving away from aging copper infrastructure and migrating customers to 4G/LTE wireless technologies, the FAA will find it difficult to replace many of their remote TDM based circuits with wireless.

American Council for Technology-Industry Advisory Council (ACT-IAC) 3040 Williams Drive, Suite 500, Fairfax, VA 22031 www.actiac.org ● (p) (703) 208.4800 (f) ● (703) 208.4805 Advancing Government Through Collaboration, Education and Action Page 3 TDM to IP White Paper  Parallel Operations: Although not a technical challenge, migrating to a new access technology typically involves parallel operations costs. These costs are incurred as new Ethernet access circuits are added while still maintaining the existing TDM access circuits. Because of the parallel operations costs, it is important to transition services from the TDM circuit to the Ethernet circuit as quickly as possible to enable the disconnection of the TDM circuit. For most commercial and government enterprises, the ROI for IP based services supports the migration, as the technology provides greater efficiencies, flexibility, and scalability.  Application Testing: Given the dependencies on timing and other characteristics that were inherent with TDM technology, extensive application-level testing will be required to ensure the FAA applications work as expected utilizing the new technology. 5. FAA Migration Strategies  The FAA understands that a shift to IP communications is needed, however, migrating FAA programs to IP is a daunting task. Some FAA systems have been in service for 30 plus years and the ability to modify them is limited. Many of these programs will require a major investment to refresh the technology.  The FAA is in the process of implementing policies to address this issue. An FAA Order has been issued mandating that new FAA systems utilize Ethernet and IP for communications. Existing systems have been told to develop plans to migrate to IP.  Examples of two large FAA programs that have plans to migrate to IP are National Airspace Voice System (NVS) and Subscriber Identity Module (SIM). NVS is the new FAA voice system that is IP based and replaces the legacy analog voice system. Voice services on the NAS represent over half of all FTI communications services. SIM is the new IP based surveillance radar modernization activity that will replace existing TDM serial connections. Surveillance radar makes up about 7% of all FTI communications services.  While the FAA is implementing policies to mandate migration to IP based systems in the future, it is understood that this will take many years to implement (5 to 15 years). In parallel with these FAA migration activities, the network must provide a mechanism to transport TDM services over the new, packet-based technologies being deployed by the carriers.

American Council for Technology-Industry Advisory Council (ACT-IAC) 3040 Williams Drive, Suite 500, Fairfax, VA 22031 www.actiac.org ● (p) (703) 208.4800 (f) ● (703) 208.4805 Advancing Government Through Collaboration, Education and Action Page 4 TDM to IP White Paper  TDM Pseudowire is a technology that simulates a TDM circuit over a packet based network and has been used successfully by carriers and non-carrier entities. TDM Pseudowire utilizes packet-based timing protocols, such as Institute of Electrical and Electronics Engineers (IEEE) 1588 Telecom profile (aka Precision Timing Protocol, or PTP) to maintain synchronization over the non- synchronous network.  Although carriers are beginning to utilize these technologies to transport customer TDM services, the carrier field-force supporting these circuits are being largely trained to support the new packet-based technologies and are not focused specifically on these circuit types.  An FAA solution would benefit from having the TDM Pseudowire technology implemented at the edge, by the FTI contractor. This enables the FTI contractor field force to become trained specifically in this technology and not rely on the carrier field force, which will be busy implementing and supporting the new, packet-based technologies. 6. Existing FTI CPE Complement and Telecommunications Access FAA Telecommunications Infrastructure (FTI) Customer Premise Equipment (CPE) was designed to interoperate with carrier-provided leased TDM-based services. The CPE delivers services with various interface types to the SDP as illustrated in Figure 6-1. This figure illustrates cases from a small site with one DS-0 service delivered directly to the Session Description Protocol (SDP) and no CPE to the largest site with 1,117 services delivered via a combination of Digital Signal (DS)-1, DS-3 and OC-x circuits and associated CPE. The CPE configuration at a site is dependent on the service mix at that location and could result in multiple instances of the equipment shown in the figure. Even the IP services delivered today with Ethernet interfaces are transported over a private TDM infrastructure. This private infrastructure is critical to maintain the closed security posture of the NAS operational network.

FAA T1 DS3 System 1 or DS x C O FAA Ethernet Digital System Router Ethernet Access FAA System Carrier System FAA VG, DDC, or DDS (DACS) Optical System VG, DDC, or DDS Multiplexer FAA Multiplexer System

VG or FAA DDC System A/B Switch

FAA Multiplexer System VG, DDC, or DDS Optical VG, DDC, or DDS Multiplexer FAA Digital System Access

FAA Ethernet Carrier System System Router Ethernet (DACS) FAA O System C x T1 D S1 o FAA r DS System 3 DDC DS0 (VG) Alternate FAA Modem System SWC VG or DDS DS0 (VG or DDS) FAA System FTI Service Delivery Point (SDP)

Figure 6-1. Typical FAA Site Service Delivery Source: Undisclosed Carrier 7. Network Adaptation Technologies

7.1 Pseudowire Pseudowire is the emulation of a point-to-point connection over a Layer 2 frame- switched or Layer 3 packet-switched network. The pseudowire technology emulates the operation of a "transparent wire" carrying the service transported by an underlying packet network such as Multi-Protocol Label Switching (MPLS), Internet Protocol (IPv4 or IPv6), or Layer 2 Tunneling Protocol version 3 (L2TPv3). The first pseudowire specifications were called the “Martini” draft for ATM pseudowires, and the TDMoIP draft for transport of E1/T1 over IP11. In 2001, the Internet

Figure 7-1. Pseudowire Implementation Source: Undisclosed Carrier 7.1.1 How Pseudowire Emulates TDM TDM pseudowire technology emulates TDM service over a packet switching network using the following techniques: a. “Packetization” and Encapsulation of TDM Traffic. The TDM traffic has to be “packetized” and encapsulated before being sent to the network (Ethernet, MPLS or IP). Specific packet connectivity information is dependent on the network type. The encapsulation process places a pseudowire control word in front of the TDM data. b. Attenuate Packet Delay Variation (PDV). Packet networks create latency and more important PDV, also known as jitter. The TDM service cannot function with the jitter inherent in packet networks and so the pseudowire emulation must be able to smooth out the jitter of the packet network. This is accomplished by using a jitter buffer, which stores packets on the receive side and transmits them smoothly to the TDM link. c. Compensate for Frame Loss and Out-of-Sequence Packets. Packet networks by their nature experience loss of frames and out of sequence TDM Pseudowire frames (as a result of congestion, routing paths, etc.). The

Table 7-1. Pseudowire Types Source: Undisclosed Carrier TDM PW Type TDM Service Support Advantages Limitations & Disadvantages

SAToP Unframed Mature standards TDM service is Low overhead more Lowest end-to-end susceptible to delay Flexible frame loss and re- packet size sequence No DS0 grooming can be performed

TDMoIP Unframed, Framed, Complete support Higher delay when Channelized of TDM services in transporting several one protocol time slots due to n x 8 byte frames

CESoPSN Framed, Lower packetization No support for Channelized delay when unframed, must use transporting several SAToP time slots (DS)

7.2 Microwave and SATCOM The FTI program currently offers microwave and dedicated satellite communications (SATCOM) services for site connectivity. These proven technologies are available for use at remote sites where telecommunications providers plan to abandon copper infrastructure serving those locations. These technologies, while available for use today, are more expensive than 4G/LTE and TDMA-VSAT wireless alternatives.

7.3 4G/LTE Wireless 4G/LTE wireless solutions are not currently used on the FTI program but offer the potential for a low-cost last mile connectivity for small FAA facilities (e.g. 1-3 services) where deteriorated or abandoned wireline copper infrastructure may be too expensive to repair or replace. A prime example is the FAA’s Ventura (VTU) Very High Frequency Omnidirectional Range (VHF VOR) facility in Ventura, California. A wildfire destroyed the aerial copper cable that provided telecommunications access to this site. The local exchange carrier for the facility estimated $1M to restore the wireline connection. Similar situations are expected to arise across the NAS as copper facilities into buildings reach capacity or deteriorate and become unusable. Security and performance are issues that need to be addressed when using any commercial service offering.

7.4 TDMA-VSAT Wireless Time Division Multiple Access – Very Small Aperture Terminal (TDMA-VSAT) is a satellite communications technology. It provides another telecommunications access solution for FAA sites where terrestrial copper infrastructure has been abandoned and there is no 4G/LTE coverage to provide service. This technology shares bandwidth among its users allowing it to be less expensive than dedicated bandwidth solutions such as implemented with FTI-SAT services. Security concerns with TDMA-VSAT are similar to those of 4G/LTE, therefore encryption is recommended. Alternatively, TDMA- VSAT technology can be implemented as a private network where the hub is dedicated

American Council for Technology-Industry Advisory Council (ACT-IAC) 3040 Williams Drive, Suite 500, Fairfax, VA 22031 www.actiac.org ● (p) (703) 208.4800 (f) ● (703) 208.4805 Advancing Government Through Collaboration, Education and Action Page 9 TDM to IP White Paper to FAA traffic. However, this alternative can be expensive and would likely require hundreds of remote sites to be a cost effective solution. 8. Network Adaptation Architecture One approach to integrating the Network Adaptation technology into the FTI network focuses on solving the problem where the problem is manifested and minimizing the changes to the existing architecture as much as possible. To that end, the issue of carriers phasing out their TDM service offerings is a last-mile problem. Therefore, within each airspace, only those last mile circuits that are affected need to transition to the new technology. The result will be a mixture of last mile circuit technologies and a transition that occurs over time based on when and where the carrier circuits themselves transition. Another key goal of this approach is to ensure transparency to NAS applications. Encryption and firewall equipment provide security protection over the Carrier Ethernet access circuit. Notice that the Network Adaption is performed at the network facing interfaces and the FAA Session Description Protocol (SDP) remains unchanged. It allows the FAA to continue ordering TDM services from the FTI provider and requiring the network to perform the appropriate conversion to deliver those services. On the FTI side of the SDP, additions to the CPE are minimized by inserting the conversion device at the connection to the carrier. This approach keeps a majority of the CPE and FAA facing interfaces unchanged. As shown previously in Figure 6-1, the FTI interface to the carrier today is either Voice Grade (VG), Digital Data Service (DDS), T1, T3 or Optical Carrier (OC)-x circuits. Figure 8-1 revisits the CPE design with Carrier Ethernet circuits and pseudo wire, encryption and firewall network adaptation equipment. The new Pseudo wire equipment will accept these various interfaces as input and provide a single Ethernet output to the carrier. Timing is a critical aspect of providing TDM Pseudo wire and therefore must be a key consideration as well as security when migrating to Carrier Ethernet access circuits.

FAA Site FTI Customer Premise Equipment (CPE)

FAA DS0 (VG or DDS) System VG or DDS 10M Enet l l Primary FAA DS0 (VG) a DDC w 10M Enet

System Modem e SWC r e i

r i F 00M FAA or 1 T1 w 10 & t

System o ne E t n 3 d r DS ne o o u E S1 i +

D e FAA Ethernet t 0M s

p 0 System Digital 1 y P

Router r

Ethernet Access c FAA Carrier n System E System x OC FAA VG, DDC, or DDS (DACS) Optical System VG, DDC, or DDS Multiplexer FAA Multiplexer Network System Adaptation

VG or CPE FAA DDC System A/B Switch and Carrier Ethernet FAA Multiplexer Circuits System VG, DDC, or DDS Optical VG, DDC, or DDS Digital Multiplexer O FAA Cx System Access Carrier FAA Ethernet l l

System a System 10 Router D w 0M e Ethernet (DACS) S1 o e + r FAA r D r E i S i n 3 et System F w 1 T1 o 0 & or

d 0 FAA 0M n u Enet o System e i s t p DDC DS0 (VG) P Alternate FAA Modem y r 10M Enet System c SWC n

VG or DDS DS0 (VG or DDS) E FAA System 10M Enet FTI Service Delivery Point (SDP)

Figure 8-1. Typical FAA Site Service Delivery with Network Adaptation Source: Undisclosed Carrier In summary, the Network Adaptation approach adds pseudo wire conversion and security devices to the FTI network in locations where telecommunications carriers will no longer offer TDM circuits. Security and timing are essential components of migrating to Carrier Ethernet. 9. Network Adaptation Migration Strategy Each FAA site typically includes several NAS systems. Some of these systems are planned for upgrade to native IP implementations, some may be targeted to change their network interfaces to IP and, for the remainder, there are no plans for change and will continue to require TDM connectivity. Additionally, those systems planned for migration to IP may do so over a multi-year period as modernized systems are qualified and as funding is available for deployment. Implementing Network Adaptation at an American Council for Technology-Industry Advisory Council (ACT-IAC) 3040 Williams Drive, Suite 500, Fairfax, VA 22031 www.actiac.org ● (p) (703) 208.4800 (f) ● (703) 208.4805 Advancing Government Through Collaboration, Education and Action Page 11 TDM to IP White Paper FAA site will allow for a controlled, low risk, migration to IP technology. For example, an ATCT will typically include an air-to-ground and ground-to-ground voice system, a flight data system, terminal automation feed, and weather sensors. It is unlikely all these systems will have converted to IP and be ready for IP telecommunications at the same time. One system may be upgraded at a site while the other systems will continue to rely on TDM technology. Network Adaptation can be deployed to a site on an as-needed basis when the Local Exchange Carrier no longer offers TDM services to that site. Alternatively, Network Adaptation can be deployed where there is a cost advantage to using Carrier Ethernet technology over TDM. The carriers are moving to Carrier Ethernet technologies on different schedules. The existing FTI network has already encountered some Carrier Ethernet implementations by a few providers. Some carriers have notified their customers that they have plans to discontinue offering TDM services by 2020, while others have no plans to stop offering TDM services as conversion to Ethernet would be a significant capital investment. The Network Adaptation approach can be used by the FAA to create a capability that meets current FTI service requirements while carriers change their transport technology from TDM to IP. This approach will minimize impact to the FAA when carriers change the technology and enable the FAA to focus on migrating NAS systems to IP. 10. Network Timing Considerations for Migrating FTI from TDM to Ethernet Transport To begin this discussion it is important to understand the timing architecture of a TDM network. The inherent nature of TDM requires that all network equipment be synchronous with the same master clock reference, without that timing slips will occur and data will be lost. Network clocks are divided into Stratum levels based on their accuracy, stability, and other parameters, according to Telcordia GR-1244-core. Stratum levels are expressed as a number, sometimes along with a letter. The better the clock source the lower the Stratum level number. The primary references used in a TDM network need to meet the Stratum 1 requirements. As a clock is distributed across a network, impairments are introduced that reduce its stability, this results in the clock being classified at a higher Stratum level. Stratum 1 clocks at major network facilities synchronize to the GPS satellite network derived reference from the master atomic clock. All Telco providers around the world synchronize to this same reference. Stratum 1 Clocks typically employ a rubidium

10.1 Network timing using TDM Transport network Figure 10-1 below illustrates timing in a TDM transport network.  Each device on the TDM network must have a timing reference that is traceable to Stratum 1 clock.  The first three DACS’s on the right-hand side of the example below receive their clock reference from the DACS on the left that directly connects to the GPS source.

Figure 10-1. Network Timing Using TDM Transport Network Source: Undisclosed Carrier 10.2 Asynchronous T1 transparency in the transport network It is important to understand that a T1 circuit starts and terminates at the T1 framer; the traditional transport network simply provides a pipe and does not provide timing. To understand why let’s look at TDMs basic building blocks: 1. 24xDS0 = DS1 2. 4xDS1 = DS2 3. 7xDS2 = DS3 4. Etc.… Rate adaptation is implemented each time a DS1 is multiplexed into a DS2; this is known as bit stuffing and creates a buffer around the DS1. This allows a transport

Figure 10-2. Asynchronous T1 Transparency in the Transport Network Source: Undisclosed Carrier 10.3 Ethernet Transport for TDM Pseudowire Ethernet transport can cause severe consequences to a TDM network timing design as it is impossible to pass transparent T1 timing through the Ethernet transport network. The Ethernet transport networks can operate untimed and passes packets without issue. If TDM Pseudowire (a.k.a. Circuit Emulation) is used, the network must be timed to a Stratum 1 traceable source, this can be accomplished via Synchronous Ethernet or

10.4 Metro Ethernet Forum MEF 3 When the TDM circuit is transported via Circuit Emulation Services, this continuous signal is broken into packets at the Metro Ethernet Network-bound Inter-Working Function of the Circuit Emulation Service connection and reassembled into a continuous signal at the Customer equipment-bound Inter-Working Function of the Circuit Emulation Service connection. In essence, the continuous frequency of the TDM service clock is disrupted when the signal is mapped into packets. In order to recover the service clock frequency at the egress of the Circuit Emulation Service connection, the interworking function must employ a process that is specific to the Circuit Emulation Service interface type. The description and requirements of the Inter-Working function service clock recovery are contained in the Implementation Agreement for that service.

10.5 Metro Ethernet Forum MEF 8 Synchronization is an important consideration in any circuit emulation scheme. Put simply, the clock used to play out the data at the TDM-bound Inter-Working Function must be the same frequency as the clock used to input the data at the Metro Ethernet Network-bound Inter-Working Function, otherwise frame slips will occur over time. Summarized, there are four basic options for the TDM clock to the TDM-bound Inter- Working Function.  use the clock from the incoming TDM line (TDM line timing)  use an external reference clock source (External timing)  use a free-running oscillator (Free run timing)  recovering the clock from the Ethernet interface (Ethernet line timing)

American Council for Technology-Industry Advisory Council (ACT-IAC) 3040 Williams Drive, Suite 500, Fairfax, VA 22031 www.actiac.org ● (p) (703) 208.4800 (f) ● (703) 208.4805 Advancing Government Through Collaboration, Education and Action Page 16 TDM to IP White Paper The last option, Ethernet line timing, covers all methods where information is extracted from the Ethernet, including:  Adaptive timing, where the clock is recovered from data in the CESoETH frames, and the arrival time of the frames.  Differential timing, where the clock is recovered from a combination of data contained in the CESoETH frames, and knowledge of a reference clock common to both the Metro Ethernet Network-bound and TDM-bound Inter-Working Functions. Such a reference may be distributed by a variety of means. For maximum applicability, it is recommended that CESoETH implementations should support at least TDM line, external and adaptive timing. This will enable the implementation to be used in the majority of timing scenarios. However, not every combination of timing options for the TDM-bound Inter-Working Functions on either side of the Metro Ethernet Network will yield performance capable of meeting R47 (see below), so care must be taken to ensure the combinations chosen are appropriate. The following synchronization requirements are placed on a CESoETH implementation. Certain applications may require the use of more stringent requirements.  R47. The method of synchronization used MUST be such that the TDM-bound Inter-Working Function meets the traffic interface requirements specified in ITU-T recommendations [G.824] for DS1 and DS3 circuits.  R48. Jitter and wander that can be tolerated at the Metro Ethernet Network- bound Inter-Working Function. TDM input MUST meet the traffic interface requirements specified in ITU-T recommendations [G.824] for DS1 and DS3 circuits. Note: The requirements set forth [in G.824] are consistent with DS1/DS3 interface requirements specified in Telcordia’s [GR-253-CORE]. The pertinent traffic interface requirement is R5-237. 11. The Internet Society Network Working Group RFC 5087

11.1 Timing Recovery TDM networks are inherently synchronous; somewhere in the network there will always be at least one extremely accurate primary reference clock, with long-term accuracy of one part in 1E-11. This node provides reference timing to secondary nodes with somewhat lower accuracy, and these in turn distribute timing information further. This hierarchy of time synchronization is essential for the proper functioning of the network

American Council for Technology-Industry Advisory Council (ACT-IAC) 3040 Williams Drive, Suite 500, Fairfax, VA 22031 www.actiac.org ● (p) (703) 208.4800 (f) ● (703) 208.4805 Advancing Government Through Collaboration, Education and Action Page 17 TDM to IP White Paper as a whole; for details see [G823] and [G824]. Figure 11-1 shows a high-level view of a Network Synchronization Reference Model. Packets in Packet Switched Networks (PSNs) reach their destination with delay that has a random component, known as packet delay variation (PDV). When emulating TDM on a PSN, extracting data from the jitter buffer at a constant rate overcomes much of the high frequency component of this randomness ("jitter"). The rate at which we extract data from the jitter buffer is determined by the destination clock, and were this to be precisely matched to the source clock proper timing would be maintained. Unfortunately, the source clock information is not disseminated through a PSN, and the destination clock frequency will only nominally equal the source clock frequency, leading to low frequency ("wander") timing inaccuracies. In broadest terms, there are four methods of overcoming this difficulty. In the first and second methods timing information is provided by some means independent of the PSN. This timing may be provided to the TDM end systems (method 1) or to the Inter- Working Functions (method 2). In a third method, a common clock is assumed available to both Inter-Working Functions and the relationship between the TDM source clock and this clock is encoded in the packet. This encoding may take the form of RTP timestamps or may utilize the synchronous residual timestamp (SRTS) bits in the AAL1 overhead. In the final method (adaptive clock recovery) the timing must be deduced solely based on the packet arrival times. Adaptive clock recovery utilizes only observable characteristics of the packets arriving from the PSN, such as the precise time of arrival of the packet at the TDM-bound Inter- Working Function, or the fill-level of the jitter buffer as a function of time. Due to the packet delay variation in the PSN, filtering processes that combat the statistical nature of the observable characteristics must be employed. Frequency Locked Loops (FLL) and Phase Locked Loops (PLL) are well suited for this task. Whatever timing recovery mechanism is employed, the output of the TDM-bound Inter-Working Function must conform to the jitter and wander specifications of TDM traffic interfaces, as defined in [G823][G824]. For some applications, more stringent jitter and wander tolerances may be imposed. The Internet Society Network Working Group RFC 4197

Exploded View PE

PHY DEC PHY

PHY ENC PHY

K G CE1 J PE1 S1 S2 PE2 H CE2

P D P P E P H E H H N H Y C Y Y C Y Carrier Ethernet Network

P E P P D P H N H H E H Y C Y Y C Y

B E A C D F

L Figure 11-1. Network Synchronization Reference Model (High level view) Source: Undisclosed Carrier  CE1, CE2 = Customer edge devices terminating TDM circuits to be emulated.  PE1, PE2 = Provider edge devices adapting these end services to TDM Pseudowire.  S1, S2 = Provider core routers  Phy = Physical interface terminating the TDM circuit.  Enc = PSN-bound interface of the TDM Pseudowire, where the encapsulation takes place.  Dec = CE-bound interface of the TDM Pseudowire, where the de-encapsulation takes place. It contains a compensation buffer (also known as the "jitter buffer") of limited size.

 = TDM attachment circuits.

 = TDM Pseudowire providing edge-to-edge emulation for the TDM circuit.  The characters "A" - "L" denote various clocks:

 "A" The clock used by CE1 for transmission of the TDM attachment circuit towards CE1.  "B" The clock recovered by PE1 from the incoming TDM attachment circuit. "A" and "B" always have the same frequency.  "G" The clock used by CE2 for transmission of the TDM attachment circuit towards CE2.  "H" The clock recovered by PE2 from the incoming TDM attachment circuit. "G" and "H" always have the same frequency.  "C", "D” Local oscillators available to PE1 and PE2, respectively.  "E" Clock used by PE2 to transmit the TDM attachment service circuit to CE2 (the recovered clock).  "F" Clock recovered by CE2 from the incoming TDM attachment service ("E and "F" have the same frequency).  "I" If the clock exists, it is the common network reference clock available to PE1 and PE2.  "J" Clock used by PE1 to transmit the TDM attachment service circuit to CE1 (the recovered clock). A requirement of edge-to-edge emulation of a TDM circuit is that clock "B" and "E", as well as clock "H" and "J", are of the same frequency. The most appropriate method will depend on the network synchronization scheme. The following groups of synchronization scenarios can be considered:

11.3 Synchronous Network Scenarios Depending on which part of the network is synchronized by a common clock, there are two scenarios: 11.3.1 Provider Edge (PE) Synchronized Network Figure 11-2 is an adapted version of the generic network reference model, and presents the PE synchronized network scenario. The common network reference clock "I" is available to all the PE devices, and local oscillators "C" and "D" are locked to "I".

Figure 11-2. Synchronous Network Scenarios Source: Undisclosed Carrier 11.3.2 Customer Edge (CE) Synchronized Network Figure 11-3 is an adapted version of the generic network reference model, and presents the CE synchronized network scenario. The common network reference clock "L" is available to all of the CE devices, and local oscillators "A" and "G" are locked to "L" No timing information has to be transferred in these cases.

CE1 PE1 S1 S2 PE2 CE2

Carrier Ethernet Network

A G

L Figure 11-3. CE Synchronized Network Source: Undisclosed Carrier 11.3.3 Relative Network Scenario

G CE1 PE1 S1 S2 PE2 CE2

Carrier Ethernet Network

I Figure 11-4. Relative Network Scenario Source: Undisclosed Carrier

11.3.4 Adaptive Network Scenario  Synchronizing clocks "A" and "E" in this scenario (Figure 11-5) is more difficult than it is in the other scenarios.  Note that the tolerance between clocks "A" and "E" must be small enough to ensure that the jitter buffer does not overflow or underflow.  In this case, timing information MAY be explicitly transferred from the ingress PE to the egress PE, e.g., by RTP.

11.3.5 Network Synchronization The encapsulation layer must provide synchronization services that are sufficient to match the ingress and egress end service clocks regardless of the specific network synchronization scenario, and keep the jitter and wander of the egress service clock within the service-specific limits defined by the appropriate normative references.

J G CE1 PE1 S1 S2 PE2 CE2

Carrier Ethernet Network

A E

Figure 11-5. Network Synchronization Source: Undisclosed Carrier 12. Summary/Recommendations Carrier Ethernet can be used on the FTI network to transport most TDM traffic needs, however, timing, must be considered in every case. This must be managed very carefully on the FTI TDM network as specific information will need to be noted on every service affected by Ethernet transport.  Traditional T1 troubleshooting practices will be affected as T1 circuit behavior will be different.  Network timing will be affected as FTI will not be able to guarantee Stratum 1 traceability.  Carrier network architecture may differ greatly across the NAS. - Network may have all or some PE devices untimed as network synchronization is generally not needed in a packet network and thus quite often ignored. - Network may have all PE devices timed from a Stratum 1 source.  FTI has no way to trace this source or to guarantee that it’s redundant as there is no clear standard followed yet by the industry.  Network PE devices may be independent point to point with no Stratum 1 reference at all.  Network PE devices may use another customer’s circuit as a timing reference for the entire system; this would put the FTI service at risk.  For these reasons the FTI network should not rely on carrier timing solutions but implement FTI specific timing solutions over the carrier network.

13. Authors and Affiliations

Michael Peterson Century Link Steve Dempsey Cisco Howard Heller Harris Corporation Kymber Weese Mitre Corporation