1 TSN Algorithms for Large Scale Networks: A Survey and Conceptual Comparison Ahmed Nasrallah, Venkatraman Balasubramanian, Akhilesh Thyagaturu, Martin Reisslein, and Hesham ElBakoury Abstract—This paper provides a comprehensive survey of to multimedia) industrial control, automotive, and avionics queueing and scheduling mechanisms for supporting large scale applications, and mobile backhaul that require just-in-time deterministic networks (LDNs). The survey finds that extensive delivery of data traffic. Similarly, the IETF Deterministic mechanism design research and standards development for LDNs has been conducted over the past few years. However, these Networking (DetNet) group is working in collaboration with mechanism design studies have not been followed up with a the TSN group to develop standardization of IP (L3) layer comprehensive rigorous evaluation. The main outcome of this deterministic forwarding services. survey is a clear organization of the various research and stan- The main enabler for synchronous traffic deterministic dardization efforts towards queueing and scheduling mechanisms forwarding services is the Cyclic Queuing and Forwarding for LDNs as well as the identification of the main strands of mechanism development and their interdependencies. Based (CQF) protocol. More specifically, the CQF protocol typically on this survey, it appears urgent to conduct a comprehensive combines a Time-Aware Shaper (TAS) at the egress port of rigorous simulation study of the main strands of mechanisms. a switch and Per-Stream Filtering and Policing (PSFP) at the ingress port of a switch, to shape and regulate the transmission selection. This transmission selection within switches utilizes I. INTRODUCTION time division multiplexing based on an underlying time syn- A. Motivation chronization in L2 bridged networks [2]. The CQF protocol Traditional services for packet switched networks involved results in delays that are a function of the Cycle Time (CT), a best effort process that handled and adequately ensured which is typically set according to the Quality of Service average latencies. Typically, different methods exist (e.g., (QoS) characteristics of all Scheduled Traffic (ST) flows, and explicit congestion notification, flow and congestion control) the number of hops. that feeds back notifications to control and to “slow” down In this paper, we provide a comprehensive survey of the the data rates of different applications. This control ensures various scheduling (forwarding) mechanisms for ensuring de- network stability and overall fairness and inter-operation be- terministic QoS in large scale networks. Following [3], we tween different stream/flows in a network of many disparate define a large-scale network as a network that covers a large applications operating on the common converged Ethernet geographic area so that there are long propagation delays be- technology networks. However, throttling data rates due to tween network nodes and switches; in particular, a large scale network fluctuations is not an option for real-time applications network is a multi-hop network with long propagation delays (e.g., cyber-physical systems) due to the nature of these ap- between adjacent switches. Moreover, a large-scale network plications [1]. Moreover, accurately determining upper latency has a large number of network nodes and switches as well bounds, and guaranteeing zero packet loss, and minimal jitter as a large number of traffic flows. More specifically, a large- (delay variation) is severely limited in traditional networks. scale deterministic network (LDN) is a large-scale network Traditional networking technologies can provide these guaran- with a large number of traffic flows requiring deterministic arXiv:1905.08478v2 [cs.NI] 19 Jun 2019 tees typically only with sophisticated highly engineered middle quality of service. Most scheduling mechanisms for LDNs boxes over small scale networks. A deterministic forwarding are variations of underlying CQF scheduling principles. We service is highly desirable for strict real-time applications comprehensively survey these variations of CQF as well as that enables the convergence of Information and Operational their advantages, as well as shortcomings and limitations. We Technology (IT/OT) under a unified Ethernet technology. outline the implication for future research. The IEEE 802.1 TSN working group (evolved from Au- dio/Video Bridging, AVB group) is working to develop Time B. Contributions Sensitive Networking (TSN) standardizations that target deter- ministic forwarding applications and bridging between layer 2 Our main contribution is in the form of a comparative networks. Specifically, applications that involve (in addition analysis of forwarding protocols for LDNs. i) We outline the main considerations in designing and Please direct correspondence to M. Reisslein ([email protected]). A. Nasrallah, V. Balasubramanian, A.S. Thyagaturu, and M. Reisslein are applying a TSN/DetNet forwarding mechanisms for large with the School of Electrical, Computer, and Energy Eng., Arizona State Uni- scale DetNets (LDNs). versity, Tempe, AZ 85287-5706, USA, Phone: 480-965-8593, Fax: 480-965- ii) We survey DetNet related standards and other information 8325, (e-mail: fahnasral, [email protected], athyagat, [email protected]). H. ElBakoury is with Futurewei Technologies Inc., 2330 Central Express- sources for approaches used to guarantee DetNet and way, Santa Clara, CA, 95050, USA (e-mail: [email protected]). TSN QoS requirements for LDNs. 2 ii) We present a comparative analysis between the main queued and credit is increased by idleSlope. When the frame proposals for LDNs and highlight the advantages and is transmitted, the credit value decreases by sendSlope. If any limitations for each. additional frame is waiting after a CBS frame is transmitted, then it can be transmitted (back to back) if the credit is greater C. Organization than or equal to zero. Otherwise, it is queued while the credit is increased by idleSlope and other traffic can use the channel. Section III provides background on the current state of the This effectively spreads out the CBS traffic and avoids traffic TSN and DetNet developments. Sections III-A and IV present burstiness that could cascade downstream to cause latency and the surveyed proposed protocols from standards and from jitter problems. Note that the SRP [4] is used in conjunction academic research. These efforts typically start from CQF and with CBS to register and reserve available bandwidth. continue to derivatives of CQF. Throughout, we discussion 2) Time-Aware Shaper (TAS): The IEEE 802.1Qbv and compare the presented approaches. Finally, Section V (TAS) [8] standard proposes to emulate time-division mul- concludes the paper. tiplexing for all the queues at a switch port egress using timed gates that open/close according to a prescribed schedule, II. BACKGROUND: IEEE 802.1 TIME SENSITIVE allowing frames full access to the egress link with zero NETWORKING (TSN) AND DETERMINISTIC NETWORKING interference from other queues. An open/close instruction is (DETNET) referred to as a Gate Control Entry (GCE) that dictates which This section provides a brief overview of the standards and queues to allow access to the transmission medium. The entire research in TSN and DetNet that are relevant to large scale cyclic sequence of GCEs is referred to as the Gate Control networks. TSN has evolved from AVB due to the growing List (GCL) which is configured by a network administrator demands industrial applications, e.g., Internet of Things (IoT) or a central management entity. After a sequence of a GCL and Industry 4.0. TSN promises to provide flows or streams, for a particular switch egress port finishes, it repeats starting which are sequences of data packets belonging to an end-to- from the first GCE again. Each GCE is opened for a limited end communication between a talker (sender) and listeners time governed by the window time (or slot time) with a (receivers), with Ultra Low Latency (ULL) with bounded desired transmission selection algorithm once the queue gate delays, zero congestion packet loss, and very small jitter. These is opened. For an example system with two traffic classes, so-called scheduled traffic (ST) flows that receive the ULL there are commonly two types of GCEs, one for high-priority service may coexist with best effort traffic flows. AVB started Scheduled Traffic (ST) and another one for low-priority Best with the enhanced clock synchronization (IEEE 1588v2), Effort traffic (BE). The ratio of the ST window duration to the 802.1Qat [4], and 802.1Qav [5], stream reservation protocol BE window duration in a cycle time governs the prioritization (SRP) and credit-based shaper (CBS), respectively. Due to the level of the ST traffic. growing popularity and success of these protocols in profes- 3) Asynchronous Traffic Shaper (ATS): The ATS sional audio/video production, research and standardization shaper [13] is based on the Urgency-based Scheduler into determinism started to grow in tandem, prompting the (UBS) by Specht et al. [14]. The UBS follows a per-flow IEEE 802.1 TSN group to start developing real-time Ethernet shaped queueing scheme under the flow’s defined Ethernet standardization for industrial and automotive applications in PCP priority and another subsequent set of FIFO shared L2 and IETF DetNet in L3. Several standards have already queues using Internal Priority Values (IPV) assigned after the been published, including, i) Frame Preemption (802.1Qbu first set of per-hop