1 Ultra-Low Latency (ULL) Networks: The IEEE TSN and IETF DetNet Standards and Related 5G ULL Research Ahmed Nasrallah, Akhilesh S. Thyagaturu, Ziyad Alharbi, Cuixiang Wang, Xing Shao, Martin Reisslein, and Hesham ElBakoury Abstract—Many network applications, e.g., industrial control, tactile Internet [2], [3], and on the order of 100 microsec- demand Ultra-Low Latency (ULL). However, traditional packet onds for the one-way fronthaul in wireless cellular networks. networks can only reduce the end-to-end latencies to the order of For example, critical healthcare applications, e.g., for tele- tens of milliseconds. The IEEE 802.1 Time Sensitive Networking (TSN) standard and related research studies have sought to surgery, and transportation applications [4] require near real- provide link layer support for ULL networking, while the time connectivity. Throughput requirements largely dependent emerging IETF Deterministic Networking (DetNet) standards on the application needs, which may vary widely from small seek to provide the complementary network layer ULL support. amounts of IoT data to large exchanges of media data transfers This article provides an up-to-date comprehensive survey of the to and from the cloud (or the fog to reduce latency) [5]. IEEE TSN and IETF DetNet standards and the related research studies. The survey of these standards and research studies is Additionally, autonomous automotive vehicles [6], augmented organized according to the main categories of flow concept, and virtual reality (AR/VR), as well as robotic applications, flow synchronization, flow management, flow control, and flow which are essential for Industrial IoT (IIoT), may require both integrity. ULL networking mechanisms play a critical role in high data rates as well as ULL [7]–[10]. The high data rates the emerging fifth generation (5G) network access chain from may be required for transporting video feeds from cameras wireless devices via access, backhaul, and core networks. We survey the studies that specifically target the support of ULL in that are used to control vehicles and robots [11]. Therefore, in 5G networks, with the main categories of fronthaul, backhaul, such heterogeneous environments and applications, a dedicated and network management. Throughout, we identify the pitfalls mechanism to universally accommodate a diverse range of and limitations of the existing standards and research studies. ULL requirements would be very helpful [12]. This survey can thus serve as a basis for the development of standards enhancements and future ULL research studies that address the identified pitfalls and limitations. B. Contributions and Organization of this Survey Index Terms—Deterministic networking (DetNet), Preemption, This article provides a comprehensive up-to-date survey of Time-sensitive networking (TSN), Time synchronization, Ultra- low delay. standards and research studies addressing networking mecha- nisms for ULL applications. Section III covers the IEEE TSN standards that have grown out of the AVB standards and focus I. INTRODUCTION primarily on the link layer, while Section IV covers the ULL A. Motivation research studies related to TSN. Section V covers the Internet Traditional networks which provide end-to-end connectivity Engineering Task Force (IETF) Deterministic Networking to the users have only been successful in reducing the operat- (DetNet) standards developments, while Section VI covers ing end-to-end latencies to the order of tens of milliseconds. the ULL research studies related to DetNet. This sequence arXiv:1803.07673v3 [cs.NI] 24 Sep 2018 However, present and future applications demand Ultra-Low of the section on standards followed by the section on related Latency (ULL). For instance, the end-to-end latencies should research studies is inspired by the temporal sequence of the be on the order of a few microseconds to a few milliseconds development of the ULL field, where standard development for industrial applications [1], around 1 millisecond for the has typically preceded research studies. A large portion of the ULL applications will likely involve Please direct correspondence to M. Reisslein. A. Nasrallah, A.S. Thyagaturu, Z. Alharbi, and M. Reisslein are with the wireless communications, whereby the fifth generation (5G) School of Electrical, Computer, and Energy Eng., Arizona State University, wireless systems will play a prominent role. In particular, Tempe, AZ 85287-5706, USA, Phone: 480-965-8593, Fax: 480-965-8325, (e- the emerging tactile Internet paradigm with end-to-end target mail: fahnasral, athyagat, zalharbi, [email protected]). C. Wang and X. Shao are with the Dept. of Internet of Things latencies below 1 ms is tightly coupled to the ongoing 5G Eng., Yancheng Inst. of Techn., Yancheng, Jiangsu Province, P.R. China (e- developments [5], [13]–[16]. The support of 5G wireless ULL mail: [email protected], [email protected]). C. Wang and X. Shao visited communications services will likely heavily rely on the TSN Arizona State Univ., Tempe, while contributing to this study. C. Wang was supported by the Overseas Training Program of the Yancheng Institute of and DetNet standards and research results. On the other hand, Technology. X. Shao was supported by a Jiangsu Provincial Government due to prevalence and importance of wireless communications Scholarship for Overseas Learning. in today’s society, the particular 5G wireless context and H. ElBakoury is with Futurewei Technologies Inc., 2330 Central Expressway, Santa Clara, CA, 95050, USA (e-mail: requirements will likely influence the future development of [email protected]). ULL standards development and research. We believe that for 2 a thorough understanding of the complete ULL research area II. BACKGROUND it is vital to comprehensively consider the ULL standards, A. Latency Terminology namely TSN and DetNet, as well as a main “application domain” of ULL standards and research results. We anticipate Generally, latency refers to the total end-to-end packet delay that 5G wireless communications will emerge as a highly from the instant of the beginning of transmission by the sender important application domain of ULL standards and research (talker) to the complete reception by the receiver (listener). results and we therefore survey ULL related standards and The term ultra-low latency (ULL) commonly refers to latencies research studies for 5G wireless systems in Section VII. that are very short, e.g., on the order of a few milliseconds or Section VIII identifies the main gaps and limitation of the less than one millisecond. ULL applications often require de- existing TSN and DetNet standards as well as ULL related terministic latency, i.e., all frames of a given application traffic 5G standards and research studies and outlines future research flow (connection) must not exceed a prescribed bound [44], directions to address these gaps and limitations. e.g., to ensure the proper functioning of industrial automation systems. It is also possible that applications may require probabilistic latency, i.e., a prescribed delay bound should be met with high probability, e.g., for multimedia streaming C. Related Literature systems [45], [46], where rare delay bound violations have negligible impact of the perceived quality of the multimedia. While to the best of our knowledge there is no prior survey Latency jitter, or jitter for short, refers to the packet latency on time sensitive networking (TSN), there are prior surveys on variations. Often ULL systems require very low jitter. Latency topics that relate to TSN. We proceed to review these related and jitter are the two main quality of service (QoS) metrics surveys and differentiate them from our survey. for ULL networking. We note that there are a wide range of A survey on general techniques for reducing latencies in ULL applications with vastly different QoS requirements, see Internet Protocol (IP) packet networks has been presented Table I. For instance, some industrial control applications have in [17]. The survey [17] gave a broad overview of the very tight delay bounds, e.g., only a few microseconds, while sources of latencies in IP networks and techniques for latency other industrial control applications have more relaxed delay reduction that have appeared in publications up to August bounds up to a millisecond. 2014. The range of considered latency sources included the network structure, e.g., aspects of content placement and service architectures, the end point interactions, e.g., aspects of B. IEEE 802.1 Overview transport initialization and secure session initialization, as well Before we delve into the standardization efforts of the as the delays inside end hosts, e.g., operating system delays. IEEE Time-Sensitive Network (TSN) Task Group (TG), we In contrast, we provide an up-to-date survey of the IEEE Time briefly explain the organizational structure of the IEEE 802.1 Sensitive Networking (TSN) standard for the link layer and the Working Group (WG). The 802.1 WG is chartered to develop Deterministic Networking (DetNet) standard for the network and maintain standards and recommended practices in the layer, and related research studies. Thus, in brief, whereas the following areas: 1) 802 LAN/MAN architecture, 2) inter- survey [17] broadly covered all latency sources up to 2014, networking among 802 LANs, MANs, and other wide area we comprehensively cover the link and network layer latency networks, 3) security, 4) 802 overall network management, and reduction standards and studies up to July 2018. 5) protocol layers above the MAC and LLC layers. Currently, A few surveys have examined specific protocol aspects that there are four active task groups in this WG: 1) Time Sensitive relate to latency, e.g., time synchronization protocols have Networking, 2) Security, 3) Data Center Bridging, and 4) been surveyed in [18], [19], routing protocols have been OmniRAN. surveyed in [20]–[22], while congestion control protocols have The main IEEE 802.1 standard that has been continuously been covered in [23], [24]. Several surveys have covered revised and updated over the years is IEEE 802.1Q-2014 [58], latency reduction through mobile edge and fog computing, formally known as the IEEE 802.1D standard.