Transport Area Working Group M. Amend Internet-Draft Deutsche Telekom Intended Status: Experimental A

Transport Area Working Group M. Amend Internet-Draft Deutsche Telekom Intended status: Experimental A. Brunstrom Expires: January 9, 2020 A. Kassler Karlstad University V. Rakocevic City University of London July 08, 2019

Lossless and overhead free DCCP - UDP header conversion (U-DCCP) draft-amend-tsvwg-dccp-udp-header-conversion-01

Abstract

The Datagram Congestion Control Protocol (DCCP) is a transport-layer protocol that provides upper layers with the ability to use non- reliable congestion-controlled flows. DCCP is not widely deployed in the Internet, and the reason for that can be defined as a typical example of a chicken-egg problem. Even if an application developer decided to use DCCP, the middle-boxes like firewalls and NATs would prevent DCCP end-to-end since they lack support for DCCP. Moreover, as long as the protocol penetration of DCCP does not increase, the middle-boxes will not handle DCCP properly. To overcome this challenge, NAT/NATP traversal and UDP encapsulation for DCCP is already defined. However, the former requires special middle-box support and the latter introduces overhead. The recent proposal of a multipath extension for DCCP further underlines the challenge of efficient middle-box passing as its main goal is to be applied over the Internet, traversing numerous uncontrolled middle-boxes. This document introduces a new solution which disguises DCCP during transmission as UDP without requiring middle-box modification or introducing any overhead.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

Amend, et al. Expires January 9, 2020 [Page 1] Internet-Draft DCCP - UDP header conversion July 2019

This Internet-Draft will expire on January 9, 2020.

This document is subject to BCP 78 and the IETF Trust’s Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

1. Introduction ...... 2 2. Terminology ...... 3 3. U-DCCP ...... 3 3.1. Overview ...... 3 3.2. The DCCP Generic header ...... 4 3.3. UDP header ...... 5 3.4. U-DCCP conversion considerations ...... 6 3.5. U-DCCP header ...... 6 3.6. Implementation ...... 7 3.7. Pseudo-code DCCP to U-DCCP conversion ...... 7 3.8. Pseudo-code U-DCCP to DCCP restoration ...... 8 3.9. U-DCCP negotiation (required????) ...... 9 4. Security Considerations ...... 9 5. IANA Considerations ...... 9 6. Notes ...... 9 7. Acknowledgments ...... 9 8. Informative References ...... 9 Authors’ Addresses ...... 10

1. Introduction

The Datagram Congestion Control Protocol (DCCP) [RFC4340] is a transport-layer protocol that provides upper layers with the ability to use non-reliable congestion-controlled flows. The current specification for DCCP [RFC4340] specifies a direct native encapsulation in IPv4 or IPv6 packets.

DCCP support has been specified for devices that use Network Address Translation (NAT) or Network Address and Port Translation (NAPT)

Amend, et al. Expires January 9, 2020 [Page 2] Internet-Draft DCCP - UDP header conversion July 2019

[RFC5597]. However, there is a significant installed base of NAT/ NAPT devices that do not support [RFC5597]. An UDP encapsulation for DCCP [RFC6773] circumvents such limitations and makes DCCP compatible with any UDP [RFC0768] compliant device that supports [RFC4787] but does not support [RFC5597]. For convenience, the standard encapsulation for DCCP [RFC4340] (including [RFC5596] and [RFC5597] as required) is referred to as DCCP-STD, whereas the UDP encapsulation for DCCP [RFC6773] is referred to as DCCP-UDP.

It can be stated that DCCP-STD and DCCP-UDP are techniques which increase the success rate of DCCP transmissions significantly. However, DCCP-STD fails on devices that block DCCP for any reasons. On the other hand, DCCP-UDP uses the well-accepted UDP to let devices assume they are handling the UDP protocol, but at the cost of a reduced goodput/throughput ratio.

To compensate for the inefficiency of DCCP-STD (device blocking) and DCCP-UDP (overhead), this document proposes a beneficial modification scheme relying on UDP (like DCCP-UDP), but with no overhead. This goal is reached by re-arranging DCCP’s extended header to make it look like UDP, without losing critical information. This solution is referred to as U-DCCP.

U-DCCP is limited to DCCP’s extended header, requiring X is set to 1. Otherwise U-DCCP relies on the NAT/NATP functionalities specified for UDP in [RFC4787], [RFC6888] and [RFC7857].

2. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].

3. U-DCCP

3.1. Overview

The basic approach of U-DCCP is to modify the extended header of a DCCP packet so that it appears like UDP [RFC0768]. In particular, this takes place without losing any header information, but requires a U-DCCP termination before the packet is delivered to the DCCP end system . This method does not change the 4-tuple of IP and port addressing, however it changes the protocol carried over IP from DCCP to UDP. As a consequence, the length of the packet remains unchanged and behaves like DCCP-STD. The solution is not a tunneling approach. It requires that the same port used by DCCP can be used by UDP.

Amend, et al. Expires January 9, 2020 [Page 3] Internet-Draft DCCP - UDP header conversion July 2019

The method is designed to support use when the IP addresses are modified by a device that implements NAT/NAPT. A NAT translates the IP addresses, which impacts the transport-layer checksum. A NAPT device may also translate the port values (usually the source port). In both cases, the outer transport header that includes these values would need to be updated by the NAT/NAPT.

U-DCCP supports IPv4 and IPv6.

The basic format of a U-DCCP packet is:

Figure 1: Format of U-DCCP packet

The U-DCCP header is described in Section 3.4 after introducing the traditional DCCP header in Section 3.1 and its target appearance of a UDP header in Section 3.2. Section 3.3 discusses considerations for building the U-DCCP header upfront.

3.2. The DCCP Generic header

The DCCP Generic Header [RFC4340] takes two forms: one with long sequence numbers (48 bits) and the other with short sequence numbers (24 bits). The short one is not part of U-DCCP’s modification.

Amend, et al. Expires January 9, 2020 [Page 4] Internet-Draft DCCP - UDP header conversion July 2019

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Port | Dest Port | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Offset | CCVal | CsCov | Checksum | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | |X| | . | Res | Type |=| Reserved | Sequence Number (high bits) . | | |1| | . +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Sequence Number (low bits) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 2: The extended DCCP Header with Long Sequence Numbers [RFC4340]

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Port | Dest Port | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Data Offset | CCVal | CsCov | Checksum | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | |X| | | Res | Type |=| Sequence Number (low bits) | | | |0| | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 3: The short DCCP Header with Short Sequence Numbers [RFC4340]

All generic header fields have the meaning specified in [RFC4340], updated by [RFC5596].

3.3. UDP header

Figure 4: The UDP Header [RFC768]

All header fields have the meaning specified in [RFC0768].

Amend, et al. Expires January 9, 2020 [Page 5] Internet-Draft DCCP - UDP header conversion July 2019

3.4. U-DCCP conversion considerations

The U-DCCP header has the goal to merge the information of DCCP’s extended header (Section 3.1) and imitates in the first 64 bits the UDP header (Section 3.2). Information required to restore a DCCP header from any conversion, which must not be lost, includes: source and destination port, Data Offset, CCVal, CsCov, Checksum, Type, X and the Sequence Number.

Compared with the UDP header, the DCCP extented header shows similarities in source and destination port and checksum. The length field of UDP (bits 33-48) is not part of the DCCP header and contains in case of DCCP the fields Data Offset, CCVal and CsCov.

For the goal of imitating UDP, the checksum must cover the whole datagram, which renders any limitation by CsCov useless. The checksum itself is required to re-calculate after conversion anyway.

If the conversion is limited to DCCP’S extended header only, X is always "1".

Thus, Data Offset, CCVal, Type and Sequence Number must be re- arranged in a way that the Length field of UDP can be applied.

3.5. U-DCCP header

The considerations of Section 3.3 leads to the following header, denoted as U-DCCP header.

0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ U | Source Port | Dest Port | D +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ P | Length | Checksum | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Type | CCVal | Data Offset | Sequence Number (high bits) . +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ . Sequence Number (low bits) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 5: The U-DCCDP Header

The first 8 bytes of the U-DCCP header corresponds to [RFC0768] and the fields are interpreted as follows:

Source and Dest(ination) Ports: 16 bits each

Amend, et al. Expires January 9, 2020 [Page 6] Internet-Draft DCCP - UDP header conversion July 2019

These fields identify the UDP ports used by the source and destination (respectively) of the packet to listen for incoming UDP packets. The UDP port values identify the DCCP source and destination ports.

Length: 16 bits

This field is the length of the UDP datagram, including the UDP header and the payload (for U-DCCP, the payload comprises the payload of the original DCCP datagram and part of its header).

Checksum: 16 bits

This field is the Internet checksum of a network-layer pseudoheader and Length bytes of the UDP packet [RFC0768]. The UDP checksum MUST NOT be zero for a U-DCCP packet.

The remaining 8 bytes of the U-DCCP header contains:

Type, CCVal, Data Offset, Seq. Number: As specified in [RFC4340]

In case U-DCCP is applied, the IP layer must be instructed to carry an UDP datagram and its checksum must be re-calculated. For detailed information see Section 3.7.

3.6. Implementation

The process of applying U-DCCP is defined as follows:

DCCP generation -> U-DCCP conversion -> UDP transmission -> U-DCCP reception and restoration -> DCCP reception

The conversion can be integrated into DCCP endpoints directly or as an additional component on the way along the transmission route. Depending on the degree of integration, especially the process of checksum calculation and validation can be optimized. Section 3.7 and Section 3.8 provide a possible pseudo-code for the conversion without any optimized integration into the sender’s network stack or into the receiver’s network stack. The pseudo-code assumes explicit knowledge on which U-DCCP flows need conversion between the sender and the receiver.

3.7. Pseudo-code DCCP to U-DCCP conversion

A possible processing of an already generated DCCP datagram for U-DCCP conversion:

1. Receive DCCP datagram.

Amend, et al. Expires January 9, 2020 [Page 7] Internet-Draft DCCP - UDP header conversion July 2019

2. Check eligibility for conversion; otherwise bypass conversion.

3. Verify consistency, e.g. checksum; otherwise drop.

4. Shift Type and CCVal field to the ninth octet.

5. Shift Data Offset field to the tenth octet.

6. Place a length information at octet 5+6 corresponding to [RFC0768].

7. Modify the IP header’s encapsulated protocol from DCCP to UDP.

8. Re-calculate IP header checksum.

9. Reset DCCP checksum field: octet 7+8 = 0.

10. Generate new checksum at octet 7+8 as described in [RFC0768].

11. Forward to destination based on the unmodified 4-tuple of IP- addresses and ports.

3.8. Pseudo-code U-DCCP to DCCP restoration

A possible processing of an already converted U-DCCP datagram for DCCP restoration:

1. Receive UDP datagram.

2. Check eligibility for restoration; otherwise bypass restoration

3. Validate UDP checksum; otherwise drop.

4. Restore Data Offset field according to [RFC4340].

5. Restore CCVal field according to [RFC4340].

6. Set CsCov field according to [RFC4340] to "0".

7. Restore Type field according to [RFC4340].

8. Set Reserved bits according to [RFC4340] to "0".

9. Set X according to [RFC4340] to "1".

10. Modify the IP header’s encapsulated protocol from UDP to DCCP.

11. Re-calculate IP header checksum.

Amend, et al. Expires January 9, 2020 [Page 8] Internet-Draft DCCP - UDP header conversion July 2019

12. Reset DCCP checksum field: octet 7+8 = 0.

13. Generate new checksum at octet 7+8 as described in [RFC0768].

14. Forward to destination based on the unmodified 4-tuple of IP- addresses and ports.

3.9. U-DCCP negotiation (required????)

Tbd later if required. Otherwise assumes explicit knowledge about the U-DCCP conversion between sender and receiver.

4. Security Considerations

TBD.

5. IANA Considerations

6. Notes

This document is inspired by [RFC6773] and some text passages for the -00 version are copied unmodified.

7. Acknowledgments

8. Informative References

[RFC0768] Postel, J., "User Datagram Protocol", STD 6, RFC 768, DOI 10.17487/RFC0768, August 1980, .

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, .

[RFC4340] Kohler, E., Handley, M., and S. Floyd, "Datagram Congestion Control Protocol (DCCP)", RFC 4340, DOI 10.17487/RFC4340, March 2006, .

[RFC4787] Audet, F., Ed. and C. Jennings, "Network Address Translation (NAT) Behavioral Requirements for Unicast UDP", BCP 127, RFC 4787, DOI 10.17487/RFC4787, January 2007, .

Amend, et al. Expires January 9, 2020 [Page 9] Internet-Draft DCCP - UDP header conversion July 2019

[RFC5596] Fairhurst, G., "Datagram Congestion Control Protocol (DCCP) Simultaneous-Open Technique to Facilitate NAT/ Middlebox Traversal", RFC 5596, DOI 10.17487/RFC5596, September 2009, .

[RFC5597] Denis-Courmont, R., "Network Address Translation (NAT) Behavioral Requirements for the Datagram Congestion Control Protocol", BCP 150, RFC 5597, DOI 10.17487/RFC5597, September 2009, .

[RFC6773] Phelan, T., Fairhurst, G., and C. Perkins, "DCCP-UDP: A Datagram Congestion Control Protocol UDP Encapsulation for NAT Traversal", RFC 6773, DOI 10.17487/RFC6773, November 2012, .

[RFC6888] Perreault, S., Ed., Yamagata, I., Miyakawa, S., Nakagawa, A., and H. Ashida, "Common Requirements for Carrier-Grade NATs (CGNs)", BCP 127, RFC 6888, DOI 10.17487/RFC6888, April 2013, .

[RFC7857] Penno, R., Perreault, S., Boucadair, M., Ed., Sivakumar, S., and K. Naito, "Updates to Network Address Translation (NAT) Behavioral Requirements", BCP 127, RFC 7857, DOI 10.17487/RFC7857, April 2016, .

Authors’ Addresses

Markus Amend Deutsche Telekom Deutsche-Telekom-Allee 7 64295 Darmstadt Germany

Email: [email protected]

Anna Brunstrom Karlstad University Universitetsgatan 2 651 88 Karlstad Sweden

Email: [email protected]

Amend, et al. Expires January 9, 2020 [Page 10] Internet-Draft DCCP - UDP header conversion July 2019

Andreas Kassler Karlstad University Universitetsgatan 2 651 88 Karlstad Sweden

Email: [email protected]

Veselin Rakocevic City University of London Northampton Square London United Kingdom

Email: [email protected]

Amend, et al. Expires January 9, 2020 [Page 11] Transport Area Working Group M. Amend Internet-Draft D. Hugo Intended status: Experimental DT Expires: 13 January 2022 A. Brunstrom A. Kassler Karlstad University V. Rakocevic City University of London S. Johnson BT 12 July 2021

DCCP Extensions for Multipath Operation with Multiple Addresses draft-amend-tsvwg-multipath-dccp-05

Abstract

DCCP communication is currently restricted to a single path per connection, yet multiple paths often exist between peers. The simultaneous use of these multiple paths for a DCCP session could improve resource usage within the network and, thus, improve user experience through higher throughput and improved resilience to network failures. Use cases for a Multipath DCCP (MP-DCCP) are mobile devices (handsets, vehicles) and residential home gateways simultaneously connected to distinct paths as, e.g., a cellular link and a WiFi link or to a mobile radio station and a fixed access network. Compared to existing multipath protocols such as MPTCP, MP- DCCP provides specific support for non-TCP user traffic as UDP or plain IP. More details on potential use cases are provided in [website], [slide] and [paper]. All this use cases profit from an Open Source Linux reference implementation provided under [website].

This document presents a set of extensions to traditional DCCP to support multipath operation. Multipath DCCP provides the ability to simultaneously use multiple paths between peers. The protocol offers the same type of service to applications as DCCP and it provides the components necessary to establish and use multiple DCCP flows across potentially disjoint paths.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Amend, et al. Expires 13 January 2022 [Page 1] Internet-Draft Multipath DCCP July 2021

This Internet-Draft will expire on 13 January 2022.

This document is subject to BCP 78 and the IETF Trust’s Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

1. Introduction ...... 3 1.1. Multipath DCCP in the Networking Stack ...... 4 1.2. Terminology ...... 4 1.3. MP-DCCP Concept ...... 5 1.4. Differences from Multipath TCP ...... 5 1.5. Requirements Language ...... 9 2. Operation Overview ...... 9 3. MP-DCCP Protocol ...... 9 3.1. Multipath Capable Feature ...... 12 3.2. Multipath Option ...... 12 3.2.1. MP_CONFIRM ...... 13 3.2.2. MP_JOIN ...... 13 3.2.3. MP_FAST_CLOSE ...... 14 3.2.4. MP_KEY ...... 14 3.2.5. MP_SEQ ...... 15 3.2.6. MP_HMAC ...... 15 3.2.7. MP_RTT ...... 16 3.2.8. MP_ADDADDR ...... 17 3.2.9. MP_REMOVEADDR ...... 18 3.2.10. MP_PRIO ...... 19

Amend, et al. Expires 13 January 2022 [Page 2] Internet-Draft Multipath DCCP July 2021

3.3. MP-DCCP Handshaking Procedure ...... 19 4. Security Considerations ...... 21 5. Interactions with Middleboxes ...... 22 6. Implementation ...... 22 7. Acknowledgments ...... 22 8. IANA Considerations ...... 23 9. Informative References ...... 25 Authors’ Addresses ...... 28

1. Introduction

Multipath DCCP (MP-DCCP) is a set of extensions to regular DCCP [RFC4340], i.e. the Datagram Congestion Control Protocol denoting a transport protocol that provides bidirectional unicast connections of congestion-controlled unreliable datagrams. A multipath extension to DCCP enables the transport of user data across multiple paths simultaneously. This is beneficial to applications that transfer fairly large amounts of data, due to the possibility to aggregate capacity of the multiple paths. In addition, it enables to tradeoff timeliness and reliability, which is important for low latency applications that do not require guaranteed delivery services such as Audio/Video streaming. DCCP multipath operation is suggested in the context of ongoing 3GPP work on 5G multi-access solutions [I-D.amend-tsvwg-multipath-framework-mpdccp] and for hybrid access networks [I-D.lhwxz-hybrid-access-network-architecture][I-D.muley-net work-based-bonding-hybrid-access]. It can be applied for load- balancing, seamless session handover, and aggregation purposes (referred to as ATSSS; Access steering, switching, and splitting in 3GPP terminology [TS23.501]).

This document presents the protocol changes required to add multipath capability to DCCP; specifically, those for signaling and setting up multiple paths ("subflows"), managing these subflows, re-assembly of data, and termination of sessions. DCCP, as stated in [RFC4340] does not provide reliable and ordered delivery. Consequently, multiple application subflows may be multiplexed over a single DCCP connection with no inherent performance penalty for flows that do not require in-ordered delivery. DCCP does not provide built-in support for those multiple application subflows.

Amend, et al. Expires 13 January 2022 [Page 3] Internet-Draft Multipath DCCP July 2021

In the following, use of the term subflow will refer to physical separate DCCP subflows transmitted via different paths, but not to application subflows. Application subflows are differing content- wise by source and destination port per application as, for example, enabled by Service Codes introduced to DCCP in [RFC5595], and those subflows can be multiplexed over a single DCCP connection. For sake of consistency we assume that only a single application is served by a DCCP connection here as shown in Figure 1 while use of that feature should not impact DCCP operation on each single path as noted in ([RFC5595], sect. 2.4).

1.1. Multipath DCCP in the Networking Stack

MP-DCCP operates at the transport layer and aims to be transparent to both higher and lower layers. It is a set of additional features on top of standard DCCP; Figure 1 illustrates this layering. MP-DCCP is designed to be used by applications in the same way as DCCP with no changes to the application itself.

+------+ | Application | +------+ +------+ | Application | | MP-DCCP | +------+ + ------+ ------+ | DCCP | |Subflow (DCCP) |Subflow (DCCP) | +------+ +------+ | IP | | IP | IP | +------+ +------+

Figure 1: Comparison of Standard DCCP and MP-DCCP Protocol Stacks

1.2. Terminology

Throughout this document we make use of terms that are either specific for multipath transport or are defined in the context of MP- DCCP, similar to [RFC8684], as follows:

Path: A sequence of links between a sender and a receiver, defined in this context by a 4-tuple of source and destination address/ port pairs.

Subflow: A flow of DCCP segments operating over an individual path, which forms part of a larger MP-DCCP connection. A subflow is started and terminated similar to a regular (single-path) DCCP connection.

Amend, et al. Expires 13 January 2022 [Page 4] Internet-Draft Multipath DCCP July 2021

(MP-DCCP) Connection: A set of one or more subflows, over which an application can communicate between two hosts. There is a one-to-one mapping between a connection and an application socket.

Token: A locally unique identifier given to a multipath connection by a host. May also be referred to as a "Connection ID".

Host: An end host operating an MP-DCCP implementation, and either initiating or accepting an MP-DCCP connection. In addition to these terms, within framework of MP-DCCP the interpretation of, and effect on, regular single-path DCCP semantics is discussed in Section 3.

1.3. MP-DCCP Concept

Host A Host B ------Address A1 Address A2 Address B1 Address B2 ------| | | | | (DCCP flow setup) | | |------>| | |<------| | | | | | | | (DCCP flow setup) | | | |------>| | | |<------| | | merge individual DCCP flows to one multipath connection | | | |

Figure 2: Example MP-DCCP Usage Scenario

1.4. Differences from Multipath TCP

Multipath DCCP is similar to Multipath TCP [RFC8684], in that it extends the related basic DCCP transport protocol [RFC4340] with multipath capabilities in the same way as Multipath TCP extends TCP [RFC0793]. However, because of the differences between the underlying TCP and DCCP protocols, the transport characteristics of MPTCP and MP-DCCP are different.

Amend, et al. Expires 13 January 2022 [Page 5] Internet-Draft Multipath DCCP July 2021

Table 1 compares the protocol characteristics of TCP and DCCP, which are by nature inherited by their respective multipath extensions. A major difference lies in the delivery of payload, which is for TCP an exact copy of the generated byte-stream. DCCP behaves in a different way and does not guarantee to deliver any payload nor the order of delivery. Since this is mainly affecting the receiving endpoint of a TCP or DCCP communication, many similarities on the sender side can be identified. Both transport protocols share the 3-way initiation of a communication and both employ congestion control to adapt the sending rate to the path characteristics.

Amend, et al. Expires 13 January 2022 [Page 6] Internet-Draft Multipath DCCP July 2021

+======+======+======+ | Feature | TCP | DCCP | +======+======+======+ | Full-Duplex | yes | yes | +------+------+------+ | Connection-Oriented | yes | yes | +------+------+------+ | Header option space | 40 bytes | < 1008 bytes or PMTU | +------+------+------+ | Data transfer | reliable | unreliable | +------+------+------+ | Packet-loss handling | re-transmission | report only | +------+------+------+ | Ordered data delivery | yes | no | +------+------+------+ | Sequence numbers | one per byte | one per PDU | +------+------+------+ | Flow control | yes | no | +------+------+------+ | Congestion control | yes | yes | +------+------+------+ | ECN support | yes | yes | +------+------+------+ | Selective ACK | yes | depends on | | | | congestion control | +------+------+------+ | Fix message | no | yes | | boundaries | | | +------+------+------+ | Path MTU discovery | yes | yes | +------+------+------+ | Fragmentation | yes | no | +------+------+------+ | SYN flood protection | yes | no | +------+------+------+ | Half-open connections | yes | no | +------+------+------+

Table 1: TCP and DCCP protocol comparison

Consequently, the multipath features, shown in Table 2, are the same, supporting volatile paths having varying capacity and latency, session handover and path aggregation capabilities. All of them profit by the existence of congestion control.

Amend, et al. Expires 13 January 2022 [Page 7] Internet-Draft Multipath DCCP July 2021

+======+======+======+ | Feature | MPTCP | MP-DCCP | +======+======+======+ | Volatile paths | yes | yes | +------+------+------+ | Session handover | yes | yes | +------+------+------+ | Path aggregation | yes | yes | +------+------+------+ | Robust session establishment | no | yes | +------+------+------+ | Data re-assembly | yes | optional / modular | +------+------+------+ | Expandability | limited by | flexible | | | TCP header | | +------+------+------+

Table 2: MPTCP and MP-DCCP protocol comparison

Therefore, the sender logic is not much different between MP-DCCP and MPTCP, even if the multipath session initiation differs. MP-DCCP inherits a robust session establishment feature, which guarantees communication establishment if at least one functional path is available. MPTCP relies on an initial path, which has to work; otherwise no communication can be established.

The receiver side for MP-DCCP has to deal with the unreliable transport character of DCCP and a possible re-assembly of the data stream while not advocating it. As many unreliable applications have built-in application support for reordering (such as adaptive audio and video buffers), those applications might not need support for re- assembly. However, for applications that benefit from partial or full support of reordering, MP-DCCP can provide flexible support for re-assembly, even if for DCCP the order of delivery is unreliable by nature. Such optional re-assembly mechanisms may account for the fact that packet loss may occur for any of the DCCP subflows. Another issue may occur as packet reordering may happen when the different DCCP subflows are routed across paths with different latencies. In theory, applications using DCCP are aware that packet reordering might happen, since DCCP has no mechanisms to prevent it.

The receiving process for MPTCP is on the other hand a rigid "just wait" approach, since TCP guarantees reliable delivery.

Amend, et al. Expires 13 January 2022 [Page 8] Internet-Draft Multipath DCCP July 2021

1.5. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].

2. Operation Overview

RFC 4340 states that some applications might want to share congestion control state among multiple DCCP flows between same source and destination addresses. This functionality could be provided by the Congestion Manager (CM) [RFC3124], a generic multiplexing facility. However, the CM would not fully support MP-DCCP without change; it does not gracefully handle multiple congestion control mechanisms, for example.

The operation of MP-DCCP for data transfer takes one input data stream from an application, and splits it into one or more subflows, with sufficient control information to allow received data to be reassembled and delivered in order to the recipient application. The following subsections define this behavior in detail.

The Multipath Capability for MP-DCCP can be negotiated with a new DCCP feature, as described in Section 3. Once negotiated, all subsequent MP-DCCP operations are signalled with a variable length multipath-related option, as described in Section 3.1.

3. MP-DCCP Protocol

The DCCP protocol feature list ([RFC4340] section 6.4) will be enhanced by a new Multipath related feature with Feature number 10, as shown in Table 3.

Amend, et al. Expires 13 January 2022 [Page 9] Internet-Draft Multipath DCCP July 2021

+======+======+======+======+======+ | Number | Meaning | Rule | Rec’n Value | Initial Req’d | +======+======+======+======+======+ | 0 | Reserved | | | | +------+------+------+------+------+ | 1 | Congestion | SP | 2 | Y | | | Control ID (CCID) | | | | +------+------+------+------+------+ | 2 | Allow Short | SP | 0 | Y | | | Seqnos | | | | +------+------+------+------+------+ | 3 | Sequence Window | NN | 100 | Y | +------+------+------+------+------+ | 4 | ECN Incapable | SP | 0 | N | +------+------+------+------+------+ | 5 | Ack Ratio | NN | 2 | N | +------+------+------+------+------+ | 6 | Send Ack Vector | SP | 0 | N | +------+------+------+------+------+ | 7 | Send NDP Count | SP | 0 | N | +------+------+------+------+------+ | 8 | Minimum Checksum | SP | 0 | N | | | Coverage | | | | +------+------+------+------+------+ | 9 | Check Data | SP | 0 | N | | | Checksum | | | | +------+------+------+------+------+ | 10 | Multipath Capable | SP | 0 | N | +------+------+------+------+------+ | 11-127 | Reserved | | | | +------+------+------+------+------+ | 128-255 | CCID-specific | | | | | | features | | | | +------+------+------+------+------+

Table 3: Proposed Feature Set

The DCCP protocol options as defined in ([RFC4340] section 5.8) and ([RFC5634] section 2.2.1) will be enhanced by a new Multipath related variable-length option with option type 46, as shown in Table 4.

Amend, et al. Expires 13 January 2022 [Page 10] Internet-Draft Multipath DCCP July 2021

+======+======+======+======+ | Type | Option Length | Meaning | DCCP-Data? | +======+======+======+======+ | 0 | 1 | Padding | Y | +------+------+------+------+ | 1 | 1 | Mandatory | N | +------+------+------+------+ | 2 | 1 | Slow Receiver | Y | +------+------+------+------+ | 3-31 | 1 | Reserved | | +------+------+------+------+ | 32 | variable | Change L | N | +------+------+------+------+ | 33 | variable | Confirm L | N | +------+------+------+------+ | 34 | variable | Change R | N | +------+------+------+------+ | 35 | variable | Confirm R | N | +------+------+------+------+ | 36 | variable | Init Cookie | N | +------+------+------+------+ | 37 | 3-8 | NDP Count | Y | +------+------+------+------+ | 38 | variable | Ack Vector [Nonce 0] | N | +------+------+------+------+ | 39 | variable | Ack Vector [Nonce 1] | N | +------+------+------+------+ | 40 | variable | Data Dropped | N | +------+------+------+------+ | 41 | 6 | Timestamp | Y | +------+------+------+------+ | 42 | 6/8/10 | Timestamp Echo | Y | +------+------+------+------+ | 43 | 4/6 | Elapsed Time | N | +------+------+------+------+ | 44 | 6 | Data Checksum | Y | +------+------+------+------+ | 45 | 8 | Quick-Start Response | ? | +------+------+------+------+ | 46 | variable | Multipath | Y | +------+------+------+------+ | 47-127 | variable | Reserved | | +------+------+------+------+ | 128-255 | variable | CCID-specific options | - | +------+------+------+------+

Table 4: Proposed Option Set

Amend, et al. Expires 13 January 2022 [Page 11] Internet-Draft Multipath DCCP July 2021

[Tbd/tbv] In addition to the multipath option, MP-DCCP requires particular considerations for:

* The minimum PMTU of the individual paths must be announced to the application. Changes of individual path PMTUs must be re- announced to the application if they result in a value lower than the currently announced PMTU.

* Overall sequencing for optional re-assembly procedure

* Congestion control

* Robust MP-DCCP session establishment (no dependency on an initial path setup)

3.1. Multipath Capable Feature

DCCP endpoints are multipath-disabled by default and multipath capability can be negotiated with the Multipath Capable Feature.

Multipath Capable has feature number 10 and is server-priority. It takes one-byte values. The first four bits are used to specify compatible versions of the MP-DCCP implementation. The following four bits are reserved for further use.

3.2. Multipath Option

+------+------+------+------+------|00101110| Length | MP_OPT | Value(s) ... +------+------+------+------+------Type=46

Amend, et al. Expires 13 January 2022 [Page 12] Internet-Draft Multipath DCCP July 2021

+======+======+======+======+ | Type | Option | MP_OPT | Meaning | | | Length | | | +======+======+======+======+ | 46 | var | 0 =MP_CONFIRM | Confirm reception and | | | | | processing of an MP_OPT option | +------+------+------+------+ | 46 | 11 | 1 =MP_JOIN | Join path to an existing MP- | | | | | DCCP flow | +------+------+------+------+ | 46 | 3 | 2 | Close MP-DCCP flow | | | | =MP_FAST_CLOSE | | +------+------+------+------+ | 46 | var | 3 =MP_KEY | Exchange key material for | | | | | MP_HMAC | +------+------+------+------+ | 46 | 7 | 4 =MP_SEQ | Multipath Sequence Number | +------+------+------+------+ | 46 | 23 | 5 =MP_HMAC | HMA Code for authentication | +------+------+------+------+ | 46 | 12 | 6 =MP_RTT | Transmit RTT values | +------+------+------+------+ | 46 | var | 7 =MP_ADDADDR | Advertise additional Address | +------+------+------+------+ | 46 | var | 8 | Remove Address | | | | =MP_REMOVEADDR | | +------+------+------+------+ | 46 | 4 | 9 =MP_PRIO | Change Subflow Priority | +------+------+------+------+

Table 5: MP_OPT Option Types

3.2.1. MP_CONFIRM

+------+------+------+------+------+------+------+ |00101110| Length |00000000| List of options ... +------+------+------+------+------+------+------+ Type=46 MP_OPT=0

MP_CONFIRM can be used to send confirmation of received and processed options. Confirmed options are copied verbatim and appended as List of options. The length varies dependent on the amount of options.

[Tbd] Encoding "list of options"

3.2.2. MP_JOIN

Amend, et al. Expires 13 January 2022 [Page 13] Internet-Draft Multipath DCCP July 2021

+------+------+------+------+------+------+------+ |00101110|00001011|00000001| Path Token | +------+------+------+------+------+------+------+ | Nonce | +------+------+------+------+ Type=46 Length=11 MP_OPT=1

The MP_JOIN option is used to add a new path to an existing MP-DCCP flow. The Path Token is the SHA-1 HASH of the derived key (d-key), which was previously exchanged with the MP_KEY option. MP_HMAC MUST be set when using MP_JOIN to provide authentication (See MP_HMAC for details). Also MP_KEY MUST be set to provide key material for authentication purposes.

3.2.3. MP_FAST_CLOSE

+------+------+------+ |00101110|00000011|00000010| +------+------+------+ Type=46 Length=3 MP_OPT=2

MP_FAST_CLOSE terminates the MP-DCCP flow and all corresponding subflows.

3.2.4. MP_KEY

+------+------+------+------+------+------+------+ |00101110| Length |00000011|Key Type| Key Data ... +------+------+------+------+------+------+------+ Type=46 MP_OPT=3

The MP_KEY suboption is used to exchange key material between hosts. The Length varies between 5 and 8 Bytes. The Key Type field is used to specify the key type. Key types are shown in Table 6.

Amend, et al. Expires 13 January 2022 [Page 14] Internet-Draft Multipath DCCP July 2021

+======+======+======+ | Key Type | Key Length | Meaning | +======+======+======+ | 0 =Plain Text | 8 | Plain Text Key | +------+------+------+ | 1 =ECDHE-C25519-SHA256 | 32 | ECDHE with SHA256 | | | | and Curve25519 | +------+------+------+ | 2 =ECDHE-C25519-SHA512 | 32 | ECDHE with SHA512 | | | | and Curve25519 | +------+------+------+ | 3-255 | | Reserved | +------+------+------+

Table 6: MP_KEY Key Types

Plain Text Key Material is exchanged in plain text between hosts, and the key parts (key-a, key-b) are used by each host to generate the derived key (d-key) by concatenating the two parts with the local key in front (e.g. hostA d-key=(key-a+key-b), hostB d-key=(key-b+key-a)).

ECDHE-SHA256-C25519 Key Material is exchanged via ECDHE key exchange with SHA256 and Curve 25519 to generate the derived key (d-key).

ECDHE-SHA512-C25519 Key Material is exchanged via ECDHE key exchange with SHA512 and Curve 25519 to generate the derived key (d-key).

3.2.5. MP_SEQ

+------+------+------+------+------+------+------+ |00101110|00000111|00000100| Multipath Sequence Number | +------+------+------+------+------+------+------+ Type=46 Length=7 MP_OPT=4

The MP_SEQ option is used for end-to-end datagram-based sequence numbers of an MP-DCCP connection. The initial data sequence number (IDSN) SHOULD be set randomly. The MP_SEQ number space is different from path individual sequence number space.

3.2.6. MP_HMAC

+------+------+------+------+------+------+ |00101110|00001011|00000101| HMAC-SHA1 (20 bytes) ... +------+------+------+------+------+------+ Type=46 Length=23 MP_OPT=5

Amend, et al. Expires 13 January 2022 [Page 15] Internet-Draft Multipath DCCP July 2021

The MP_HMAC option is used to provide authentication for the MP_JOIN option. The HMAC is built using the derived key (d-key) calculated previously from the handshake key material exchanged with the MP_KEY option. The Message for the HMAC is the header of the MP_JOIN for which authentication shall be performed. By including a nonce in these datagrams, possible replay-attacks are remedied.

3.2.7. MP_RTT

+------+------+------+------+------+------+------+ |00101110|00001100|00000110|RTT Type| RTT +------+------+------+------+------+------+------+ | | Age | +------+------+------+------+------+ Type=46 Length=12 MP_OPT=6

The MP_RTT option is used to transmit RTT values in milliseconds and MUST belong to the path over which this information is transmitted. Additionally, the age of the measurement is specified in milliseconds.

Raw RTT (=0) Raw RTT value of the last Datagram Round-Trip. The Age parameter is set to the age of when the Ack for the datagram was received.

Min RTT (=1) Min RTT value. The period for computing the Minimum can be specified by the Age parameter.

Max RTT (=2) Max RTT value. The period for computing the Maximum can be specified by the Age parameter.

Smooth RTT (=3) Averaged RTT value. The period for computing the smoothed RTT can be specified by the Age parameter.

Age (=4) The Age parameter is a 4-byte value which is set to the age or timestamp when the Ack for the datagram was received in case of RTT type = 0 and may contain the periods for computing of derived RTT values depending on other RTT types, i.e., the Minimum (=1) and Maximum (=2) as well as the averaged smoothed RTT value (=3). [TBD/TBV]

Amend, et al. Expires 13 January 2022 [Page 16] Internet-Draft Multipath DCCP July 2021

3.2.8. MP_ADDADDR

The MP_ADDADDR option announces additional addresses (and, optionally, ports) on which a host can be reached. This option can be used at any time during an existing DCCP connection, when the sender wishes to enable multiple paths and/or when additional paths become available. Length is variable depending on IPv4 or IPv6 and whether port number is used and is in range between 28 and 42 bytes.

1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +------+------+------+------+------+ | Kind | Length |Subtype| IPVer | Address ID | +------+------+------+------+------+ | Address (IPv4 - 4 bytes / IPv6 - 16 bytes) | +------+------+ | Port (2 bytes, optional) | | +------+ | | HMAC (20 bytes) | | | | | | | | | | +------+ | | +------+

Every address has an Address ID that can be used for uniquely identifying the address within a connection for address removal. The Address ID is also used to identify MP_JOIN options (see Section 3.2.2) relating to the same address, even when address translators are in use. The Address ID MUST uniquely identify the address for the sender of the option (within the scope of the connection); the mechanism for allocating such IDs is implementation specific.

All Address IDs learned via either MP_JOIN or ADD_ADDR SHOULD be stored by the receiver in a data structure that gathers all the Address-ID-to-address mappings for a connection (identified by a token pair). In this way, there is a stored mapping between the Address ID, observed source address, and token pair for future processing of control information for a connection.

Ideally, ADD_ADDR and REMOVE_ADDR options would be sent reliably, and in order, to the other end. This would ensure that this address management does not unnecessarily cause an outage in the connection when remove/add addresses are processed in reverse order, and also to ensure that all possible paths are used. Note, however, that losing

Amend, et al. Expires 13 January 2022 [Page 17] Internet-Draft Multipath DCCP July 2021

reliability and ordering will not break the multipath connections, it will just reduce the opportunity to open new paths and to survive different patterns of path failures.

Therefore, implementing reliability signals for these DCCP options is not necessary. In order to minimize the impact of the loss of these options, however, it is RECOMMENDED that a sender should send these options on all available subflows. If these options need to be received in order, an implementation SHOULD only send one ADD_ADDR/ REMOVE_ADDR option per RTT, to minimize the risk of misordering. A host that receives an ADD_ADDR but finds a connection set up to that IP address and port number is unsuccessful SHOULD NOT perform further connection attempts to this address/port combination for this connection. A sender that wants to trigger a new incoming connection attempt on a previously advertised address/port combination can therefore refresh ADD_ADDR information by sending the option again.

[TBD/TBV]

3.2.9. MP_REMOVEADDR

If, during the lifetime of an MP-DCCP connection, a previously announced address becomes invalid (e.g., if the interface disappears), the affected host SHOULD announce this so that the peer can remove subflows related to this address.

This is achieved through the Remove Address (REMOVE_ADDR) option which will remove a previously added address (or list of addresses) from a connection and terminate any subflows currently using that address.

For security purposes, if a host receives a REMOVE_ADDR option, it must ensure the affected path(s) are no longer in use before it instigates closure. Typical DCCP validity tests on the subflow (e.g., packet type specific sequence and acknowledgement number check) MUST also be undertaken. An implementation can use indications of these test failures as part of intrusion detection or error logging.

The sending and receipt of this message SHOULD trigger the sending of DCCP-Close and DCCP-Reset by client and server, respectively on the affected subflow(s) (if possible), as a courtesy to cleaning up middlebox state, before cleaning up any local state.

Address removal is undertaken by ID, so as to permit the use of NATs and other middleboxes that rewrite source addresses. If there is no address at the requested ID, the receiver will silently ignore the request.

Amend, et al. Expires 13 January 2022 [Page 18] Internet-Draft Multipath DCCP July 2021

Minimum length of this option is 4 bytes (for one address to remove).

[TBD/TBV]

3.2.10. MP_PRIO

In the event that a single specific path out of the set of available paths shall be treated with higher priority compared to the others, a host may wish to signal such change in priority of subflows to the peer. Therefore, the MP_PRIO option, shown below, can be used to set a priority flag for the subflow on which it is sent.

Whether more than two values for priority (e.g., B for backup and P for prioritized path) are defined in case of more than two parallel paths is for further consideration.

[TBD/TBV]

3.3. MP-DCCP Handshaking Procedure

Amend, et al. Expires 13 January 2022 [Page 19] Internet-Draft Multipath DCCP July 2021

Host A Host B ------Address A1 Address A2 Address B1 ------| | | | DCCP-Request + | |------MP_KEY(Key-A) ------>| |<------MP_KEY(Key-B) ------| | DCCP-Response + agreed | | | | | DCCP-Ack | | |------MP_KEY(Key-A) + MP_KEY(Key-B) ------>| | | | | | DCCP-Request + | | |--- MP_JOIN(TB,RA) ------>| | |<------MP_JOIN(TB,RB) + MP_HMAC(A)-----| | |DCCP-Response | | | | | |DCCP-Ack | | |------MP_HMAC(B) ------>| | |<------| | |DCCP-ACK |

Figure 3: Example MP-DCCP Handshake

The basic initial handshake for the first flow is as follows:

* Host A sends a DCCP-Request with the MP-Capable feature Change request and the MP_KEY option with Host-specific Key-A

* Host B sends a DCCP-Response with Confirm feature for MP-Capable and the MP_Key option with Host-specific Key-B

* Host A sends a DCCP-Ack with both Keys echoed to Host B.

The handshake for subsequent flows based on a successful initial handshake is as follows:

* Host A sends a DCCP-Request with the MP-Capable feature Change request and the MP_JOIN option with Host B’s Token TB, generated from the derived key by applying a SHA-1 hash and truncating to the first 32 bits. Additionally, an own random nonce RA is transmitted with the MP_JOIN.

* Host B computes the HMAC of the DCCP-Request and sends a DCCP- Response with Confirm feature option for MP-Capable and the MP_JOIN option with the Token TB and a random nonce RB together with the computed MP_HMAC.

Amend, et al. Expires 13 January 2022 [Page 20] Internet-Draft Multipath DCCP July 2021

* Host A sends a DCCP-Ack with the HMAC computed for the DCCP- Response.

* Host B sends a DCCP-Ack confirm the HMAC and to conclude the handshaking.

4. Security Considerations

Similar to DCCP, MP-DCCP does not provide cryptographic security guarantees inherently. Thus, if applications need cryptographic security (integrity, authentication, confidentiality, access control, and anti-replay protection) the use of IPsec or some other kind of end-to-end security is recommended; Secure Real-time Transport Protocol (SRTP) [RFC3711] is one candidate protocol for authentication. Together with Encryption of Header Extensions in SRTP, as provided by [RFC6904], also integrity would be provided.

As described in [RFC4340], DCCP provides protection against hijacking and limits the potential impact of some denial-of-service attacks, but DCCP provides no inherent protection against attackers’ snooping on data packets. Regarding the security of MP-DCCP no additional risks should be introduced compared to regular DCCP of today. Thereof derived are the following key security requirements to be fulfilled by MP-DCCP:

* Provide a mechanism to confirm that parties involved in a subflow handshake are identical to those in the original connection setup.

* Provide verification that the new address to be included in a MP connection is valid for a peer to receive traffic at before using it.

* Provide replay protection, i.e., ensure that a request to add/ remove a subflow is ’fresh’.

In order to achieve these goals, MP-DCCP includes a hash-based handshake algorithm documented in Sections Section 3.2.4 and Section 3.3. The security of the MP-DCCP connection depends on the use of keys that are shared once at the start of the first subflow and are never sent again over the network. To ease demultiplexing while not giving away any cryptographic material, future subflows use a truncated cryptographic hash of this key as the connection identification "token". The keys are concatenated and used as keys for creating Hash-based Message Authentication Codes (HMACs) used on subflow setup, in order to verify that the parties in the handshake are the same as in the original connection setup. It also provides verification that the peer can receive traffic at this new address. Replay attacks would still be possible when only keys are used;

Amend, et al. Expires 13 January 2022 [Page 21] Internet-Draft Multipath DCCP July 2021

therefore, the handshakes use single-use random numbers (nonces) at both ends -- this ensures that the HMAC will never be the same on two handshakes. Guidance on generating random numbers suitable for use as keys is given in [RFC4086]. During normal operation, regular DCCP protection mechanisms (such as header checksum to protect DCCP headers against corruption) will provide the same level of protection against attacks on individual DCCP subflows as exists for regular DCCP today.

5. Interactions with Middleboxes

Issues from interaction with on-path middleboxes such as NATs, firewalls, proxies, intrusion detection systems (IDSs), and others have to be considered for all extensions to standard protocols since otherwise unexpected reactions of middleboxes may hinder its deployment. DCCP already provides means to mitigate the potential impact of middleboxes, also in comparison to TCP (see [RFC4043], sect. 16). In case, however, both hosts are located behind a NAT or firewall entity, specific measures have to be applied such as the [RFC5596]-specified simultaneous-open technique that update the (traditionally asymmetric) connection-establishment procedures for DCCP. Further standardized technologies addressing NAT type middleboxes are covered by [RFC5597].

[RFC6773] specifies UDP Encapsulation for NAT Traversal of DCCP sessions, similar to other UDP encapsulations such as for SCTP [RFC6951]. The alternative U-DCCP approach proposed in [I-D.amend-tsvwg-dccp-udp-header-conversion] would reduce tunneling overhead. The handshaking procedure for DCCP-UDP header conversion or use of a DCCP-UDP negotiation procedure to signal support for DCCP-UDP header conversion would require encapsulation during the handshakes and use of two additional port numbers out of the UDP port number space, but would require zero overhead afterwards.

6. Implementation

The approach described above has been implemented in open source across different testbeds and a new scheduling algorithm has been extensively tested. Also demonstrations of a laboratory setup have been executed and have been published at [website].

7. Acknowledgments

1. Notes

This document is inspired by Multipath TCP [RFC6824]/[RFC8684] and some text passages for the -00 version of the draft are copied almost unmodified.

Amend, et al. Expires 13 January 2022 [Page 22] Internet-Draft Multipath DCCP July 2021

8. IANA Considerations

This document defines one new value to DCCP feature list and one new DCCP Option with ten corresponding Subtypes as follows. This document defines a new DCCP feature parameter for negotiating the support of multipath capability for DCCP sessions between hosts as described in Section 3. The following entry in Table 7 should be added to the "Feature Numbers Registry" according to [RFC4340], Section 19.4. under the "DCCP Protocol" heading.

+======+======+======+ | Value | Feature Name | Specification | +======+======+======+ | 0x10 | MP-DCCP capability feature | Section 3.1 | +------+------+------+

Table 7: Addition to DCCP Feature list Entries

This document defines a new DCCP protocol option of type=46 as described in Section 3.2 together with 10 additional sub-options. The following entries in Table 8 should be added to the "DCCP Protocol options" and assigned as "MP-DCCP sub-options", respectively.

Amend, et al. Expires 13 January 2022 [Page 23] Internet-Draft Multipath DCCP July 2021

Table 8: Addition to DCCP Protocol options and corresponding sub-options

[Tbd], must include options for:

Amend, et al. Expires 13 January 2022 [Page 24] Internet-Draft Multipath DCCP July 2021

* handshaking procedure to indicate MP support

* handshaking procedure to indicate JOINING of an existing MP connection

* signaling of new or changed addresses

* setting handover or aggregation mode

* setting reordering on/off

should include options carrying:

* overall sequence number for restoring purposes

* sender time measurements for restoring purposes

* scheduler preferences

* reordering preferences

9. Informative References

[I-D.amend-tsvwg-dccp-udp-header-conversion] Amend, M., Brunstrom, A., Kassler, A., and V. Rakocevic, "Lossless and overhead free DCCP - UDP header conversion (U-DCCP)", Work in Progress, Internet-Draft, draft-amend- tsvwg-dccp-udp-header-conversion-01, 8 July 2019, .

[I-D.amend-tsvwg-multipath-framework-mpdccp] Amend, M., Bogenfeld, E., Brunstrom, A., Kassler, A., and V. Rakocevic, "A multipath framework for UDP traffic over heterogeneous access networks", Work in Progress, Internet-Draft, draft-amend-tsvwg-multipath-framework- mpdccp-01, 8 July 2019, .

[I-D.lhwxz-hybrid-access-network-architecture] Leymann, N., Heidemann, C., Wesserman, M., Xue, L., and M. Zhang, "Hybrid Access Network Architecture", Work in Progress, Internet-Draft, draft-lhwxz-hybrid-access- network-architecture-02, 13 January 2015, .

Amend, et al. Expires 13 January 2022 [Page 25] Internet-Draft Multipath DCCP July 2021

[I-D.muley-network-based-bonding-hybrid-access] Muley, P., Henderickx, W., Liang, G., Liu, H., Cardullo, L., Newton, J., Seo, S., Draznin, S., and B. Patil, "Network based Bonding solution for Hybrid Access", Work in Progress, Internet-Draft, draft-muley-network-based- bonding-hybrid-access-03, 22 October 2018, .

[paper] Amend, M., Bogenfeld, E., Cvjetkovic, M., Rakocevic, V., Pieska, M., Kassler, A., and A. Brunstrom, "A Framework for Multiaccess Support for Unreliable Internet Traffic using Multipath DCCP", DOI 10.1109/LCN44214.2019.8990746, October 2019, .

[RFC0793] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, DOI 10.17487/RFC0793, September 1981, .

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, .

[RFC3124] Balakrishnan, H. and S. Seshan, "The Congestion Manager", RFC 3124, DOI 10.17487/RFC3124, June 2001, .

[RFC3711] Baugher, M., McGrew, D., Naslund, M., Carrara, E., and K. Norrman, "The Secure Real-time Transport Protocol (SRTP)", RFC 3711, DOI 10.17487/RFC3711, March 2004, .

[RFC4043] Pinkas, D. and T. Gindin, "Internet X.509 Public Key Infrastructure Permanent Identifier", RFC 4043, DOI 10.17487/RFC4043, May 2005, .

[RFC4086] Eastlake 3rd, D., Schiller, J., and S. Crocker, "Randomness Requirements for Security", BCP 106, RFC 4086, DOI 10.17487/RFC4086, June 2005, .

[RFC4340] Kohler, E., Handley, M., and S. Floyd, "Datagram Congestion Control Protocol (DCCP)", RFC 4340, DOI 10.17487/RFC4340, March 2006, .

Amend, et al. Expires 13 January 2022 [Page 26] Internet-Draft Multipath DCCP July 2021

[RFC5595] Fairhurst, G., "The Datagram Congestion Control Protocol (DCCP) Service Codes", RFC 5595, DOI 10.17487/RFC5595, September 2009, .

[RFC5596] Fairhurst, G., "Datagram Congestion Control Protocol (DCCP) Simultaneous-Open Technique to Facilitate NAT/ Middlebox Traversal", RFC 5596, DOI 10.17487/RFC5596, September 2009, .

[RFC5597] Denis-Courmont, R., "Network Address Translation (NAT) Behavioral Requirements for the Datagram Congestion Control Protocol", BCP 150, RFC 5597, DOI 10.17487/RFC5597, September 2009, .

[RFC5634] Fairhurst, G. and A. Sathiaseelan, "Quick-Start for the Datagram Congestion Control Protocol (DCCP)", RFC 5634, DOI 10.17487/RFC5634, August 2009, .

[RFC6773] Phelan, T., Fairhurst, G., and C. Perkins, "DCCP-UDP: A Datagram Congestion Control Protocol UDP Encapsulation for NAT Traversal", RFC 6773, DOI 10.17487/RFC6773, November 2012, .

[RFC6824] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure, "TCP Extensions for Multipath Operation with Multiple Addresses", RFC 6824, DOI 10.17487/RFC6824, January 2013, .

[RFC6904] Lennox, J., "Encryption of Header Extensions in the Secure Real-time Transport Protocol (SRTP)", RFC 6904, DOI 10.17487/RFC6904, April 2013, .

[RFC6951] Tuexen, M. and R. Stewart, "UDP Encapsulation of Stream Control Transmission Protocol (SCTP) Packets for End-Host to End-Host Communication", RFC 6951, DOI 10.17487/RFC6951, May 2013, .

[RFC8684] Ford, A., Raiciu, C., Handley, M., Bonaventure, O., and C. Paasch, "TCP Extensions for Multipath Operation with Multiple Addresses", RFC 8684, DOI 10.17487/RFC8684, March 2020, .

Amend, et al. Expires 13 January 2022 [Page 27] Internet-Draft Multipath DCCP July 2021

[slide] Amend, M., "MP-DCCP for enabling transfer of UDP/IP traffic over multiple data paths in multi-connectivity networks", IETF105 , n.d., .

[TS23.501] 3GPP, "System architecture for the 5G System; Stage 2; Release 16", December 2020, .

[website] "Multipath extension for DCCP", n.d., .

Authors’ Addresses

Markus Amend Deutsche Telekom Deutsche-Telekom-Allee 9 64295 Darmstadt Germany

Email: [email protected]

Dirk von Hugo Deutsche Telekom Deutsche-Telekom-Allee 9 64295 Darmstadt Germany

Email: [email protected]

Anna Brunstrom Karlstad University Universitetsgatan 2 SE-651 88 Karlstad Sweden

Email: [email protected]

Amend, et al. Expires 13 January 2022 [Page 28] Internet-Draft Multipath DCCP July 2021

Andreas Kassler Karlstad University Universitetsgatan 2 SE-651 88 Karlstad Sweden

Email: [email protected]

Veselin Rakocevic City University of London Northampton Square London United Kingdom

Email: [email protected]

Stephen Johnson BT Adastral Park Martlesham Heath IP5 3RE United Kingdom

Email: [email protected]

Amend, et al. Expires 13 January 2022 [Page 29] Transport Area Working Group M. Amend Internet-Draft E. Bogenfeld Intended status: Informational Deutsche Telekom Expires: January 9, 2020 A. Brunstrom A. Kassler Karlstad University V. Rakocevic City University of London July 08, 2019

A multipath framework for UDP traffic over heterogeneous access networks draft-amend-tsvwg-multipath-framework-mpdccp-01

Abstract

More and more of today’s devices are multi-homing capable, in particular 3GPP user equipment like smartphones. In the current standardization of the next upcoming mobile network generation 5G Rel.16, this is especially targeted in the study group Access Traffic Steering Switching Splitting [TR23.793]. ATSSS describes the flexible selection or combination of 3GPP untrusted access like Wi-Fi and cellular access, overcoming the single-access limitation of today’s devices and services. Another multi-connectivity scenario is the Hybrid Access [I-D.lhwxz-hybrid-access-network-architecture][I-D. muley-network-based-bonding-hybrid-access], providing multiple access for CPEs, which extends the traditional way of single access connectivity at home to dual-connectivity over 3GPP and fixed access. A missing piece in the ATSSS and Hybrid Access is the access and path measurement, which is required for efficient and beneficial traffic steering decisions. This becomes particularly important in heterogeneous access networks with a multitude of volatile access paths. While MP-TCP has been proposed to be used within ATSSS, there are drawbacks when being used to encapsulate unreliable traffic as it blindly retransmits each lost frame leading to excessive delay and potential head-of-line blocking. A decision for MP-TCP though leaves the increasing share of UDP in today’s traffic mix () unconsidered. In this document, a multi-access framework is proposed leveraging the MP-DCCP network protocol, which enables flexible traffic steering, switching and splitting also for unreliable traffic. A benefit is the support for pluggable congestion control which enables our framework to be used either independent or complementary to MP-TCP.

Amend, et al. Expires January 9, 2020 [Page 1] Internet-Draft Multipath framework for UDP July 2019

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

This Internet-Draft will expire on January 9, 2020.

Table of Contents

1. Introduction ...... 3 2. Requirements ...... 3 3. IP compatible multipath framework based on MP-DCCP . . . . . 4 4. Application and placement ...... 6 5. Conclusion ...... 7 6. Security Considerations ...... 8 7. Acknowledgments ...... 8 8. Informative References ...... 8 Authors’ Addresses ...... 9

Amend, et al. Expires January 9, 2020 [Page 2] Internet-Draft Multipath framework for UDP July 2019

1. Introduction

Multi-connectivity access networks are evolving. Ongoing standardization at 3GPP for 5G mobile networks [TR23.793] or the so called Hybrid Access networks [I-D.lhwxz-hybrid-access-network-archit ecture][I-D.muley-network-based-bonding-hybrid-access] already provides or will enable in the near future the possibility to use multi-connectivity for a very large number of mobile users. Multi- connectivity solutions come with many user benefits including superior resilience against network outages, higher capacities for user traffic and network cost optimizations. Since multi- connectivity architectures are almost mature, new network protocols are required to fully exploit multi-connectivity and maximise its potential. In the simplest case, multi-connectivity is used for load-balancing decisions in order to balance the user flows over multiple paths. However, this has no effect on resilience or capacity gain on those load balanced individual flows. More complex scenarios include the dynamic shifting of traffic flows seamlessly between multiple paths or even aggregating those paths for leveraging the available capacity of multiple individual paths. Like [TR23.793] this document refers to the three distribution schemes Steering (load balancing), Switching (seamlesshandover) and Splitting (capacity aggregation).

MP-TCP [RFC6824] is a protocol, which can be applied in the above mentioned use cases and supports load-balancing, traffic shifting among the multiple paths and also capacity aggregation. Further, it leverages the inherent congestion control from TCP which adapts the sending rate by observing congestion signals from the network. By design, MP-TCP is limited to TCP services as it blindly re-transmits lost packets. Consequently, when MP-TCP is used as a framework for ATSSS, it may re-transmit packets sent from unreliable services such as e.g. UDP unnecessary. This may lead to head-of-line blocking and increased latency, which is detremental to real-time services. As future multi-connectivity systems must support latency sensitive traffic that might be transported over unreliable transport, it is not sufficient anymore to rely on supporting only TCP. The increasing share of UDP traffic, mainly impacted by the QUIC introduction, may significantly reduce the share from TCP. It might be expected that with HTTP/3 carried over QUIC [I-D.ietf-quic-http], the previous strong dominance of TCP will be challenged by UDP.

2. Requirements

A multiaccess framework shall meet the following requirements:

Amend, et al. Expires January 9, 2020 [Page 3] Internet-Draft Multipath framework for UDP July 2019

o IP compatibility: A multiaccess framework shall be able to transport IP packets and not make any assumptions on which transport protocol is encapsulated.

o Support for unreliable traffic: A multiaccess framework should provide support for transporting unreliable traffic, such as QUIC or UDP based flows. Therefore, unreliable transmission should besupported.

o Support for flexible re-ordering: A multiaccess framework should support flexible re-ordering of user traffic, including no reordering at all. This requirements is important to support low latency traffic, where the re-creation of packet order may negatively impact delivery latency.

o Support for flexible congestion control: A multiaccess framework should support flexible congestion control, including the disabling of the congestion control, if the inner traffic is known to be congestion controlled.

o Support for flexible packet scheduling: A multiaccess framework should support different packet scheduling mechanisms, which should be configurable from the control plane. Examples are cheapest path first, or other more sophisticated schedulers.

o Lightweight: A multiaccess framework should be lightweight in computational resources and limit the encapsulaton overhead.

To use QUIC as part of a multiaccess framework, by for example providing multipath support for QUIC, it could be beneficial if unreliable transmission is supported as well as being able to influence or disable QUICs congestion control. In addition, it would be beneficial if the encryption of QUIC can be disabled. This is because for ATSSS, it is foreseen that the underlying tunnel from the mobile over public WLANs is baseed on IPSec.

3. IP compatible multipath framework based on MP-DCCP

We propose a new multiaccess framework, which overcomes MP-TCP’s restriction to TCP services and provides IP compatibility in Figure 1. The framework employs MP-DCCP [I-D.amend-tsvwg-multipath-dccp] in combination with an encapsulation scheme. For simplification, Figure 1 assumes a traffic direction from the left (sender) to the right (receiver) and requires application in each direction for bi-directional transmission. The framework in Figure 1 can replace or complement MP-TCP to reach IP compatibility.

Amend, et al. Expires January 9, 2020 [Page 4] Internet-Draft Multipath framework for UDP July 2019

Service |<- MP-DCCP >| Service or Device or Device +------+ ___ +-----+ DCCP Flow 1 +------+ +------+ | | _ |Seq||Path |------|Re- | _ | | | Sender|___| \__\ /_| | : |order |____/ |___|Receiver| | | IP|_/ |Sched| : | | \_|IP | | | | VNIF_in |uler |------|engine| VNIF_out | | +------+ +-----+ DCCP Flow n +------+ +------+

Figure 1: IP compatible multipath framework based on MP-DCCP

PDUs generated from the sender and travelling through the framework in Figure 1 pass the components in the following order:

1. Sender: Generates any PDU based on the IP protocol.

2. VNIF_in: IP based Virtual Network Interface as entry point to the multipath framework. A simple routing logic in front (between (1)and (2)) can act as gatekeeper and decides upon redirecting traffic through the VNIF or bypassing it. The VNIF adds an extra IP header to reach the multi-connectivity termination point.

3. Seq: Sequencing of the PDUs passed through (2) depending on the incoming order. Adds an incrementing number, which is later added to the DCCP encapsulation in (4).

4. Path Scheduler: Decision logic for scheduling sequenced PDUs over the individual connected DCCP flows for multipath transmission. The path scheduler can use the information from the DCCP flows (see (5)) inherent congestion control information like CWND, packet loss, RTT, Jitter, etc.. After selection of a DCCP flow, the PDU is encapsulated into the individual flow. Further information, at least the sequencing, is added on top as DCCP option.

5. DCCP Flow(s): Responsible to transmit the encapsulated PDUs to the MP-DCCP exit point.

6. Reorder engine: Depending on the sequencing information of (3), a re-assembly of the PDU stream can be applied. Different re-order algorithms should be supported in a configurable way, including no re-ordering.

7. VNIF_out: Releases PDUs that have passed the re-ordering engine and strips the DCCP specific overhead. Again, routing is responsible to deliver the PDUs to the receiver based on the destination information in the PDU.

Amend, et al. Expires January 9, 2020 [Page 5] Internet-Draft Multipath framework for UDP July 2019

8. Receiver: Receive the PDU as generated in (1).

The simple enclosing of the MP-DCCP with Virtual Network Interface (VNIF) provides the IP compatibility. However, a service or protocol classifier between sender and VNIF can reduce the scope to particular traffic, e.g. UDP, by simple routing decisions. The MP-DCCP takes over responsibility for the multi-path transfer of the traffic, which is directed through the VNIF_in. For possible re-assembly operations, the IP packets may be stamped with a continuously incremented sequence number. This is not mandatory, but assumed required in most seamless handover and capacity aggregation use cases. The path scheduler decides for each IP packet, which DCCP flow it should use for encapsulation, based on a configurable decision logic and supported by the congestion control information of the DCCP flows available for transmission. A DCCP flow selection for a PDU leads to its encapsulation into the respective DCCP flow and adding extra information required for the multipath transmission, e.g. the sequence number. Encapsulation also means, that a DCCP and IP header is added to the original PDU to reach the multi- connectivity end-point. When the encapsulated PDUs arrive at the multi-path termination point, they are re-ordered depending on the carried sequence number and a configurable logic. The re-ordering engine may also include a logic in which packets are just forwarded (no re-ordering). Re-ordering needs to be considered carefully since any active intervention changes the latency responsiveness. The multi-path termination is finally completed when the DCCP overhead is stripped and the PDU leaves VNIF_out. Further routing depends again on the IP layer of the original PDU.

4. Application and placement

The framework of Figure 1 is very flexible in applying multipath support in different architectures and allows MP-DCCP to be applied at any place between sender and receiver. Figure 2 to Figure 5 provide several architectural options for the deployment of the framework.

Device Middlebox 1 Middlebox 2 Device +------+ +------+ +------+ +------+ |Sender| -> |MP-DCCP entry| -> |MP-DCCP exit| -> |Receiver| +------+ +------+ +------+ +------+

Figure 2: Sender and receiver independent MP-DCCP

Amend, et al. Expires January 9, 2020 [Page 6] Internet-Draft Multipath framework for UDP July 2019

Device Middlebox Device +------+ +------+ +------+ |Sender + MP-DCCP entry| -> |MP-DCCP exit| -> |Receiver| +------+ +------+ +------+

Figure 3: Sender integrated but receiver independent MP-DCCP

Device Middlebox Device +------+ +------+ +------+ |Sender| -> |MP-DCCP entry| -> |MP-DCCP exit + Receiver| +------+ +------+ +------+

Figure 4: Sender independent and receiver integrated MP-DCCP

Device Device +------+ +------+ |Sender + MP-DCCP entry| -> |MP-DCCP exit + Receiver| +------+ +------+

Figure 5: Sender and receiver integrated MP-DCCP

5. Conclusion

The specified IP compatible multipath framework based on MP-DCCP in this document comprises several benefits:

o Pure routing

o Inherent path estimation and measurement

o Imposes no constraints on reliability or in-order delivery of application PDUs

o Modular re-ordering

o Modular scheduling

o IP compatible

o Based on the standardized DCCP.

Middle-box traversing, when the framework is applied in uncontrolled environments, is addressed in [RFC6733] and [I-D.amend-tsvwg-dccp-udp-header-conversion].

Amend, et al. Expires January 9, 2020 [Page 7] Internet-Draft Multipath framework for UDP July 2019

6. Security Considerations

[Tbd]

7. Acknowledgments

8. Informative References

[I-D.amend-tsvwg-dccp-udp-header-conversion] Amend, M., Brunstrom, A., Kassler, A., and V. Rakocevic, "Lossless and overhead free DCCP - UDP header conversion (U-DCCP)", draft-amend-tsvwg-dccp-udp-header-conversion-01 (work in progress), July 2019.

[I-D.amend-tsvwg-multipath-dccp] Amend, M., Brunstrom, A., Kassler, A., and V. Rakocevic, "DCCP Extensions for Multipath Operation with Multiple Addresses", draft-amend-tsvwg-multipath-dccp-01 (work in progress), March 2019.

[I-D.ietf-quic-http] Bishop, M., "Hypertext Transfer Protocol Version 3 (HTTP/3)", draft-ietf-quic-http-18 (work in progress), January 2019.

[I-D.lhwxz-hybrid-access-network-architecture] Leymann, N., Heidemann, C., Wasserman, M., Xue, L., and M. Zhang, "Hybrid Access Network Architecture", draft-lhwxz- hybrid-access-network-architecture-02 (work in progress), January 2015.

[I-D.muley-network-based-bonding-hybrid-access] Muley, P., Henderickx, W., Geng, L., Liu, H., Cardullo, L., Newton, J., Seo, S., Draznin, S., and B. Patil, "Network based Bonding solution for Hybrid Access", draft- muley-network-based-bonding-hybrid-access-03 (work in progress), October 2018.

[RFC6733] Fajardo, V., Ed., Arkko, J., Loughney, J., and G. Zorn, Ed., "Diameter Base Protocol", RFC 6733, DOI 10.17487/RFC6733, October 2012, .

[RFC6824] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure, "TCP Extensions for Multipath Operation with Multiple Addresses", RFC 6824, DOI 10.17487/RFC6824, January 2013, .

Amend, et al. Expires January 9, 2020 [Page 8] Internet-Draft Multipath framework for UDP July 2019

[TR23.793] 3GPP, "Study on access traffic steering, switch and splitting support in the 5G System (5GS) architecture", December 2018.

Authors’ Addresses

Markus Amend Deutsche Telekom Deutsche-Telekom-Allee 7 64295 Darmstadt Germany

Email: [email protected]

Eckard Bogenfeld Deutsche Telekom Deutsche-Telekom-Allee 7 64295 Darmstadt Germany

Email: [email protected]

Anna Brunstrom Karlstad University Universitetsgatan 2 651 88 Karlstad Sweden

Email: [email protected]

Andreas Kassler Karlstad University Universitetsgatan 2 651 88 Karlstad Sweden

Email: [email protected]

Amend, et al. Expires January 9, 2020 [Page 9] Internet-Draft Multipath framework for UDP July 2019

Veselin Rakocevic City University of London Northampton Square London United Kingdom

Email: [email protected]

Amend, et al. Expires January 9, 2020 [Page 10] Network Working Group A. Choudhary Internet-Draft M. Jethanandani Intended status: Standards Track Cisco Systems Expires: March 23, 2020 N. Strahle E. Aries Juniper Networks I. Chen Jabil Sep 20, 2019

YANG Model for QoS draft-asechoud-rtgwg-qos-model-11

Abstract

This document describes a YANG model for Quality of Service (QoS) configuration and operational parameters.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

This Internet-Draft will expire on March 23, 2020.

Choudhary, et al. Expires March 23, 2020 [Page 1] Internet-Draft YANG Model For QoS Sep 2019

the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

1. Introduction ...... 2 1.1. Tree Diagrams ...... 3 2. Terminology ...... 3 3. QoS Model Design ...... 3 4. DiffServ Model Design ...... 4 5. Modules Tree Structure ...... 4 6. Modules ...... 13 6.1. IETF-QOS-CLASSIFIER ...... 14 6.2. IETF-QOS-POLICY ...... 17 6.3. IETF-QOS-ACTION ...... 20 6.4. IETF-QOS-TARGET ...... 38 6.5. IETF-DIFFSERV ...... 40 6.6. IETF-QUEUE-POLICY ...... 50 6.7. IETF-SCHEDULER-POLICY ...... 53 7. IANA Considerations ...... 56 8. Security Considerations ...... 57 9. Acknowledgement ...... 57 10. References ...... 57 10.1. Normative References ...... 57 10.2. Informative References ...... 58 Appendix A. Company A, Company B and Company C examples . . . . 58 A.1. Example of Company A Diffserv Model ...... 58 A.2. Example of Company B Diffserv Model ...... 68 A.3. Example of Company C Diffserv Model ...... 82 Authors’ Addresses ...... 88

1. Introduction

This document defines a base YANG [RFC6020] [RFC7950] data module for Quality of Service (QoS) configuration parameters. Differentiated Services (DiffServ) module is an augmentation of the base QoS model. Remote Procedure Calls (RPC) or notification definition is not part of this document. QoS base modules define a basic building blocks to define a classifier, policy, action and target. The base modules have been augmented to include packet match fields and action parameters to define the DiffServ module. Queues and schedulers are stitched as part of diffserv policy itself or separate modules are defined for creating Queue policy and Scheduling policy. The DiffServ model is based on DiffServ architecture, and various references have been made to available standard architecture documents.

Choudhary, et al. Expires March 23, 2020 [Page 2] Internet-Draft YANG Model For QoS Sep 2019

DiffServ is a preferred approach for network service providers to offer services to different customers based on their network Quality- of-Service (QoS) objectives. The traffic streams are differentiated based on DiffServ Code Points (DSCP) carried in the IP header of each packet. The DSCP markings are applied by upstream node or by the edge router on entry to the DiffServ network.

Editorial Note: (To be removed by RFC Editor)

This draft contains several placeholder values that need to be replaced with finalized values at the time of publication. Please apply the following replacements: o "XXXX" --> the assigned RFC value for this draft both in this draft and in the YANG models under the revision statement. o The "revision" date in model, in the format XXXX-XX-XX, needs to be updated with the date the draft gets approved.

The YANG modules in this document conform to the Network Management Datastore Architecture (NMDA) [RFC8342 [RFC8342]].

1.1. Tree Diagrams

Tree diagrams used in this document follow the notation defined in [RFC8340 [RFC8340]]

2. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

3. QoS Model Design

A classifier consists of packets which may be grouped when a logical set of rules are applied on different packet header fields. The grouping may be based on different values or range of values of same packet header field, presence or absence of some values or range of values of a packet field or a combination thereof. The QoS classifier is defined in the ietf-qos-classifier module.

A classifier entry contains one or more packet conditioning functions. A packet conditioning function is typically based on direction of traffic and may drop, mark or delay network packets. A set of classifier entries with corresponding conditioning functions when arranged in order of priority represents a QoS policy. A QoS

Choudhary, et al. Expires March 23, 2020 [Page 3] Internet-Draft YANG Model For QoS Sep 2019

policy may contain one or more classifier entries. These are defined in ietf-qos-policy module.

Actions are configured in line with respect to the policy module. These include marking, dropping or shaping. Actions are defined in the ietf-qos-action module.

A meter qualifies if the traffic arrival rate is based on agreed upon rate and variability. A meter is modeled based on commonly used alogrithms in industry, Single Rate Tri Color Marking (srTCM) [RFC2697] meter, Two Rate Tri Color Marking (trTCM) [RFC2698] meter, and Single Rate Two Color Marking meter. Different vendors can extend it with other types of meters as well.

4. DiffServ Model Design

DiffServ architecture [RFC3289] and [RFC2475] describe the architecture as a simple model where traffic entering a network is classified and possibly conditioned at the boundary of the network and assigned a different Behavior Aggregate (BA). Each BA is identified by a specific value of DSCP, and is used to select a Per Hop Behavior (PHB).

The packet classification policy identifies the subset of traffic which may receive a DiffServ by being conditioned or mapped. Packet classifiers select packets within a stream based on the content of some portion of the packet header. There are two types of classifiers, the BA classifier, and the Multi-Field (MF) classifier which selects packets based on a value which is combination of one or more header fields. In the ietf-diffserv module, this is realized by augmenting the QoS classification module.

Traffic conditioning includes metering, shaping and/or marking. A meter is used to measure the traffic against a given traffic profile. The traffic profile specifies the temporal property of the traffic. A packet that arrives is first determined to be in or out of the profile, which will result in the action of marked, dropped or shaped. This is realized in vendor specific modules based on the parameters defined in action module. The metering parameters are augmented to the QoS policy module when metering is defined inline, and to the metering template when metering profile is referred in policy module.

5. Modules Tree Structure

This document defines seven YANG modules - four QoS base modules, a scheduler policy module, a queuing policy module and one DiffServ module.

Choudhary, et al. Expires March 23, 2020 [Page 4] Internet-Draft YANG Model For QoS Sep 2019

ietf-qos-classifier consists of classifier entries identified by a classifier entry name. Each entry MAY contain a list of filter entries. When no filter entry is present in a classifier entry, it matches all traffic.

module: ietf-qos-classifier +--rw classifiers +--rw classifier-entry* [classifier-entry-name] +--rw classifier-entry-name string +--rw classifier-entry-descr? string +--rw classifier-entry-filter-operation? identityref +--rw filter-entry* [filter-type filter-logical-not] +--rw filter-type identityref +--rw filter-logical-not boolean

An ietf-qos-policy module contains list of policy objects identified by a policy name and policy type which MUST be provided. With different values of policy types, each vendor MAY define their own construct of policy for different QoS functionalities. Each vendor MAY augment classifier entry in a policy definition with a set of actions.

module: ietf-qos-policy +--rw policies +--rw policy-entry* [policy-name policy-type] +--rw policy-name string +--rw policy-type identityref +--rw policy-descr? string +--rw classifier-entry* [classifier-entry-name] +--rw classifier-entry-name string +--rw classifier-entry-inline? boolean +--rw classifier-entry-filter-oper? identityref +--rw filter-entry* [filter-type filter-logical-not] {policy-inline-classifier-config}? | +--rw filter-type identityref | +--rw filter-logical-not boolean +--rw classifier-action-entry-cfg* [action-type] +--rw action-type identityref +--rw (action-cfg-params)?

ietf-qos-action module contains grouping of set of QoS actions. These include metering, marking, dropping and shaping. Marking sets DiffServ codepoint value in the classified packet. Color-aware and Color-blind meters are augmented by vendor specific modules based on the parameters defined in action module.

Choudhary, et al. Expires March 23, 2020 [Page 5] Internet-Draft YANG Model For QoS Sep 2019

module: ietf-qos-action +--rw meter-template +--rw meter-entry* [meter-name] {meter-template-support}? +--rw meter-name string +--rw (meter-type)? +--:(one-rate-two-color-meter-type) | +--rw one-rate-two-color-meter | +--rw committed-rate-value? uint64 | +--rw committed-rate-unit? identityref | +--rw committed-burst-value? uint64 | +--rw committed-burst-unit? identityref | +--rw conform-action | | +--rw conform-2color-meter-action-params* [conform-2color-meter-action-type] | | +--rw conform-2color-meter-action-type identityref | | +--rw (conform-2color-meter-action-val)? | +--rw exceed-action | +--rw exceed-2color-meter-action-params* [exceed-2color-meter-action-type] | +--rw exceed-2color-meter-action-type identityref | +--rw (exceed-2color-meter-action-val)? +--:(one-rate-tri-color-meter-type) | +--rw one-rate-tri-color-meter | +--rw committed-rate-value? uint64 | +--rw committed-rate-unit? identityref | +--rw committed-burst-value? uint64 | +--rw committed-burst-unit? identityref | +--rw excess-burst-value? uint64 | +--rw excess-burst-unit? identityref | +--rw conform-action | | +--rw conform-3color-meter-action-params* [conform-3color-meter-action-type] | | +--rw conform-3color-meter-action-type identityref | | +--rw (conform-3color-meter-action-val)? | +--rw exceed-action | | +--rw exceed-3color-meter-action-params* [exceed-3color-meter-action-type] | | +--rw exceed-3color-meter-action-type identityref | | +--rw (exceed-3color-meter-action-val)? | +--rw violate-action | +--rw violate-3color-meter-action-params* [violate-3color-meter-action-type] | +--rw violate-3color-meter-action-type identityref

Choudhary, et al. Expires March 23, 2020 [Page 6] Internet-Draft YANG Model For QoS Sep 2019

| +--rw (violate-3color-meter-action-val)? +--:(two-rate-tri-color-meter-type) +--rw two-rate-tri-color-meter +--rw committed-rate-value? uint64 +--rw committed-rate-unit? identityref +--rw committed-burst-value? uint64 +--rw committed-burst-unit? identityref +--rw peak-rate-value? uint64 +--rw peak-rate-unit? identityref +--rw peak-burst-value? uint64 +--rw peak-burst-unit? identityref +--rw conform-action | +--rw conform-3color-meter-action-params* [conform-3color-meter-action-type] | +--rw conform-3color-meter-action-type identityref | +--rw (conform-3color-meter-action-val)? +--rw exceed-action | +--rw exceed-3color-meter-action-params* [exceed-3color-meter-action-type] | +--rw exceed-3color-meter-action-type identityref | +--rw (exceed-3color-meter-action-val)? +--rw violate-action +--rw violate-3color-meter-action-params* [violate-3color-meter-action-type] +--rw violate-3color-meter-action-type identityref +--rw (violate-3color-meter-action-val)?

ietf-qos-target module contains reference of qos-policy and augments ietf-interfaces [RFC8343] module. A single policy of a particular policy-type can be applied on an interface in each direction of traffic. Policy-type is of type identity and is populated in a vendor specific manner. This way it provides greater flexibility for each vendor to define different policy types each with its own capabilities and restrictions.

Classifier, metering and queuing counters are associated with a target.

module: ietf-qos-target augment /if:interfaces/if:interface: +--rw qos-target-entry* [direction policy-type] +--rw direction identityref +--rw policy-type identityref +--rw policy-name string

Choudhary, et al. Expires March 23, 2020 [Page 7] Internet-Draft YANG Model For QoS Sep 2019

Diffserv module augments QoS classifier module. Many of the YANG types defined in [RFC6991] are represented as leafs in the classifier module.

Metering and marking actions are realized by augmenting the QoS policy-module. Any queuing, AQM and scheduling actions are part of vendor specific augmentation. Statistics are realized by augmenting the QoS target module.

module: ietf-diffserv augment /classifier:classifiers/classifier:classifier-entry + /classifier:filter-entry: +--rw (filter-param)? +--:(dscp) | +--rw dscp-cfg* [dscp-min dscp-max] | +--rw dscp-min inet:dscp | +--rw dscp-max inet:dscp +--:(source-ipv4-address) | +--rw source-ipv4-address-cfg* [source-ipv4-addr] | +--rw source-ipv4-addr inet:ipv4-prefix +--:(destination-ipv4-address) | +--rw destination-ipv4-address-cfg* [destination-ipv4-addr] | +--rw destination-ipv4-addr inet:ipv4-prefix +--:(source-ipv6-address) | +--rw source-ipv6-address-cfg* [source-ipv6-addr] | +--rw source-ipv6-addr inet:ipv6-prefix +--:(destination-ipv6-address) | +--rw destination-ipv6-address-cfg* [destination-ipv6-addr] | +--rw destination-ipv6-addr inet:ipv6-prefix +--:(source-port) | +--rw source-port-cfg* [source-port-min source-port-max] | +--rw source-port-min inet:port-number | +--rw source-port-max inet:port-number +--:(destination-port) | +--rw destination-port-cfg* [destination-port-min destination-port-max] | +--rw destination-port-min inet:port-number | +--rw destination-port-max inet:port-number +--:(protocol) | +--rw protocol-cfg* [protocol-min protocol-max] | +--rw protocol-min uint8 | +--rw protocol-max uint8 +--:(traffic-group) +--rw traffic-group-cfg +--rw traffic-group-name? string augment /policy:policies/policy:policy-entry + /policy:classifier-entry/policy:filter-entry: +--rw (filter-params)?

Choudhary, et al. Expires March 23, 2020 [Page 8] Internet-Draft YANG Model For QoS Sep 2019

+--:(dscp) | +--rw dscp-cfg* [dscp-min dscp-max] | +--rw dscp-min inet:dscp | +--rw dscp-max inet:dscp +--:(source-ipv4-address) | +--rw source-ipv4-address-cfg* [source-ipv4-addr] | +--rw source-ipv4-addr inet:ipv4-prefix +--:(destination-ipv4-address) | +--rw destination-ipv4-address-cfg* [destination-ipv4-addr] | +--rw destination-ipv4-addr inet:ipv4-prefix +--:(source-ipv6-address) | +--rw source-ipv6-address-cfg* [source-ipv6-addr] | +--rw source-ipv6-addr inet:ipv6-prefix +--:(destination-ipv6-address) | +--rw destination-ipv6-address-cfg* [destination-ipv6-addr] | +--rw destination-ipv6-addr inet:ipv6-prefix +--:(source-port) | +--rw source-port-cfg* [source-port-min source-port-max] | +--rw source-port-min inet:port-number | +--rw source-port-max inet:port-number +--:(destination-port) | +--rw destination-port-cfg* [destination-port-min destination-port-max] | +--rw destination-port-min inet:port-number | +--rw destination-port-max inet:port-number +--:(protocol) | +--rw protocol-cfg* [protocol-min protocol-max] | +--rw protocol-min uint8 | +--rw protocol-max uint8 +--:(traffic-group) +--rw traffic-group-cfg +--rw traffic-group-name? string augment /policy:policies/policy:policy-entry + /policy:classifier-entry + /policy:classifier-action-entry-cfg + /policy:action-cfg-params: +--:(dscp-marking) | +--rw dscp-cfg | +--rw dscp? inet:dscp +--:(meter-inline) {action:meter-inline-feature}? | +--rw (meter-type)? | +--:(one-rate-two-color-meter-type) | | +--rw one-rate-two-color-meter | | +--rw committed-rate-value? uint64 | | +--rw committed-rate-unit? identityref | | +--rw committed-burst-value? uint64 | | +--rw committed-burst-unit? identityref | | +--rw conform-action

Choudhary, et al. Expires March 23, 2020 [Page 9] Internet-Draft YANG Model For QoS Sep 2019

Choudhary, et al. Expires March 23, 2020 [Page 10] Internet-Draft YANG Model For QoS Sep 2019

| | +--rw conform-3color-meter-action-params* [conform-3color-meter-action-type] | | +--rw conform-3color-meter-action-type identityref | | +--rw (conform-3color-meter-action-val)? | +--rw exceed-action | | +--rw exceed-3color-meter-action-params* [exceed-3color-meter-action-type] | | +--rw exceed-3color-meter-action-type identityref | | +--rw (exceed-3color-meter-action-val)? | +--rw violate-action | +--rw violate-3color-meter-action-params* [violate-3color-meter-action-type] | +--rw violate-3color-meter-action-type identityref | +--rw (violate-3color-meter-action-val)? +--:(meter-reference) {action:meter-reference-feature}? | +--rw meter-reference-cfg | +--rw meter-reference-name string | +--rw meter-type identityref +--:(traffic-group-marking) {action:traffic-group-feature}? | +--rw traffic-group-cfg | +--rw traffic-group? string +--:(child-policy) {action:child-policy-feature}? | +--rw child-policy-cfg {child-policy-feature}? | +--rw policy-name? string +--:(count) {action:count-feature}? | +--rw count-cfg {count-feature}? | +--rw count-action? empty +--:(named-count) {action:named-counter-feature}? | +--rw named-counter-cfg {named-counter-feature}? | +--rw count-name-action? string +--:(queue-inline) {diffserv-queue-inline-support}? | +--rw queue-cfg | +--rw priority-cfg | | +--rw priority-level? uint8 | +--rw min-rate-cfg | | +--rw rate-value? uint64 | | +--rw rate-unit? identityref | +--rw max-rate-cfg | | +--rw rate-value? uint64 | | +--rw rate-unit? identityref | | +--rw burst-value? uint64 | | +--rw burst-unit? identityref | +--rw algorithmic-drop-cfg | +--rw (drop-algorithm)? | +--:(tail-drop)

Choudhary, et al. Expires March 23, 2020 [Page 11] Internet-Draft YANG Model For QoS Sep 2019

| +--rw tail-drop-cfg | +--rw tail-drop-alg? empty +--:(scheduler-inline) {diffserv-scheduler-inline-support}? +--rw scheduler-cfg +--rw min-rate-cfg | +--rw rate-value? uint64 | +--rw rate-unit? identityref +--rw max-rate-cfg +--rw rate-value? uint64 +--rw rate-unit? identityref +--rw burst-value? uint64 +--rw burst-unit? identityref

module: ietf-queue-policy +--rw queue-template {queue-policy-support}? +--rw name? string +--rw queue-cfg +--rw priority-cfg | +--rw priority-level? uint8 +--rw min-rate-cfg | +--rw rate-value? uint64 | +--rw rate-unit? identityref +--rw max-rate-cfg | +--rw rate-value? uint64 | +--rw rate-unit? identityref | +--rw burst-value? uint64 | +--rw burst-unit? identityref +--rw algorithmic-drop-cfg +--rw (drop-algorithm)? +--:(tail-drop) +--rw tail-drop-cfg +--rw tail-drop-alg? empty augment /policy:policies/policy:policy-entry + /policy:classifier-entry/policy:filter-entry: +--rw (filter-params)? {queue-policy-support}? +--:(traffic-group-name) +--rw traffic-group-reference-cfg +--rw traffic-group-name string augment /policy:policies/policy:policy-entry + /policy:classifier-entry + /policy:classifier-action-entry-cfg + /policy:action-cfg-params: +--:(queue-template-name) {queue-template-support,queue-policy-support}? | +--rw queue-template-reference-cfg | +--rw queue-template-name string +--:(queue-inline) {queue-inline-support,queue-policy-support}?

Choudhary, et al. Expires March 23, 2020 [Page 12] Internet-Draft YANG Model For QoS Sep 2019

+--rw queue-cfg +--rw priority-cfg | +--rw priority-level? uint8 +--rw min-rate-cfg | +--rw rate-value? uint64 | +--rw rate-unit? identityref +--rw max-rate-cfg | +--rw rate-value? uint64 | +--rw rate-unit? identityref | +--rw burst-value? uint64 | +--rw burst-unit? identityref +--rw algorithmic-drop-cfg +--rw (drop-algorithm)? +--:(tail-drop) +--rw tail-drop-cfg +--rw tail-drop-alg? empty

module: ietf-scheduler-policy augment /policy:policies/policy:policy-entry + /policy:classifier-entry/policy:filter-entry: +--rw (filter-params)? +--:(filter-match-all) +--rw match-all-cfg +--rw match-all-action? empty augment /policy:policies/policy:policy-entry + /policy:classifier-entry + /policy:classifier-action-entry-cfg + /policy:action-cfg-params: +--:(scheduler) | +--rw scheduler-cfg | +--rw min-rate-cfg | | +--rw rate-value? uint64 | | +--rw rate-unit? identityref | +--rw max-rate-cfg | +--rw rate-value? uint64 | +--rw rate-unit? identityref | +--rw burst-value? uint64 | +--rw burst-unit? identityref +--:(queue-policy-name) +--rw queue-policy-name +--rw queue-policy string

6. Modules

Choudhary, et al. Expires March 23, 2020 [Page 13] Internet-Draft YANG Model For QoS Sep 2019

6.1. IETF-QOS-CLASSIFIER

file "[email protected]" module ietf-qos-classifier { yang-version 1.1; namespace "urn:ietf:params:xml:ns:yang:ietf-qos-classifier"; prefix classifier;

revision 2019-03-13 { description "Latest revision of qos base classifier module"; reference "RFC XXXX: YANG Model for QoS"; }

feature policy-inline-classifier-config { description " This feature allows classifier configuration directly under policy."; }

Choudhary, et al. Expires March 23, 2020 [Page 14] Internet-Draft YANG Model For QoS Sep 2019

feature classifier-template-feature { description " This feature allows classifier as template configuration in a policy."; }

feature match-any-filter-type-support { description " This feature allows classifier configuration directly under policy."; }

identity filter-type { description "This is identity of base filter-type"; }

identity classifier-entry-filter-operation-type { description "Classifier entry filter logical operation"; }

identity match-all-filter { base classifier-entry-filter-operation-type; description "Classifier entry filter logical AND operation"; }

identity match-any-filter { base classifier-entry-filter-operation-type; if-feature "match-any-filter-type-support"; description "Classifier entry filter logical OR operation"; }

grouping filters { description "Filters types in a Classifier entry"; leaf filter-type { type identityref { base filter-type; } description "This leaf defines type of the filter"; } leaf filter-logical-not { type boolean; description

Choudhary, et al. Expires March 23, 2020 [Page 15] Internet-Draft YANG Model For QoS Sep 2019

" This is logical-not operator for a filter. When true, it indicates filter looks for absence of a pattern defined by the filter "; } }

grouping classifier-entry-generic-attr { description " Classifier generic attributes like name, description, operation type "; leaf classifier-entry-name { type string; description "classifier entry name"; } leaf classifier-entry-descr { type string; description "classifier entry description statement"; } leaf classifier-entry-filter-operation { type identityref { base classifier-entry-filter-operation-type; } default "match-all-filter"; description "Filters are applicable as match-any or match-all filters"; } }

grouping classifier-entry-inline-attr { description "attributes of inline classifier in a policy"; leaf classifier-entry-inline { type boolean; default "false"; description "Indication of inline classifier entry"; } leaf classifier-entry-filter-oper { type identityref { base classifier-entry-filter-operation-type; } default "match-all-filter";

Choudhary, et al. Expires March 23, 2020 [Page 16] Internet-Draft YANG Model For QoS Sep 2019

description "Filters are applicable as match-any or match-all filters"; } list filter-entry { if-feature "policy-inline-classifier-config"; must " ../classifier-entry-inline = ’true’ " { description "For inline filter configuration, inline attributemust be true"; } key "filter-type filter-logical-not"; uses filters; description "Filters configured inline in a policy"; } }

container classifiers { if-feature "classifier-template-feature"; description "list of classifier entry"; list classifier-entry { key "classifier-entry-name"; description "each classifier entry contains a list of filters"; uses classifier-entry-generic-attr; list filter-entry { key "filter-type filter-logical-not"; uses filters; description "Filter entry configuration"; } } } }

6.2. IETF-QOS-POLICY

file "[email protected]" module ietf-qos-policy { yang-version 1.1; namespace "urn:ietf:params:xml:ns:yang:ietf-qos-policy"; prefix policy; import ietf-qos-classifier { prefix classifier; reference "RFC XXXX: YANG Model for QoS"; }

Choudhary, et al. Expires March 23, 2020 [Page 17] Internet-Draft YANG Model For QoS Sep 2019

organization "IETF RTG (Routing Area) Working Group"; contact "WG Web: WG List: WG Chair: Chris Bowers WG Chair: Jeff Tantsura Editor: Aseem Choudhary Editor: Mahesh Jethanandani Editor: Norm Strahle "; description "This module contains a collection of YANG definitions for configuring qos specification implementations. Copyright (c) 2019 IETF Trust and the persons identified as authors of the code. All rights reserved. Redistribution and use in source and binary forms, with or without modification, is permitted pursuant to, and subject to the license terms contained in, the Simplified BSD License set forth in Section 4.c of the IETF Trust’s Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info). This version of this YANG module is part of RFC XXXX; see the RFC itself for full legal notices."; revision 2019-03-13 { description "Latest revision of qos policy"; reference "RFC XXXX: YANG Model for QoS"; } identity policy-type { description "This base identity type defines policy-types"; } grouping policy-generic-attr { description "Policy Attributes"; leaf policy-name { type string; description "policy name"; } leaf policy-type { type identityref { base policy-type; }

Choudhary, et al. Expires March 23, 2020 [Page 18] Internet-Draft YANG Model For QoS Sep 2019

description "policy type"; } leaf policy-descr { type string; description "policy description"; } } identity action-type { description "This base identity type defines action-types"; } grouping classifier-action-entry-cfg { description "List of Configuration of classifier & associated actions"; list classifier-action-entry-cfg { key "action-type"; ordered-by user; description "Configuration of classifier & associated actions"; leaf action-type { type identityref { base action-type; } description "This defines action type "; } choice action-cfg-params { description "Choice of action types"; } } } container policies { description "list of policy templates"; list policy-entry { key "policy-name policy-type"; description "policy template"; uses policy-generic-attr; list classifier-entry { key "classifier-entry-name"; ordered-by user; description "Classifier entry configuration in a policy"; leaf classifier-entry-name {

Choudhary, et al. Expires March 23, 2020 [Page 19] Internet-Draft YANG Model For QoS Sep 2019

type string; description "classifier entry name"; } uses classifier:classifier-entry-inline-attr; uses classifier-action-entry-cfg; } } } }

6.3. IETF-QOS-ACTION

file "[email protected]" module ietf-qos-action { yang-version 1.1; namespace "urn:ietf:params:xml:ns:yang:ietf-qos-action"; prefix action; import ietf-inet-types { prefix inet; reference "RFC 6991: Common YANG Data Types"; } import ietf-qos-policy { prefix policy; reference "RFC XXXX: YANG Model for QoS"; } organization "IETF RTG (Routing Area) Working Group"; contact "WG Web: WG List: WG Chair: Chris Bowers WG Chair: Jeff Tantsura Editor: Aseem Choudhary Editor: Mahesh Jethanandani Editor: Norm Strahle "; description "This module contains a collection of YANG definitions for configuring qos specification implementations. Copyright (c) 2019 IETF Trust and the persons identified as authors of the code. All rights reserved. Redistribution and use in source and binary forms, with or without modification, is permitted pursuant to, and subject

Choudhary, et al. Expires March 23, 2020 [Page 20] Internet-Draft YANG Model For QoS Sep 2019

to the license terms contained in, the Simplified BSD License set forth in Section 4.c of the IETF Trust’s Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info). This version of this YANG module is part of RFC XXXX; see the RFC itself for full legal notices."; revision 2019-03-13 { description "Latest revision for qos actions"; reference "RFC XXXX: YANG Model for QoS"; } feature meter-template-support { description " This feature allows support of meter-template."; } feature meter-inline-feature { description "This feature allows support of meter-inline configuration."; } feature meter-reference-feature { description "This feature allows support of meter by reference configuration."; } feature queue-action-support { description " This feature allows support of queue action configuration in policy."; } feature scheduler-action-support { description " This feature allows support of scheduler configuration in policy."; } feature child-policy-feature { description " This feature allows configuration of hierarchical policy."; } feature count-feature { description "This feature allows action configuration to enable counter in a classifier"; } feature named-counter-feature { description "This feature allows action configuration to enable named counter in a classifier"; }

Choudhary, et al. Expires March 23, 2020 [Page 21] Internet-Draft YANG Model For QoS Sep 2019

feature traffic-group-feature { description "traffic-group action support"; } feature burst-time-unit-support { description "This feature allows burst unit to be configured as time duration."; }

identity rate-unit-type { description "base rate-unit type"; } identity bits-per-second { base rate-unit-type; description "bits per second identity"; } identity kilo-bits-per-second { base rate-unit-type; description "kilo bits per second identity"; } identity mega-bits-per-second { base rate-unit-type; description "mega bits per second identity"; } identity giga-bits-per-second { base rate-unit-type; description "mega bits per second identity"; } identity percent { base rate-unit-type; description "percentage"; } identity burst-unit-type { description "base burst-unit type"; } identity bytes { base burst-unit-type; description "bytes"; }

Choudhary, et al. Expires March 23, 2020 [Page 22] Internet-Draft YANG Model For QoS Sep 2019

identity kilo-bytes { base burst-unit-type; description "kilo bytes"; } identity mega-bytes { base burst-unit-type; description "mega bytes"; } identity millisecond { base burst-unit-type; if-feature burst-time-unit-support; description "milli seconds"; } identity microsecond { base burst-unit-type; if-feature burst-time-unit-support; description "micro seconds"; } identity dscp-marking { base policy:action-type; description "dscp marking action type"; } identity meter-inline { base policy:action-type; if-feature meter-inline-feature; description "meter-inline action type"; } identity meter-reference { base policy:action-type; if-feature meter-reference-feature; description "meter reference action type"; } identity queue { base policy:action-type; if-feature queue-action-support; description "queue action type"; } identity scheduler { base policy:action-type; if-feature scheduler-action-support;

Choudhary, et al. Expires March 23, 2020 [Page 23] Internet-Draft YANG Model For QoS Sep 2019

description "scheduler action type"; } identity discard { base policy:action-type; description "discard action type"; } identity child-policy { base policy:action-type; if-feature child-policy-feature; description "child-policy action type"; } identity count { base policy:action-type; if-feature count-feature; description "count action type"; } identity named-counter { base policy:action-type; if-feature named-counter-feature; description "name counter action type"; }

identity meter-type { description "This base identity type defines meter types"; } identity one-rate-two-color-meter-type { base meter-type; description "one rate two color meter type"; } identity one-rate-tri-color-meter-type { base meter-type; description "one rate three color meter type"; reference "RFC2697: A Single Rate Three Color Marker"; } identity two-rate-tri-color-meter-type { base meter-type; description "two rate three color meter action type"; reference

Choudhary, et al. Expires March 23, 2020 [Page 24] Internet-Draft YANG Model For QoS Sep 2019

"RFC2698: A Two Rate Three Color Marker"; }

identity drop-type { description "drop algorithm"; } identity tail-drop { base drop-type; description "tail drop algorithm"; }

identity conform-2color-meter-action-type { description "action type in a meter"; } identity exceed-2color-meter-action-type { description "action type in a meter"; } identity conform-3color-meter-action-type { description "action type in a meter"; } identity exceed-3color-meter-action-type { description "action type in a meter"; } identity violate-3color-meter-action-type { description "action type in a meter"; }

grouping rate-value-unit { leaf rate-value { type uint64; description "rate value"; } leaf rate-unit { type identityref { base rate-unit-type; } description "rate unit"; } description

Choudhary, et al. Expires March 23, 2020 [Page 25] Internet-Draft YANG Model For QoS Sep 2019

"rate value and unit grouping"; } grouping burst { description "burst value and unit configuration"; leaf burst-value { type uint64; description "burst value"; } leaf burst-unit { type identityref { base burst-unit-type; } description "burst unit"; } }

grouping threshold { description "Threshold Parameters"; container threshold { description "threshold"; choice threshold-type { case size { leaf threshold-size { type uint64; units "bytes"; description "Threshold size"; } } case interval { leaf threshold-interval { type uint64; units "microsecond"; description "Threshold interval"; } } description "Choice of threshold type"; } } }

Choudhary, et al. Expires March 23, 2020 [Page 26] Internet-Draft YANG Model For QoS Sep 2019

grouping drop { container drop-cfg { leaf drop-action { type empty; description "always drop algorithm"; } description "the drop action"; } description "always drop grouping"; }

grouping queuelimit { container qlimit-thresh { uses threshold; description "the queue limit"; } description "the queue limit beyond which queue will not hold any packet"; }

grouping conform-2color-meter-action-params { description "meter action parameters"; list conform-2color-meter-action-params { key "conform-2color-meter-action-type"; ordered-by user; description "Configuration of basic-meter & associated actions"; leaf conform-2color-meter-action-type { type identityref { base conform-2color-meter-action-type; } description "meter action type"; } choice conform-2color-meter-action-val { description " meter action based on choice of meter action type"; } } }

grouping exceed-2color-meter-action-params { description

Choudhary, et al. Expires March 23, 2020 [Page 27] Internet-Draft YANG Model For QoS Sep 2019

"meter action parameters"; list exceed-2color-meter-action-params { key "exceed-2color-meter-action-type"; ordered-by user; description "Configuration of basic-meter & associated actions"; leaf exceed-2color-meter-action-type { type identityref { base exceed-2color-meter-action-type; } description "meter action type"; } choice exceed-2color-meter-action-val { description " meter action based on choice of meter action type"; } } }

grouping conform-3color-meter-action-params { description "meter action parameters"; list conform-3color-meter-action-params { key "conform-3color-meter-action-type"; ordered-by user; description "Configuration of basic-meter & associated actions"; leaf conform-3color-meter-action-type { type identityref { base conform-3color-meter-action-type; } description "meter action type"; } choice conform-3color-meter-action-val { description " meter action based on choice of meter action type"; } } }

grouping exceed-3color-meter-action-params { description "meter action parameters"; list exceed-3color-meter-action-params { key "exceed-3color-meter-action-type";

Choudhary, et al. Expires March 23, 2020 [Page 28] Internet-Draft YANG Model For QoS Sep 2019

ordered-by user; description "Configuration of basic-meter & associated actions"; leaf exceed-3color-meter-action-type { type identityref { base exceed-3color-meter-action-type; } description "meter action type"; } choice exceed-3color-meter-action-val { description " meter action based on choice of meter action type"; } } }

grouping violate-3color-meter-action-params { description "meter action parameters"; list violate-3color-meter-action-params { key "violate-3color-meter-action-type"; ordered-by user; description "Configuration of basic-meter & associated actions"; leaf violate-3color-meter-action-type { type identityref { base violate-3color-meter-action-type; } description "meter action type"; } choice violate-3color-meter-action-val { description " meter action based on choice of meter action type"; } } }

grouping one-rate-two-color-meter { container one-rate-two-color-meter { description "single rate two color marker meter"; leaf committed-rate-value { type uint64; description "committed rate value"; }

Choudhary, et al. Expires March 23, 2020 [Page 29] Internet-Draft YANG Model For QoS Sep 2019

leaf committed-rate-unit { type identityref { base rate-unit-type; } description "committed rate unit"; } leaf committed-burst-value { type uint64; description "burst value"; } leaf committed-burst-unit { type identityref { base burst-unit-type; } description "committed burst unit"; } container conform-action { uses conform-2color-meter-action-params; description "conform action"; } container exceed-action { uses exceed-2color-meter-action-params; description "exceed action"; } } description "single rate two color marker meter attributes"; }

grouping one-rate-tri-color-meter { container one-rate-tri-color-meter { description "single rate three color meter"; reference "RFC2697: A Single Rate Three Color Marker"; leaf committed-rate-value { type uint64; description "meter rate"; } leaf committed-rate-unit { type identityref { base rate-unit-type;

Choudhary, et al. Expires March 23, 2020 [Page 30] Internet-Draft YANG Model For QoS Sep 2019

} description "committed rate unit"; } leaf committed-burst-value { type uint64; description "committed burst size"; } leaf committed-burst-unit { type identityref { base burst-unit-type; } description "committed burst unit"; } leaf excess-burst-value { type uint64; description "excess burst size"; } leaf excess-burst-unit { type identityref { base burst-unit-type; } description "excess burst unit"; } container conform-action { uses conform-3color-meter-action-params; description "conform, or green action"; } container exceed-action { uses exceed-3color-meter-action-params; description "exceed, or yellow action"; } container violate-action { uses violate-3color-meter-action-params; description "violate, or red action"; } } description "one-rate-tri-color-meter attributes"; }

Choudhary, et al. Expires March 23, 2020 [Page 31] Internet-Draft YANG Model For QoS Sep 2019

grouping two-rate-tri-color-meter { container two-rate-tri-color-meter { description "two rate three color meter"; reference "RFC2698: A Two Rate Three Color Marker"; leaf committed-rate-value { type uint64; units "bits-per-second"; description "committed rate"; } leaf committed-rate-unit { type identityref { base rate-unit-type; } description "committed rate unit"; } leaf committed-burst-value { type uint64; description "commited burst size"; } leaf committed-burst-unit { type identityref { base burst-unit-type; } description "committed burst unit"; } leaf peak-rate-value { type uint64; description "peak rate"; } leaf peak-rate-unit { type identityref { base rate-unit-type; } description "committed rate unit"; } leaf peak-burst-value { type uint64; description "commited burst size"; }

Choudhary, et al. Expires March 23, 2020 [Page 32] Internet-Draft YANG Model For QoS Sep 2019

leaf peak-burst-unit { type identityref { base burst-unit-type; } description "peak burst unit"; } container conform-action { uses conform-3color-meter-action-params; description "conform, or green action"; } container exceed-action { uses exceed-3color-meter-action-params; description "exceed, or yellow action"; } container violate-action { uses violate-3color-meter-action-params; description "exceed, or red action"; } } description "two-rate-tri-color-meter attributes"; }

grouping meter { choice meter-type { case one-rate-two-color-meter-type { uses one-rate-two-color-meter; description "basic meter"; } case one-rate-tri-color-meter-type { uses one-rate-tri-color-meter; description "one rate tri-color meter"; } case two-rate-tri-color-meter-type { uses two-rate-tri-color-meter; description "two rate tri-color meter"; } description " meter action based on choice of meter action type"; } description

Choudhary, et al. Expires March 23, 2020 [Page 33] Internet-Draft YANG Model For QoS Sep 2019

"meter attributes"; }

container meter-template { description "list of meter templates"; list meter-entry { if-feature meter-template-support; key "meter-name"; description "meter entry template"; leaf meter-name { type string; description "meter identifier"; } uses meter; } }

grouping meter-reference { container meter-reference-cfg { leaf meter-reference-name { type string ; mandatory true; description "This leaf defines name of the meter referenced"; } leaf meter-type { type identityref { base meter-type; } mandatory true; description "This leaf defines type of the meter"; } description "meter reference name"; } description "meter reference"; }

grouping count { container count-cfg { if-feature count-feature; leaf count-action { type empty;

Choudhary, et al. Expires March 23, 2020 [Page 34] Internet-Draft YANG Model For QoS Sep 2019

description "count action"; } description "the count action"; } description "the count action grouping"; }

grouping named-counter { container named-counter-cfg { if-feature named-counter-feature; leaf count-name-action { type string; description "count action"; } description "the count action"; } description "the count action grouping"; }

grouping discard { container discard-cfg { leaf discard { type empty; description "discard action"; } description "discard action"; } description "discard grouping"; }

grouping priority { container priority-cfg { leaf priority-level { type uint8; description "priority level"; } description "priority attributes";

Choudhary, et al. Expires March 23, 2020 [Page 35] Internet-Draft YANG Model For QoS Sep 2019

} description "priority attributes grouping"; } grouping min-rate { container min-rate-cfg { uses rate-value-unit; description "min guaranteed bandwidth"; reference "RFC3289, section 3.5.3"; } description "minimum rate grouping"; } grouping dscp-marking { container dscp-cfg { leaf dscp { type inet:dscp; description "dscp marking"; } description "dscp marking container"; } description "dscp marking grouping"; } grouping traffic-group-marking { container traffic-group-cfg { leaf traffic-group { type string; description "traffic group marking"; } description "traffic group marking container"; } description "traffic group marking grouping"; } grouping child-policy { container child-policy-cfg { if-feature child-policy-feature; leaf policy-name { type string; description "Hierarchical Policy";

Choudhary, et al. Expires March 23, 2020 [Page 36] Internet-Draft YANG Model For QoS Sep 2019

} description "Hierarchical Policy configuration container"; } description "Grouping of Hierarchical Policy configuration"; } grouping max-rate { container max-rate-cfg { uses rate-value-unit; uses burst; description "maximum rate attributes container"; reference "RFC3289, section 3.5.4"; } description "maximum rate attributes"; } grouping queue { container queue-cfg { uses priority; uses min-rate; uses max-rate; container algorithmic-drop-cfg { choice drop-algorithm { case tail-drop { container tail-drop-cfg { leaf tail-drop-alg { type empty; description "tail drop algorithm"; } description "Tail Drop configuration container"; } description "Tail Drop choice"; } description "Choice of Drop Algorithm"; } description "Algorithmic Drop configuration container"; } description "Queue configuration container"; }

Choudhary, et al. Expires March 23, 2020 [Page 37] Internet-Draft YANG Model For QoS Sep 2019

description "Queue grouping"; } grouping scheduler { container scheduler-cfg { uses min-rate; uses max-rate; description "Schedular configuration container"; } description "Schedular configuration grouping"; } }

6.4. IETF-QOS-TARGET

file "[email protected]" module ietf-qos-target { yang-version 1.1; namespace "urn:ietf:params:xml:ns:yang:ietf-qos-target"; prefix target;

import ietf-interfaces { prefix if; reference "RFC8343: A YANG Data Model for Interface Management"; } import ietf-qos-policy { prefix policy; reference "RFC XXXX: YANG Model for QoS"; }

organization "IETF RTG (Routing Area) Working Group"; contact "WG Web: WG List: WG Chair: Chris Bowers WG Chair: Jeff Tantsura Editor: Aseem Choudhary Editor: Mahesh Jethanandani Editor: Norm Strahle "; description

Choudhary, et al. Expires March 23, 2020 [Page 38] Internet-Draft YANG Model For QoS Sep 2019

"This module contains a collection of YANG definitions for configuring qos specification implementations. Copyright (c) 2019 IETF Trust and the persons identified as authors of the code. All rights reserved. Redistribution and use in source and binary forms, with or without modification, is permitted pursuant to, and subject to the license terms contained in, the Simplified BSD License set forth in Section 4.c of the IETF Trust’s Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info). This version of this YANG module is part of RFC XXXX; see the RFC itself for full legal notices.";

revision 2019-03-13 { description "Latest revision qos based policy applied to a target"; reference "RFC XXXX: YANG Model for QoS"; }

identity direction { description "This is identity of traffic direction"; }

identity inbound { base direction; description "Direction of traffic coming into the network entry"; }

identity outbound { base direction; description "Direction of traffic going out of the network entry"; }

augment "/if:interfaces/if:interface" { description "Augments Diffserv Target Entry to Interface module"; list qos-target-entry { key "direction policy-type"; description "policy target for inbound or outbound direction"; leaf direction { type identityref { base direction; } description

Choudhary, et al. Expires March 23, 2020 [Page 39] Internet-Draft YANG Model For QoS Sep 2019

"Direction fo the traffic flow either inbound or outbound"; } leaf policy-type { type identityref { base policy:policy-type; } description "Policy entry type"; } leaf policy-name { type string; mandatory true; description "Policy entry name"; } } } }

6.5. IETF-DIFFSERV

file "[email protected]" module ietf-diffserv { yang-version 1.1; namespace "urn:ietf:params:xml:ns:yang:ietf-diffserv"; prefix diffserv;

import ietf-qos-classifier { prefix classifier; reference "RFC XXXX: YANG Model for QoS"; } import ietf-qos-policy { prefix policy; reference "RFC XXXX: YANG Model for QoS"; } import ietf-qos-action { prefix action; reference "RFC XXXX: YANG Model for QoS"; } import ietf-inet-types { prefix inet; reference "RFC 6991: Common YANG Data Types"; }

organization "IETF RTG (Routing Area) Working Group"; contact "WG Web:

Choudhary, et al. Expires March 23, 2020 [Page 40] Internet-Draft YANG Model For QoS Sep 2019

WG List: WG Chair: Chris Bowers WG Chair: Jeff Tantsura Editor: Aseem Choudhary Editor: Mahesh Jethanandani Editor: Norm Strahle "; description "This module contains a collection of YANG definitions for configuring diffserv specification implementations. Copyright (c) 2019 IETF Trust and the persons identified as authors of the code. All rights reserved. Redistribution and use in source and binary forms, with or without modification, is permitted pursuant to, and subject to the license terms contained in, the Simplified BSD License set forth in Section 4.c of the IETF Trust’s Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info). This version of this YANG module is part of RFC XXXX; see the RFC itself for full legal notices.";

revision 2019-03-13 { description "Latest revision of diffserv based classifier"; reference "RFC XXXX: YANG Model for QoS"; }

feature diffserv-queue-inline-support { description "Queue inline support in diffserv policy"; } feature diffserv-scheduler-inline-support { description "scheduler inline support in diffserv policy"; } identity diffserv-policy-type { base policy:policy-type; description "This defines ip policy-type"; } identity ipv4-diffserv-policy-type { base policy:policy-type; description "This defines ipv4 policy-type";

Choudhary, et al. Expires March 23, 2020 [Page 41] Internet-Draft YANG Model For QoS Sep 2019

} identity ipv6-diffserv-policy-type { base policy:policy-type; description "This defines ipv6 policy-type"; }

identity dscp { base classifier:filter-type; description "Differentiated services code point filter-type"; } identity source-ipv4-address { base classifier:filter-type; description "source ipv4 address filter-type"; } identity destination-ipv4-address { base classifier:filter-type; description "destination ipv4 address filter-type"; } identity source-ipv6-address { base classifier:filter-type; description "source ipv6 address filter-type"; } identity destination-ipv6-address { base classifier:filter-type; description "destination ipv6 address filter-type"; } identity source-port { base classifier:filter-type; description "source port filter-type"; } identity destination-port { base classifier:filter-type; description "destination port filter-type"; } identity protocol { base classifier:filter-type; description "protocol type filter-type"; } identity traffic-group-name {

Choudhary, et al. Expires March 23, 2020 [Page 42] Internet-Draft YANG Model For QoS Sep 2019

base classifier:filter-type; description "traffic-group filter type"; }

identity meter-type { description "This base identity type defines meter types"; } identity one-rate-two-color-meter-type { base meter-type; description "one rate two color meter type"; } identity one-rate-tri-color-meter-type { base meter-type; description "one rate three color meter type"; } identity two-rate-tri-color-meter-type { base meter-type; description "two rate three color meter action type"; } grouping dscp-cfg { list dscp-cfg { key "dscp-min dscp-max"; description "list of dscp ranges"; leaf dscp-min { type inet:dscp; description "Minimum value of dscp min-max range"; } leaf dscp-max { type inet:dscp; description "maximum value of dscp min-max range"; } } description "Filter grouping containing list of dscp ranges"; } grouping source-ipv4-address-cfg { list source-ipv4-address-cfg { key "source-ipv4-addr"; description "list of source ipv4 address";

Choudhary, et al. Expires March 23, 2020 [Page 43] Internet-Draft YANG Model For QoS Sep 2019

leaf source-ipv4-addr { type inet:ipv4-prefix; description "source ipv4 prefix"; } } description "Filter grouping containing list of source ipv4 addresses"; } grouping destination-ipv4-address-cfg { list destination-ipv4-address-cfg { key "destination-ipv4-addr"; description "list of destination ipv4 address"; leaf destination-ipv4-addr { type inet:ipv4-prefix; description "destination ipv4 prefix"; } } description "Filter grouping containing list of destination ipv4 address"; } grouping source-ipv6-address-cfg { list source-ipv6-address-cfg { key "source-ipv6-addr"; description "list of source ipv6 address"; leaf source-ipv6-addr { type inet:ipv6-prefix; description "source ipv6 prefix"; } } description "Filter grouping containing list of source ipv6 addresses"; } grouping destination-ipv6-address-cfg { list destination-ipv6-address-cfg { key "destination-ipv6-addr"; description "list of destination ipv4 or ipv6 address"; leaf destination-ipv6-addr { type inet:ipv6-prefix; description "destination ipv6 prefix"; } }

Choudhary, et al. Expires March 23, 2020 [Page 44] Internet-Draft YANG Model For QoS Sep 2019

description "Filter grouping containing list of destination ipv6 address"; } grouping source-port-cfg { list source-port-cfg { key "source-port-min source-port-max"; description "list of ranges of source port"; leaf source-port-min { type inet:port-number; description "minimum value of source port range"; } leaf source-port-max { type inet:port-number; description "maximum value of source port range"; } } description "Filter grouping containing list of source port ranges"; } grouping destination-port-cfg { list destination-port-cfg { key "destination-port-min destination-port-max"; description "list of ranges of destination port"; leaf destination-port-min { type inet:port-number; description "minimum value of destination port range"; } leaf destination-port-max { type inet:port-number; description "maximum value of destination port range"; } } description "Filter grouping containing list of destination port ranges"; } grouping protocol-cfg { list protocol-cfg { key "protocol-min protocol-max"; description "list of ranges of protocol values"; leaf protocol-min { type uint8 {

Choudhary, et al. Expires March 23, 2020 [Page 45] Internet-Draft YANG Model For QoS Sep 2019

range "0..255"; } description "minimum value of protocol range"; } leaf protocol-max { type uint8 { range "0..255"; } description "maximum value of protocol range"; } } description "Filter grouping containing list of Protocol ranges"; } grouping traffic-group-cfg { container traffic-group-cfg { leaf traffic-group-name { type string ; description "This leaf defines name of the traffic group referenced"; } description "traffic group container"; } description "traffic group grouping"; }

augment "/classifier:classifiers/classifier:classifier-entry" + "/classifier:filter-entry" { choice filter-param { description "Choice of filter types"; case dscp { uses dscp-cfg; description "Filter containing list of dscp ranges"; } case source-ipv4-address { uses source-ipv4-address-cfg; description "Filter containing list of source ipv4 addresses"; } case destination-ipv4-address { uses destination-ipv4-address-cfg; description

Choudhary, et al. Expires March 23, 2020 [Page 46] Internet-Draft YANG Model For QoS Sep 2019

"Filter containing list of destination ipv4 address"; } case source-ipv6-address { uses source-ipv6-address-cfg; description "Filter containing list of source ipv6 addresses"; } case destination-ipv6-address { uses destination-ipv6-address-cfg; description "Filter containing list of destination ipv6 address"; } case source-port { uses source-port-cfg; description "Filter containing list of source-port ranges"; } case destination-port { uses destination-port-cfg; description "Filter containing list of destination-port ranges"; } case protocol { uses protocol-cfg; description "Filter Type Protocol"; } case traffic-group { uses traffic-group-cfg; description "Filter Type traffic-group"; } } description "augments diffserv filters to qos classifier"; } augment "/policy:policies/policy:policy-entry" + "/policy:classifier-entry/policy:filter-entry" { when "../../policy:policy-type = ’diffserv:ipv4-diffserv-policy-type’ or ../../policy:policy-type = ’diffserv:ipv6-diffserv-policy-type’ or ../../policy:policy-type = ’diffserv:diffserv-policy-type’" { description "Filters can be augmented if policy type is ipv4, ipv6 or default diffserv policy types "; }

Choudhary, et al. Expires March 23, 2020 [Page 47] Internet-Draft YANG Model For QoS Sep 2019

description "Augments Diffserv Classifier with common filter types"; choice filter-params { description "Choice of action types"; case dscp { uses dscp-cfg; description "Filter containing list of dscp ranges"; } case source-ipv4-address { when "../../policy:policy-type != ’diffserv:ipv6-diffserv-policy-type’" { description "If policy type is v6, this filter cannot be used."; } uses source-ipv4-address-cfg; description "Filter containing list of source ipv4 addresses"; } case destination-ipv4-address { when "../../policy:policy-type != ’diffserv:ipv6-diffserv-policy-type’" { description "If policy type is v6, this filter cannot be used."; } uses destination-ipv4-address-cfg; description "Filter containing list of destination ipv4 address"; } case source-ipv6-address { when "../../policy:policy-type != ’diffserv:ipv4-diffserv-policy-type’" { description "If policy type is v4, this filter cannot be used."; } uses source-ipv6-address-cfg; description "Filter containing list of source ipv6 addresses"; } case destination-ipv6-address { when "../../policy:policy-type != ’diffserv:ipv4-diffserv-policy-type’" { description "If policy type is v4, this filter cannot be used."; } uses destination-ipv6-address-cfg; description

Choudhary, et al. Expires March 23, 2020 [Page 48] Internet-Draft YANG Model For QoS Sep 2019

"Filter containing list of destination ipv6 address"; } case source-port { uses source-port-cfg; description "Filter containing list of source-port ranges"; } case destination-port { uses destination-port-cfg; description "Filter containing list of destination-port ranges"; } case protocol { uses protocol-cfg; description "Filter Type Protocol"; } case traffic-group { uses traffic-group-cfg; description "Filter Type traffic-group"; } } } augment "/policy:policies/policy:policy-entry" + "/policy:classifier-entry" + "/policy:classifier-action-entry-cfg" + "/policy:action-cfg-params" { when "../../policy:policy-type = ’diffserv:ipv4-diffserv-policy-type’ or ../../policy:policy-type = ’diffserv:ipv6-diffserv-policy-type’ or ../../policy:policy-type = ’diffserv:diffserv-policy-type’ " { description "Actions can be augmented if policy type is ipv4, ipv6 or default diffserv policy types "; } description "Augments Diffserv Policy with action configuration"; case dscp-marking { uses action:dscp-marking; } case meter-inline { if-feature action:meter-inline-feature; uses action:meter; } case meter-reference {

Choudhary, et al. Expires March 23, 2020 [Page 49] Internet-Draft YANG Model For QoS Sep 2019

if-feature action:meter-reference-feature; uses action:meter-reference; } case child-policy { if-feature action:child-policy-feature; uses action:child-policy; } case count { if-feature action:count-feature; uses action:count; } case named-count { if-feature action:named-counter-feature; uses action:named-counter; } case queue-inline { if-feature diffserv-queue-inline-support; uses action:queue; } case scheduler-inline { if-feature diffserv-scheduler-inline-support; uses action:scheduler; } } }

6.6. IETF-QUEUE-POLICY

file "[email protected]" module ietf-queue-policy { yang-version 1.1; namespace "urn:ietf:params:xml:ns:yang:ietf-queue-policy"; prefix queue-policy;

import ietf-qos-policy { prefix policy; reference "RFC XXXX: YANG Model for QoS"; } import ietf-qos-action { prefix action; reference "RFC XXXX: YANG Model for QoS"; } import ietf-diffserv { prefix diffserv; reference "RFC XXXX: YANG Model for QoS"; }

Choudhary, et al. Expires March 23, 2020 [Page 50] Internet-Draft YANG Model For QoS Sep 2019

revision 2019-03-13 { description "Latest revision of queuing policy module"; reference "RFC XXXX: YANG Model for QoS"; }

feature queue-policy-support { description " This feature allows queue policy configuration as a separate policy type support."; }

feature queue-inline-support { description "Queue inline support in Queue policy"; }

feature queue-template-support { description "Queue template support in Queue policy";

Choudhary, et al. Expires March 23, 2020 [Page 51] Internet-Draft YANG Model For QoS Sep 2019

}

identity queue-policy-type { base policy:policy-type; description "This defines queue policy-type"; }

augment "/policy:policies/policy:policy-entry" + "/policy:classifier-entry/policy:filter-entry" { when "../../policy:policy-type = ’queue-policy:queue-policy-type’" { description "If policy type is v6, this filter cannot be used."; } if-feature queue-policy-support; choice filter-params { description "Choice of action types"; case traffic-group-name { uses diffserv:traffic-group-cfg; description "traffic group name"; } } description "Augments Queue policy Classifier with common filter types"; }

identity queue-template-name { base policy:action-type; description "queue template name"; }

grouping queue-template-reference { container queue-template-reference-cfg { leaf queue-template-name { type string ; mandatory true; description "This leaf defines name of the queue template referenced"; } description "queue template reference"; } description "queue template reference grouping";

Choudhary, et al. Expires March 23, 2020 [Page 52] Internet-Draft YANG Model For QoS Sep 2019

}

container queue-template { if-feature queue-policy-support; description "Queue template"; leaf name { type string; description "A unique name identifying this queue template"; } uses action:queue; }

augment "/policy:policies/policy:policy-entry" + "/policy:classifier-entry" + "/policy:classifier-action-entry-cfg" + "/policy:action-cfg-params" { when "../../policy:policy-type = ’queue-policy:queue-policy-type’" { description "queue policy actions."; } if-feature queue-policy-support; case queue-template-name { if-feature queue-template-support; uses queue-template-reference; } case queue-inline { if-feature queue-inline-support; uses action:queue; } description "augments queue template reference to queue policy"; } }

6.7. IETF-SCHEDULER-POLICY

file "[email protected]" module ietf-scheduler-policy { yang-version 1.1; namespace "urn:ietf:params:xml:ns:yang:ietf-scheduler-policy"; prefix scheduler-policy;

import ietf-qos-classifier {

Choudhary, et al. Expires March 23, 2020 [Page 53] Internet-Draft YANG Model For QoS Sep 2019

prefix classifier; reference "RFC XXXX: YANG Model for QoS"; } import ietf-qos-policy { prefix policy; reference "RFC XXXX: YANG Model for QoS"; } import ietf-qos-action { prefix action; reference "RFC XXXX: YANG Model for QoS"; }

organization "IETF RTG (Routing Area) Working Group"; contact "WG Web: WG List: WG Chair: Chris Bowers WG Chair: Jeff Tantsura Editor: Norm Strahle Editor: Aseem Choudhary "; description "This module contains a collection of YANG definitions for configuring diffserv specification implementations. Copyright (c) 2019 IETF Trust and the persons identified as authors of the code. All rights reserved. Redistribution and use in source and binary forms, with or without modification, is permitted pursuant to, and subject to the license terms contained in, the Simplified BSD License set forth in Section 4.c of the IETF Trust’s Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info). This version of this YANG module is part of RFC XXXX; see the RFC itself for full legal notices.";

revision 2019-03-13 { description "Latest revision of scheduler policy module"; reference "RFC XXXX: YANG Model for QoS"; } feature scheduler-policy-support { description " This feature allows sheduler policy configuration as a separate policy type support."; }

Choudhary, et al. Expires March 23, 2020 [Page 54] Internet-Draft YANG Model For QoS Sep 2019

identity scheduler-policy-type { base policy:policy-type; description "This defines scheduler policy-type"; }

identity filter-match-all { base classifier:filter-type; description "Traffic-group filter type"; }

grouping filter-match-all-cfg { container match-all-cfg { leaf match-all-action { type empty; description "match all packets"; } description "the match-all action"; } description "the match-all filter grouping"; }

augment "/policy:policies/policy:policy-entry" + "/policy:classifier-entry/policy:filter-entry" { when "../../policy:policy-type = ’scheduler-policy:scheduler-policy-type’" { description "Only when policy type is scheduler-policy"; } choice filter-params { description "Choice of action types"; case filter-match-all { uses filter-match-all-cfg; description "filter match-all"; } } description "Augments Queue policy Classifier with common filter types"; }

identity queue-policy-name { base policy:action-type;

Choudhary, et al. Expires March 23, 2020 [Page 55] Internet-Draft YANG Model For QoS Sep 2019

description "queue policy name"; }

grouping queue-policy-name-cfg { container queue-policy-name { leaf queue-policy { type string ; mandatory true; description "This leaf defines name of the queue-policy"; } description "container for queue-policy name"; } description "queue-policy name grouping"; }

augment "/policy:policies/policy:policy-entry" + "/policy:classifier-entry" + "/policy:classifier-action-entry-cfg" + "/policy:action-cfg-params" { when "../../policy:policy-type = ’scheduler-policy:scheduler-policy-type’" { description "Only when policy type is scheduler-policy"; } case scheduler { uses action:scheduler; } case queue-policy-name { uses queue-policy-name-cfg; } description "augments scheduler template reference to scheduler policy"; } }

7. IANA Considerations

TBD

Choudhary, et al. Expires March 23, 2020 [Page 56] Internet-Draft YANG Model For QoS Sep 2019

8. Security Considerations

9. Acknowledgement

The authors wish to thank Ruediger Geib, Fred Baker, Greg Misky, Tom Petch, many others for their helpful comments.

10. References

10.1. Normative References

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, .

[RFC2697] Heinanen, J. and R. Guerin, "A Single Rate Three Color Marker", RFC 2697, DOI 10.17487/RFC2697, September 1999, .

[RFC2698] Heinanen, J. and R. Guerin, "A Two Rate Three Color Marker", RFC 2698, DOI 10.17487/RFC2698, September 1999, .

[RFC3289] Baker, F., Chan, K., and A. Smith, "Management Information Base for the Differentiated Services Architecture", RFC 3289, DOI 10.17487/RFC3289, May 2002, .

[RFC6020] Bjorklund, M., Ed., "YANG - A Data Modeling Language for the Network Configuration Protocol (NETCONF)", RFC 6020, DOI 10.17487/RFC6020, October 2010, .

[RFC6991] Schoenwaelder, J., Ed., "Common YANG Data Types", RFC 6991, DOI 10.17487/RFC6991, July 2013, .

[RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language", RFC 7950, DOI 10.17487/RFC7950, August 2016, .

[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, .

Choudhary, et al. Expires March 23, 2020 [Page 57] Internet-Draft YANG Model For QoS Sep 2019

[RFC8342] Bjorklund, M., Schoenwaelder, J., Shafer, P., Watsen, K., and R. Wilton, "Network Management Datastore Architecture (NMDA)", RFC 8342, DOI 10.17487/RFC8342, March 2018, .

[RFC8343] Bjorklund, M., "A YANG Data Model for Interface Management", RFC 8343, DOI 10.17487/RFC8343, March 2018, .

10.2. Informative References

[RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., and W. Weiss, "An Architecture for Differentiated Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, .

[RFC8340] Bjorklund, M. and L. Berger, Ed., "YANG Tree Diagrams", BCP 215, RFC 8340, DOI 10.17487/RFC8340, March 2018, .

Appendix A. Company A, Company B and Company C examples

Company A, Company B and Company C Diffserv modules augments all the filter types of the QoS classifier module as well as the QoS policy module that allow it to define marking, metering, min-rate, max-rate actions. Queuing and metering counters are realized by augmenting of the QoS target module.

A.1. Example of Company A Diffserv Model

The following Company A vendor example augments the qos and diffserv model, demonstrating some of the following functionality:

- use of template based classifier definitions

- use of single policy type modelling queue, scheduler policy, and a filter policy. All of these policies either augment the qos policy or the diffserv modules

- use of inline actions in a policy

- flexibility in marking dscp or metadata at ingress and/or egress.

module example-compa-diffserv { yang-version 1.1; namespace "urn:ietf:params:xml:ns:yang:example-compa-diffserv"; prefix example;

Choudhary, et al. Expires March 23, 2020 [Page 58] Internet-Draft YANG Model For QoS Sep 2019

import ietf-qos-classifier { prefix classifier; reference "RFC XXXX: YANG Model for QoS"; } import ietf-qos-policy { prefix policy; reference "RFC XXXX: YANG Model for QoS"; } import ietf-qos-action { prefix action; reference "RFC XXXX: YANG Model for QoS"; } import ietf-diffserv { prefix diffserv; reference "RFC XXXX: YANG Model for QoS"; }

organization "Company A"; contact "Editor: XYZ "; description "This module contains a collection of YANG definitions of companyA diffserv specification extension."; Copyright (c) 2019 IETF Trust and the persons identified as authors of the code. All rights reserved. Redistribution and use in source and binary forms, with or without modification, is permitted pursuant to, and subject to the license terms contained in, the Simplified BSD License set forth in Section 4.c of the IETF Trust’s Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info).

This version of this YANG module is part of RFC XXXX; see the RFC itself for full legal notices.";

revision 2019-03-13 { description "Initial revision for diffserv actions on network packets"; reference "RFC 6020: YANG - A Data Modeling Language for the Network Configuration Protocol (NETCONF)"; }

identity default-policy-type { base policy:policy-type; description "This defines default policy-type";

Choudhary, et al. Expires March 23, 2020 [Page 59] Internet-Draft YANG Model For QoS Sep 2019

}

identity qos-group { base classifier:filter-type; description "qos-group filter-type"; }

grouping qos-group-cfg { list qos-group-cfg { key "qos-group-min qos-group-max"; description "list of dscp ranges"; leaf qos-group-min { type uint8; description "Minimum value of qos-group range"; } leaf qos-group-max { type uint8; description "maximum value of qos-group range"; } } description "Filter containing list of qos-group ranges"; }

grouping wred-threshold { container wred-min-thresh { uses action:threshold; description "Minimum threshold"; } container wred-max-thresh { uses action:threshold; description "Maximum threshold"; } leaf mark-probability { type uint32 { range "1..1000"; } description "Mark probability"; } description "WRED threshold attributes";

Choudhary, et al. Expires March 23, 2020 [Page 60] Internet-Draft YANG Model For QoS Sep 2019

}

grouping randomdetect { leaf exp-weighting-const { type uint32; description "Exponential weighting constant factor for wred profile"; } uses wred-threshold; description "Random detect attributes"; }

augment "/classifier:classifiers/" + "classifier:classifier-entry/" + "classifier:filter-entry/diffserv:filter-param" { case qos-group { uses qos-group-cfg; description "Filter containing list of qos-group ranges. Qos-group represent packet metadata information in a device. "; } description "augmentation of classifier filters"; } augment "/policy:policies/policy:policy-entry/" + "policy:classifier-entry/" + "policy:classifier-action-entry-cfg/" + "policy:action-cfg-params" { case random-detect { uses randomdetect; } description "Augment the actions to policy entry"; }

augment "/policy:policies" + "/policy:policy-entry" + "/policy:classifier-entry" + "/policy:classifier-action-entry-cfg" + "/policy:action-cfg-params" + "/diffserv:meter-inline" + "/diffserv:meter-type" + "/diffserv:one-rate-two-color-meter-type" + "/diffserv:one-rate-two-color-meter" + "/diffserv:conform-action" + "/diffserv:conform-2color-meter-action-params" +

Choudhary, et al. Expires March 23, 2020 [Page 61] Internet-Draft YANG Model For QoS Sep 2019

"/diffserv:conform-2color-meter-action-val" {

description "augment the one-rate-two-color meter conform with actions"; case meter-action-drop { description "meter drop"; uses action:drop; } case meter-action-mark-dscp { description "meter action dscp marking"; uses action:dscp-marking; } } augment "/policy:policies" + "/policy:policy-entry" + "/policy:classifier-entry" + "/policy:classifier-action-entry-cfg" + "/policy:action-cfg-params" + "/diffserv:meter-inline" + "/diffserv:meter-type" + "/diffserv:one-rate-two-color-meter-type" + "/diffserv:one-rate-two-color-meter" + "/diffserv:exceed-action" + "/diffserv:exceed-2color-meter-action-params" + "/diffserv:exceed-2color-meter-action-val" {

description "augment the one-rate-two-color meter exceed with actions"; case meter-action-drop { description "meter drop"; uses action:drop; } case meter-action-mark-dscp { description "meter action dscp marking"; uses action:dscp-marking; } } augment "/policy:policies" + "/policy:policy-entry" + "/policy:classifier-entry" + "/policy:classifier-action-entry-cfg" + "/policy:action-cfg-params" +

Choudhary, et al. Expires March 23, 2020 [Page 62] Internet-Draft YANG Model For QoS Sep 2019

"/diffserv:meter-inline" + "/diffserv:meter-type" + "/diffserv:one-rate-tri-color-meter-type" + "/diffserv:one-rate-tri-color-meter" + "/diffserv:conform-action" + "/diffserv:conform-3color-meter-action-params" + "/diffserv:conform-3color-meter-action-val" {

description "augment the one-rate-tri-color meter conform with actions"; case meter-action-drop { description "meter drop"; uses action:drop; } case meter-action-mark-dscp { description "meter action dscp marking"; uses action:dscp-marking; } } augment "/policy:policies" + "/policy:policy-entry" + "/policy:classifier-entry" + "/policy:classifier-action-entry-cfg" + "/policy:action-cfg-params" + "/diffserv:meter-inline" + "/diffserv:meter-type" + "/diffserv:one-rate-tri-color-meter-type" + "/diffserv:one-rate-tri-color-meter" + "/diffserv:exceed-action" + "/diffserv:exceed-3color-meter-action-params" + "/diffserv:exceed-3color-meter-action-val" {

description "augment the one-rate-tri-color meter exceed with actions"; case meter-action-drop { description "meter drop"; uses action:drop; } case meter-action-mark-dscp { description "meter action dscp marking"; uses action:dscp-marking; }

Choudhary, et al. Expires March 23, 2020 [Page 63] Internet-Draft YANG Model For QoS Sep 2019

} augment "/policy:policies" + "/policy:policy-entry" + "/policy:classifier-entry" + "/policy:classifier-action-entry-cfg" + "/policy:action-cfg-params" + "/diffserv:meter-inline" + "/diffserv:meter-type" + "/diffserv:one-rate-tri-color-meter-type" + "/diffserv:one-rate-tri-color-meter" + "/diffserv:violate-action" + "/diffserv:violate-3color-meter-action-params" + "/diffserv:violate-3color-meter-action-val" { description "augment the one-rate-tri-color meter conform with actions"; case meter-action-drop { description "meter drop"; uses action:drop; } case meter-action-mark-dscp { description "meter action dscp marking"; uses action:dscp-marking; } }

augment "/policy:policies" + "/policy:policy-entry" + "/policy:classifier-entry" + "/policy:classifier-action-entry-cfg" + "/policy:action-cfg-params" + "/diffserv:meter-inline" + "/diffserv:meter-type" + "/diffserv:two-rate-tri-color-meter-type" + "/diffserv:two-rate-tri-color-meter" + "/diffserv:conform-action" + "/diffserv:conform-3color-meter-action-params" + "/diffserv:conform-3color-meter-action-val" {

description "augment the one-rate-tri-color meter conform with actions"; case meter-action-drop { description "meter drop"; uses action:drop;

Choudhary, et al. Expires March 23, 2020 [Page 64] Internet-Draft YANG Model For QoS Sep 2019

} case meter-action-mark-dscp { description "meter action dscp marking"; uses action:dscp-marking; } } augment "/policy:policies" + "/policy:policy-entry" + "/policy:classifier-entry" + "/policy:classifier-action-entry-cfg" + "/policy:action-cfg-params" + "/diffserv:meter-inline" + "/diffserv:meter-type" + "/diffserv:two-rate-tri-color-meter-type" + "/diffserv:two-rate-tri-color-meter" + "/diffserv:exceed-action" + "/diffserv:exceed-3color-meter-action-params" + "/diffserv:exceed-3color-meter-action-val" {

description "augment the two-rate-tri-color meter exceed with actions"; case meter-action-drop { description "meter drop"; uses action:drop; } case meter-action-mark-dscp { description "meter action dscp marking"; uses action:dscp-marking; } } augment "/policy:policies" + "/policy:policy-entry" + "/policy:classifier-entry" + "/policy:classifier-action-entry-cfg" + "/policy:action-cfg-params" + "/diffserv:meter-inline" + "/diffserv:meter-type" + "/diffserv:two-rate-tri-color-meter-type" + "/diffserv:two-rate-tri-color-meter" + "/diffserv:violate-action" + "/diffserv:violate-3color-meter-action-params" + "/diffserv:violate-3color-meter-action-val" { description "augment the two-rate-tri-color meter violate

Choudhary, et al. Expires March 23, 2020 [Page 65] Internet-Draft YANG Model For QoS Sep 2019

with actions"; case meter-action-drop { description "meter drop"; uses action:drop; } case meter-action-mark-dscp { description "meter action dscp marking"; uses action:dscp-marking; } } augment "/policy:policies" + "/policy:policy-entry" + "/policy:classifier-entry" + "/policy:classifier-action-entry-cfg" + "/policy:action-cfg-params" + "/diffserv:meter-inline" + "/diffserv:meter-type" + "/diffserv:one-rate-two-color-meter-type" + "/diffserv:one-rate-two-color-meter" { description "augment the one-rate-two-color meter with" + "color classifiers"; container conform-color { uses classifier:classifier-entry-generic-attr; description "conform color classifier container"; } container exceed-color { uses classifier:classifier-entry-generic-attr; description "exceed color classifier container"; } } augment "/policy:policies" + "/policy:policy-entry" + "/policy:classifier-entry" + "/policy:classifier-action-entry-cfg" + "/policy:action-cfg-params" + "/diffserv:meter-inline" + "/diffserv:meter-type" + "/diffserv:one-rate-tri-color-meter-type" + "/diffserv:one-rate-tri-color-meter" { description "augment the one-rate-tri-color meter with" + "color classifiers"; container conform-color {

Choudhary, et al. Expires March 23, 2020 [Page 66] Internet-Draft YANG Model For QoS Sep 2019

uses classifier:classifier-entry-generic-attr; description "conform color classifier container"; } container exceed-color { uses classifier:classifier-entry-generic-attr; description "exceed color classifier container"; } container violate-color { uses classifier:classifier-entry-generic-attr; description "violate color classifier container"; } } augment "/policy:policies" + "/policy:policy-entry" + "/policy:classifier-entry" + "/policy:classifier-action-entry-cfg" + "/policy:action-cfg-params" + "/diffserv:meter-inline" + "/diffserv:meter-type" + "/diffserv:two-rate-tri-color-meter-type" + "/diffserv:two-rate-tri-color-meter" { description "augment the two-rate-tri-color meter with" + "color classifiers"; container conform-color { uses classifier:classifier-entry-generic-attr; description "conform color classifier container"; } container exceed-color { uses classifier:classifier-entry-generic-attr; description "exceed color classifier container"; } container violate-color { uses classifier:classifier-entry-generic-attr; description "violate color classifier container"; } } }

Choudhary, et al. Expires March 23, 2020 [Page 67] Internet-Draft YANG Model For QoS Sep 2019

A.2. Example of Company B Diffserv Model

The following vendor example augments the qos and diffserv model, demonstrating some of the following functionality:

- use of inline classifier definitions (defined inline in the policy vs referencing an externally defined classifier)

- use of mulitple policy types, e.g. a queue policy, a scheduler policy, and a filter policy. All of these policies either augment the qos policy or the diffserv modules

- use of a queue module, which uses and extends the queue grouping from the ietf-qos-action module

- use of meter templates (v.s. meter inline)

- use of internal meta data for classification and marking

module example-compb-diffserv-filter-policy { yang-version 1.1; namespace "urn:ietf:params:xml:ns:yang:" + "example-compb-diffserv-filter-policy"; prefix compb-filter-policy;

import ietf-qos-classifier { prefix classifier; reference "RFC XXXX: YANG Model for QoS"; } import ietf-qos-policy { prefix policy; reference "RFC XXXX: YANG Model for QoS"; } import ietf-qos-action { prefix action; reference "RFC XXXX: YANG Model for QoS"; } import ietf-diffserv { prefix diffserv; reference "RFC XXXX: YANG Model for QoS"; }

organization "Company B"; contact "Editor: XYZ ";

description

Choudhary, et al. Expires March 23, 2020 [Page 68] Internet-Draft YANG Model For QoS Sep 2019

"This module contains a collection of YANG definitions for configuring diffserv specification implementations. Copyright (c) 2019 IETF Trust and the persons identified as authors of the code. All rights reserved. Redistribution and use in source and binary forms, with or without modification, is permitted pursuant to, and subject to the license terms contained in, the Simplified BSD License set forth in Section 4.c of the IETF Trust’s Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info).

This version of this YANG module is part of RFC XXXX; see the RFC itself for full legal notices.";

revision 2019-03-13 { description "Latest revision of diffserv policy"; reference "RFC XXXX"; }

/************************************************* * Classification types *************************************************/

identity forwarding-class { base classifier:filter-type; description "Forwarding class filter type"; }

identity internal-loss-priority { base classifier:filter-type; description "Internal loss priority filter type"; }

grouping forwarding-class-cfg { list forwarding-class-cfg { key "forwarding-class"; description "list of forwarding-classes"; leaf forwarding-class { type string; description "Forwarding class name"; } }

Choudhary, et al. Expires March 23, 2020 [Page 69] Internet-Draft YANG Model For QoS Sep 2019

description "Filter containing list of forwarding classes"; }

grouping loss-priority-cfg { list loss-priority-cfg { key "loss-priority"; description "list of loss-priorities"; leaf loss-priority { type enumeration { enum high { description "High Loss Priority"; } enum medium-high { description "Medium-high Loss Priority"; } enum medium-low { description "Medium-low Loss Priority"; } enum low { description "Low Loss Priority"; } } description "Loss-priority"; } } description "Filter containing list of loss priorities"; }

augment "/policy:policies" + "/policy:policy-entry" + "/policy:classifier-entry" + "/policy:filter-entry" + "/diffserv:filter-params" { case forwarding-class { uses forwarding-class-cfg; description "Filter Type Internal-loss-priority"; } case internal-loss-priority { uses loss-priority-cfg; description "Filter Type Internal-loss-priority"; } description

Choudhary, et al. Expires March 23, 2020 [Page 70] Internet-Draft YANG Model For QoS Sep 2019

"Augments Diffserv Classifier with vendor" + " specific types"; }

/************************************************* * Actions *************************************************/

identity mark-fwd-class { base policy:action-type; description "mark forwarding class action type"; }

identity mark-loss-priority { base policy:action-type; description "mark loss-priority action type"; }

grouping mark-fwd-class { container mark-fwd-class-cfg { leaf forwarding-class { type string; description "Forwarding class name"; } description "mark-fwd-class container"; } description "mark-fwd-class grouping"; }

grouping mark-loss-priority { container mark-loss-priority-cfg { leaf loss-priority { type enumeration { enum high { description "High Loss Priority"; } enum medium-high { description "Medium-high Loss Priority"; } enum medium-low { description "Medium-low Loss Priority"; } enum low {

Choudhary, et al. Expires March 23, 2020 [Page 71] Internet-Draft YANG Model For QoS Sep 2019

description "Low Loss Priority"; } } description "Loss-priority"; } description "mark-loss-priority container"; } description "mark-loss-priority grouping"; }

identity exceed-2color-meter-action-drop { base action:exceed-2color-meter-action-type; description "drop action type in a meter"; }

identity meter-action-mark-fwd-class { base action:exceed-2color-meter-action-type; description "mark forwarding class action type"; }

identity meter-action-mark-loss-priority { base action:exceed-2color-meter-action-type; description "mark loss-priority action type"; }

identity violate-3color-meter-action-drop { base action:violate-3color-meter-action-type; description "drop action type in a meter"; }

augment "/policy:policies/policy:policy-entry/" + "policy:classifier-entry/" + "policy:classifier-action-entry-cfg/" + "policy:action-cfg-params" { case mark-fwd-class { uses mark-fwd-class; description "Mark forwarding class in the packet"; } case mark-loss-priority { uses mark-loss-priority;

Choudhary, et al. Expires March 23, 2020 [Page 72] Internet-Draft YANG Model For QoS Sep 2019

description "Mark loss priority in the packet"; } case discard { uses action:discard; description "Discard action"; } description "Augments common diffserv policy actions"; }

augment "/action:meter-template" + "/action:meter-entry" + "/action:meter-type" + "/action:one-rate-tri-color-meter-type" + "/action:one-rate-tri-color-meter" { leaf one-rate-color-aware { type boolean; description "This defines if the meter is color-aware"; } } augment "/action:meter-template" + "/action:meter-entry" + "/action:meter-type" + "/action:two-rate-tri-color-meter-type" + "/action:two-rate-tri-color-meter" { leaf two-rate-color-aware { type boolean; description "This defines if the meter is color-aware"; } }

/* example of augmenting a meter template with a /* vendor specific action */ augment "/action:meter-template" + "/action:meter-entry" + "/action:meter-type" + "/action:one-rate-two-color-meter-type" + "/action:one-rate-two-color-meter" + "/action:exceed-action" + "/action:exceed-2color-meter-action-params" + "/action:exceed-2color-meter-action-val" {

case exceed-2color-meter-action-drop {

Choudhary, et al. Expires March 23, 2020 [Page 73] Internet-Draft YANG Model For QoS Sep 2019

description "meter drop"; uses action:drop; } case meter-action-mark-fwd-class { uses mark-fwd-class; description "Mark forwarding class in the packet"; } case meter-action-mark-loss-priority { uses mark-loss-priority; description "Mark loss priority in the packet"; } }

augment "/action:meter-template" + "/action:meter-entry" + "/action:meter-type" + "/action:two-rate-tri-color-meter-type" + "/action:two-rate-tri-color-meter" + "/action:violate-action" + "/action:violate-3color-meter-action-params" + "/action:violate-3color-meter-action-val" { case exceed-3color-meter-action-drop { description "meter drop"; uses action:drop; }

description "Augment the actions to the two-color meter"; }

augment "/action:meter-template" + "/action:meter-entry" + "/action:meter-type" + "/action:one-rate-tri-color-meter-type" + "/action:one-rate-tri-color-meter" + "/action:violate-action" + "/action:violate-3color-meter-action-params" + "/action:violate-3color-meter-action-val" { case exceed-3color-meter-action-drop { description "meter drop"; uses action:drop; }

Choudhary, et al. Expires March 23, 2020 [Page 74] Internet-Draft YANG Model For QoS Sep 2019

description "Augment the actions to basic meter"; }

} module example-compb-queue-policy { yang-version 1.1; namespace "urn:ietf:params:xml:ns:yang:example-compb-queue-policy"; prefix queue-plcy;

import ietf-qos-classifier { prefix classifier; reference "RFC XXXX: YANG Model for QoS"; } import ietf-qos-policy { prefix policy; reference "RFC XXXX: YANG Model for QoS"; }

organization "Company B"; contact "Editor: XYZ ";

description "This module defines a queue policy. The classification is based on aforwarding class, and the actions are queues. Copyright (c) 2019 IETF Trust and the persons identified as authors of the code. All rights reserved. Redistribution and use in source and binary forms, with or without modification, is permitted pursuant to, and subject to the license terms contained in, the Simplified BSD License set forth in Section 4.c of the IETF Trust’s Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info). This version of this YANG module is part of RFC XXXX; see the RFC itself for full legal notices.";

revision 2019-03-13 { description "Latest revision of diffserv policy"; reference "RFC XXXX"; }

identity forwarding-class { base classifier:filter-type; description "Forwarding class filter type";

Choudhary, et al. Expires March 23, 2020 [Page 75] Internet-Draft YANG Model For QoS Sep 2019

}

grouping forwarding-class-cfg { leaf forwarding-class-cfg { type string; description "forwarding-class name"; } description "Forwarding class filter"; }

augment "/policy:policies" + "/policy:policy-entry" + "/policy:classifier-entry" + "/policy:filter-entry" { /* Does NOT support "logical-not" of forwarding class. Use "must"? */ choice filter-params { description "Choice of filters"; case forwarding-class-cfg { uses forwarding-class-cfg; description "Filter Type Internal-loss-priority"; } } description "Augments Diffserv Classifier with fwd class filter"; }

identity compb-queue { base policy:action-type; description "compb-queue action type"; }

grouping compb-queue-name { container queue-name { leaf name { type string; description "Queue class name"; } description "compb queue container"; } description

Choudhary, et al. Expires March 23, 2020 [Page 76] Internet-Draft YANG Model For QoS Sep 2019

"compb-queue grouping"; }

augment "/policy:policies" + "/policy:policy-entry" + "/policy:classifier-entry" + "/policy:classifier-action-entry-cfg" { choice action-cfg-params { description "Choice of action types"; case compb-queue { uses compb-queue-name; } } description "Augment the queue actions to queue policy entry"; } }

module example-compb-queue { yang-version 1.1; namespace "urn:ietf:params:xml:ns:yang:ietf-compb-queue"; prefix compb-queue;

import ietf-qos-action { prefix action; reference "RFC XXXX: YANG Model for QoS"; }

organization "Company B"; contact "Editor: XYZ ";

description "This module describes a compb queue module. This is a template for a queue within a queue policy, referenced by name.

This version of this YANG module is part of RFC XXXX; see the RFC itself for full legal notices.";

revision 2019-03-13 { description "Latest revision of diffserv based classifier"; reference "RFC XXXX"; }

Choudhary, et al. Expires March 23, 2020 [Page 77] Internet-Draft YANG Model For QoS Sep 2019

container compb-queue { description "Queue used in compb architecture"; leaf name { type string; description "A unique name identifying this queue"; } uses action:queue; container excess-rate { choice excess-rate-type { case percent { leaf excess-rate-percent { type uint32 { range "1..100"; } description "excess-rate-percent"; } } case proportion { leaf excess-rate-poroportion { type uint32 { range "1..1000"; } description "excess-rate-poroportion"; } } description "Choice of excess-rate type"; } description "Excess rate value"; } leaf excess-priority { type enumeration { enum high { description "High Loss Priority"; } enum medium-high { description "Medium-high Loss Priority"; } enum medium-low { description "Medium-low Loss Priority"; } enum low { description "Low Loss Priority";

Choudhary, et al. Expires March 23, 2020 [Page 78] Internet-Draft YANG Model For QoS Sep 2019

} enum none { description "No excess priority"; } } description "Priority of excess (above guaranted rate) traffic"; } container buffer-size { choice buffer-size-type { case percent { leaf buffer-size-percent { type uint32 { range "1..100"; } description "buffer-size-percent"; } } case temporal { leaf buffer-size-temporal { type uint64; units "microsecond"; description "buffer-size-temporal"; } } case remainder { leaf buffer-size-remainder { type empty; description "use remaining of buffer"; } } description "Choice of buffer size type"; } description "Buffer size value"; } }

augment "/compb-queue" + "/queue-cfg" + "/algorithmic-drop-cfg" + "/drop-algorithm" { case random-detect {

Choudhary, et al. Expires March 23, 2020 [Page 79] Internet-Draft YANG Model For QoS Sep 2019

list drop-profile-list { key "priority"; description "map of priorities to drop-algorithms"; leaf priority { type enumeration { enum any { description "Any priority mapped here"; } enum high { description "High Priority Packet"; } enum medium-high { description "Medium-high Priority Packet"; } enum medium-low { description "Medium-low Priority Packet"; } enum low { description "Low Priority Packet"; } } description "Priority of guaranteed traffic"; } leaf drop-profile { type string; description "drop profile to use for this priority"; } } } description "compb random detect drop algorithm config"; } }

module example-compb-scheduler-policy { yang-version 1.1; namespace "urn:ietf:params:xml:ns:yang:" + "example-compb-scheduler-policy"; prefix scheduler-plcy;

import ietf-qos-action { prefix action; reference "RFC XXXX: YANG Model for QoS"; }

Choudhary, et al. Expires March 23, 2020 [Page 80] Internet-Draft YANG Model For QoS Sep 2019

import ietf-qos-policy { prefix policy; reference "RFC XXXX: YANG Model for QoS"; }

organization "Company B"; contact "Editor: XYZ ";

description "This module defines a scheduler policy. The classification is based on classifier-any, and the action is a scheduler.";

revision 2019-03-13 { description "Latest revision of diffserv policy"; reference "RFC XXXX"; }

identity queue-policy { base policy:action-type; description "forwarding-class-queue action type"; }

grouping queue-policy-name { container compb-queue-policy-name { leaf name { type string; description "Queue policy name"; } description "compb-queue-policy container"; } description "compb-queue policy grouping"; }

augment "/policy:policies" + "/policy:policy-entry" + "/policy:classifier-entry" + "/policy:classifier-action-entry-cfg" { choice action-cfg-params { case schedular { uses action:schedular; }

Choudhary, et al. Expires March 23, 2020 [Page 81] Internet-Draft YANG Model For QoS Sep 2019

case queue-policy { uses queue-policy-name; } description "Augment the scheduler policy with a queue policy"; } } }

A.3. Example of Company C Diffserv Model

Company C vendor augmentation is based on Ericsson’s implementation differentiated QoS. This implementation first sorts traffic based on a classifier, which can sort traffic into one or more traffic forwarding classes. Then, a policer or meter policy references the classifier and its traffic forwarding classes to specify different service levels for each traffic forwarding class.

Because each classifier sorts traffic into one or more traffic forwarding classes, this type of classifier does not align with ietf- qos-classifier.yang, which defines one traffic forwarding class per classifier. Additionally, Company C’s policing and metering policies relies on the classifier’s pre-defined traffic forwarding classes to provide differentiated services, rather than redefining the patterns within a policing or metering policy, as is defined in ietf- diffserv.yang.

Due to these differences, even though Company C uses all the building blocks of classifier and policy, Company C’s augmentation does not use ietf-diffserv.yang to provide differentiated service levels. Instead, Company C’s augmentation uses the basic building blocks, ietf-qos-policy.yang to provide differentiated services.

module example-compc-qos-policy { yang-version 1.1; namespace "urn:example-compc-qos-policy"; prefix "compcqos";

import ietf-qos-policy { prefix "pol"; reference "RFC XXXX: YANG Model for QoS"; }

import ietf-qos-action { prefix "action"; reference "RFC XXXX: YANG Model for QoS"; }

Choudhary, et al. Expires March 23, 2020 [Page 82] Internet-Draft YANG Model For QoS Sep 2019

organization ""; contact ""; description "";

revision 2019-03-13 { description ""; reference ""; }

/* identities */

identity compc-qos-policy { base pol:policy-type; }

identity mdrr-queuing-policy { base compc-qos-policy; }

identity pwfq-queuing-policy { base compc-qos-policy; }

identity policing-policy { base compc-qos-policy; }

identity metering-policy { base compc-qos-policy; }

identity forwarding-policy { base compc-qos-policy; }

identity overhead-profile-policy { base compc-qos-policy; }

identity resource-profile-policy { base compc-qos-policy; }

identity protocol-rate-limit-policy { base compc-qos-policy; }

identity compc-qos-action {

Choudhary, et al. Expires March 23, 2020 [Page 83] Internet-Draft YANG Model For QoS Sep 2019

base pol:action-type; }

/* groupings */

grouping redirect-action-grp { container redirect { /* Redirect options */ } }

/* deviations */

deviation "/pol:policies/pol:policy-entry" { deviate add { must "pol:type = compc-qos-policy" { description "Only policy types drived from compc-qos-policy " + "are supported"; } } }

deviation "/pol:policies/pol:policy-entry/pol:classifier-entry" { deviate add { must "../per-class-action = ’true’" { description "Only policies with per-class actions have classifiers"; } must "((../sub-type != ’mdrr-queuing-policy’) and " + " (../sub-type != ’pwfq-queuing-policy’)) or " + "(((../sub-type = ’mdrr-queuing-policy’) or " + " (../sub-type = ’pwfq-queueing-policy’)) and " + " ((classifier-entry-name = ’0’) or " + " (classifier-entry-name = ’1’) or " + " (classifier-entry-name = ’2’) or " + " (classifier-entry-name = ’3’) or " + " (classifier-entry-name = ’4’) or " + " (classifier-entry-name = ’5’) or " + " (classifier-entry-name = ’6’) or " + " (classifier-entry-name = ’7’) or " + " (classifier-entry-name = ’8’)))" { description "MDRR queuing policy’s or PWFQ queuing policy’s " + "classifier-entry-name is limited to the listed values"; } } }

Choudhary, et al. Expires March 23, 2020 [Page 84] Internet-Draft YANG Model For QoS Sep 2019

deviation "/pol:policies/pol:policy-entry/pol:classifier-entry" + "/pol:classifier-action-entry-cfg" { deviate add { max-elements 1; must "action-type = ’compc-qos-action’" { description "Only compc-qos-action is allowed"; } } }

/* augments */

augment "/pol:policies/pol:policy-entry" { when "pol:type = ’compc-qos-policy’)" { description "Additional nodes only for diffserv-policy"; } leaf sub-type { type identityref { base compc-qos-policy; } mandatory true; /* The value of this leaf must not change once configured */ } leaf per-class-action { mandatory true; type boolean; must "(((. = ’true’) and " + " ((../sub-type = ’policing-policy’) or " + " (../sub-type = ’metering-policy’) or " + " (../sub-type = ’mdrr-queuing-policy’) or " + " (../sub-type = ’pwfq-queuing-policy’) or " + " (../sub-type = ’forwarding-policy’))) or " + " ((. = ’false’) and " + " ((../sub-type = ’overhead-profile-policy’) or " + " (../sub-type = ’resource-profile-policy’) or " + " (../sub-type = ’protocol-rate-limit-policy’)))" { description "Only certain policies have per-class action"; } } container traffic-classifier { presence true; when "../sub-type = ’policing-policy’ or " + "../sub-type = ’metering-policy’ or " + "../sub-type = ’forwarding-policy’" { description

Choudhary, et al. Expires March 23, 2020 [Page 85] Internet-Draft YANG Model For QoS Sep 2019

"A classifier for policing-policy or metering-policy"; } leaf name { type string; mandatory true; description "Traffic classifier name"; } leaf type { type enumeration { enum ’internal-dscp-only-classifier’ { value 0; description "Classify traffic based on (internal) dscp only"; } enum ’ipv4-header-based-classifier’ { value 1; description "Classify traffic based on IPv4 packet header fields"; } enum ’ipv6-header-based-classifier’ { value 2; description "Classify traffic based on IPv6 packet header fields"; } } mandatory true; description "Traffic classifier type"; } } container traffic-queue { when "(../sub-type = ’mdrr-queuing-policy’) or " + "(../sub-type = ’pwfq-queuing-policy’)" { description "Queuing policy properties"; } leaf queue-map { type string; description "Traffic queue map for queuing policy"; } } container overhead-profile { when "../sub-type = ’overhead-profile-policy’" { description "Overhead profile policy properties"; }

Choudhary, et al. Expires March 23, 2020 [Page 86] Internet-Draft YANG Model For QoS Sep 2019

} container resource-profile { when "../sub-type = ’resource-profile-policy’" { description "Resource profile policy properties"; } } container protocol-rate-limit { when "../sub-type = ’protocol-rate-limit-policy’" { description "Protocol rate limit policy properties"; } } }

augment "/pol:policies/pol:policy-entry/pol:classifier-entry" + "/pol:classifier-action-entry-cfg/pol:action-cfg-params" { when "../../../pol:type = ’compc-qos-policy’)" { description "Configurations for a classifier-policy-type policy"; } case metering-or-policing-policy { when "../../../sub-type = ’policing-policy’ or " + "../../../sub-type = ’metering-policy’" { } container dscp-marking { uses action:dscp-marking; } container precedence-marking { uses action:dscp-marking; } container priority-marking { uses action:priority; } container rate-limiting { uses action:one-rate-two-color-meter; } } case mdrr-queuing-policy { when "../../../sub-type = ’mdrr-queuing-policy’" { description "MDRR queue handling properties for the traffic " + "classified into current queue"; } leaf mdrr-queue-weight { type uint8 { range "20..100"; }

Choudhary, et al. Expires March 23, 2020 [Page 87] Internet-Draft YANG Model For QoS Sep 2019

units percentage; } } case pwfq-queuing-policy { when "../../../sub-type = ’pwfq-queuing-policy’" { description "PWFQ queue handling properties for traffic " + "classified into current queue"; } leaf pwfq-queue-weight { type uint8 { range "20..100"; } units percentage; } leaf pwfq-queue-priority { type uint8; } leaf pwfq-queue-rate { type uint8; } } case forwarding-policy { when "../../../sub-type = ’forwarding-policy’" { description "Forward policy handling properties for traffic " + "in this classifier"; } uses redirect-action-grp; } description "Add the classify action configuration"; }

}

Authors’ Addresses

Aseem Choudhary Cisco Systems 170 W. Tasman Drive San Jose, CA 95134 US

Email: [email protected]

Choudhary, et al. Expires March 23, 2020 [Page 88] Internet-Draft YANG Model For QoS Sep 2019

Mahesh Jethanandani Cisco Systems 170 W. Tasman Drive San Jose, CA 95134 US

Email: [email protected]

Norm Strahle Juniper Networks 1194 North Mathilda Avenue Sunnyvale, CA 94089 US

Email: [email protected]

Ebben Aries Juniper Networks 1194 North Mathilda Avenue Sunnyvale, CA 94089 US

Email: [email protected]

Ing-Wher Chen Jabil

Email: [email protected]

Choudhary, et al. Expires March 23, 2020 [Page 89] Network Working Group A. Choudhary Internet-Draft Cisco Systems Intended status: Standards Track I. Chen Expires: January 13, 2022 The MITRE Corporation July 12, 2021

YANG Model for QoS Operational Parameters draft-asechoud-rtgwg-qos-oper-model-09

Abstract

This document describes a YANG model for Quality of Service (QoS) operational parameters.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on January 13, 2022.

Copyright Notice

Choudhary & Chen Expires January 13, 2022 [Page 1] Internet-Draft YANG Model For QoS Operational Parameters July 2021

Table of Contents

1. Introduction ...... 2 1.1. Tree Diagrams ...... 2 2. Terminology ...... 3 3. QoS Operational Model Design ...... 3 4. Modules Tree Structure ...... 4 5. Modules ...... 6 5.1. ietf-qos-oper ...... 6 6. Security Considerations ...... 13 7. Acknowledgement ...... 13 8. References ...... 13 8.1. Normative References ...... 13 8.2. Informative References ...... 14 Authors’ Addresses ...... 14

1. Introduction

This document defines a base YANG [RFC6020] [RFC7950] data module for Quality of Service (QoS) operational parameters. Remote Procedure Calls (RPC) or notification definition is currently not part of this document and will be added later if necessary. QoS configuration modules are defined by [I-D.ietf-rtgwg-qos-model].

This document doesn’t include operational parameters for randomdetect (RED), which is left to individual vendor to augment it.

Editorial Note: (To be removed by RFC Editor)

This draft contains several placeholder values that need to be replaced with finalized values at the time of publication. Please apply the following replacements: o "XXXX" --> the assigned RFC value for this draft both in this draft and in the YANG models under the revision statement. o The "revision" date in model, in the format XXXX-XX-XX, needs to be updated with the date the draft gets approved.

The YANG modules in this document conform to the Network Management Datastore Architecture (NMDA) [RFC8342].

1.1. Tree Diagrams

Tree diagrams used in this document follow the notation defined in [RFC8340]

Choudhary & Chen Expires January 13, 2022 [Page 2] Internet-Draft YANG Model For QoS Operational Parameters July 2021

2. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

3. QoS Operational Model Design

QoS operational model include QoS policy applied to an interface in each direction of traffic. For each QoS policy applied to an interface the model further includes counters for associated Classifiers, Meters and Queues in a particular direction. To modularize and for reusability, grouping have been defined for various counters of classifier, Meters and Queues. The target is assumed to be interface but the groupings can be used for any other target type where QoS policy is applied.

[I-D.ietf-rtgwg-qos-model] defines various building blocks for applying a QoS Policy on a target. It includes QoS Policy configuration, which is a container of various classifiers and corresponding actions which are configured for traffic conditioning. This drafts defines the various counters for these building blocks. ietf-qos-oper module defined in this draft augments ietf-interfaces [RFC8343] module.

Classifier statistics contains counters for packets and bytes matched to the traffic in a direction and also average rate at which traffic is hitting a classifier. Classification criterion may be based on IP, MPLS or Ethernet. Counters defined in this draft are agnostic to underlying data plane technology.

Statistics of meter is modeled based on commonly used algorithms in industry, Single Rate Tri Color Marking (srTCM) [RFC2697] meter, Two Rate Tri Color Marking (trTCM) [RFC2698] meter. Metering statistics includes counters corresponding to various rates configured. A metering container is referred by a metering identifier. This identifier could be a classifier name if the metering configuration is inline with classifier or it could be metering template name if the metering is configured as separate entity and associated with the classifier.

Queuing statistics includes counters corresponding to various queues associated with the policy. A queuing container is referred by queuing identifier. This identifier could be a classifier name if the queuing configuration is inline with classifier and hence there is one-to-one mapping between a classifier and a queue or it could be

Choudhary & Chen Expires January 13, 2022 [Page 3] Internet-Draft YANG Model For QoS Operational Parameters July 2021

a separate queue identifier if one or more than one classifiers are associated with a queue.

4. Modules Tree Structure

This document defines counters for classifiers, meters and queues.

Classifier statistics consists of list of classifier entries identified by a classifier entry name. Classifier counters include matched packets, bytes and average rate of traffic matching a particular classifier.

Metering statistics consists of meters identified by an identifier. Metering counters include conform, exceed, violate and drop packets and bytes.

Queuing counters include instantaneous, peak, average queue length, as well as output conform, exceed, tail drop packets and bytes.

Named statistics is defined as statistics which are tagged by a name. This could be aggregated or non-aggregated. Aggregated named statistics is defined as counters which are aggregated across classifier entries in a policy applied to an interface in a particular direction. Non-aggregated named statistics are counters of classifier, metering or queuing which have the same tag name but maintained separately.

module: ietf-qos-oper augment /if:interfaces/if:interface: +--ro qos-interface-statistics +--ro stats-per-direction* [] +--ro direction? identityref +--ro policy-name? string +--ro classifier-statistics* [] | +--ro classifier-entry-name? string | +--ro classified-pkts? uint64 | +--ro classified-bytes? uint64 | +--ro classified-rate? uint64 +--ro named-statistics* [] | +--ro stats-name? string | +--ro aggregated | | +--ro pkts? uint64 | | +--ro bytes? uint64 | | +--ro rate? uint64 | +--ro non-aggregated | +--ro classifier-statistics* [] | | +--ro classifier-entry-name? string

Choudhary & Chen Expires January 13, 2022 [Page 4] Internet-Draft YANG Model For QoS Operational Parameters July 2021

Choudhary & Chen Expires January 13, 2022 [Page 5] Internet-Draft YANG Model For QoS Operational Parameters July 2021

+--ro queue-peak-size-bytes? uint64 +--ro tailed-drop-pkts? uint64 +--ro tailed-drop-bytes? uint64

5. Modules

5.1. ietf-qos-oper

file "ietf-qos-oper.yang"

module ietf-qos-oper { yang-version 1.1; namespace "urn:ietf:params:xml:ns:yang:ietf-qos-oper"; prefix oper; import ietf-interfaces { prefix if; reference "RFC8343: A YANG Data Model for Interface Management"; } organization "IETF RTG (Routing Area) Working Group"; contact "WG Web: WG List: Editor: Aseem Choudhary "; description "This module contains a collection of YANG definitions for qos operational specification. Copyright (c) 2021 IETF Trust and the persons identified as authors of the code. All rights reserved. Redistribution and use in source and binary forms, with or without modification, is permitted pursuant to, and subject to the license terms contained in, the Simplified BSD License set forth in Section 4.c of the IETF Trust’s Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info). This version of this YANG module is part of RFC XXXX; see the RFC itself for full legal notices."; revision 2021-07-12 { description "Initial revision for qos operational statistics"; reference "RFC XXXX: YANG Model for QOS Operational Parameters"; } identity direction { description "This is identity of traffic direction";

Choudhary & Chen Expires January 13, 2022 [Page 6] Internet-Draft YANG Model For QoS Operational Parameters July 2021

} identity inbound { base direction; description "Direction of traffic coming into the network entry"; } identity outbound { base direction; description "Direction of traffic going out of the network entry"; } grouping classifier-entry-stats { description " This group defines the classifier filter counters of each classifier entry "; leaf classified-pkts { type uint64; description " Number of total packets which filtered to a classifier-entry"; } leaf classified-bytes { type uint64; description " Number of total bytes which filtered to a classifier-entry"; } leaf classified-rate { type uint64; units "bits-per-second"; description " Rate of average data flow through a classifier-entry"; } } grouping named-stats { description "QoS matching statistics associated with a stats-name"; leaf pkts { type uint64; description " Number of total matched packets associated to a statistics name"; } leaf bytes { type uint64;

Choudhary & Chen Expires January 13, 2022 [Page 7] Internet-Draft YANG Model For QoS Operational Parameters July 2021

description " Number of total matched bytes associated to a statistics name"; } leaf rate { type uint64; units "bits-per-second"; description " Rate of average matched data which is associated to a statistics name"; } } grouping queue-stats { description "Queuing Counters"; leaf output-conform-pkts { type uint64; description "Number of packets transmitted from queue "; } leaf output-conform-bytes { type uint64; description "Number of bytes transmitted from queue "; } leaf output-exceed-pkts { type uint64; description "Number of packets transmitted from queue "; } leaf output-exceed-bytes { type uint64; description "Number of bytes transmitted from queue "; } leaf queue-current-size-bytes { type uint64; description "Number of bytes currently buffered "; } leaf queue-average-size-bytes { type uint64; description "Average queue size in number of bytes"; } leaf queue-peak-size-bytes { type uint64; description

Choudhary & Chen Expires January 13, 2022 [Page 8] Internet-Draft YANG Model For QoS Operational Parameters July 2021

"Peak buffer queue size in bytes "; } leaf tailed-drop-pkts { type uint64; description "Total number of packets tail-dropped "; } leaf tailed-drop-bytes { type uint64; description "Total number of bytes tail-dropped "; } } grouping meter-stats { description "Metering counters"; leaf conform-pkts { type uint64; description "Number of conform packets"; } leaf conform-bytes { type uint64; description "Bytes of conform packets"; } leaf conform-rate { type uint64; units "bits-per-second"; description "Traffic Rate measured as conformimg"; } leaf exceed-pkts { type uint64; description "Number of packets counted as exceeding"; } leaf exceed-bytes { type uint64; description "Bytes of packets counted as exceeding"; } leaf exceed-rate { type uint64; units "bits-per-second"; description "Traffic Rate measured as exceeding"; }

Choudhary & Chen Expires January 13, 2022 [Page 9] Internet-Draft YANG Model For QoS Operational Parameters July 2021

leaf violate-pkts { type uint64; description "Number of packets counted as violating"; } leaf violate-bytes { type uint64; description "Bytes of packets counted as violating"; } leaf violate-rate { type uint64; units "bits-per-second"; description "Traffic Rate measured as violating"; } leaf meter-drop-pkts { type uint64; description "Number of packets dropped by meter"; } leaf meter-drop-bytes { type uint64; description "Bytes of packets dropped by meter"; } } grouping classifier-entry-statistics { description "Statistics for a classifier entry"; leaf classifier-entry-name { type string; description "Classifier Entry Name"; } uses classifier-entry-stats; }

grouping queuing-stats { description "Statistics for a queue"; leaf queue-id { type string; description "Queue Identifier"; } uses queue-stats; }

Choudhary & Chen Expires January 13, 2022 [Page 10] Internet-Draft YANG Model For QoS Operational Parameters July 2021

grouping metering-stats { description "Statistics for a meter"; leaf meter-id { type string; description "Meter Identifier"; } uses meter-stats; }

augment "/if:interfaces/if:interface" { description "Augments Qos Target Entry to Interface module";

container qos-interface-statistics { config false; description "Qos Interface statistics";

list stats-per-direction { description "Qos Interface statistics for ingress or egress direction";

leaf direction { type identityref { base direction; } description "Direction fo the traffic flow either inbound or outbound"; } leaf policy-name { type string; description "Policy entry name for single level policy as well as for Hierarchical policies. For Hierarchical policies, this represent relative path as well as the last level policy name."; }

list classifier-statistics { description "Classifier Statistics for each Classifier Entry in a Policy applied in a particular direction"; reference "RFC3289: Section 6"; uses classifier-entry-statistics;

Choudhary & Chen Expires January 13, 2022 [Page 11] Internet-Draft YANG Model For QoS Operational Parameters July 2021

} list named-statistics { config false; description "Statistics for a statistics-name"; leaf stats-name { type string; description "stats-name represents classifier, metering and/or queuing name. Classifier statistics may be aggregated or non-aggregated type. Metering and queuing statistics is of non-aggregated type only representing counters for meter and queue respectively. Stats-name may include hierarchical path as well if the counters represent hierarchical policy statistics"; } container aggregated { description "Matched aggregated statistics for a statistics-name"; uses named-stats; } container non-aggregated { description "Statistics for non-aggregated statistics-name"; list classifier-statistics { description "Classifier Statistics for each Classifier Entry in a Policy applied in a particular direction"; uses classifier-entry-statistics; } list metering-statistics { config false; description "Statistics for each Meter associated with the Policy"; reference "RFC2697: A Single Rate Three Color Marker RFC2698: A Two Rate Three Color Marker"; uses metering-stats; } list queueing-statistics { config false; description "Statistics for each Queue associated with the Policy"; uses queuing-stats; } }

Choudhary & Chen Expires January 13, 2022 [Page 12] Internet-Draft YANG Model For QoS Operational Parameters July 2021

} list metering-statistics { config false; description "Statistics for each Meter associated with the Policy"; reference "RFC2697: A Single Rate Three Color Marker RFC2698: A Two Rate Three Color Marker"; uses metering-stats; } list queueing-statistics { config false; description "Statistics for each Queue associated with the Policy"; uses queuing-stats; } } } } }

6. Security Considerations

7. Acknowledgement

MITRE has approved this document for Public Release, Distribution Unlimited, with Public Release Case Number 20-0518. The author’s affiliation with The MITRE Corporation is provided for identification purposes only, and is not intended to convey or imply MITRE’s concurrence with, or support for, the positions, opinions or viewpoints expressed by the author.

8. References

8.1. Normative References

[I-D.ietf-rtgwg-qos-model] Choudhary, A., Jethanandani, M., Strahle, N., Aries, E., and I. Chen, "YANG Model for QoS", draft-ietf-rtgwg-qos- model-03 (work in progress), February 2021.

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, .

Choudhary & Chen Expires January 13, 2022 [Page 13] Internet-Draft YANG Model For QoS Operational Parameters July 2021

[RFC2697] Heinanen, J. and R. Guerin, "A Single Rate Three Color Marker", RFC 2697, DOI 10.17487/RFC2697, September 1999, .

[RFC2698] Heinanen, J. and R. Guerin, "A Two Rate Three Color Marker", RFC 2698, DOI 10.17487/RFC2698, September 1999, .

[RFC6020] Bjorklund, M., Ed., "YANG - A Data Modeling Language for the Network Configuration Protocol (NETCONF)", RFC 6020, DOI 10.17487/RFC6020, October 2010, .

[RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language", RFC 7950, DOI 10.17487/RFC7950, August 2016, .

[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, .

[RFC8342] Bjorklund, M., Schoenwaelder, J., Shafer, P., Watsen, K., and R. Wilton, "Network Management Datastore Architecture (NMDA)", RFC 8342, DOI 10.17487/RFC8342, March 2018, .

[RFC8343] Bjorklund, M., "A YANG Data Model for Interface Management", RFC 8343, DOI 10.17487/RFC8343, March 2018, .

8.2. Informative References

[RFC8340] Bjorklund, M. and L. Berger, Ed., "YANG Tree Diagrams", BCP 215, RFC 8340, DOI 10.17487/RFC8340, March 2018, .

Authors’ Addresses

Aseem Choudhary Cisco Systems 170 W. Tasman Drive San Jose, CA 95134 US

Email: [email protected]

Choudhary & Chen Expires January 13, 2022 [Page 14] Internet-Draft YANG Model For QoS Operational Parameters July 2021

Ing-Wher Chen The MITRE Corporation

Email: [email protected]

Choudhary & Chen Expires January 13, 2022 [Page 15] TSVWG A. Ferrieux, Ed. Internet-Draft I. Hamchaoui, Ed. Intended status: Informational Orange Labs Expires: January 29, 2021 I. Lubashev, Ed. Akamai Technologies D. Tikhonov, Ed. LiteSpeed Technologies July 28, 2020

Packet Loss Signaling for Encrypted Protocols draft-ferrieuxhamchaoui-tsvwg-lossbits-03

Abstract

This document describes a protocol-independent method that employs two bits to allow endpoints to signal packet loss in a way that can be used by network devices to measure and locate the source of the loss. The signaling method applies to all protocols with a protocol- specific way to identify packet loss. The method is especially valuable when applied to protocols that encrypt transport header and do not allow an alternative method for loss detection.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

This Internet-Draft will expire on January 29, 2021.

Copyright Notice

This document is subject to BCP 78 and the IETF Trust’s Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of

Ferrieux, et al. Expires January 29, 2021 [Page 1] Internet-Draft loss-bits July 2020

publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

1. Introduction ...... 3 1.1. Motivation for Passive On-Path Loss Observation . . . . . 3 1.2. On-Path Loss Observation ...... 3 1.3. On-Path Loss Signaling ...... 4 1.4. Recommended Use of the Signals ...... 4 2. Notational Conventions ...... 4 3. Loss Bits ...... 4 3.1. Setting the sQuare Bit on Outgoing Packets ...... 5 3.1.1. Q Run Length Selection ...... 5 3.2. Setting the Loss Event Bit on Outgoing Packets . . . . . 5 4. Using the Loss Bits for Passive Loss Measurement ...... 6 4.1. End-To-End Loss ...... 6 4.2. Upstream Loss ...... 6 4.3. Correlating End-to-End and Upstream Loss ...... 7 4.4. Downstream Loss ...... 7 4.5. Observer Loss ...... 7 5. ECN-Echo Event Bit ...... 8 5.1. Setting the ECN-Echo Event Bit on Outgoing Packets . . . 8 5.2. Using E Bit for Passive ECN-Reported Congestion Measurement ...... 9 6. Protocol Ossification Considerations ...... 9 7. Security Considerations ...... 9 7.1. Optimistic ACK Attack ...... 10 8. Privacy Considerations ...... 10 9. IANA Considerations ...... 10 10. Change Log ...... 10 10.1. Since version 02 ...... 10 10.2. Since version 01 ...... 11 10.3. Since version 00 ...... 11 11. Acknowledgments ...... 11 12. References ...... 11 12.1. Normative References ...... 11 12.2. Informative References ...... 12 Authors’ Addresses ...... 13

Ferrieux, et al. Expires January 29, 2021 [Page 2] Internet-Draft loss-bits July 2020

1. Introduction

1.1. Motivation for Passive On-Path Loss Observation

Packet loss is hard and pervasive problem of day-to-day network operation. Proactively detecting, measuring, and locating it is crucial to maintaining high QoS and timely resolution of crippling end-to-end throughput issues. To this effect, in a TCP-dominated world, network operators have been heavily relying on information present in the clear in TCP headers: sequence and acknowledgment numbers and SACKs when enabled (see [RFC8517]). These allow for quantitative estimation of packet loss by passive on-path observation. Additionally, the lossy segment (upstream or downstream from the observation point) can be quickly identified by moving the passive observer around.

With encrypted protocols, the equivalent transport headers are encrypted and passive packet loss observation is not possible, as described in [TRANSPORT-ENCRYPT].

Measuring TCP loss between similar endpoints cannot be relied upon to evaluate encrypted protocol loss. Different protools could be routed by the network differently and the fraction of Internet traffic delivered using protocols other than TCP is increasing every year. It is imperative to measure packet loss experienced by encrypted protocol users directly.

1.2. On-Path Loss Observation

There are three sources of loss that network operators need to observe to guarantee high QoS:

- _upstream loss_ - loss between the sender and the observation point (Section 4.2)

- _downstream loss_ - loss between the observation point and the destination (Section 4.4)

- _observer loss_ - loss by the observer itself that does not cause downstream loss (Section 4.5)

The upstream and downstream loss together constitute _end-to-end loss_ (Section 4.1).

Ferrieux, et al. Expires January 29, 2021 [Page 3] Internet-Draft loss-bits July 2020

1.3. On-Path Loss Signaling

Following the recommendation in [RFC8558] of making path signals explicit, this document proposes adding two explicit loss bits to the clear portion of the protocol headers to restore network operators’ ability to maintain high QoS. These bits can be added to an unencrypted portion of a header belonging to any protocol layer, e.g. IP (see [IP]) and IPv6 (see [IPv6]) headers or extensions, such as [IPv6AltMark], UDP surplus space (see [UDP-OPTIONS] and [UDP-SURPLUS]), reserved bits in a QUIC v1 header (see [QUIC-TRANSPORT]).

1.4. Recommended Use of the Signals

The loss signal is not designed for use in automated control of the network in environments where loss bits are set by untrusted hosts, Instead, the signal is to be used for troubleshooting individual flows as well as for monitoring the network by aggregating information from multiple flows and raising operator alarms if aggregate statistics indicate a potential problem.

2. Notational Conventions

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119].

3. Loss Bits

The draft introduces two bits that are to be present in packets capable of loss reporting. These are packets that include protocol headers with the loss bits. Only loss of packets capable of loss reporting is reported using loss bits.

Whenever this specification refers to packets, it is referring only to packets capable of loss reporting.

- Q: The "sQuare signal" bit is toggled every N outgoing packets as explained below in Section 3.1.

- L: The "Loss event" bit is set to 0 or 1 according to the Unreported Loss counter, as explained below in Section 3.2.

Each endpoint maintains appropriate counters independently and separately for each separately identifiable flow (each subflow for multipath connections).

Ferrieux, et al. Expires January 29, 2021 [Page 4] Internet-Draft loss-bits July 2020

3.1. Setting the sQuare Bit on Outgoing Packets

The sQuare Value is initialized to the Initial Q Value (0) and is reflected in the Q bit of every outgoing packet. The sQuare value is inverted after sending every N packets (a Q Run). Hence, Q Period is 2*N. The Q bit represents "packet color" as defined by [RFC8321]. The sQuare Bit can also be called an Alernate Marking bit.

Observation points can estimate the upstream losses by counting the number of packets during a half period of the square signal, as described in Section 4.

3.1.1. Q Run Length Selection

The sender is expected to choose N (Q run length) based on the expected amount of loss and reordering on the path. The choice of N strikes a compromise - the observation could become too unreliable in case of packet reordering and/or severe loss if N is too small, while short flows may not yield a useful upstream loss measurement if N is too large (see Section 4.2).

The value of N MUST be at least 64 and be a power of 2. This requirement allows an Observer to infer the Q run length by observing one period of the square signal. It also allows the Observer to identify flows that set the loss bits to arbitrary values (see Section 6).

If the sender does not have sufficient information to make an informed decision about Q run length, the sender SHOULD use N=64, since this value has been extensively tried in large-scale field tests and yielded good results. Alternatively, the sender MAY also choose a random N for each flow, increasing the chances of using a Q run length that gives the best signal for some flows.

The sender MUST keep the value of N constant for a given flow.

3.2. Setting the Loss Event Bit on Outgoing Packets

The Unreported Loss counter is initialized to 0, and L bit of every outgoing packet indicates whether the Unreported Loss counter is positive (L=1 if the counter is positive, and L=0 otherwise). The value of the Unreported Loss counter is decremented every time a packet with L=1 is sent.

The value of the Unreported Loss counter is incremented for every packet that the protocol declares lost, using whatever loss detection machinery the protocol employs. If the protocol is able to rescind the loss determination later, a positive Unreported Loss counter MAY

Ferrieux, et al. Expires January 29, 2021 [Page 5] Internet-Draft loss-bits July 2020

be decremented due to the rescission, but it SHOULD NOT become negative due to the rescission.

This loss signaling is similar to loss signaling in [ConEx], except the Loss Event bit is reporting the exact number of lost packets, whereas Echo Loss bit in [ConEx] is reporting an approximate number of lost bytes.

For protocols, such as TCP ([TCP]), that allow network devices to change data segmentation, it is possible that only a part of the packet is lost. In these cases, the sender MUST increment Unreported Loss counter by the fraction of the packet data lost (so Unreported Loss counter may become negative when a packet with L=1 is sent after a partial packet has been lost).

Observation points can estimate the end-to-end loss, as determined by the upstream endpoint, by counting packets in this direction with the L bit equal to 1, as described in Section 4.

4. Using the Loss Bits for Passive Loss Measurement

4.1. End-To-End Loss

The Loss Event bit allows an observer to calculate the end-to-end loss rate by counting packets with L bit value of 0 and 1 for a given flow. The end-to-end loss rate is the fraction of packets with L=1.

The assumption here is that upstream loss affects packets with L=0 and L=1 equally. If some loss is caused by tail-drop in a network device, this may be a simplification. If the sender’s congestion controller reduces the packet send rate after loss, there may be a sufficient delay before sending packets with L=1 that they have a greater chance of arriving at the observer.

4.2. Upstream Loss

Blocks of N (Q Run length) consecutive packets are sent with the same value of the Q bit, followed by another block of N packets with an inverted value of the Q bit. Hence, knowing the value of N, an on- path observer can estimate the amount of upstream loss after observing at least N packets. The upstream loss rate ("u") is one minus the average number of packets in a block of packets with the same Q value ("p") divided by N ("u=1-avg(p)/N").

The observer needs to be able to tolerate packet reordering that can blur the edges of the square signal.

Ferrieux, et al. Expires January 29, 2021 [Page 6] Internet-Draft loss-bits July 2020

The observer needs to differentiate packets as belonging to different flows, since they use independent counters.

4.3. Correlating End-to-End and Upstream Loss

Upstream loss is calculated by observing packets that did not suffer the upstream loss. End-to-end loss, however, is calculated by observing subsequent packets after the sender’s protocol detected the loss. Hence, end-to-end loss is generally observed with a delay of between 1 RTT (loss declared due to multiple duplicate acknowledgments) and 1 RTO (loss declared due to a timeout) relative to the upstream loss.

The flow RTT can sometimes be estimated by timing protocol handshake messages. This RTT estimate can be greatly improved by observing a dedicated protocol mechanism for conveying RTT information, such as the Latency Spin bit of [QUIC-TRANSPORT].

Whenever the observer needs to perform a computation that uses both upstream and end-to-end loss rate measurements, it SHOULD use upstream loss rate leading the end-to-end loss rate by approximately 1 RTT. If the observer is unable to estimate RTT of the flow, it should accumulate loss measurements over time periods of at least 4 times the typical RTT for the observed flows.

If the calculated upstream loss rate exceeds the end-to-end loss rate calculated in Section 4.1, then either the Q Period is too short for the amount of packet reordering or there is observer loss, described in Section 4.5. If this happens, the observer SHOULD adjust the calculated upstream loss rate to match end-to-end loss rate.

4.4. Downstream Loss

Because downstream loss affects only those packets that did not suffer upstream loss, the end-to-end loss rate ("e") relates to the upstream loss rate ("u") and downstream loss rate ("d") as "(1-u)(1-d)=1-e". Hence, "d=(e-u)/(1-u)".

4.5. Observer Loss

A typical deployment of a passive observation system includes a network tap device that mirrors network packets of interest to a device that performs analysis and measurement on the mirrored packets. The observer loss is the loss that occurs on the mirror path.

Observer loss affects upstream loss rate measurement since it causes the observer to account for fewer packets in a block of identical Q

Ferrieux, et al. Expires January 29, 2021 [Page 7] Internet-Draft loss-bits July 2020

bit values (see {{upstreamloss)}). The end-to-end loss rate measurement, however, is unaffected by the observer loss, since it is a measurement of the fraction of packets with the set L bit value, and the observer loss would affect all packets equally (see Section 4.1).

The need to adjust the upstream loss rate down to match end-to-end loss rate as described in Section 4.3 is a strong indication of the observer loss, whose magnitude is between the amount of such adjustment and the entirety of the upstream loss measured in Section 4.2. Alternatively, a high apparent upstream loss rate could be an indication of significant reordering, possibly due to packets belonging to a single flow being multiplexed over several upstream paths with different latency characteristics.

5. ECN-Echo Event Bit

While the primary focus of the draft is on exposing packet loss, modern networks can report congestion before they are forced to drop packets, as described in [ECN]. When transport protocols keep ECN- Echo feedback under encryption, this signal cannot be observed by the network operators. When tasked with diagnosing network performance problems, knowledge of a congestion downstream of an observation point can be instrumental.

If downstream congestion information is desired, this information can be signaled with an additional bit.

- E: The "ECN-Echo Event" bit is set to 0 or 1 according to the Unreported ECN Echo counter, as explained below in Section 5.1.

5.1. Setting the ECN-Echo Event Bit on Outgoing Packets

The Unreported ECN-Echo counter operates identicaly to Unreported Loss counter (Section 3.2), except it counts packets delivered by the network with CE markings, according to the ECN-Echo feedback from the receiver.

This ECN-Echo signaling is similar to ECN signaling in [ConEx]. ECN- Echo mechanism in QUIC provides the number of packets received with CE marks. For protocols like TCP, the method described in [ConEx-TCP] can be employed. As stated in [ConEx-TCP], such feedback can be further improved using a method described in [ACCURATE].

Ferrieux, et al. Expires January 29, 2021 [Page 8] Internet-Draft loss-bits July 2020

5.2. Using E Bit for Passive ECN-Reported Congestion Measurement

A network observer can count packets with CE codepoint and determine the upstream CE-marking rate directly.

Observation points can also estimate ECN-reported end-to-end congestion by counting packets in this direction with a E bit equal to 1.

The upstream CE-marking rate and end-to-end ECN-reported congestion can provide information about downstream CE-marking rate. Presence of E bits along with L bits, however, can somewhat confound precise estimates of upstream and downstream CE-markings in case the flow contains packets that are not ECN-capable.

6. Protocol Ossification Considerations

Accurate loss information is not critical to the operation of any protocol, though its presence for a sufficient number of flows is important for the operation of networks.

The loss bits are amenable to "greasing" described in [RFC8701], if the protocol designers are not ready to dedicate (and ossify) bits used for loss reporting to this function. The greasing could be accomplished similarly to the Latency Spin bit greasing in [QUIC-TRANSPORT]. Namely, implementations could decide that a fraction of flows should not encode loss information in the loss bits and, instead, the bits would be set to arbitrary values. The observers would need to be ready to ignore flows with loss information more resembling noise than the expected signal.

7. Security Considerations

Passive loss observation has been a part of the network operations for a long time, so exposing loss information to the network does not add new security concerns for protocols that are currently observable.

In the absence of upstream packet loss, the Q bit signal does not provide any information that cannot be observed by simply counting packets transiting a network path. In the presence of upstream packet loss, the Q bit will disclose the loss, but this is information about the environment and not the endpoint state. The L bit signal discloses internal state of the protocol’s loss detection machinery, but this state can often be gleamed by timing packets and observing congestion controller response. Hence, loss bits do not provide a viable new mechanism to attack data integrity and secrecy.

Ferrieux, et al. Expires January 29, 2021 [Page 9] Internet-Draft loss-bits July 2020

7.1. Optimistic ACK Attack

A defense against an Optimistic ACK Attack, decribed in [QUIC-TRANSPORT], involves a sender randomly skipping packet numbers to detect a receiver acknowledging packet numbers that have never been received. The Q bit signal may inform the attacker which packet numbers were skipped on purpose and which had been actually lost (and are, therefore, safe for the attacker to acknowledge). To use the Q bit for this purpose, the attacker must first receive at least an entire Q Run of packets, which renders the attack ineffective against a delay-sensitive congestion controller.

A protocol that is more susceptible to an Optimistic ACK Attack with the loss signal provided by Q bit and uses a loss-based congestion controller, SHOULD shorten the current Q Run by the number of skipped packets numbers. For example, skipping a single packet number will invert the sQuare signal one outgoing packet sooner.

8. Privacy Considerations

To minimize unintentional exposure of information, loss bits provide an explicit loss signal - a preferred way to share information per [RFC8558].

New protocols commonly have specific privacy goals, and loss reporting must ensure that loss information does not compromise those privacy goals. For example, [QUIC-TRANSPORT] allows changing Connection IDs in the middle of a connection to reduce the likelihood of a passive observer linking old and new subflows to the same device. A QUIC implementation would need to reset all counters when it changes the destination (IP address or UDP port) or the Connection ID used for outgoing packets. It would also need to avoid incrementing Unreported Loss counter for loss of packets sent to a different destination or with a different Connection ID.

9. IANA Considerations

This document makes no request of IANA.

10. Change Log

10.1. Since version 02

- Minor improvement and clarifications

Ferrieux, et al. Expires January 29, 2021 [Page 10] Internet-Draft loss-bits July 2020

10.2. Since version 01

- Clarified Q Period selection

- Added an optional E (ECN-Echo Event) bit

- Clarified L bit calculation for protocols that allow partial data loss due to a change in segmentation (such as TCP)

10.3. Since version 00

- Addressed review comments

- Improved guidelines for privacy protections for QIUC

11. Acknowledgments

The sQuare bit was originally suggested by Kazuho Oku in early proposals for loss measurement and is an instance of the "alternate marking" as defined in [RFC8321].

12. References

12.1. Normative References

[ConEx] Mathis, M. and B. Briscoe, "Congestion Exposure (ConEx) Concepts, Abstract Mechanism, and Requirements", RFC 7713, DOI 10.17487/RFC7713, December 2015, .

[ConEx-TCP] Kuehlewind, M., Ed. and R. Scheffenegger, "TCP Modifications for Congestion Exposure (ConEx)", RFC 7786, DOI 10.17487/RFC7786, May 2016, .

[ECN] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition of Explicit Congestion Notification (ECN) to IP", RFC 3168, DOI 10.17487/RFC3168, September 2001, .

[IP] Postel, J., "Internet Protocol", STD 5, RFC 791, DOI 10.17487/RFC0791, September 1981, .

Ferrieux, et al. Expires January 29, 2021 [Page 11] Internet-Draft loss-bits July 2020

[IPv6] Deering, S. and R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", STD 86, RFC 8200, DOI 10.17487/RFC8200, July 2017, .

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, .

[RFC8321] Fioccola, G., Ed., Capello, A., Cociglio, M., Castaldelli, L., Chen, M., Zheng, L., Mirsky, G., and T. Mizrahi, "Alternate-Marking Method for Passive and Hybrid Performance Monitoring", RFC 8321, DOI 10.17487/RFC8321, January 2018, .

[RFC8558] Hardie, T., Ed., "Transport Protocol Path Signals", RFC 8558, DOI 10.17487/RFC8558, April 2019, .

[TCP] Postel, J., "Transmission Control Protocol", STD 7, RFC 793, DOI 10.17487/RFC0793, September 1981, .

12.2. Informative References

[ACCURATE] Briscoe, B., Kuehlewind, M., and R. Scheffenegger, "More Accurate ECN Feedback in TCP", draft-ietf-tcpm-accurate- ecn-11 (work in progress), March 2020.

[IPv6AltMark] Fioccola, G., Zhou, T., Cociglio, M., Qin, F., and R. Pang, "IPv6 Application of the Alternate Marking Method", draft-ietf-6man-ipv6-alt-mark-01 (work in progress), June 2020.

[QUIC-TRANSPORT] Iyengar, J. and M. Thomson, "QUIC: A UDP-Based Multiplexed and Secure Transport", draft-ietf-quic-transport-29 (work in progress), June 2020.

[RFC8517] Dolson, D., Ed., Snellman, J., Boucadair, M., Ed., and C. Jacquenet, "An Inventory of Transport-Centric Functions Provided by Middleboxes: An Operator Perspective", RFC 8517, DOI 10.17487/RFC8517, February 2019, .

Ferrieux, et al. Expires January 29, 2021 [Page 12] Internet-Draft loss-bits July 2020

[RFC8701] Benjamin, D., "Applying Generate Random Extensions And Sustain Extensibility (GREASE) to TLS Extensibility", RFC 8701, DOI 10.17487/RFC8701, January 2020, .

[TRANSPORT-ENCRYPT] Fairhurst, G. and C. Perkins, "Considerations around Transport Header Confidentiality, Network Operations, and the Evolution of Internet Transport Protocols", draft- ietf-tsvwg-transport-encrypt-16 (work in progress), July 2020.

[UDP-OPTIONS] Touch, J., "Transport Options for UDP", draft-ietf-tsvwg- udp-options-08 (work in progress), September 2019.

[UDP-SURPLUS] Herbert, T., "UDP Surplus Header", draft-herbert-udp- space-hdr-01 (work in progress), July 2019.

Authors’ Addresses

Alexandre Ferrieux (editor) Orange Labs

EMail: [email protected]

Isabelle Hamchaoui (editor) Orange Labs

EMail: [email protected]

Igor Lubashev (editor) Akamai Technologies

EMail: [email protected]

Dmitri Tikhonov (editor) LiteSpeed Technologies

EMail: [email protected]

Ferrieux, et al. Expires January 29, 2021 [Page 13] Transport Working Group P. Heist Internet-Draft R.W. Grimes Intended status: Informational J. Morton Expires: 4 January 2020 3 July 2019

Some Congestion Experienced One and Two-Flow Tests draft-heist-tsvwg-sce-one-and-two-flow-tests-00

Abstract

This note presents one and two-flow test results for the SCE (Some Congestion Experienced) reference implementation. These tests are not intended to be a comprehensive real-world evaluation of SCE, but an illustration of SCE’s influence on basic TCP metrics in a controlled environment.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

This Internet-Draft will expire on 4 January 2020.

Copyright Notice

This document is subject to BCP 78 and the IETF Trust’s Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Heist, et al. Expires 4 January 2020 [Page 1] Internet-Draft sceonetwotests July 2019

Table of Contents

1. Introduction ...... 2 2. Terminology ...... 3 3. Test Tools and Environment ...... 3 4. Tests ...... 4 5. Results and Analysis ...... 4 5.1. One-Flow Tests ...... 4 5.1.1. Reno-SCE TCP Throughput ...... 5 5.1.2. Reno-SCE TCP RTT ...... 6 5.1.3. DCTCP-SCE TCP Throughput ...... 7 5.1.4. DCTCP-SCE TCP RTT ...... 8 5.2. Two-Flow Tests ...... 9 5.2.1. Single Queue (Cake "flowblind") ...... 9 5.2.2. Single Queue (Cake "sce-single") ...... 11 5.2.3. Fair Queue (Cake "triple-isolate") ...... 13 6. Security Considerations ...... 15 7. IANA Considerations ...... 15 8. Acknowledgments ...... 15 9. Informative References ...... 15 Appendix A. Appendix (Raw Results Tables) ...... 15 A.1. One-Flow TCP Throughput ...... 16 A.2. One-Flow TCP RTT ...... 18 A.3. Two-Flow TCP Throughput (Cake "flowblind") ...... 21 A.4. Two-Flow TCP Throughput (Cake "flowblind sce-single") ...... 28 A.5. Two-Flow TCP Throughput (Cake "triple-isolate") . . . . . 36 A.6. Two-Flow TCP RTT (Cake "flowblind") ...... 43 A.7. Two-Flow TCP RTT (Cake "flowblind sce-single") . . . . . 50 A.8. Two-Flow TCP RTT (Cake "triple-isolate") ...... 58 Authors’ Addresses ...... 65

1. Introduction

SCE provides early and proportional feedback to the CC (congestion control) algorithms for transport protocols, including but not limited to TCP. The [sce-repo] is a Linux kernel modified to support SCE, including:

* Enhancements to Linux’s Cake (Common Applications Kept Enhanced) AQM to support SCE signaling

* Modifications to the TCP receive path to reflect SCE signals back to the sender

* The addition of three new TCP CC algorithms that modify the originals to add SCE support: Reno-SCE, DCTCP-SCE and Cubic-SCE (work in progress as of this writing)

Heist, et al. Expires 4 January 2020 [Page 2] Internet-Draft sceonetwotests July 2019

In this note we run one and two-flow TCP tests across a range of simulated path bandwidths and RTTs. One-flow tests measure SCE’s impact on TCP throughput and TCP RTT. Two-flow tests evaluate fairness between and among several SCE and non-SCE TCP implementations, while making several adjustments to Cake’s SCE and fair queueing parameters.

It is recognized that these tests do not simulate real-world conditions, and will not be an indication of how SCE will perform in all situations. However, they serve as fundamental tests for the SCE reference implementation. Once the behavior in these tests is well understood and theory and experiment are in agreement, additional complexity can be added to the test procedures with the confidence that the reference implementation’s fundamentals are sound.

2. Terminology

The following terminology is used in this document:

* Path Bandwidth or Cake-limited Bandwidth: The available bandwidth between the sender and receiver, as controlled by Cake on the middlebox, and set by Cake’s bandwidth parameter.

* Path RTT or just RTT (in context): The approximate minimum round- trip time of packets that go from the sender to receiver and back when the path is unloaded.

* Netem bi-directional delay: The total path RTT added by netem in both directions.

* TCP throughput and TCP RTT: Well-known terms that apply specifically to the flows under test.

3. Test Tools and Environment

The [Flent] tool is used for all tests. Flent uses netperf for its TCP tests, and allows for test batches, plotting, the recording of results and the collection of metadata in JSON format [RFC8259]. Flent both captures the measured TCP throughput from netperf, and simultaneously uses the [ss] tool in Linux to passively monitor TCP RTT.

All tests are performed using a three node dumbbell topology:

+------+ +------+ +------+ | Sender |------| Middlebox |------| Receiver | +------+ +------+ +------+

Heist, et al. Expires 4 January 2020 [Page 3] Internet-Draft sceonetwotests July 2019

Figure 1: Test topology

* Sender: Runs Flent and sends data to the receiver

* Middlebox:

- Acts as a router between the sender and receiver

- Runs Cake on egress of both interfaces for queue management and SCE signaling

- Runs netem on the ingress of both interfaces for delay simulation, splitting the total delay in half for each interface

* Receiver: Receives data from and reflects SCE signals back to the sender via the ESCE (Echo Some Congestion Experienced) bit

All nodes run the SCE reference implementation kernel as of commit 56915a82 (2019-06-20), and are connected directly via Gigabit Ethernet.

4. Tests

The tests are implemented with a Flent batch file to drive netperf and re-configure Cake and netem on the middlebox with various parameters. Scripts post-process the results and create csv and markdown tables for external use, including by this document.

Unless otherwise mentioned, measurements are obtained from TCP flows from start to finish, not at steady state. This allows for some discussion of the differences in TCP CC algorithm behavior during slow start and congestion avoidance. Typically, each test is run long enough to obtain a reasonable approximation of steady state throughput, but in a few high BDP cases slow start accounts for a significant portion of the test length. When relevant to the analysis of the results, this is stated in the text.

5. Results and Analysis

5.1. One-Flow Tests

The goal of the one-flow tests is to analyze the impact of SCE on the TCP throughput and TCP RTT of single TCP flows across a range of simulated path bandwidths and RTTs. What follows is an analysis of the results. See Section a.1 and Section a.2 for the raw results for TCP throughput and TCP RTT, respectively.

Heist, et al. Expires 4 January 2020 [Page 4] Internet-Draft sceonetwotests July 2019

5.1.1. Reno-SCE TCP Throughput

The following table shows the difference in TCP throughput for Reno- SCE vs Reno across the tested range of simulated path bandwidths and RTTs:

+---+-----+------+------+------+------+------+------+------+ | | 0| 2| 5| 10| 20 | 40 | 80 | 160 | +===+=====+======+======+======+======+======+======+======+ |0.5|0.000| 0.000| 0.000| 0.000| 0.000 | -0.100 | -0.180 | -0.160 | +---+-----+------+------+------+------+------+------+------+ |1 |0.000| 0.000| 0.000| 0.000| -0.110 | -0.210 | -0.280 | -0.280 | +---+-----+------+------+------+------+------+------+------+ |2 |0.000| 0.000|-0.010| -0.120| -0.210 | -0.375 | -0.370 | -0.405 | +---+-----+------+------+------+------+------+------+------+ |5 |0.000|-0.052|-0.142| -0.228| -0.178 | -0.032 | -0.006 | 0.016 | +---+-----+------+------+------+------+------+------+------+ |10 |0.000|-0.071|-0.139| -0.143| -0.046 | 0.062 | 0.116 | 0.168 | +---+-----+------+------+------+------+------+------+------+ |25 |0.000|-0.003|-0.006| 0.010| 0.073 | 0.143 | 0.196 | 0.259 | +---+-----+------+------+------+------+------+------+------+ |50 |0.000| 0.000| 0.001| 0.029| 0.102 | 0.169 | 0.235 | 0.272 | +---+-----+------+------+------+------+------+------+------+ |100|0.000| 0.000| 0.004| 0.043| 0.103 | 0.215 | 0.238 | 0.289 | +---+-----+------+------+------+------+------+------+------+

Table 1: Difference in TCP Throughput (reno-sce - reno), normalized to Cake- limited Bandwidth; Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

From the above TCP throughput differences we can observe:

1. Improved utilization for SCE at sufficiently high BDPs. This is due to SCE’s proportional congestion signals, which can significantly reduce the classic Reno throughput sawtooth by making drops or CE marks rare to non-existent. The utilization improvement increases with BDP largely because the TCP window recovery time after a drop or CE mark increases with BDP, deepening the sawtooth.

2. Significant under-utilization at bandwidths <= 10Mbit, which tends to worsen as path RTT increases. Investigation is underway as to the source of this. These drops in utilization are however also accompanied by drops in TCP RTT (see below).

Heist, et al. Expires 4 January 2020 [Page 5] Internet-Draft sceonetwotests July 2019

5.1.2. Reno-SCE TCP RTT

The following table shows the difference in TCP RTT for Reno-SCE vs Reno across the tested range of simulated path bandwidths and RTTs:

+---+------+------+------+------+------+------+------+------+ | | 0| 2| 5| 10| 20 | 40 | 80 | 160 | +===+======+======+======+======+======+======+======+======+ |0.5|-88.41|-87.51|-88.43|-88.59| -62.37 | -52.56 | -42.23 | -31.93 | +---+------+------+------+------+------+------+------+------+ |1 |-42.30|-41.94|-42.65|-33.03| -32.92 | -27.92 | -19.13 | -13.20 | +---+------+------+------+------+------+------+------+------+ |2 |-29.09|-27.67|-22.19|-19.23| -14.54 | -10.12 | -3.66 | -1.50 | +---+------+------+------+------+------+------+------+------+ |5 | -9.18| -9.47| -7.51| -8.37| -5.63 | -3.49 | -1.69 | -1.24 | +---+------+------+------+------+------+------+------+------+ |10 | -2.46| -2.75| -2.95| -3.91| -2.44 | -1.42 | -0.54 | -0.41 | +---+------+------+------+------+------+------+------+------+ |25 | -1.87| -2.09| -2.53| -1.74| -0.42 | 0.01 | 0.19 | -0.01 | +---+------+------+------+------+------+------+------+------+ |50 | -2.00| -2.18| -1.48| -0.31| 0.59 | 0.98 | 0.86 | 0.57 | +---+------+------+------+------+------+------+------+------+ |100| -1.87| -1.67| -1.04| 0.06| 0.95 | 1.42 | 1.41 | 1.15 | +---+------+------+------+------+------+------+------+------+

Table 2: Difference in TCP RTT (reno-sce - reno); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

From the above TCP RTT differences we can observe:

1. Significant reductions in TCP RTT for Reno-SCE at most BDPs. SCE’s proportional congestion signals are aiding the sender in managing queue lengths.

2. Greater reductions in TCP RTT at lower path bandwidths and RTTs. This is because Reno-linear growth becomes relatively larger as the path bandwidth decreases. This results in greater queue growth in the interval before AQM activates, and a subsequently longer drain time after congestion is signaled.

3. A slight increase in TCP RTT for SCE at high BDPs. This has been observed to have a possible connection to the ESCE (Echo Some Congestion Experienced) feedback strategy implemented in on the TCP receive side. Research and optimization in this area is ongoing.

Heist, et al. Expires 4 January 2020 [Page 6] Internet-Draft sceonetwotests July 2019

5.1.3. DCTCP-SCE TCP Throughput

The following table shows the difference in TCP throughput for DCTCP- SCE vs DCTCP across the tested range of simulated path bandwidths and RTTs:

+---+------+------+------+------+------+------+------+------+ | | 0| 2| 5| 10| 20 | 40 | 80 | 160 | +===+======+======+======+======+======+======+======+======+ |0.5| 0.000| 0.000| 0.000| 0.000| 0.000 | -0.120 | -0.220 | -0.400 | +---+------+------+------+------+------+------+------+------+ |1 | 0.000| 0.000| 0.000| 0.000| -0.120 | -0.260 | -0.420 | -0.570 | +---+------+------+------+------+------+------+------+------+ |2 | 0.000| 0.000|-0.015|-0.130| -0.285 | -0.505 | -0.565 | -0.705 | +---+------+------+------+------+------+------+------+------+ |5 | 0.000|-0.038|-0.150|-0.270| -0.374 | -0.588 | -0.652 | -0.620 | +---+------+------+------+------+------+------+------+------+ |10 | 0.000|-0.087|-0.197|-0.265| -0.266 | -0.257 | -0.271 | -0.134 | +---+------+------+------+------+------+------+------+------+ |25 |-0.006|-0.060|-0.131|-0.167| -0.154 | -0.133 | -0.067 | 0.110 | +---+------+------+------+------+------+------+------+------+ |50 | 0.000|-0.001|-0.006|-0.008| 0.005 | 0.012 | 0.143 | 0.180 | +---+------+------+------+------+------+------+------+------+ |100| 0.000| 0.007| 0.018| 0.031| 0.040 | 0.087 | 0.151 | 0.246 | +---+------+------+------+------+------+------+------+------+

Table 3: Difference in TCP Throughput (dctcp-sce - dctcp), normalized to Cake- limited Bandwidth; Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

From the above TCP throughput differences we can observe:

1. Improved utilization for SCE at higher BDPs. At first glance we might assume that this is due to SCE’s feedback signals improving upon DCTCP’s window recovery time, as with Reno, but another significant part of this increase is due to DCTCP-SCE’s steeper ramp during slow start, and a test length that is short relative to the time spent in slow start. The test is 300 seconds long, and at a path bandwidth of 100Mbit and RTT of 160ms, DCTCP takes a full 225 seconds to ramp up to the BDP, while DCTCP-SCE, with its steeper ramp, takes only 80 seconds. That said, steady state throughput at this BDP is around 87Mbit for DCTCP, and around 96Mbit for DCTCP-SCE, an increase of around 10%, so the proportional congestion control signals also play a part in increasing utilization.

2. Significant under-utilization at bandwidths <= 25Mbit, which tends to worsen as path RTT increases. Investigation is underway

Heist, et al. Expires 4 January 2020 [Page 7] Internet-Draft sceonetwotests July 2019

as to the source of this. These drops in utilization are however also accompanied by drops in TCP RTT (see below).

5.1.4. DCTCP-SCE TCP RTT

The following table shows the difference in TCP RTT for DCTCP-SCE vs DCTCP across the tested range of simulated path bandwidths and RTTs:

+---+------+------+------+------+------+------+------+------+ | | 0| 2| 5| 10| 20 | 40 | 80 | 160 | +===+======+======+======+======+======+======+======+======+ |0.5|-82.57|-81.21|-81.37|-81.53| -60.53 | -70.29 | -74.76 | -54.33 | +---+------+------+------+------+------+------+------+------+ |1 |-41.78|-42.31|-40.43|-30.98| -35.26 | -39.27 | -29.95 | -15.62 | +---+------+------+------+------+------+------+------+------+ |2 |-21.94|-20.29|-20.86|-17.58| -19.93 | -12.70 | -7.37 | -3.97 | +---+------+------+------+------+------+------+------+------+ |5 | -6.65| -7.27| -7.08| -6.45| -5.57 | -3.59 | -2.19 | -1.31 | +---+------+------+------+------+------+------+------+------+ |10 | -1.85| -2.11| -2.85| -2.25| -2.16 | -1.51 | -1.28 | -1.12 | +---+------+------+------+------+------+------+------+------+ |25 | -2.47| -2.52| -2.94| -3.13| -2.93 | -1.63 | -0.98 | -0.78 | +---+------+------+------+------+------+------+------+------+ |50 | -2.63| -3.15| -2.99| -3.12| -3.37 | -3.57 | -0.81 | -0.63 | +---+------+------+------+------+------+------+------+------+ |100| -2.79| -2.81| -2.82| -3.01| -3.26 | -2.52 | -0.14 | 0.02 | +---+------+------+------+------+------+------+------+------+

Table 4: Difference in TCP RTT (dctcp-sce - dctcp); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

From the above TCP RTT differences we can observe:

1. A nearly across the board reduction in TCP RTT for DCTCP-SCE. SCE’s proportional congestion signals are aiding the sender in managing queue lengths.

2. Greater reductions in TCP RTT at lower bandwidths and path RTTs. See Section 5.1.2 for an explanation of this.

3. A slight increase in TCP RTT for DCTCP-SCE at 100Mbit / 160ms. See Section 5.1.2 for a likely explanation of this.

Heist, et al. Expires 4 January 2020 [Page 8] Internet-Draft sceonetwotests July 2019

5.2. Two-Flow Tests

The goal of the two-flow tests is to measure fairness between and among SCE and non-SCE TCP flows, through either a single queue or with fair queueing. What follows is a partial analysis of the results, and see Section a.3 through Section a.8 for the raw results tables.

5.2.1. Single Queue (Cake "flowblind")

Cake’s flowblind parameter disables fair queueing, so that Cake uses only a single queue. This can be used to evaluate single-queue throughput fairness between SCE and non-SCE flows.

5.2.1.1. Reno vs Reno

It might be useful to remind ourselves that competition between two (or more) flows of the same TCP CC algorithm is usually designed to yield a high degree of throughput fairness in a single queue. Looking at Jain’s fairness index [RFC5166] for two flows across a range of simulated path bandwidths and RTTs shows this to be true for two TCP Reno flows, for example:

+-----+------+------+------+------+ | | 0 | 10 | 20 | 80 | +=====+======+======+======+======+ | 1 | 0.993 | 1.000 | 1.000 | 0.998 | +-----+------+------+------+------+ | 5 | 1.000 | 1.000 | 0.998 | 0.998 | +-----+------+------+------+------+ | 10 | 1.000 | 0.999 | 0.998 | 1.000 | +-----+------+------+------+------+ | 50 | 1.000 | 0.998 | 0.999 | 0.999 | +-----+------+------+------+------+ | 100 | 0.998 | 0.999 | 0.999 | 1.000 | +-----+------+------+------+------+

Table 5: reno vs reno Jain’s fairness index; Columns: netem bi- dir Delay (ms); Rows: Cake Bandwidth (Mbit)

5.2.1.2. Reno vs Reno-SCE

Now, we do a comparison of Reno vs Reno-SCE in a single queue, while still using Cake’s default SCE signaling ramp:

Heist, et al. Expires 4 January 2020 [Page 9] Internet-Draft sceonetwotests July 2019

+-----+------+------+------+------+ | | 0 | 10 | 20 | 80 | +=====+======+======+======+======+ | 1 | 0.969 | 0.969 | 0.915 | 0.821 | +-----+------+------+------+------+ | 5 | 0.958 | 0.824 | 0.714 | 0.632 | +-----+------+------+------+------+ | 10 | 0.959 | 0.719 | 0.669 | 0.698 | +-----+------+------+------+------+ | 50 | 0.608 | 0.621 | 0.673 | 0.907 | +-----+------+------+------+------+ | 100 | 0.560 | 0.627 | 0.668 | 0.782 | +-----+------+------+------+------+

Table 6: reno vs reno-sce Jain’s fairness index; Columns: netem bi- dir Delay (ms); Rows: Cake Bandwidth (Mbit)

From the above Jain’s fairness index numbers we can observe that while there can be reasonable fairness at very low BDPs, and while there is no starvation, fairness degrades quickly at higher throughputs and RTTs. This is due to the Cake’s default SCE signaling ramp being tuned to provide an early signal of congestion, to avoid CE marks and packet drops. As a result, SCE enabled flows back off in the face of competition, whereas non-SCE flows fill the queue until a drop or CE mark occurs.

5.2.1.3. Cubic vs DCTCP-SCE

Of particular interest to the congestion control community is competition between the commonly used TCP Cubic algorithm and DCTCP- SCE. It is a well-established fact that classic DCTCP will typically out-compete Cubic in a single queue. It would be valuable if there were a way to improve that fairness with SCE.

When comparing Cubic vs DCTCP-SCE using Cake’s default SCE signaling ramp, we can see that while there is no starvation, fairness does degrade fairly rapidly as BDP increases:

Heist, et al. Expires 4 January 2020 [Page 10] Internet-Draft sceonetwotests July 2019

+-----+------+------+------+------+ | | 0 | 10 | 20 | 80 | +=====+======+======+======+======+ | 1 | 0.979 | 0.976 | 0.922 | 0.821 | +-----+------+------+------+------+ | 5 | 0.977 | 0.797 | 0.708 | 0.604 | +-----+------+------+------+------+ | 10 | 0.979 | 0.693 | 0.645 | 0.627 | +-----+------+------+------+------+ | 50 | 0.599 | 0.570 | 0.586 | 0.607 | +-----+------+------+------+------+ | 100 | 0.547 | 0.552 | 0.563 | 0.599 | +-----+------+------+------+------+

Table 7: cubic vs dctcp-sce Jain’s fairness index; Columns: netem bi- dir Delay (ms); Rows: Cake Bandwidth (Mbit)

As of today, SCE by default does not lead to fairness at all BDPs between SCE and non-SCE flows. However, efforts are ongoing to improve this, and as we can see in Section 5.2.2, Cake’s signaling ramp can be tuned to improve this fairness.

5.2.2. Single Queue (Cake "sce-single")

As we saw in Section 5.2.1, there is room to improve SCE vs non-SCE fairness in a single queue. One way to do this is to change the SCE signaling ramp to reduce or delay SCE signals until closer to the point where CE signals occur. This is the motivation behind the sce- single and sce-thresh Cake parameters.

By default, Cake begins proportionally signaling SCE when a packet’s sojourn time in the queue is greater than half the CoDel target, and reaches 100% SCE signaling when the sojourn time is greater than or equal to the CoDel target. The sce-single parameter delays the start of the ramp until the sojourn reaches the CoDel target itself, while keeping the ramp slope the same. The sce-thresh parameter, while not evaluated here, allows an intermediate ramp between the default and sce-single, using values from 2-1024, with sce-thresh set to 8 yielding a ramp that’s halfway in-between.

It is expected that the result of signaling SCE later is a subsequent increase in TCP RTT, including for single flows. Thus, the adjustment of the SCE signaling ramp is a tradeoff between the increased utilization and reduced TCP RTT possible with SCE, and fairness with non-SCE flows in a single queue. By using sce-single,

Heist, et al. Expires 4 January 2020 [Page 11] Internet-Draft sceonetwotests July 2019

we can show the maximum fairness that can be achieved by tuning the SCE signaling ramp in this way.

5.2.2.1. Reno vs Reno-SCE

Revisiting the Reno vs Reno-SCE comparison that we did in Section 5.2.1.2, we now run the same test with Cake’s sce-single parameter set:

+-----+------+------+------+------+ | | 0 | 10 | 20 | 80 | +=====+======+======+======+======+ | 1 | 1.000 | 0.998 | 1.000 | 0.987 | +-----+------+------+------+------+ | 5 | 0.997 | 0.952 | 0.785 | 0.819 | +-----+------+------+------+------+ | 10 | 0.997 | 0.950 | 0.790 | 0.994 | +-----+------+------+------+------+ | 50 | 1.000 | 0.986 | 0.980 | 0.977 | +-----+------+------+------+------+ | 100 | 0.988 | 0.901 | 0.932 | 0.987 | +-----+------+------+------+------+

Table 8: reno vs reno-sce Jain’s fairness index; Columns: netem bi- dir Delay (ms); Rows: Cake Bandwidth (Mbit)

We can see that single queue fairness has improved considerably. While there are results at a few bandwidth-delay combinations that are still under investigation, single queue fairness between Reno and Reno-SCE has largely been achieved.

5.2.2.2. Cubic vs DCTCP-SCE

Revisiting the Cubic vs DCTCP-SCE comparison that we did in Section 5.2.1.3, we now run the same test with Cake’s sce-single parameter set:

Heist, et al. Expires 4 January 2020 [Page 12] Internet-Draft sceonetwotests July 2019

+-----+------+------+------+------+ | | 0 | 10 | 20 | 80 | +=====+======+======+======+======+ | 1 | 0.993 | 0.999 | 0.995 | 0.963 | +-----+------+------+------+------+ | 5 | 1.000 | 0.898 | 0.765 | 0.741 | +-----+------+------+------+------+ | 10 | 0.975 | 0.893 | 0.733 | 0.885 | +-----+------+------+------+------+ | 50 | 0.999 | 0.752 | 0.799 | 0.923 | +-----+------+------+------+------+ | 100 | 0.944 | 0.714 | 0.661 | 0.650 | +-----+------+------+------+------+

Table 9: cubic vs dctcp-sce Jain’s fairness index; Columns: netem bi- dir Delay (ms); Rows: Cake Bandwidth (Mbit)

While it can be seen that there is an almost across the board improvement vs Cake’s default SCE ramp, there are still bandwidth- delay combinations that do not yield a sufficient level of fairness in a single queue. Work is ongoing to improve this.

5.2.3. Fair Queue (Cake "triple-isolate")

Cake’s default triple-isolate fairness mode provides fairness among flows and a combination of source and destination IP addresses. For our purposes, this will effectively serve as fair queueing among flows, as we are only using one IP on each of the source and destination hosts.

With fair queueing, we expect to achieve a high level of throughput fairness at most bandwidth-delay combinations.

5.2.3.1. Cubic vs DCTCP-SCE

Revisiting the Cubic vs DCTCP-SCE comparisons that we did in Section 5.2.1.3 and Section 5.2.2.2, we now run the same test with Cake’s fair queueing enabled via triple-isolate:

Heist, et al. Expires 4 January 2020 [Page 13] Internet-Draft sceonetwotests July 2019

+-----+------+------+------+------+ | | 0 | 10 | 20 | 80 | +=====+======+======+======+======+ | 1 | 1.000 | 1.000 | 1.000 | 0.916 | +-----+------+------+------+------+ | 5 | 1.000 | 0.980 | 0.931 | 0.698 | +-----+------+------+------+------+ | 10 | 1.000 | 0.946 | 0.896 | 0.722 | +-----+------+------+------+------+ | 50 | 1.000 | 0.978 | 0.977 | 1.000 | +-----+------+------+------+------+ | 100 | 1.000 | 0.999 | 1.000 | 1.000 | +-----+------+------+------+------+

Table 10: cubic vs dctcp-sce Jain’s fairness index; Columns: netem bi- dir Delay (ms); Rows: Cake Bandwidth (Mbit)

With fair queueing, fairness is achieved for a broad range of bandwidth-delay combinations. We do see a consistent and narrow deviation at around 160ms between 5 and 10Mbit, which happens in competition between SCE and non-SCE flows, regardless of the exact algorithms in use. The cause of this will be investigated.

5.2.3.2. DCTCP vs DCTCP-SCE

As another example of SCE vs non-SCE competition with fair queueing enabled, here we compare DCTCP vs DCTCP-SCE:

+-----+------+------+------+------+ | | 0 | 10 | 20 | 80 | +=====+======+======+======+======+ | 1 | 1.000 | 1.000 | 1.000 | 0.900 | +-----+------+------+------+------+ | 5 | 1.000 | 0.973 | 0.925 | 0.661 | +-----+------+------+------+------+ | 10 | 1.000 | 0.950 | 0.901 | 0.739 | +-----+------+------+------+------+ | 50 | 1.000 | 0.976 | 0.978 | 1.000 | +-----+------+------+------+------+ | 100 | 1.000 | 0.999 | 0.999 | 0.999 | +-----+------+------+------+------+

Table 11: dctcp vs dctcp-sce Jain’s fairness index; Columns: netem bi- dir Delay (ms); Rows: Cake Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 14] Internet-Draft sceonetwotests July 2019

As with Cubic vs DCTCP-SCE, a high degree of fairness is achieved for most bandwidth-delay combinations, yet we do see the same small gap in fairness at around 160ms between 5 and 10Mbit as we see with Cubic vs DCTCP-SCE and other SCE vs non-SCE flow tests with fair queueing. It is expected that further investigation into this will lead to a solution.

6. Security Considerations

There are no known security considerations introduced by this note.

7. IANA Considerations

This document has no IANA actions.

8. Acknowledgments

Many thanks go out to Toke Hoiland-Jorgensen for making several key changes to the Flent tool.

9. Informative References

[Flent] "The FLExible Network Tester Home Page", July 2019, .

[RFC5166] Floyd, S., Ed., "Metrics for the Evaluation of Congestion Control Mechanisms", RFC 5166, DOI 10.17487/RFC5166, March 2008, .

[RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data Interchange Format", STD 90, RFC 8259, DOI 10.17487/RFC8259, December 2017, .

[sce-repo] "Some Congestion Experienced Reference Implementation GitHub Repository", July 2019, .

[ss] "ss man page", July 2019, .

Appendix A. Appendix (Raw Results Tables)

Heist, et al. Expires 4 January 2020 [Page 15] Internet-Draft sceonetwotests July 2019

A.1. One-Flow TCP Throughput

+---+------+------+------+------+------+------+------+------+ | | 0 | 2 | 5 | 10 | 20 | 40 | 80 | 160 | +===+======+======+======+======+======+======+======+======+ |0.5| 0.48 | 0.48 | 0.48 | 0.48 | 0.48 | 0.47 | 0.48 | 0.44 | +---+------+------+------+------+------+------+------+------+ |1 | 0.96 | 0.96 | 0.96 | 0.96 | 0.95 | 0.94 | 0.87 | 0.82 | +---+------+------+------+------+------+------+------+------+ |2 | 1.91 | 1.91 | 1.91 | 1.91 | 1.91 | 1.83 | 1.82 | 1.52 | +---+------+------+------+------+------+------+------+------+ |5 | 4.78 | 4.78 | 4.78 | 4.78 | 4.76 | 4.57 | 4.32 | 3.28 | +---+------+------+------+------+------+------+------+------+ |10 | 9.56 | 9.56 | 9.56 | 9.55 | 9.43 | 8.85 | 8.23 | 6.51 | +---+------+------+------+------+------+------+------+------+ |25 | 23.91 | 23.91 | 23.89 | 23.82 | 23.17 | 22.13 | 20.83 | 16.07 | +---+------+------+------+------+------+------+------+------+ |50 | 47.81 | 47.80 | 47.71 | 47.43 | 45.87 | 44.12 | 40.44 | 31.64 | +---+------+------+------+------+------+------+------+------+ |100| 95.64 | 95.50 | 95.17 | 94.39 | 92.52 | 87.73 | 77.25 | 82.38 | +---+------+------+------+------+------+------+------+------+

Table 12: cubic Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+------+------+------+------+------+------+------+ | | 0 | 2 | 5 | 10 | 20 | 40 | 80 | 160 | +===+======+======+======+======+======+======+======+======+ |0.5| 0.48 | 0.48 | 0.48 | 0.48 | 0.48 | 0.47 | 0.45 | 0.40 | +---+------+------+------+------+------+------+------+------+ |1 | 0.96 | 0.96 | 0.96 | 0.96 | 0.95 | 0.94 | 0.89 | 0.82 | +---+------+------+------+------+------+------+------+------+ |2 | 1.91 | 1.91 | 1.91 | 1.90 | 1.84 | 1.85 | 1.64 | 1.54 | +---+------+------+------+------+------+------+------+------+ |5 | 4.78 | 4.78 | 4.78 | 4.78 | 4.64 | 3.91 | 3.88 | 3.29 | +---+------+------+------+------+------+------+------+------+ |10 | 9.56 | 9.56 | 9.56 | 9.52 | 8.90 | 8.04 | 7.44 | 6.43 | +---+------+------+------+------+------+------+------+------+ |25 | 23.91 | 23.91 | 23.89 | 23.38 | 21.65 | 19.51 | 17.63 | 14.50 | +---+------+------+------+------+------+------+------+------+ |50 | 47.82 | 47.81 | 47.74 | 46.21 | 42.17 | 38.31 | 31.97 | 28.85 | +---+------+------+------+------+------+------+------+------+ |100| 95.63 | 95.53 | 94.91 | 90.65 | 83.51 | 69.92 | 63.70 | 53.89 | +---+------+------+------+------+------+------+------+------+

Table 13: reno Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 16] Internet-Draft sceonetwotests July 2019

+---+------+------+------+------+------+------+------+------+ | | 0 | 2 | 5 | 10 | 20 | 40 | 80 | 160 | +===+======+======+======+======+======+======+======+======+ |0.5| 0.48 | 0.48 | 0.48 | 0.48 | 0.48 | 0.42 | 0.36 | 0.32 | +---+------+------+------+------+------+------+------+------+ |1 | 0.96 | 0.96 | 0.96 | 0.96 | 0.84 | 0.73 | 0.61 | 0.54 | +---+------+------+------+------+------+------+------+------+ |2 | 1.91 | 1.91 | 1.89 | 1.66 | 1.42 | 1.10 | 0.90 | 0.73 | +---+------+------+------+------+------+------+------+------+ |5 | 4.78 | 4.52 | 4.07 | 3.64 | 3.75 | 3.75 | 3.85 | 3.37 | +---+------+------+------+------+------+------+------+------+ |10 | 9.56 | 8.85 | 8.17 | 8.09 | 8.44 | 8.66 | 8.60 | 8.11 | +---+------+------+------+------+------+------+------+------+ |25 | 23.90 | 23.84 | 23.74 | 23.62 | 23.47 | 23.09 | 22.54 | 20.97 | +---+------+------+------+------+------+------+------+------+ |50 | 47.81 | 47.81 | 47.78 | 47.66 | 47.25 | 46.74 | 43.70 | 42.44 | +---+------+------+------+------+------+------+------+------+ |100| 95.64 | 95.58 | 95.30 | 94.98 | 93.82 | 91.37 | 87.46 | 82.83 | +---+------+------+------+------+------+------+------+------+

Table 14: reno-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+------+------+------+------+------+------+------+ | | 0 | 2 | 5 | 10 | 20 | 40 | 80 | 160 | +===+======+======+======+======+======+======+======+======+ |0.5| 0.48 | 0.48 | 0.48 | 0.48 | 0.48 | 0.48 | 0.47 | 0.47 | +---+------+------+------+------+------+------+------+------+ |1 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.95 | 0.93 | 0.91 | +---+------+------+------+------+------+------+------+------+ |2 | 1.91 | 1.91 | 1.91 | 1.91 | 1.91 | 1.88 | 1.78 | 1.78 | +---+------+------+------+------+------+------+------+------+ |5 | 4.78 | 4.78 | 4.78 | 4.77 | 4.74 | 4.68 | 4.54 | 4.10 | +---+------+------+------+------+------+------+------+------+ |10 | 9.56 | 9.56 | 9.55 | 9.52 | 9.41 | 9.23 | 8.86 | 7.23 | +---+------+------+------+------+------+------+------+------+ |25 | 23.91 | 23.90 | 23.84 | 23.68 | 23.25 | 22.34 | 20.33 | 15.86 | +---+------+------+------+------+------+------+------+------+ |50 | 47.82 | 47.75 | 47.55 | 46.96 | 45.33 | 43.93 | 34.63 | 31.54 | +---+------+------+------+------+------+------+------+------+ |100| 95.62 | 94.89 | 93.59 | 91.78 | 90.26 | 81.84 | 70.42 | 57.89 | +---+------+------+------+------+------+------+------+------+

Table 15: dctcp Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 17] Internet-Draft sceonetwotests July 2019

+---+------+------+------+------+------+------+------+------+ | | 0 | 2 | 5 | 10 | 20 | 40 | 80 | 160 | +===+======+======+======+======+======+======+======+======+ |0.5| 0.48 | 0.48 | 0.48 | 0.48 | 0.48 | 0.42 | 0.36 | 0.27 | +---+------+------+------+------+------+------+------+------+ |1 | 0.96 | 0.96 | 0.96 | 0.96 | 0.84 | 0.69 | 0.51 | 0.34 | +---+------+------+------+------+------+------+------+------+ |2 | 1.91 | 1.91 | 1.88 | 1.65 | 1.34 | 0.87 | 0.65 | 0.37 | +---+------+------+------+------+------+------+------+------+ |5 | 4.78 | 4.59 | 4.03 | 3.42 | 2.87 | 1.74 | 1.28 | 1.00 | +---+------+------+------+------+------+------+------+------+ |10 | 9.56 | 8.69 | 7.58 | 6.87 | 6.75 | 6.66 | 6.15 | 5.89 | +---+------+------+------+------+------+------+------+------+ |25 | 23.75 | 22.39 | 20.56 | 19.51 | 19.39 | 19.02 | 18.66 | 18.62 | +---+------+------+------+------+------+------+------+------+ |50 | 47.81 | 47.68 | 47.27 | 46.54 | 45.59 | 44.52 | 41.77 | 40.53 | +---+------+------+------+------+------+------+------+------+ |100| 95.64 | 95.59 | 95.34 | 94.90 | 94.30 | 90.53 | 85.48 | 82.53 | +---+------+------+------+------+------+------+------+------+

Table 16: dctcp-sce Mean TCP Throughput (Mbit); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

A.2. One-Flow TCP RTT

+---+------+------+------+------+------+------+------+------+ | |0 |2 |5 |10 | 20 | 40 | 80 | 160 | +===+======+======+======+======+======+======+======+======+ |0.5|121.38|120.07|119.91|121.37| 108.48 | 128.70 | 189.50 | 230.34 | +---+------+------+------+------+------+------+------+------+ |1 |68.40 |68.16 |68.10 |69.33 | 72.97 | 88.11 | 116.66 | 191.18 | +---+------+------+------+------+------+------+------+------+ |2 |34.80 |35.01 |40.33 |40.17 | 52.74 | 62.05 | 100.71 | 175.00 | +---+------+------+------+------+------+------+------+------+ |5 |13.79 |15.92 |18.43 |23.83 | 31.04 | 49.07 | 87.79 | 167.49 | +---+------+------+------+------+------+------+------+------+ |10 |7.08 |8.95 |12.58 |17.39 | 26.12 | 44.73 | 84.60 | 164.49 | +---+------+------+------+------+------+------+------+------+ |25 |6.36 |8.29 |11.31 |15.33 | 24.06 | 43.37 | 82.68 | 162.72 | +---+------+------+------+------+------+------+------+------+ |50 |5.88 |8.16 |10.70 |14.76 | 23.51 | 42.82 | 82.10 | 161.81 | +---+------+------+------+------+------+------+------+------+ |100|5.99 |8.08 |10.38 |14.42 | 23.69 | 42.90 | 82.16 | 162.01 | +---+------+------+------+------+------+------+------+------+

Table 17: cubic Mean TCP RTT (ms); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 18] Internet-Draft sceonetwotests July 2019

+---+------+------+------+------+------+------+------+------+ | |0 |2 |5 |10 | 20 | 40 | 80 | 160 | +===+======+======+======+======+======+======+======+======+ |0.5|138.77|137.84|138.91|138.92| 132.94 | 140.51 | 170.48 | 235.32 | +---+------+------+------+------+------+------+------+------+ |1 |66.82 |66.68 |68.93 |69.02 | 77.11 | 91.69 | 123.15 | 196.59 | +---+------+------+------+------+------+------+------+------+ |2 |41.00 |41.25 |40.81 |42.24 | 47.24 | 65.51 | 99.34 | 176.74 | +---+------+------+------+------+------+------+------+------+ |5 |16.25 |18.37 |19.18 |25.13 | 32.06 | 49.61 | 87.61 | 167.39 | +---+------+------+------+------+------+------+------+------+ |10 |7.68 |9.86 |13.14 |18.75 | 26.74 | 45.54 | 84.50 | 164.45 | +---+------+------+------+------+------+------+------+------+ |25 |6.27 |8.26 |11.46 |15.22 | 23.65 | 42.84 | 82.59 | 162.69 | +---+------+------+------+------+------+------+------+------+ |50 |5.99 |8.08 |10.34 |14.11 | 22.86 | 42.22 | 81.97 | 162.08 | +---+------+------+------+------+------+------+------+------+ |100|5.87 |7.64 |9.85 |13.62 | 22.64 | 41.96 | 81.73 | 161.76 | +---+------+------+------+------+------+------+------+------+

Table 18: reno Mean TCP RTT (ms); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+------+------+------+------+------+------+------+ | |0 | 2 | 5 | 10 | 20 | 40 | 80 | 160 | +===+=====+======+======+======+======+======+======+======+ |0.5|50.36| 50.33 | 50.48 | 50.33 | 70.57 | 87.95 | 128.25 | 203.39 | +---+-----+------+------+------+------+------+------+------+ |1 |24.52| 24.74 | 26.28 | 35.99 | 44.19 | 63.77 | 104.02 | 183.39 | +---+-----+------+------+------+------+------+------+------+ |2 |11.91| 13.58 | 18.62 | 23.01 | 32.70 | 55.39 | 95.68 | 175.24 | +---+-----+------+------+------+------+------+------+------+ |5 |7.07 | 8.90 | 11.67 | 16.76 | 26.43 | 46.12 | 85.92 | 166.15 | +---+-----+------+------+------+------+------+------+------+ |10 |5.22 | 7.11 | 10.19 | 14.84 | 24.30 | 44.12 | 83.96 | 164.04 | +---+-----+------+------+------+------+------+------+------+ |25 |4.40 | 6.17 | 8.93 | 13.48 | 23.23 | 42.85 | 82.78 | 162.68 | +---+-----+------+------+------+------+------+------+------+ |50 |3.99 | 5.90 | 8.86 | 13.80 | 23.45 | 43.20 | 82.83 | 162.65 | +---+-----+------+------+------+------+------+------+------+ |100|4.00 | 5.97 | 8.81 | 13.68 | 23.59 | 43.38 | 83.14 | 162.91 | +---+-----+------+------+------+------+------+------+------+

Table 19: reno-sce Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 19] Internet-Draft sceonetwotests July 2019

+---+------+------+------+------+------+------+------+------+ | |0 |2 |5 |10 | 20 | 40 | 80 | 160 | +===+======+======+======+======+======+======+======+======+ |0.5|132.20|131.17|131.14|130.92| 130.17 | 157.56 | 199.65 | 260.24 | +---+------+------+------+------+------+------+------+------+ |1 |66.23 |66.69 |66.65 |66.60 | 79.38 | 103.86 | 135.31 | 203.32 | +---+------+------+------+------+------+------+------+------+ |2 |33.84 |34.07 |39.03 |40.24 | 52.76 | 69.25 | 104.09 | 183.85 | +---+------+------+------+------+------+------+------+------+ |5 |13.67 |15.87 |18.50 |23.05 | 32.73 | 53.12 | 89.63 | 168.33 | +---+------+------+------+------+------+------+------+------+ |10 |6.99 |9.06 |12.82 |17.26 | 26.75 | 45.79 | 85.43 | 165.10 | +---+------+------+------+------+------+------+------+------+ |25 |6.57 |8.26 |11.25 |16.07 | 25.65 | 44.11 | 83.45 | 163.30 | +---+------+------+------+------+------+------+------+------+ |50 |6.22 |8.39 |10.98 |15.83 | 25.65 | 45.56 | 82.73 | 162.51 | +---+------+------+------+------+------+------+------+------+ |100|6.31 |8.27 |11.16 |16.12 | 25.88 | 44.90 | 82.43 | 162.05 | +---+------+------+------+------+------+------+------+------+

Table 20: dctcp Mean TCP RTT (ms); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+------+------+------+------+------+------+------+ | |0 | 2 | 5 | 10 | 20 | 40 | 80 | 160 | +===+=====+======+======+======+======+======+======+======+ |0.5|49.63| 49.96 | 49.77 | 49.39 | 69.64 | 87.27 | 124.89 | 205.91 | +---+-----+------+------+------+------+------+------+------+ |1 |24.45| 24.38 | 26.22 | 35.62 | 44.12 | 64.59 | 105.36 | 187.70 | +---+-----+------+------+------+------+------+------+------+ |2 |11.90| 13.78 | 18.17 | 22.66 | 32.83 | 56.55 | 96.72 | 179.88 | +---+-----+------+------+------+------+------+------+------+ |5 |7.02 | 8.60 | 11.42 | 16.60 | 27.16 | 49.53 | 90.11 | 170.64 | +---+-----+------+------+------+------+------+------+------+ |10 |5.14 | 6.95 | 9.97 | 15.01 | 24.59 | 44.28 | 84.15 | 163.98 | +---+-----+------+------+------+------+------+------+------+ |25 |4.10 | 5.74 | 8.31 | 12.94 | 22.72 | 42.48 | 82.47 | 162.52 | +---+-----+------+------+------+------+------+------+------+ |50 |3.59 | 5.24 | 7.99 | 12.71 | 22.28 | 41.99 | 81.92 | 161.88 | +---+-----+------+------+------+------+------+------+------+ |100|3.52 | 5.46 | 8.34 | 13.11 | 22.62 | 42.38 | 82.29 | 162.07 | +---+-----+------+------+------+------+------+------+------+

Table 21: dctcp-sce Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 20] Internet-Draft sceonetwotests July 2019

A.3. Two-Flow TCP Throughput (Cake "flowblind")

+---+------+------+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+======+======+======+ | | cubic | | | | cubic | | | | +---+------+------+------+------+------+------+------+------+ |1 | 0.48 | 0.48 | 0.48 | 0.46 | 0.48 | 0.48 | 0.47 | 0.46 | +---+------+------+------+------+------+------+------+------+ |5 | 2.41 | 2.39 | 2.37 | 2.27 | 2.37 | 2.38 | 2.37 | 2.19 | +---+------+------+------+------+------+------+------+------+ |10 | 4.82 | 4.57 | 4.70 | 3.83 | 4.74 | 4.93 | 4.77 | 4.83 | +---+------+------+------+------+------+------+------+------+ |50 | 22.64 | 23.46 | 24.09 | 22.26 | 25.18 | 24.19 | 23.07 | 20.68 | +---+------+------+------+------+------+------+------+------+ |100| 47.66 | 42.88 | 46.29 | 44.10 | 48.01 | 52.03 | 48.29 | 43.72 | +---+------+------+------+------+------+------+------+------+

Table 22: cubic-cubic Mean TCP Throughput (Mbit); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+------+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+======+======+======+ | | cubic | | | | reno | | | | +---+------+------+------+------+------+------+------+------+ |1 | 0.43 | 0.46 | 0.45 | 0.41 | 0.53 | 0.50 | 0.51 | 0.51 | +---+------+------+------+------+------+------+------+------+ |5 | 2.10 | 2.04 | 2.46 | 2.30 | 2.68 | 2.71 | 2.22 | 2.04 | +---+------+------+------+------+------+------+------+------+ |10 | 4.23 | 4.61 | 4.50 | 4.16 | 5.34 | 4.81 | 4.83 | 4.17 | +---+------+------+------+------+------+------+------+------+ |50 | 22.79 | 24.60 | 23.55 | 26.18 | 25.03 | 23.05 | 23.32 | 14.70 | +---+------+------+------+------+------+------+------+------+ |100| 46.82 | 51.06 | 48.73 | 55.96 | 48.83 | 43.85 | 44.96 | 29.02 | +---+------+------+------+------+------+------+------+------+

Table 23: cubic-reno Mean TCP Throughput (Mbit); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 21] Internet-Draft sceonetwotests July 2019

+---+------+------+------+------+------+-----+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+=====+======+======+ | | cubic | | | | reno-sce | | | | +---+------+------+------+------+------+-----+------+------+ |1 | 0.55 | 0.55 | 0.62 | 0.66 | 0.41 | 0.41| 0.34 | 0.24 | +---+------+------+------+------+------+-----+------+------+ |5 | 2.75 | 3.63 | 3.96 | 3.92 | 2.03 | 1.14| 0.80 | 0.46 | +---+------+------+------+------+------+-----+------+------+ |10 | 5.50 | 7.97 | 8.24 | 7.57 | 4.06 | 1.57| 1.26 | 1.37 | +---+------+------+------+------+------+-----+------+------+ |50 | 43.53 | 44.29 | 42.98 | 39.36 | 4.30 | 3.44| 4.25 | 4.89 | +---+------+------+------+------+------+-----+------+------+ |100| 91.16 | 89.89 | 88.69 | 76.76 | 4.46 | 5.28| 5.77 | 11.79 | +---+------+------+------+------+------+-----+------+------+

Table 24: cubic-reno-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+------+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+======+======+======+ | | cubic | | | | dctcp | | | | +---+------+------+------+------+------+------+------+------+ |1 | 0.48 | 0.47 | 0.40 | 0.33 | 0.48 | 0.48 | 0.56 | 0.60 | +---+------+------+------+------+------+------+------+------+ |5 | 2.30 | 1.53 | 1.49 | 1.70 | 2.49 | 3.25 | 3.26 | 2.87 | +---+------+------+------+------+------+------+------+------+ |10 | 3.69 | 2.77 | 2.81 | 2.99 | 5.88 | 6.77 | 6.68 | 5.96 | +---+------+------+------+------+------+------+------+------+ |50 | 14.99 | 16.93 | 15.25 | 18.99 | 32.85 | 30.62 | 31.72 | 24.58 | +---+------+------+------+------+------+------+------+------+ |100| 31.33 | 30.67 | 33.63 | 32.23 | 64.35 | 63.57 | 59.81 | 55.50 | +---+------+------+------+------+------+------+------+------+

Table 25: cubic-dctcp Mean TCP Throughput (Mbit); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 22] Internet-Draft sceonetwotests July 2019

+---+------+------+------+------+------+-----+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+=====+======+======+ | | cubic | | | | dctcp-sce | | | | +---+------+------+------+------+------+-----+------+------+ |1 | 0.55 | 0.55 | 0.62 | 0.66 | 0.41 | 0.40| 0.34 | 0.24 | +---+------+------+------+------+------+-----+------+------+ |5 | 2.76 | 3.59 | 3.91 | 4.01 | 2.02 | 1.18| 0.85 | 0.42 | +---+------+------+------+------+------+-----+------+------+ |10 | 5.48 | 7.95 | 8.24 | 7.81 | 4.08 | 1.60| 1.22 | 1.01 | +---+------+------+------+------+------+-----+------+------+ |50 | 43.46 | 44.60 | 43.46 | 39.78 | 4.36 | 3.12| 3.75 | 4.29 | +---+------+------+------+------+------+-----+------+------+ |100| 91.37 | 90.47 | 88.47 | 77.27 | 4.28 | 4.69| 5.59 | 7.74 | +---+------+------+------+------+------+-----+------+------+

Table 26: cubic-dctcp-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+------+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+======+======+======+ | | reno | | | | reno | | | | +---+------+------+------+------+------+------+------+------+ |1 | 0.52 | 0.48 | 0.47 | 0.44 | 0.44 | 0.48 | 0.48 | 0.48 | +---+------+------+------+------+------+------+------+------+ |5 | 2.39 | 2.36 | 2.23 | 1.98 | 2.39 | 2.37 | 2.41 | 2.15 | +---+------+------+------+------+------+------+------+------+ |10 | 4.74 | 4.86 | 4.79 | 3.92 | 4.82 | 4.51 | 4.40 | 3.93 | +---+------+------+------+------+------+------+------+------+ |50 | 23.88 | 22.77 | 22.88 | 18.78 | 23.95 | 24.90 | 24.03 | 17.50 | +---+------+------+------+------+------+------+------+------+ |100| 49.89 | 45.68 | 48.07 | 36.99 | 45.76 | 49.17 | 44.65 | 37.37 | +---+------+------+------+------+------+------+------+------+

Table 27: reno-reno Mean TCP Throughput (Mbit); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 23] Internet-Draft sceonetwotests July 2019

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |reno | | | | reno-sce | | | | +---+-----+------+------+------+------+------+------+------+ |1 |0.56 | 0.56 | 0.62 | 0.66 | 0.39 | 0.39 | 0.33 | 0.24 | +---+-----+------+------+------+------+------+------+------+ |5 |2.89 | 3.43 | 3.82 | 3.64 | 1.89 | 1.26 | 0.86 | 0.49 | +---+-----+------+------+------+------+------+------+------+ |10 |5.77 | 7.71 | 7.91 | 6.96 | 3.80 | 1.78 | 1.38 | 1.44 | +---+-----+------+------+------+------+------+------+------+ |50 |43.10| 42.13| 39.08 | 28.40 | 4.72 | 5.18 | 6.99 | 14.62 | +---+-----+------+------+------+------+------+------+------+ |100|90.26| 83.33| 78.25 | 62.70 | 5.40 | 10.72 | 13.52 | 19.41 | +---+-----+------+------+------+------+------+------+------+

Table 28: reno-reno-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+------+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+======+======+======+ | | reno | | | | dctcp | | | | +---+------+------+------+------+------+------+------+------+ |1 | 0.55 | 0.54 | 0.42 | 0.37 | 0.41 | 0.42 | 0.53 | 0.57 | +---+------+------+------+------+------+------+------+------+ |5 | 2.60 | 1.65 | 1.58 | 1.44 | 2.18 | 3.12 | 3.15 | 3.09 | +---+------+------+------+------+------+------+------+------+ |10 | 4.12 | 3.28 | 2.98 | 2.80 | 5.44 | 6.24 | 6.46 | 5.99 | +---+------+------+------+------+------+------+------+------+ |50 | 17.17 | 17.86 | 16.27 | 11.57 | 30.66 | 29.71 | 30.63 | 28.80 | +---+------+------+------+------+------+------+------+------+ |100| 36.09 | 33.70 | 28.55 | 18.60 | 59.58 | 60.61 | 64.04 | 59.50 | +---+------+------+------+------+------+------+------+------+

Table 29: reno-dctcp Mean TCP Throughput (Mbit); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 24] Internet-Draft sceonetwotests July 2019

+---+------+------+------+------+------+----+------+------+ | | 0 | 10 | 20 | 80 | 0 |10 | 20 | 80 | +===+======+======+======+======+======+====+======+======+ | | reno | | | | dctcp-sce | | | | +---+------+------+------+------+------+----+------+------+ |1 | 0.57 | 0.56 | 0.63 | 0.66 | 0.39 |0.39| 0.33 | 0.24 | +---+------+------+------+------+------+----+------+------+ |5 | 2.89 | 3.74 | 3.88 | 3.68 | 1.89 |1.03| 0.83 | 0.47 | +---+------+------+------+------+------+----+------+------+ |10 | 5.70 | 7.72 | 7.94 | 7.03 | 3.86 |1.77| 1.32 | 1.18 | +---+------+------+------+------+------+----+------+------+ |50 | 43.12| 42.73 | 39.77 | 31.55 | 4.70 |4.57| 6.11 | 9.93 | +---+------+------+------+------+------+----+------+------+ |100| 90.46| 84.75 | 79.64 | 63.20 | 5.18 |9.21| 11.03 | 11.65 | +---+------+------+------+------+------+----+------+------+

Table 30: reno-dctcp-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+-----+-----+-----+------+------+------+------+ | | 0 |10 |20 |80 | 0 | 10 | 20 | 80 | +===+======+=====+=====+=====+======+======+======+======+ | | reno-sce | | | | reno-sce | | | | +---+------+-----+-----+-----+------+------+------+------+ |1 | 0.48 |0.48 |0.48 |0.32 | 0.48 | 0.48 | 0.48 | 0.34 | +---+------+-----+-----+-----+------+------+------+------+ |5 | 2.39 |1.99 |1.75 |1.77 | 2.39 | 2.04 | 1.75 | 1.52 | +---+------+-----+-----+-----+------+------+------+------+ |10 | 4.78 |3.80 |3.82 |4.23 | 4.78 | 4.00 | 3.79 | 4.06 | +---+------+-----+-----+-----+------+------+------+------+ |50 | 24.24 |21.17|24.08|24.69| 23.60 | 26.11 | 22.91 | 20.34 | +---+------+-----+-----+-----+------+------+------+------+ |100| 42.25 |47.10|37.69|44.31| 53.39 | 48.22 | 57.14 | 40.92 | +---+------+-----+-----+-----+------+------+------+------+

Table 31: reno-sce-reno-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 25] Internet-Draft sceonetwotests July 2019

+---+------+-----+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+=====+======+======+======+======+======+======+ | | reno-sce | | | | dctcp | | | | +---+------+-----+------+------+------+------+------+------+ |1 | 0.46 | 0.46| 0.31 | 0.19 | 0.50 | 0.50 | 0.65 | 0.75 | +---+------+-----+------+------+------+------+------+------+ |5 | 2.16 | 1.12| 0.78 | 0.38 | 2.63 | 3.66 | 4.00 | 4.20 | +---+------+-----+------+------+------+------+------+------+ |10 | 3.62 | 1.57| 1.15 | 0.90 | 5.94 | 7.98 | 8.35 | 8.33 | +---+------+-----+------+------+------+------+------+------+ |50 | 4.03 | 2.33| 2.98 | 10.00 | 43.81 | 45.32 | 44.04 | 32.95 | +---+------+-----+------+------+------+------+------+------+ |100| 4.38 | 4.10| 3.77 | 15.34 | 91.30 | 89.54 | 88.99 | 67.81 | +---+------+-----+------+------+------+------+------+------+

Table 32: reno-sce-dctcp Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+------+------+------+------+------+----+----+ | | 0 | 10 | 20 | 80 | 0 | 10 |20 |80 | +===+======+======+======+======+======+======+====+====+ | | reno-sce | | | | dctcp-sce | | | | +---+------+------+------+------+------+------+----+----+ |1 | 0.48 | 0.48 | 0.48 | 0.34 | 0.48 | 0.48 |0.48|0.33| +---+------+------+------+------+------+------+----+----+ |5 | 2.39 | 1.97 | 1.65 | 1.78 | 2.39 | 2.05 |1.73|0.96| +---+------+------+------+------+------+------+----+----+ |10 | 4.78 | 3.97 | 3.93 | 5.01 | 4.78 | 3.73 |3.29|2.60| +---+------+------+------+------+------+------+----+----+ |50 | 31.92 | 39.92| 40.62 | 39.76 | 15.90 | 7.24 |5.80|4.70| +---+------+------+------+------+------+------+----+----+ |100| 62.33 | 82.41| 85.50 | 82.05 | 33.30 | 12.61 |9.49|6.82| +---+------+------+------+------+------+------+----+----+

Table 33: reno-sce-dctcp-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 26] Internet-Draft sceonetwotests July 2019

+---+------+------+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+======+======+======+ | | dctcp | | | | dctcp | | | | +---+------+------+------+------+------+------+------+------+ |1 | 0.48 | 0.48 | 0.50 | 0.47 | 0.48 | 0.48 | 0.45 | 0.47 | +---+------+------+------+------+------+------+------+------+ |5 | 2.41 | 2.40 | 2.39 | 2.32 | 2.37 | 2.38 | 2.37 | 2.27 | +---+------+------+------+------+------+------+------+------+ |10 | 4.89 | 4.88 | 4.77 | 4.66 | 4.68 | 4.66 | 4.70 | 4.38 | +---+------+------+------+------+------+------+------+------+ |50 | 23.87 | 23.68 | 25.27 | 19.63 | 23.95 | 23.72 | 21.30 | 21.54 | +---+------+------+------+------+------+------+------+------+ |100| 49.20 | 45.78 | 47.85 | 41.79 | 46.43 | 47.44 | 45.23 | 39.41 | +---+------+------+------+------+------+------+------+------+

Table 34: dctcp-dctcp Mean TCP Throughput (Mbit); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+------+------+------+------+-----+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+=====+======+======+ | | dctcp | | | | dctcp-sce | | | | +---+------+------+------+------+------+-----+------+------+ |1 | 0.50 | 0.50 | 0.63 | 0.75 | 0.45 | 0.46| 0.33 | 0.19 | +---+------+------+------+------+------+-----+------+------+ |5 | 2.62 | 3.67 | 4.00 | 4.26 | 2.16 | 1.12| 0.76 | 0.34 | +---+------+------+------+------+------+-----+------+------+ |10 | 5.98 | 7.98 | 8.38 | 8.47 | 3.59 | 1.56| 1.12 | 0.69 | +---+------+------+------+------+------+-----+------+------+ |50 | 43.80 | 45.41 | 44.22 | 34.01 | 4.02 | 2.21| 2.72 | 9.50 | +---+------+------+------+------+------+-----+------+------+ |100| 91.58 | 89.85 | 89.10 | 69.89 | 4.07 | 3.82| 3.61 | 8.74 | +---+------+------+------+------+------+-----+------+------+

Table 35: dctcp-dctcp-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 27] Internet-Draft sceonetwotests July 2019

+---+------+-----+-----+-----+------+-----+------+------+ | | 0 |10 |20 |80 | 0 |10 | 20 | 80 | +===+======+=====+=====+=====+======+=====+======+======+ | | dctcp-sce | | | | dctcp-sce | | | | +---+------+-----+-----+-----+------+-----+------+------+ |1 | 0.48 |0.48 |0.48 |0.34 | 0.48 |0.48 | 0.48 | 0.33 | +---+------+-----+-----+-----+------+-----+------+------+ |5 | 2.39 |1.99 |1.70 |1.02 | 2.39 |2.04 | 1.66 | 1.01 | +---+------+-----+-----+-----+------+-----+------+------+ |10 | 4.78 |3.78 |3.58 |3.04 | 4.78 |3.71 | 3.23 | 3.33 | +---+------+-----+-----+-----+------+-----+------+------+ |50 | 24.16 |19.48|19.33|18.15| 23.61 |21.42| 19.76 | 20.18 | +---+------+-----+-----+-----+------+-----+------+------+ |100| 48.08 |43.17|54.14|19.58| 47.56 |49.83| 36.85 | 19.25 | +---+------+-----+-----+-----+------+-----+------+------+

Table 36: dctcp-sce-dctcp-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

A.4. Two-Flow TCP Throughput (Cake "flowblind sce-single")

+---+------+------+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+======+======+======+ | | cubic | | | | cubic | | | | +---+------+------+------+------+------+------+------+------+ |1 | 0.48 | 0.48 | 0.48 | 0.42 | 0.48 | 0.48 | 0.48 | 0.51 | +---+------+------+------+------+------+------+------+------+ |5 | 2.42 | 2.31 | 2.44 | 2.21 | 2.36 | 2.46 | 2.29 | 2.30 | +---+------+------+------+------+------+------+------+------+ |10 | 4.83 | 4.95 | 4.62 | 4.15 | 4.74 | 4.59 | 4.88 | 4.63 | +---+------+------+------+------+------+------+------+------+ |50 | 23.88 | 23.31 | 24.19 | 23.67 | 23.95 | 24.34 | 22.99 | 19.38 | +---+------+------+------+------+------+------+------+------+ |100| 50.40 | 47.62 | 48.40 | 38.13 | 45.25 | 47.30 | 46.14 | 48.92 | +---+------+------+------+------+------+------+------+------+

Table 37: cubic-cubic Mean TCP Throughput (Mbit); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 28] Internet-Draft sceonetwotests July 2019

+---+------+------+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+======+======+======+ | | cubic | | | | reno | | | | +---+------+------+------+------+------+------+------+------+ |1 | 0.55 | 0.46 | 0.44 | 0.40 | 0.41 | 0.50 | 0.51 | 0.52 | +---+------+------+------+------+------+------+------+------+ |5 | 2.14 | 2.06 | 2.23 | 2.08 | 2.64 | 2.68 | 2.46 | 2.18 | +---+------+------+------+------+------+------+------+------+ |10 | 4.39 | 4.63 | 4.92 | 4.28 | 5.18 | 4.78 | 4.44 | 4.06 | +---+------+------+------+------+------+------+------+------+ |50 | 22.61 | 24.49 | 25.41 | 29.22 | 25.22 | 23.18 | 21.59 | 12.14 | +---+------+------+------+------+------+------+------+------+ |100| 46.37 | 48.43 | 44.51 | 52.13 | 49.30 | 46.45 | 48.88 | 31.90 | +---+------+------+------+------+------+------+------+------+

Table 38: cubic-reno Mean TCP Throughput (Mbit); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |cubic| | | | reno-sce | | | | +---+-----+------+------+------+------+------+------+------+ |1 |0.45 | 0.46 | 0.51 | 0.48 | 0.50 | 0.50 | 0.44 | 0.42 | +---+-----+------+------+------+------+------+------+------+ |5 |2.24 | 3.07 | 3.78 | 3.24 | 2.54 | 1.68 | 0.97 | 1.13 | +---+-----+------+------+------+------+------+------+------+ |10 |4.16 | 5.59 | 7.04 | 4.21 | 5.40 | 3.78 | 2.36 | 4.18 | +---+-----+------+------+------+------+------+------+------+ |50 |23.95| 24.71| 25.16 | 17.98 | 23.87 | 22.53 | 21.18 | 20.74 | +---+-----+------+------+------+------+------+------+------+ |100|50.40| 66.75| 65.82 | 64.67 | 45.26 | 28.41 | 28.70 | 22.52 | +---+-----+------+------+------+------+------+------+------+

Table 39: cubic-reno-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 29] Internet-Draft sceonetwotests July 2019

+---+------+------+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+======+======+======+ | | cubic | | | | dctcp | | | | +---+------+------+------+------+------+------+------+------+ |1 | 0.40 | 0.39 | 0.37 | 0.30 | 0.56 | 0.57 | 0.59 | 0.65 | +---+------+------+------+------+------+------+------+------+ |5 | 1.87 | 1.38 | 1.25 | 1.57 | 2.91 | 3.40 | 3.50 | 3.03 | +---+------+------+------+------+------+------+------+------+ |10 | 3.37 | 1.88 | 1.56 | 2.00 | 6.20 | 7.66 | 7.89 | 7.04 | +---+------+------+------+------+------+------+------+------+ |50 | 6.74 | 5.79 | 6.91 | 11.76 | 41.11 | 39.67 | 37.24 | 30.38 | +---+------+------+------+------+------+------+------+------+ |100| 11.73 | 12.49 | 11.51 | 22.96 | 83.95 | 71.89 | 69.95 | 60.58 | +---+------+------+------+------+------+------+------+------+

Table 40: cubic-dctcp Mean TCP Throughput (Mbit); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+-----+------+------+------+------+------+------+ | |0 |10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+======+======+======+======+======+======+ | |cubic| | | | dctcp-sce | | | | +---+-----+-----+------+------+------+------+------+------+ |1 |0.44 |0.46 | 0.51 | 0.55 | 0.52 | 0.49 | 0.44 | 0.37 | +---+-----+-----+------+------+------+------+------+------+ |5 |2.40 |3.18 | 3.69 | 3.46 | 2.39 | 1.58 | 1.06 | 0.89 | +---+-----+-----+------+------+------+------+------+------+ |10 |4.02 |6.35 | 7.58 | 5.48 | 5.54 | 3.09 | 1.87 | 2.58 | +---+-----+-----+------+------+------+------+------+------+ |50 |24.54|37.56| 35.51 | 27.95 | 23.29 | 10.16 | 11.77 | 15.42 | +---+-----+-----+------+------+------+------+------+------+ |100|59.52|77.63| 80.96 | 75.79 | 36.12 | 17.42 | 13.35 | 11.62 | +---+-----+-----+------+------+------+------+------+------+

Table 41: cubic-dctcp-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 30] Internet-Draft sceonetwotests July 2019

+---+------+------+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+======+======+======+ | | reno | | | | reno | | | | +---+------+------+------+------+------+------+------+------+ |1 | 0.49 | 0.47 | 0.49 | 0.46 | 0.47 | 0.48 | 0.47 | 0.45 | +---+------+------+------+------+------+------+------+------+ |5 | 2.38 | 2.39 | 2.33 | 1.96 | 2.40 | 2.35 | 2.33 | 2.16 | +---+------+------+------+------+------+------+------+------+ |10 | 4.78 | 4.58 | 4.63 | 3.79 | 4.79 | 4.84 | 4.61 | 4.23 | +---+------+------+------+------+------+------+------+------+ |50 | 24.05 | 23.35 | 24.20 | 19.51 | 23.78 | 24.33 | 22.69 | 16.79 | +---+------+------+------+------+------+------+------+------+ |100| 46.69 | 50.26 | 46.11 | 35.97 | 48.97 | 44.58 | 46.86 | 38.56 | +---+------+------+------+------+------+------+------+------+

Table 42: reno-reno Mean TCP Throughput (Mbit); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |reno | | | | reno-sce | | | | +---+-----+------+------+------+------+------+------+------+ |1 |0.47 | 0.50 | 0.49 | 0.49 | 0.49 | 0.46 | 0.47 | 0.39 | +---+-----+------+------+------+------+------+------+------+ |5 |2.51 | 2.89 | 3.58 | 3.08 | 2.27 | 1.83 | 1.12 | 1.11 | +---+-----+------+------+------+------+------+------+------+ |10 |4.51 | 5.67 | 6.96 | 4.39 | 5.06 | 3.55 | 2.22 | 3.74 | +---+-----+------+------+------+------+------+------+------+ |50 |23.88| 26.49| 26.33 | 15.66 | 23.94 | 20.93 | 19.68 | 21.41 | +---+-----+------+------+------+------+------+------+------+ |100|53.06| 62.99| 58.85 | 48.15 | 42.59 | 31.69 | 33.83 | 38.41 | +---+-----+------+------+------+------+------+------+------+

Table 43: reno-reno-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 31] Internet-Draft sceonetwotests July 2019

+---+------+------+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+======+======+======+ | | reno | | | | dctcp | | | | +---+------+------+------+------+------+------+------+------+ |1 | 0.42 | 0.42 | 0.38 | 0.33 | 0.54 | 0.53 | 0.58 | 0.62 | +---+------+------+------+------+------+------+------+------+ |5 | 2.00 | 1.50 | 1.47 | 1.49 | 2.79 | 3.28 | 3.28 | 3.06 | +---+------+------+------+------+------+------+------+------+ |10 | 3.62 | 2.19 | 1.92 | 2.07 | 5.95 | 7.35 | 7.53 | 6.87 | +---+------+------+------+------+------+------+------+------+ |50 | 7.80 | 6.96 | 6.91 | 8.15 | 40.04 | 38.98 | 37.41 | 31.25 | +---+------+------+------+------+------+------+------+------+ |100| 13.40 | 13.20 | 10.37 | 16.04 | 82.28 | 72.23 | 70.72 | 61.88 | +---+------+------+------+------+------+------+------+------+

Table 44: reno-dctcp Mean TCP Throughput (Mbit); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+-----+------+------+------+------+------+------+ | |0 |10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+======+======+======+======+======+======+ | |reno | | | | dctcp-sce | | | | +---+-----+-----+------+------+------+------+------+------+ |1 |0.47 |0.48 | 0.48 | 0.24 | 0.49 | 0.47 | 0.47 | 0.66 | +---+-----+-----+------+------+------+------+------+------+ |5 |2.50 |3.15 | 3.64 | 2.92 | 2.28 | 1.59 | 1.07 | 1.10 | +---+-----+-----+------+------+------+------+------+------+ |10 |4.53 |6.77 | 7.30 | 5.18 | 5.04 | 2.65 | 1.91 | 2.55 | +---+-----+-----+------+------+------+------+------+------+ |50 |26.20|35.85| 34.81 | 22.99 | 21.62 | 11.46 | 11.45 | 19.58 | +---+-----+-----+------+------+------+------+------+------+ |100|64.59|71.71| 71.03 | 59.15 | 31.06 | 22.61 | 20.74 | 17.50 | +---+-----+-----+------+------+------+------+------+------+

Table 45: reno-dctcp-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 32] Internet-Draft sceonetwotests July 2019

+---+------+-----+-----+-----+------+------+------+------+ | | 0 |10 |20 |80 | 0 | 10 | 20 | 80 | +===+======+=====+=====+=====+======+======+======+======+ | | reno-sce | | | | reno-sce | | | | +---+------+-----+-----+-----+------+------+------+------+ |1 | 0.47 |0.48 |0.50 |0.41 | 0.49 | 0.48 | 0.45 | 0.41 | +---+------+-----+-----+-----+------+------+------+------+ |5 | 2.39 |2.19 |2.04 |1.81 | 2.39 | 2.15 | 1.85 | 1.97 | +---+------+-----+-----+-----+------+------+------+------+ |10 | 4.77 |4.13 |3.90 |3.65 | 4.79 | 4.21 | 4.05 | 4.16 | +---+------+-----+-----+-----+------+------+------+------+ |50 | 23.90 |23.79|21.48|17.40| 23.92 | 23.94 | 25.97 | 21.52 | +---+------+-----+-----+-----+------+------+------+------+ |100| 48.57 |54.33|48.58|47.14| 47.10 | 40.97 | 46.49 | 41.08 | +---+------+-----+-----+-----+------+------+------+------+

Table 46: reno-sce-reno-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+-----+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+=====+======+======+======+======+======+======+ | | reno-sce | | | | dctcp | | | | +---+------+-----+------+------+------+------+------+------+ |1 | 0.41 | 0.41| 0.38 | 0.28 | 0.55 | 0.55 | 0.58 | 0.66 | +---+------+-----+------+------+------+------+------+------+ |5 | 1.87 | 1.39| 1.06 | 0.67 | 2.91 | 3.38 | 3.69 | 3.96 | +---+------+-----+------+------+------+------+------+------+ |10 | 3.69 | 2.43| 2.03 | 1.17 | 5.88 | 7.08 | 7.37 | 7.96 | +---+------+-----+------+------+------+------+------+------+ |50 | 6.49 | 3.69| 3.98 | 9.37 | 41.35 | 41.62 | 39.72 | 30.90 | +---+------+-----+------+------+------+------+------+------+ |100| 7.56 | 6.25| 6.13 | 16.00 | 88.12 | 76.49 | 72.46 | 62.73 | +---+------+-----+------+------+------+------+------+------+

Table 47: reno-sce-dctcp Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 33] Internet-Draft sceonetwotests July 2019

+---+------+-----+-----+-----+------+------+------+------+ | | 0 |10 |20 |80 | 0 | 10 | 20 | 80 | +===+======+=====+=====+=====+======+======+======+======+ | | reno-sce | | | | dctcp-sce | | | | +---+------+-----+-----+-----+------+------+------+------+ |1 | 0.47 |0.47 |0.49 |0.42 | 0.48 | 0.48 | 0.46 | 0.40 | +---+------+-----+-----+-----+------+------+------+------+ |5 | 2.35 |2.10 |1.91 |2.27 | 2.43 | 2.20 | 1.94 | 1.38 | +---+------+-----+-----+-----+------+------+------+------+ |10 | 4.78 |4.37 |4.89 |4.74 | 4.78 | 3.78 | 2.97 | 2.62 | +---+------+-----+-----+-----+------+------+------+------+ |50 | 27.23 |39.56|40.95|25.72| 20.59 | 8.16 | 6.56 | 13.56 | +---+------+-----+-----+-----+------+------+------+------+ |100| 74.00 |82.84|79.48|67.86| 21.65 | 12.49| 15.53 | 11.94 | +---+------+-----+-----+-----+------+------+------+------+

Table 48: reno-sce-dctcp-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+------+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+======+======+======+ | | dctcp | | | | dctcp | | | | +---+------+------+------+------+------+------+------+------+ |1 | 0.48 | 0.48 | 0.48 | 0.47 | 0.48 | 0.48 | 0.47 | 0.47 | +---+------+------+------+------+------+------+------+------+ |5 | 2.38 | 2.35 | 2.35 | 2.30 | 2.40 | 2.43 | 2.42 | 2.32 | +---+------+------+------+------+------+------+------+------+ |10 | 4.74 | 4.76 | 4.73 | 4.33 | 4.82 | 4.79 | 4.72 | 4.63 | +---+------+------+------+------+------+------+------+------+ |50 | 24.18 | 22.36 | 22.40 | 13.88 | 23.63 | 24.65 | 23.05 | 26.04 | +---+------+------+------+------+------+------+------+------+ |100| 42.61 | 32.42 | 25.55 | 43.30 | 53.04 | 56.18 | 59.67 | 35.60 | +---+------+------+------+------+------+------+------+------+

Table 49: dctcp-dctcp Mean TCP Throughput (Mbit); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 34] Internet-Draft sceonetwotests July 2019

+---+------+------+------+------+------+-----+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+=====+======+======+ | | dctcp | | | | dctcp-sce | | | | +---+------+------+------+------+------+-----+------+------+ |1 | 0.56 | 0.56 | 0.58 | 0.68 | 0.40 | 0.41| 0.37 | 0.27 | +---+------+------+------+------+------+-----+------+------+ |5 | 2.89 | 2.38 | 3.66 | 3.91 | 1.90 | 2.54| 1.09 | 0.67 | +---+------+------+------+------+------+-----+------+------+ |10 | 5.97 | 5.93 | 7.55 | 8.00 | 3.60 | 3.58| 1.85 | 1.05 | +---+------+------+------+------+------+-----+------+------+ |50 | 41.35 | 41.32 | 39.50 | 31.56 | 6.47 | 3.79| 3.92 | 9.96 | +---+------+------+------+------+------+-----+------+------+ |100| 89.09 | 76.31 | 73.53 | 63.92 | 6.60 | 6.36| 4.97 | 9.40 | +---+------+------+------+------+------+-----+------+------+

Table 50: dctcp-dctcp-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+-----+-----+-----+------+-----+------+------+ | | 0 |10 |20 |80 | 0 |10 | 20 | 80 | +===+======+=====+=====+=====+======+=====+======+======+ | | dctcp-sce | | | | dctcp-sce | | | | +---+------+-----+-----+-----+------+-----+------+------+ |1 | 0.50 |0.48 |0.49 |0.40 | 0.46 |0.48 | 0.47 | 0.41 | +---+------+-----+-----+-----+------+-----+------+------+ |5 | 2.40 |2.12 |1.90 |1.65 | 2.39 |2.14 | 1.92 | 1.58 | +---+------+-----+-----+-----+------+-----+------+------+ |10 | 4.78 |3.91 |3.45 |3.71 | 4.79 |3.92 | 3.59 | 3.33 | +---+------+-----+-----+-----+------+-----+------+------+ |50 | 23.65 |24.33|22.34|21.85| 24.19 |23.17| 24.15 | 21.08 | +---+------+-----+-----+-----+------+-----+------+------+ |100| 53.54 |59.02|54.68|46.28| 42.10 |36.29| 40.24 | 43.95 | +---+------+-----+-----+-----+------+-----+------+------+

Table 51: dctcp-sce-dctcp-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 35] Internet-Draft sceonetwotests July 2019

A.5. Two-Flow TCP Throughput (Cake "triple-isolate")

+---+------+------+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+======+======+======+ | | cubic | | | | cubic | | | | +---+------+------+------+------+------+------+------+------+ |1 | 0.48 | 0.48 | 0.48 | 0.46 | 0.48 | 0.48 | 0.48 | 0.46 | +---+------+------+------+------+------+------+------+------+ |5 | 2.39 | 2.39 | 2.39 | 2.20 | 2.39 | 2.39 | 2.36 | 2.21 | +---+------+------+------+------+------+------+------+------+ |10 | 4.78 | 4.74 | 4.80 | 4.42 | 4.78 | 4.79 | 4.70 | 4.19 | +---+------+------+------+------+------+------+------+------+ |50 | 23.91 | 23.82 | 23.59 | 20.89 | 23.91 | 23.83 | 23.56 | 22.55 | +---+------+------+------+------+------+------+------+------+ |100| 47.83 | 47.40 | 46.89 | 43.30 | 47.82 | 47.40 | 47.48 | 43.60 | +---+------+------+------+------+------+------+------+------+

Table 52: cubic-cubic Mean TCP Throughput (Mbit); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+------+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+======+======+======+ | | cubic | | | | reno | | | | +---+------+------+------+------+------+------+------+------+ |1 | 0.48 | 0.48 | 0.48 | 0.46 | 0.48 | 0.48 | 0.48 | 0.45 | +---+------+------+------+------+------+------+------+------+ |5 | 2.39 | 2.39 | 2.48 | 2.11 | 2.39 | 2.39 | 2.18 | 2.16 | +---+------+------+------+------+------+------+------+------+ |10 | 4.78 | 4.71 | 4.81 | 4.43 | 4.78 | 4.80 | 4.60 | 4.20 | +---+------+------+------+------+------+------+------+------+ |50 | 23.91 | 24.24 | 24.11 | 23.11 | 23.91 | 23.42 | 22.23 | 17.99 | +---+------+------+------+------+------+------+------+------+ |100| 47.83 | 48.65 | 48.14 | 48.87 | 47.83 | 46.14 | 43.84 | 35.92 | +---+------+------+------+------+------+------+------+------+

Table 53: cubic-reno Mean TCP Throughput (Mbit); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 36] Internet-Draft sceonetwotests July 2019

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |cubic| | | | reno-sce | | | | +---+-----+------+------+------+------+------+------+------+ |1 |0.48 | 0.48 | 0.48 | 0.57 | 0.48 | 0.48 | 0.48 | 0.32 | +---+-----+------+------+------+------+------+------+------+ |5 |2.39 | 2.70 | 2.87 | 3.40 | 2.39 | 2.00 | 1.75 | 1.00 | +---+-----+------+------+------+------+------+------+------+ |10 |4.78 | 5.60 | 5.38 | 4.56 | 4.78 | 3.75 | 3.79 | 4.09 | +---+-----+------+------+------+------+------+------+------+ |50 |23.92| 24.11| 23.51 | 20.30 | 23.90 | 23.56 | 23.72 | 24.27 | +---+-----+------+------+------+------+------+------+------+ |100|47.83| 47.45| 46.41 | 42.39 | 47.81 | 47.71 | 48.22 | 48.49 | +---+-----+------+------+------+------+------+------+------+

Table 54: cubic-reno-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+------+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+======+======+======+ | | cubic | | | | dctcp | | | | +---+------+------+------+------+------+------+------+------+ |1 | 0.48 | 0.48 | 0.48 | 0.44 | 0.48 | 0.48 | 0.48 | 0.50 | +---+------+------+------+------+------+------+------+------+ |5 | 2.39 | 2.38 | 2.34 | 2.16 | 2.39 | 2.40 | 2.41 | 2.35 | +---+------+------+------+------+------+------+------+------+ |10 | 4.78 | 4.68 | 4.77 | 4.38 | 4.78 | 4.85 | 4.71 | 4.44 | +---+------+------+------+------+------+------+------+------+ |50 | 23.91 | 23.86 | 23.36 | 21.81 | 23.91 | 23.68 | 23.58 | 21.10 | +---+------+------+------+------+------+------+------+------+ |100| 47.84 | 47.77 | 46.77 | 47.27 | 47.82 | 46.94 | 47.30 | 40.48 | +---+------+------+------+------+------+------+------+------+

Table 55: cubic-dctcp Mean TCP Throughput (Mbit); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 37] Internet-Draft sceonetwotests July 2019

+---+-----+-----+------+------+------+------+------+------+ | |0 |10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+======+======+======+======+======+======+ | |cubic| | | | dctcp-sce | | | | +---+-----+-----+------+------+------+------+------+------+ |1 |0.48 |0.48 | 0.48 | 0.58 | 0.48 | 0.48 | 0.48 | 0.31 | +---+-----+-----+------+------+------+------+------+------+ |5 |2.39 |2.69 | 2.94 | 3.58 | 2.39 | 2.02 | 1.68 | 0.74 | +---+-----+-----+------+------+------+------+------+------+ |10 |4.78 |5.81 | 6.15 | 6.93 | 4.78 | 3.57 | 3.03 | 1.62 | +---+-----+-----+------+------+------+------+------+------+ |50 |24.02|27.16| 26.07 | 20.95 | 23.80 | 20.14 | 19.16 | 21.20 | +---+-----+-----+------+------+------+------+------+------+ |100|47.85|49.02| 47.67 | 42.50 | 47.80 | 46.02 | 45.94 | 42.97 | +---+-----+-----+------+------+------+------+------+------+

Table 56: cubic-dctcp-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+------+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+======+======+======+ | | reno | | | | reno | | | | +---+------+------+------+------+------+------+------+------+ |1 | 0.48 | 0.48 | 0.48 | 0.46 | 0.48 | 0.48 | 0.48 | 0.46 | +---+------+------+------+------+------+------+------+------+ |5 | 2.39 | 2.37 | 2.31 | 2.04 | 2.39 | 2.35 | 2.33 | 2.01 | +---+------+------+------+------+------+------+------+------+ |10 | 4.78 | 4.81 | 4.60 | 4.25 | 4.78 | 4.74 | 4.60 | 4.33 | +---+------+------+------+------+------+------+------+------+ |50 | 23.91 | 23.77 | 23.07 | 17.70 | 23.91 | 23.77 | 22.92 | 17.72 | +---+------+------+------+------+------+------+------+------+ |100| 47.83 | 47.39 | 45.06 | 37.23 | 47.83 | 47.54 | 44.58 | 35.40 | +---+------+------+------+------+------+------+------+------+

Table 57: reno-reno Mean TCP Throughput (Mbit); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 38] Internet-Draft sceonetwotests July 2019

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |reno | | | | reno-sce | | | | +---+-----+------+------+------+------+------+------+------+ |1 |0.48 | 0.48 | 0.48 | 0.55 | 0.48 | 0.48 | 0.48 | 0.32 | +---+-----+------+------+------+------+------+------+------+ |5 |2.39 | 2.61 | 2.76 | 2.91 | 2.39 | 2.01 | 1.76 | 1.16 | +---+-----+------+------+------+------+------+------+------+ |10 |4.78 | 5.39 | 5.44 | 4.34 | 4.78 | 3.84 | 3.76 | 4.24 | +---+-----+------+------+------+------+------+------+------+ |50 |23.93| 23.81| 22.02 | 17.75 | 23.90 | 23.72 | 24.79 | 25.80 | +---+-----+------+------+------+------+------+------+------+ |100|47.83| 46.09| 42.73 | 35.26 | 47.82 | 49.03 | 50.97 | 50.07 | +---+-----+------+------+------+------+------+------+------+

Table 58: reno-reno-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+------+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+======+======+======+ | | reno | | | | dctcp | | | | +---+------+------+------+------+------+------+------+------+ |1 | 0.48 | 0.48 | 0.48 | 0.43 | 0.48 | 0.48 | 0.48 | 0.50 | +---+------+------+------+------+------+------+------+------+ |5 | 2.39 | 2.30 | 2.27 | 1.88 | 2.39 | 2.47 | 2.47 | 2.44 | +---+------+------+------+------+------+------+------+------+ |10 | 4.78 | 4.73 | 4.66 | 4.19 | 4.78 | 4.79 | 4.78 | 4.30 | +---+------+------+------+------+------+------+------+------+ |50 | 23.91 | 23.45 | 21.71 | 17.98 | 23.91 | 24.13 | 24.75 | 21.47 | +---+------+------+------+------+------+------+------+------+ |100| 47.82 | 46.42 | 43.70 | 35.62 | 47.81 | 48.18 | 49.48 | 42.19 | +---+------+------+------+------+------+------+------+------+

Table 59: reno-dctcp Mean TCP Throughput (Mbit); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 39] Internet-Draft sceonetwotests July 2019

+---+-----+-----+------+------+------+------+------+------+ | |0 |10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+======+======+======+======+======+======+ | |reno | | | | dctcp-sce | | | | +---+-----+-----+------+------+------+------+------+------+ |1 |0.48 |0.48 | 0.48 | 0.54 | 0.48 | 0.48 | 0.48 | 0.33 | +---+-----+-----+------+------+------+------+------+------+ |5 |2.39 |2.60 | 2.81 | 3.25 | 2.39 | 2.02 | 1.69 | 0.75 | +---+-----+-----+------+------+------+------+------+------+ |10 |4.78 |5.65 | 6.18 | 6.09 | 4.78 | 3.60 | 2.97 | 1.90 | +---+-----+-----+------+------+------+------+------+------+ |50 |24.05|27.29| 24.37 | 18.32 | 23.77 | 19.78 | 19.76 | 22.57 | +---+-----+-----+------+------+------+------+------+------+ |100|47.87|47.75| 44.23 | 35.55 | 47.79 | 46.27 | 47.20 | 42.70 | +---+-----+-----+------+------+------+------+------+------+

Table 60: reno-dctcp-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+-----+-----+-----+------+------+------+------+ | | 0 |10 |20 |80 | 0 | 10 | 20 | 80 | +===+======+=====+=====+=====+======+======+======+======+ | | reno-sce | | | | reno-sce | | | | +---+------+-----+-----+-----+------+------+------+------+ |1 | 0.48 |0.48 |0.48 |0.34 | 0.48 | 0.48 | 0.48 | 0.33 | +---+------+-----+-----+-----+------+------+------+------+ |5 | 2.39 |2.05 |1.80 |1.75 | 2.39 | 2.03 | 1.89 | 1.62 | +---+------+-----+-----+-----+------+------+------+------+ |10 | 4.78 |4.15 |4.14 |4.22 | 4.78 | 4.15 | 4.12 | 4.31 | +---+------+-----+-----+-----+------+------+------+------+ |50 | 23.91 |23.66|23.54|22.01| 23.90 | 23.69 | 23.24 | 22.75 | +---+------+-----+-----+-----+------+------+------+------+ |100| 47.82 |47.66|47.52|45.05| 47.82 | 47.66 | 47.54 | 44.97 | +---+------+-----+-----+-----+------+------+------+------+

Table 61: reno-sce-reno-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 40] Internet-Draft sceonetwotests July 2019

+---+------+-----+------+------+------+------+------+------+ | | 0 |10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+=====+======+======+======+======+======+======+ | | reno-sce | | | | dctcp | | | | +---+------+-----+------+------+------+------+------+------+ |1 | 0.48 |0.48 | 0.48 | 0.32 | 0.48 | 0.48 | 0.48 | 0.61 | +---+------+-----+------+------+------+------+------+------+ |5 | 2.39 |1.98 | 1.71 | 0.91 | 2.39 | 2.78 | 2.95 | 3.60 | +---+------+-----+------+------+------+------+------+------+ |10 | 4.78 |3.75 | 3.85 | 3.99 | 4.78 | 5.64 | 5.41 | 4.90 | +---+------+-----+------+------+------+------+------+------+ |50 | 23.91 |23.71| 23.51| 22.56 | 23.91 | 23.99 | 23.57 | 20.79 | +---+------+-----+------+------+------+------+------+------+ |100| 47.83 |48.07| 47.63| 46.93 | 47.83 | 47.03 | 46.40 | 39.48 | +---+------+-----+------+------+------+------+------+------+

Table 62: reno-sce-dctcp Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+-----+-----+-----+------+------+------+------+ | | 0 |10 |20 |80 | 0 | 10 | 20 | 80 | +===+======+=====+=====+=====+======+======+======+======+ | | reno-sce | | | | dctcp-sce | | | | +---+------+-----+-----+-----+------+------+------+------+ |1 | 0.48 |0.48 |0.48 |0.33 | 0.48 | 0.48 | 0.48 | 0.34 | +---+------+-----+-----+-----+------+------+------+------+ |5 | 2.39 |2.01 |1.86 |1.82 | 2.39 | 2.05 | 1.73 | 1.03 | +---+------+-----+-----+-----+------+------+------+------+ |10 | 4.78 |4.24 |4.60 |6.33 | 4.78 | 3.95 | 3.36 | 1.97 | +---+------+-----+-----+-----+------+------+------+------+ |50 | 24.01 |26.91|27.21|23.75| 23.81 | 18.74| 16.94 | 18.75 | +---+------+-----+-----+-----+------+------+------+------+ |100| 47.85 |49.17|49.32|46.19| 47.81 | 45.83| 44.78 | 40.59 | +---+------+-----+-----+-----+------+------+------+------+

Table 63: reno-sce-dctcp-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 41] Internet-Draft sceonetwotests July 2019

+---+------+------+------+------+------+------+------+------+ | | 0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+======+======+======+======+======+======+======+======+ | | dctcp | | | | dctcp | | | | +---+------+------+------+------+------+------+------+------+ |1 | 0.48 | 0.48 | 0.48 | 0.47 | 0.48 | 0.48 | 0.48 | 0.47 | +---+------+------+------+------+------+------+------+------+ |5 | 2.39 | 2.39 | 2.38 | 2.24 | 2.39 | 2.39 | 2.38 | 2.24 | +---+------+------+------+------+------+------+------+------+ |10 | 4.78 | 4.76 | 4.73 | 4.45 | 4.78 | 4.77 | 4.71 | 4.47 | +---+------+------+------+------+------+------+------+------+ |50 | 23.91 | 23.70 | 23.29 | 20.37 | 23.91 | 23.70 | 23.30 | 20.80 | +---+------+------+------+------+------+------+------+------+ |100| 47.73 | 47.13 | 46.71 | 39.67 | 47.73 | 47.17 | 46.73 | 41.12 | +---+------+------+------+------+------+------+------+------+

Table 64: dctcp-dctcp Mean TCP Throughput (Mbit); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+-----+------+------+------+------+------+------+ | |0 |10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+======+======+======+======+======+======+ | |dctcp| | | | dctcp-sce | | | | +---+-----+-----+------+------+------+------+------+------+ |1 |0.48 |0.48 | 0.48 | 0.62 | 0.48 | 0.48 | 0.48 | 0.31 | +---+-----+-----+------+------+------+------+------+------+ |5 |2.39 |2.77 | 3.00 | 3.86 | 2.39 | 1.98 | 1.67 | 0.64 | +---+-----+-----+------+------+------+------+------+------+ |10 |4.78 |5.76 | 6.17 | 6.85 | 4.78 | 3.61 | 3.10 | 1.74 | +---+-----+-----+------+------+------+------+------+------+ |50 |24.01|27.30| 26.52 | 20.66 | 23.81 | 19.96 | 19.56 | 19.92 | +---+-----+-----+------+------+------+------+------+------+ |100|47.85|48.63| 48.11 | 40.69 | 47.77 | 46.10 | 45.60 | 38.33 | +---+-----+-----+------+------+------+------+------+------+

Table 65: dctcp-dctcp-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 42] Internet-Draft sceonetwotests July 2019

+---+------+-----+-----+-----+------+-----+------+------+ | | 0 |10 |20 |80 | 0 |10 | 20 | 80 | +===+======+=====+=====+=====+======+=====+======+======+ | | dctcp-sce | | | | dctcp-sce | | | | +---+------+-----+-----+-----+------+-----+------+------+ |1 | 0.48 |0.48 |0.48 |0.34 | 0.48 |0.48 | 0.48 | 0.34 | +---+------+-----+-----+-----+------+-----+------+------+ |5 | 2.39 |2.04 |1.71 |1.09 | 2.39 |2.03 | 1.76 | 1.06 | +---+------+-----+-----+-----+------+-----+------+------+ |10 | 4.78 |3.91 |3.55 |2.97 | 4.78 |4.01 | 3.59 | 2.84 | +---+------+-----+-----+-----+------+-----+------+------+ |50 | 23.88 |21.37|20.20|19.68| 23.91 |21.09| 20.51 | 20.44 | +---+------+-----+-----+-----+------+-----+------+------+ |100| 47.82 |45.19|43.83|36.16| 47.82 |46.37| 44.59 | 37.07 | +---+------+-----+-----+-----+------+-----+------+------+

Table 66: dctcp-sce-dctcp-sce Mean TCP Throughput (Mbit); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

A.6. Two-Flow TCP RTT (Cake "flowblind")

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |cubic| | | | cubic | | | | +---+-----+------+------+------+------+------+------+------+ |1 |70.73| 69.97 | 83.70 | 139.06 | 69.07 | 70.38 | 86.15 | 140.64 | +---+-----+------+------+------+------+------+------+------+ |5 |14.09| 25.54 | 35.72 | 91.62 | 14.26 | 25.99 | 35.59 | 91.28 | +---+-----+------+------+------+------+------+------+------+ |10 |7.88 | 18.69 | 28.67 | 86.54 | 7.79 | 18.46 | 28.60 | 85.74 | +---+-----+------+------+------+------+------+------+------+ |50 |6.53 | 16.30 | 25.55 | 82.98 | 6.47 | 16.37 | 25.66 | 83.06 | +---+-----+------+------+------+------+------+------+------+ |100|6.35 | 16.40 | 25.49 | 83.27 | 6.43 | 16.29 | 25.57 | 83.36 | +---+-----+------+------+------+------+------+------+------+

Table 67: cubic-cubic Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 43] Internet-Draft sceonetwotests July 2019

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |cubic| | | | reno | | | | +---+-----+------+------+------+------+------+------+------+ |1 |65.97| 71.14 | 83.63 | 139.14 | 66.25 | 72.11 | 87.70 | 140.12 | +---+-----+------+------+------+------+------+------+------+ |5 |14.00| 25.94 | 35.16 | 91.73 | 14.87 | 25.03 | 35.98 | 92.22 | +---+-----+------+------+------+------+------+------+------+ |10 |7.60 | 18.82 | 28.77 | 86.37 | 7.99 | 19.39 | 29.11 | 86.39 | +---+-----+------+------+------+------+------+------+------+ |50 |6.43 | 16.26 | 25.45 | 82.72 | 6.41 | 16.38 | 25.46 | 83.06 | +---+-----+------+------+------+------+------+------+------+ |100|6.31 | 16.20 | 25.09 | 83.07 | 6.47 | 16.09 | 25.11 | 83.19 | +---+-----+------+------+------+------+------+------+------+

Table 68: cubic-reno Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+-----+------+------+------+------+------+------+ | |0 |10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+======+======+======+======+======+======+ | |cubic| | | | reno-sce | | | | +---+-----+-----+------+------+------+------+------+------+ |1 |71.98|73.29| 93.38| 127.68 | 61.37 | 61.69 | 71.07 | 121.98 | +---+-----+-----+------+------+------+------+------+------+ |5 |14.58|25.06| 33.50| 88.85 | 11.80 | 21.16 | 32.19 | 94.96 | +---+-----+-----+------+------+------+------+------+------+ |10 |7.70 |17.86| 27.34| 85.73 | 6.98 | 18.59 | 30.43 | 90.51 | +---+-----+-----+------+------+------+------+------+------+ |50 |5.95 |15.00| 24.47| 83.61 | 6.01 | 16.71 | 26.93 | 86.54 | +---+-----+-----+------+------+------+------+------+------+ |100|6.01 |14.78| 24.44| 83.56 | 6.06 | 15.99 | 26.21 | 85.14 | +---+-----+-----+------+------+------+------+------+------+

Table 69: cubic-reno-sce Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 44] Internet-Draft sceonetwotests July 2019

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |cubic| | | | dctcp | | | | +---+-----+------+------+------+------+------+------+------+ |1 |61.63| 62.74 | 79.63 | 144.41 | 81.28 | 81.99 | 88.30 | 142.69 | +---+-----+------+------+------+------+------+------+------+ |5 |12.75| 27.02 | 38.94 | 94.68 | 18.98 | 24.23 | 33.58 | 92.59 | +---+-----+------+------+------+------+------+------+------+ |10 |7.19 | 20.98 | 30.61 | 87.70 | 8.12 | 17.94 | 27.88 | 86.25 | +---+-----+------+------+------+------+------+------+------+ |50 |6.84 | 16.43 | 26.24 | 83.90 | 6.38 | 16.05 | 26.01 | 84.39 | +---+-----+------+------+------+------+------+------+------+ |100|6.49 | 16.34 | 26.42 | 85.08 | 6.46 | 16.42 | 26.19 | 84.83 | +---+-----+------+------+------+------+------+------+------+

Table 70: cubic-dctcp Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+-----+-----+------+------+------+------+------+ | |0 |10 |20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+=====+======+======+======+======+======+ | |cubic| | | | dctcp-sce | | | | +---+-----+-----+-----+------+------+------+------+------+ |1 |73.64|72.10|92.35| 127.57 | 60.95 | 63.18 | 73.14 | 117.81 | +---+-----+-----+-----+------+------+------+------+------+ |5 |14.10|23.88|32.68| 89.27 | 11.73 | 21.25 | 32.24 | 95.10 | +---+-----+-----+-----+------+------+------+------+------+ |10 |7.99 |17.94|27.11| 85.58 | 6.94 | 18.23 | 30.11 | 91.01 | +---+-----+-----+-----+------+------+------+------+------+ |50 |5.91 |14.97|24.16| 83.11 | 5.97 | 16.67 | 26.92 | 86.08 | +---+-----+-----+-----+------+------+------+------+------+ |100|6.13 |14.98|24.27| 82.93 | 6.08 | 16.17 | 26.17 | 84.85 | +---+-----+-----+-----+------+------+------+------+------+

Table 71: cubic-dctcp-sce Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 45] Internet-Draft sceonetwotests July 2019

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |reno | | | | reno | | | | +---+-----+------+------+------+------+------+------+------+ |1 |64.31| 68.79 | 84.19 | 141.79 | 66.79 | 71.57 | 83.55 | 140.03 | +---+-----+------+------+------+------+------+------+------+ |5 |14.46| 25.39 | 35.62 | 92.66 | 14.61 | 25.49 | 35.57 | 92.29 | +---+-----+------+------+------+------+------+------+------+ |10 |8.15 | 18.91 | 29.00 | 86.57 | 8.08 | 18.83 | 29.11 | 86.26 | +---+-----+------+------+------+------+------+------+------+ |50 |6.49 | 16.43 | 24.97 | 82.44 | 6.44 | 16.23 | 25.06 | 82.48 | +---+-----+------+------+------+------+------+------+------+ |100|6.43 | 16.20 | 24.68 | 82.39 | 6.50 | 16.18 | 24.91 | 82.39 | +---+-----+------+------+------+------+------+------+------+

Table 72: reno-reno Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+-----+------+------+------+------+------+------+ | |0 |10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+======+======+======+======+======+======+ | |reno | | | | reno-sce | | | | +---+-----+-----+------+------+------+------+------+------+ |1 |68.15|70.79| 91.27| 130.38 | 62.79 | 62.52 | 75.28 | 122.99 | +---+-----+-----+------+------+------+------+------+------+ |5 |14.95|23.90| 34.12| 89.47 | 12.89 | 21.95 | 33.51 | 95.43 | +---+-----+-----+------+------+------+------+------+------+ |10 |8.05 |18.27| 28.12| 85.51 | 7.22 | 18.71 | 29.40 | 90.04 | +---+-----+-----+------+------+------+------+------+------+ |50 |5.99 |14.84| 24.17| 83.36 | 6.09 | 16.08 | 25.91 | 84.37 | +---+-----+-----+------+------+------+------+------+------+ |100|6.19 |14.61| 24.02| 83.03 | 6.05 | 15.50 | 25.06 | 83.81 | +---+-----+-----+------+------+------+------+------+------+

Table 73: reno-reno-sce Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 46] Internet-Draft sceonetwotests July 2019

+---+-----+-----+------+------+------+------+------+------+ | |0 |10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+======+======+======+======+======+======+ | |reno | | | | dctcp | | | | +---+-----+-----+------+------+------+------+------+------+ |1 |59.96|62.40| 83.02 | 144.86 | 114.59 | 108.05 | 87.97 | 143.70 | +---+-----+-----+------+------+------+------+------+------+ |5 |12.78|27.64| 38.54 | 95.38 | 22.21 | 24.00 | 33.12 | 92.06 | +---+-----+-----+------+------+------+------+------+------+ |10 |8.10 |20.18| 30.66 | 88.42 | 8.31 | 18.10 | 27.56 | 86.64 | +---+-----+-----+------+------+------+------+------+------+ |50 |6.87 |16.58| 26.04 | 83.56 | 6.48 | 16.36 | 25.81 | 83.38 | +---+-----+-----+------+------+------+------+------+------+ |100|6.38 |16.54| 26.00 | 84.56 | 6.48 | 16.43 | 25.93 | 83.75 | +---+-----+-----+------+------+------+------+------+------+

Table 74: reno-dctcp Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+-----+-----+------+------+------+------+------+ | |0 |10 |20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+=====+======+======+======+======+======+ | |reno | | | | dctcp-sce | | | | +---+-----+-----+-----+------+------+------+------+------+ |1 |67.25|70.19|93.80| 131.16 | 63.59 | 63.29 | 75.04 | 120.28 | +---+-----+-----+-----+------+------+------+------+------+ |5 |15.01|26.28|34.17| 89.37 | 12.68 | 23.72 | 33.43 | 94.57 | +---+-----+-----+-----+------+------+------+------+------+ |10 |7.89 |18.30|27.91| 85.30 | 7.11 | 18.84 | 29.84 | 90.39 | +---+-----+-----+-----+------+------+------+------+------+ |50 |6.09 |14.74|23.74| 82.58 | 6.07 | 16.17 | 25.95 | 84.17 | +---+-----+-----+-----+------+------+------+------+------+ |100|5.78 |14.32|23.62| 82.36 | 5.95 | 15.20 | 24.81 | 83.98 | +---+-----+-----+-----+------+------+------+------+------+

Table 75: reno-dctcp-sce Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 47] Internet-Draft sceonetwotests July 2019

+---+------+-----+-----+------+------+-----+-----+------+ | | 0 |10 |20 | 80 | 0 |10 |20 | 80 | +===+======+=====+=====+======+======+=====+=====+======+ | | reno-sce | | | | reno-sce | | | | +---+------+-----+-----+------+------+-----+-----+------+ |1 | 49.45 |50.66|50.71| 114.04 | 50.74 |50.87|50.54| 113.68 | +---+------+-----+-----+------+------+-----+-----+------+ |5 | 9.58 |19.47|30.20| 88.97 | 9.66 |19.40|30.39| 89.81 | +---+------+-----+-----+------+------+-----+-----+------+ |10 | 6.55 |16.99|27.00| 85.35 | 6.55 |16.61|26.91| 85.29 | +---+------+-----+-----+------+------+-----+-----+------+ |50 | 4.46 |13.52|23.27| 82.93 | 4.54 |13.35|23.26| 83.11 | +---+------+-----+-----+------+------+-----+-----+------+ |100| 4.40 |14.06|23.83| 82.74 | 4.21 |14.07|23.74| 82.71 | +---+------+-----+-----+------+------+-----+-----+------+

Table 76: reno-sce-reno-sce Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+-----+-----+------+------+------+------+------+ | | 0 |10 |20 | 80 | 0 | 10 | 20 | 80 | +===+======+=====+=====+======+======+======+======+======+ | | reno-sce | | | | dctcp| | | | +---+------+-----+-----+------+------+------+------+------+ |1 | 53.35 |54.29|82.98| 132.00 | 93.33| 94.62 | 84.60 | 141.33 | +---+------+-----+-----+------+------+------+------+------+ |5 | 11.15 |21.45|31.72| 93.81 | 18.29| 23.96 | 33.21 | 90.14 | +---+------+-----+-----+------+------+------+------+------+ |10 | 7.04 |18.09|30.04| 92.23 | 8.00 | 17.48 | 27.41 | 86.25 | +---+------+-----+-----+------+------+------+------+------+ |50 | 6.15 |16.43|27.42| 85.57 | 6.32 | 15.91 | 25.72 | 83.84 | +---+------+-----+-----+------+------+------+------+------+ |100| 6.29 |16.54|26.52| 85.06 | 6.41 | 16.26 | 26.07 | 83.57 | +---+------+-----+-----+------+------+------+------+------+

Table 77: reno-sce-dctcp Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 48] Internet-Draft sceonetwotests July 2019

+---+------+-----+-----+------+------+-----+-----+------+ | | 0 |10 |20 | 80 | 0 |10 |20 | 80 | +===+======+=====+=====+======+======+=====+=====+======+ | | reno-sce | | | | dctcp-sce | | | | +---+------+-----+-----+------+------+-----+-----+------+ |1 | 49.87 |51.12|50.45| 113.39| 50.23 |50.54|50.13| 112.50 | +---+------+-----+-----+------+------+-----+-----+------+ |5 | 9.57 |19.48|30.47| 88.94 | 9.64 |19.11|29.76| 91.18 | +---+------+-----+-----+------+------+-----+-----+------+ |10 | 6.58 |16.86|27.26| 84.81 | 6.63 |16.66|27.39| 85.95 | +---+------+-----+-----+------+------+-----+-----+------+ |50 | 4.20 |12.98|22.81| 82.45 | 4.59 |14.89|24.87| 84.83 | +---+------+-----+-----+------+------+-----+-----+------+ |100| 4.15 |13.47|23.29| 82.64 | 4.11 |14.16|24.11| 83.93 | +---+------+-----+-----+------+------+-----+-----+------+

Table 78: reno-sce-dctcp-sce Mean TCP RTT (ms); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |dctcp| | | | dctcp | | | | +---+-----+------+------+------+------+------+------+------+ |1 |94.39| 94.71 | 87.26 | 145.73 | 95.80 | 95.81 | 92.26 | 144.97 | +---+-----+------+------+------+------+------+------+------+ |5 |18.87| 25.91 | 35.08 | 93.41 | 19.48 | 26.10 | 34.76 | 93.83 | +---+-----+------+------+------+------+------+------+------+ |10 |9.29 | 18.56 | 27.94 | 87.31 | 9.40 | 18.65 | 28.15 | 87.12 | +---+-----+------+------+------+------+------+------+------+ |50 |6.78 | 16.54 | 26.18 | 84.83 | 6.74 | 16.48 | 26.21 | 84.80 | +---+-----+------+------+------+------+------+------+------+ |100|6.69 | 16.43 | 25.89 | 85.15 | 6.75 | 16.30 | 25.94 | 85.35 | +---+-----+------+------+------+------+------+------+------+

Table 79: dctcp-dctcp Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 49] Internet-Draft sceonetwotests July 2019

+---+-----+-----+-----+------+------+------+------+------+ | |0 |10 |20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+=====+======+======+======+======+======+ | |dctcp| | | | dctcp-sce | | | | +---+-----+-----+-----+------+------+------+------+------+ |1 |91.81|93.17|83.18| 141.45 | 53.62 | 53.10 | 76.65 | 133.28 | +---+-----+-----+-----+------+------+------+------+------+ |5 |18.22|23.79|33.56| 91.14 | 11.04 | 21.38 | 31.87 | 93.16 | +---+-----+-----+-----+------+------+------+------+------+ |10 |8.06 |17.46|27.21| 86.24 | 7.25 | 18.04 | 29.56 | 92.76 | +---+-----+-----+-----+------+------+------+------+------+ |50 |6.14 |15.92|25.60| 83.50 | 6.17 | 16.47 | 27.02 | 86.00 | +---+-----+-----+-----+------+------+------+------+------+ |100|6.35 |16.04|25.95| 83.06 | 6.43 | 16.35 | 26.71 | 85.56 | +---+-----+-----+-----+------+------+------+------+------+

Table 80: dctcp-dctcp-sce Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+-----+-----+------+------+-----+-----+------+ | | 0 |10 |20 |80 | 0 |10 |20 | 80 | +===+======+=====+=====+======+======+=====+=====+======+ | | dctcp-sce | | | | dctcp-sce | | | | +---+------+-----+-----+------+------+-----+-----+------+ |1 | 48.67 |49.51|50.22|115.03| 49.42 |49.95|49.96| 113.68 | +---+------+-----+-----+------+------+-----+-----+------+ |5 | 9.56 |19.62|29.79|91.63 | 9.63 |19.19|30.02| 91.44 | +---+------+-----+-----+------+------+-----+-----+------+ |10 | 6.52 |16.78|27.09|85.64 | 6.61 |16.65|27.42| 85.73 | +---+------+-----+-----+------+------+-----+-----+------+ |50 | 4.16 |13.13|22.39|82.21 | 4.18 |12.96|22.47| 82.09 | +---+------+-----+-----+------+------+-----+-----+------+ |100| 3.90 |13.02|22.61|82.16 | 3.97 |12.94|22.72| 82.04 | +---+------+-----+-----+------+------+-----+-----+------+

Table 81: dctcp-sce-dctcp-sce Mean TCP RTT (ms); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

A.7. Two-Flow TCP RTT (Cake "flowblind sce-single")

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |cubic| | | | cubic | | | | +---+-----+------+------+------+------+------+------+------+ |1 |68.26| 69.38 | 83.80 | 142.98 | 69.18 | 68.64 | 83.08 | 142.21 |

Heist, et al. Expires 4 January 2020 [Page 50] Internet-Draft sceonetwotests July 2019

+---+-----+------+------+------+------+------+------+------+ |5 |13.86| 25.36 | 35.54 | 92.39 | 13.92 | 24.90 | 36.61 | 92.30 | +---+-----+------+------+------+------+------+------+------+ |10 |8.39 | 18.95 | 28.76 | 86.67 | 8.19 | 19.22 | 28.77 | 86.23 | +---+-----+------+------+------+------+------+------+------+ |50 |6.52 | 16.30 | 25.85 | 83.11 | 6.59 | 16.26 | 25.84 | 83.18 | +---+-----+------+------+------+------+------+------+------+ |100|6.42 | 16.49 | 25.57 | 83.19 | 6.40 | 16.38 | 25.65 | 83.22 | +---+-----+------+------+------+------+------+------+------+

Table 82: cubic-cubic Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |cubic| | | | reno | | | | +---+-----+------+------+------+------+------+------+------+ |1 |65.10| 69.82 | 83.36 | 144.19 | 64.96 | 69.20 | 85.01 | 144.48 | +---+-----+------+------+------+------+------+------+------+ |5 |13.62| 25.89 | 35.86 | 92.45 | 14.26 | 25.25 | 36.40 | 93.45 | +---+-----+------+------+------+------+------+------+------+ |10 |7.82 | 18.99 | 28.74 | 86.24 | 8.23 | 19.64 | 29.36 | 86.42 | +---+-----+------+------+------+------+------+------+------+ |50 |6.57 | 16.15 | 25.30 | 82.60 | 6.53 | 16.17 | 25.47 | 83.21 | +---+-----+------+------+------+------+------+------+------+ |100|6.46 | 16.14 | 24.89 | 83.04 | 6.54 | 16.23 | 24.88 | 83.28 | +---+-----+------+------+------+------+------+------+------+

Table 83: cubic-reno Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 51] Internet-Draft sceonetwotests July 2019

+---+-----+-----+------+------+------+------+------+------+ | |0 |10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+======+======+======+======+======+======+ | |cubic| | | | reno-sce | | | | +---+-----+-----+------+------+------+------+------+------+ |1 |67.62|68.51| 83.34| 138.15 | 68.37 | 69.98 | 84.17 | 139.38 | +---+-----+-----+------+------+------+------+------+------+ |5 |13.56|24.05| 34.64| 91.07 | 13.81 | 24.46 | 37.77 | 96.38 | +---+-----+-----+------+------+------+------+------+------+ |10 |8.16 |18.42| 28.22| 86.47 | 8.07 | 19.70 | 31.56 | 86.75 | +---+-----+-----+------+------+------+------+------+------+ |50 |6.44 |15.91| 25.22| 82.90 | 6.81 | 16.17 | 25.55 | 82.74 | +---+-----+-----+------+------+------+------+------+------+ |100|6.47 |16.73| 26.43| 84.25 | 6.62 | 16.63 | 26.62 | 84.78 | +---+-----+-----+------+------+------+------+------+------+

Table 84: cubic-reno-sce Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |cubic| | | | dctcp | | | | +---+-----+------+------+------+------+------+------+------+ |1 |64.14| 64.56 | 77.80 | 145.02 | 65.75 | 64.93 | 80.94 | 142.91 | +---+-----+------+------+------+------+------+------+------+ |5 |13.23| 25.90 | 38.90 | 96.18 | 13.84 | 23.58 | 33.08 | 93.08 | +---+-----+------+------+------+------+------+------+------+ |10 |7.73 | 21.30 | 34.10 | 90.52 | 7.86 | 18.08 | 27.45 | 87.45 | +---+-----+------+------+------+------+------+------+------+ |50 |7.93 | 18.01 | 28.17 | 86.63 | 7.02 | 16.02 | 26.69 | 85.82 | +---+-----+------+------+------+------+------+------+------+ |100|7.66 | 17.36 | 27.50 | 86.24 | 6.87 | 16.47 | 26.59 | 85.79 | +---+-----+------+------+------+------+------+------+------+

Table 85: cubic-dctcp Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 52] Internet-Draft sceonetwotests July 2019

+---+-----+-----+-----+------+------+------+------+------+ | |0 |10 |20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+=====+======+======+======+======+======+ | |cubic| | | | dctcp-sce | | | | +---+-----+-----+-----+------+------+------+------+------+ |1 |67.09|68.20|83.79| 137.00 | 66.85 | 68.78 | 84.89 | 142.55 | +---+-----+-----+-----+------+------+------+------+------+ |5 |14.22|24.20|34.59| 90.97 | 13.60 | 24.91 | 36.51 | 96.69 | +---+-----+-----+-----+------+------+------+------+------+ |10 |8.16 |18.76|28.16| 86.07 | 8.16 | 20.29 | 31.97 | 88.89 | +---+-----+-----+-----+------+------+------+------+------+ |50 |6.42 |16.27|25.71| 83.55 | 6.69 | 17.27 | 26.47 | 84.08 | +---+-----+-----+-----+------+------+------+------+------+ |100|6.54 |16.35|25.54| 83.88 | 6.79 | 16.65 | 26.21 | 84.75 | +---+-----+-----+-----+------+------+------+------+------+

Table 86: cubic-dctcp-sce Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |reno | | | | reno | | | | +---+-----+------+------+------+------+------+------+------+ |1 |67.27| 69.27 | 83.70 | 145.08 | 67.81 | 68.53 | 84.48 | 145.65 | +---+-----+------+------+------+------+------+------+------+ |5 |14.07| 26.21 | 36.48 | 93.63 | 14.28 | 26.00 | 36.21 | 93.05 | +---+-----+------+------+------+------+------+------+------+ |10 |8.21 | 19.06 | 29.68 | 86.73 | 8.20 | 18.96 | 29.25 | 86.39 | +---+-----+------+------+------+------+------+------+------+ |50 |6.62 | 16.41 | 24.97 | 82.36 | 6.68 | 16.32 | 25.01 | 82.48 | +---+-----+------+------+------+------+------+------+------+ |100|6.57 | 15.91 | 24.64 | 82.45 | 6.45 | 15.90 | 24.65 | 82.56 | +---+-----+------+------+------+------+------+------+------+

Table 87: reno-reno Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 53] Internet-Draft sceonetwotests July 2019

+---+-----+-----+------+------+------+------+------+------+ | |0 |10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+======+======+======+======+======+======+ | |reno | | | | reno-sce | | | | +---+-----+-----+------+------+------+------+------+------+ |1 |67.65|68.24| 83.30| 139.85 | 68.25 | 70.37 | 79.56 | 142.52 | +---+-----+-----+------+------+------+------+------+------+ |5 |14.09|24.97| 35.97| 91.33 | 14.55 | 24.35 | 37.58 | 96.81 | +---+-----+-----+------+------+------+------+------+------+ |10 |8.54 |19.17| 28.84| 86.91 | 8.21 | 19.96 | 32.27 | 87.41 | +---+-----+-----+------+------+------+------+------+------+ |50 |6.44 |15.55| 24.88| 82.79 | 6.56 | 16.08 | 25.22 | 82.59 | +---+-----+-----+------+------+------+------+------+------+ |100|6.40 |16.27| 25.71| 85.25 | 6.74 | 16.42 | 25.90 | 85.13 | +---+-----+-----+------+------+------+------+------+------+

Table 88: reno-reno-sce Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |reno | | | | dctcp | | | | +---+-----+------+------+------+------+------+------+------+ |1 |62.85| 65.76 | 79.20 | 146.75 | 65.00 | 66.92 | 78.64 | 140.83 | +---+-----+------+------+------+------+------+------+------+ |5 |13.65| 26.18 | 38.66 | 97.29 | 14.02 | 23.92 | 33.54 | 93.24 | +---+-----+------+------+------+------+------+------+------+ |10 |8.12 | 21.46 | 32.84 | 91.43 | 7.89 | 17.89 | 27.67 | 87.40 | +---+-----+------+------+------+------+------+------+------+ |50 |7.90 | 18.20 | 28.50 | 85.93 | 6.86 | 16.53 | 26.78 | 84.89 | +---+-----+------+------+------+------+------+------+------+ |100|7.58 | 17.41 | 27.77 | 85.89 | 7.06 | 16.76 | 27.43 | 85.22 | +---+-----+------+------+------+------+------+------+------+

Table 89: reno-dctcp Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 54] Internet-Draft sceonetwotests July 2019

+---+-----+-----+-----+------+------+------+------+------+ | |0 |10 |20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+=====+======+======+======+======+======+ | |reno | | | | dctcp-sce | | | | +---+-----+-----+-----+------+------+------+------+------+ |1 |66.94|68.38|84.13| 443.94 | 68.15 | 70.50| 80.58 | 1036.03 | +---+-----+-----+-----+------+------+------+------+------+ |5 |14.59|25.96|35.41| 91.34 | 14.17 | 25.50| 36.79 | 96.14 | +---+-----+-----+-----+------+------+------+------+------+ |10 |8.76 |19.46|28.55| 86.17 | 8.35 | 21.14| 31.51 | 88.70 | +---+-----+-----+-----+------+------+------+------+------+ |50 |6.32 |15.99|25.40| 83.48 | 6.63 | 17.15| 26.42 | 83.55 | +---+-----+-----+-----+------+------+------+------+------+ |100|6.33 |15.98|25.24| 83.42 | 6.67 | 16.22| 25.74 | 84.21 | +---+-----+-----+-----+------+------+------+------+------+

Table 90: reno-dctcp-sce Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+-----+-----+------+------+-----+-----+------+ | | 0 |10 |20 | 80 | 0 |10 |20 | 80 | +===+======+=====+=====+======+======+=====+=====+======+ | | reno-sce | | | | reno-sce | | | | +---+------+-----+-----+------+------+-----+-----+------+ |1 | 71.92 |72.60|85.12| 140.88 | 71.48 |71.82|82.18| 138.74 | +---+------+-----+-----+------+------+-----+-----+------+ |5 | 14.11 |24.23|34.56| 92.98 | 14.22 |24.21|35.30| 92.20 | +---+------+-----+-----+------+------+-----+-----+------+ |10 | 9.86 |19.83|29.38| 86.98 | 9.89 |19.80|29.32| 86.79 | +---+------+-----+-----+------+------+-----+-----+------+ |50 | 6.63 |17.01|26.57| 82.78 | 6.69 |16.95|26.48| 82.69 | +---+------+-----+-----+------+------+-----+-----+------+ |100| 6.49 |16.69|26.68| 85.00 | 6.53 |16.93|26.69| 85.15 | +---+------+-----+-----+------+------+-----+-----+------+

Table 91: reno-sce-reno-sce Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 55] Internet-Draft sceonetwotests July 2019

+---+------+-----+-----+------+------+------+------+------+ | | 0 |10 |20 | 80 | 0 | 10 | 20 | 80 | +===+======+=====+=====+======+======+======+======+======+ | | reno-sce | | | | dctcp| | | | +---+------+-----+-----+------+------+------+------+------+ |1 | 69.23 |66.60|78.75| 146.62 | 67.72| 67.64 | 79.32 | 139.43 | +---+------+-----+-----+------+------+------+------+------+ |5 | 13.35 |24.63|36.42| 98.92 | 13.90| 23.35 | 32.95 | 92.69 | +---+------+-----+-----+------+------+------+------+------+ |10 | 8.00 |19.99|30.77| 94.50 | 7.85 | 18.12 | 27.44 | 87.54 | +---+------+-----+-----+------+------+------+------+------+ |50 | 8.06 |19.71|31.42| 88.87 | 7.06 | 16.79 | 26.70 | 85.95 | +---+------+-----+-----+------+------+------+------+------+ |100| 8.34 |19.77|32.99| 91.65 | 7.55 | 16.70 | 27.21 | 87.35 | +---+------+-----+-----+------+------+------+------+------+

Table 92: reno-sce-dctcp Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+-----+-----+------+------+-----+-----+------+ | | 0 |10 |20 | 80 | 0 |10 |20 | 80 | +===+======+=====+=====+======+======+=====+=====+======+ | | reno-sce | | | | dctcp-sce | | | | +---+------+-----+-----+------+------+-----+-----+------+ |1 | 71.87 |72.66|83.24| 139.33| 72.10 |73.13|86.11| 136.98 | +---+------+-----+-----+------+------+-----+-----+------+ |5 | 15.51 |24.77|34.98| 91.65 | 15.39 |23.89|34.00| 93.84 | +---+------+-----+-----+------+------+-----+-----+------+ |10 | 9.89 |19.46|28.37| 87.12 | 9.73 |19.48|30.06| 89.37 | +---+------+-----+-----+------+------+-----+-----+------+ |50 | 6.53 |16.33|25.97| 82.70 | 6.66 |17.52|27.41| 83.22 | +---+------+-----+-----+------+------+-----+-----+------+ |100| 6.81 |16.30|26.07| 83.48 | 6.96 |16.88|26.35| 84.50 | +---+------+-----+-----+------+------+-----+-----+------+

Table 93: reno-sce-dctcp-sce Mean TCP RTT (ms); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 56] Internet-Draft sceonetwotests July 2019

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |dctcp| | | | dctcp | | | | +---+-----+------+------+------+------+------+------+------+ |1 |68.18| 68.95 | 80.54 | 143.93 | 69.62 | 69.18 | 80.52 | 141.76 | +---+-----+------+------+------+------+------+------+------+ |5 |15.31| 24.50 | 34.27 | 94.45 | 15.27 | 24.16 | 33.93 | 95.11 | +---+-----+------+------+------+------+------+------+------+ |10 |8.30 | 18.85 | 28.20 | 87.75 | 8.44 | 18.78 | 28.46 | 87.50 | +---+-----+------+------+------+------+------+------+------+ |50 |7.47 | 16.58 | 26.42 | 85.35 | 7.49 | 16.75 | 26.67 | 85.34 | +---+-----+------+------+------+------+------+------+------+ |100|7.64 | 16.55 | 26.77 | 86.26 | 7.52 | 17.00 | 26.75 | 86.42 | +---+-----+------+------+------+------+------+------+------+

Table 94: dctcp-dctcp Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+------+-----+------+------+------+-----+------+ | |0 |10 |20 | 80 | 0 | 10 |20 | 80 | +===+=====+======+=====+======+======+======+=====+======+ | |dctcp| | | | dctcp-sce | | | | +---+-----+------+-----+------+------+------+-----+------+ |1 |66.02|66.25 |78.29| 139.85 | 67.98 | 65.82 |77.89| 145.81 | +---+-----+------+-----+------+------+------+-----+------+ |5 |13.88|207.36|32.89| 92.04 | 13.55 | 380.51 |35.59| 98.73 | +---+-----+------+-----+------+------+------+-----+------+ |10 |7.81 |80.88 |27.60| 87.58 | 7.95 | 154.57 |30.74| 93.93 | +---+-----+------+-----+------+------+------+-----+------+ |50 |6.98 |16.12 |26.46| 85.87 | 8.00 | 19.22 |32.18| 88.74 | +---+-----+------+-----+------+------+------+-----+------+ |100|7.72 |16.92 |27.67| 86.88 | 8.67 | 19.60 |32.95| 91.79 | +---+-----+------+-----+------+------+------+-----+------+

Table 95: dctcp-dctcp-sce Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 57] Internet-Draft sceonetwotests July 2019

+---+------+-----+-----+------+------+-----+-----+------+ | | 0 |10 |20 |80 | 0 |10 |20 | 80 | +===+======+=====+=====+======+======+=====+=====+======+ | | dctcp-sce | | | | dctcp-sce | | | | +---+------+-----+-----+------+------+-----+-----+------+ |1 | 69.76 |71.34|84.74|139.62| 72.85 |71.92|86.45| 137.08 | +---+------+-----+-----+------+------+-----+-----+------+ |5 | 15.58 |23.92|33.94|92.72 | 15.65 |23.80|33.77| 93.02 | +---+------+-----+-----+------+------+-----+-----+------+ |10 | 9.78 |19.14|29.55|90.22 | 9.73 |19.36|29.58| 89.44 | +---+------+-----+-----+------+------+-----+-----+------+ |50 | 6.75 |15.21|24.86|82.92 | 6.80 |15.27|24.83| 83.08 | +---+------+-----+-----+------+------+-----+-----+------+ |100| 6.41 |15.71|25.16|83.67 | 6.59 |15.95|25.31| 83.71 | +---+------+-----+-----+------+------+-----+-----+------+

Table 96: dctcp-sce-dctcp-sce Mean TCP RTT (ms); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

A.8. Two-Flow TCP RTT (Cake "triple-isolate")

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |cubic| | | | cubic | | | | +---+-----+------+------+------+------+------+------+------+ |1 |92.48| 88.69 | 88.22 | 131.91 | 92.05 | 90.01 | 87.87 | 131.57 | +---+-----+------+------+------+------+------+------+------+ |5 |18.67| 28.80 | 36.43 | 90.88 | 18.74 | 28.85 | 36.72 | 90.74 | +---+-----+------+------+------+------+------+------+------+ |10 |8.48 | 18.56 | 28.64 | 86.24 | 8.53 | 18.63 | 28.72 | 86.10 | +---+-----+------+------+------+------+------+------+------+ |50 |6.51 | 15.85 | 24.13 | 82.81 | 6.60 | 15.85 | 24.00 | 82.69 | +---+-----+------+------+------+------+------+------+------+ |100|6.04 | 15.21 | 23.73 | 82.62 | 5.92 | 15.29 | 23.71 | 82.59 | +---+-----+------+------+------+------+------+------+------+

Table 97: cubic-cubic Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 58] Internet-Draft sceonetwotests July 2019

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |cubic| | | | reno | | | | +---+-----+------+------+------+------+------+------+------+ |1 |90.34| 89.98 | 87.87 | 133.32 | 68.49 | 66.53 | 73.84 | 138.30 | +---+-----+------+------+------+------+------+------+------+ |5 |17.62| 29.12 | 36.16 | 90.80 | 14.13 | 30.47 | 37.59 | 92.04 | +---+-----+------+------+------+------+------+------+------+ |10 |8.49 | 18.41 | 28.65 | 86.04 | 8.90 | 20.00 | 30.17 | 86.63 | +---+-----+------+------+------+------+------+------+------+ |50 |6.48 | 15.92 | 23.71 | 82.53 | 6.56 | 15.44 | 23.98 | 82.33 | +---+-----+------+------+------+------+------+------+------+ |100|6.15 | 15.14 | 23.53 | 82.86 | 6.22 | 14.57 | 23.50 | 82.20 | +---+-----+------+------+------+------+------+------+------+

Table 98: cubic-reno Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+-----+------+------+------+------+------+------+ | |0 |10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+======+======+======+======+======+======+ | |cubic| | | | reno-sce | | | | +---+-----+-----+------+------+------+------+------+------+ |1 |92.71|90.37| 87.84| 129.22 | 50.64 | 50.51 | 50.24 | 117.80 | +---+-----+-----+------+------+------+------+------+------+ |5 |16.87|24.55| 35.20| 90.00 | 9.62 | 19.86 | 30.07 | 92.04 | +---+-----+-----+------+------+------+------+------+------+ |10 |8.53 |18.97| 29.09| 86.20 | 6.59 | 16.56 | 26.35 | 85.23 | +---+-----+-----+------+------+------+------+------+------+ |50 |6.55 |15.97| 24.28| 82.65 | 4.41 | 13.89 | 23.32 | 82.85 | +---+-----+-----+------+------+------+------+------+------+ |100|6.03 |15.27| 23.98| 82.61 | 3.93 | 13.85 | 23.58 | 82.96 | +---+-----+-----+------+------+------+------+------+------+

Table 99: cubic-reno-sce Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 59] Internet-Draft sceonetwotests July 2019

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |cubic| | | | dctcp | | | | +---+-----+------+------+------+------+------+------+------+ |1 |92.60| 89.79 | 87.75 | 135.20 | 97.84 | 96.66 | 96.75 | 149.61 | +---+-----+------+------+------+------+------+------+------+ |5 |14.45| 28.19 | 37.86 | 91.20 | 19.07 | 25.58 | 35.17 | 94.03 | +---+-----+------+------+------+------+------+------+------+ |10 |8.63 | 18.12 | 28.34 | 86.15 | 8.16 | 17.58 | 27.57 | 87.68 | +---+-----+------+------+------+------+------+------+------+ |50 |6.57 | 15.82 | 24.40 | 82.81 | 6.80 | 16.42 | 25.94 | 84.05 | +---+-----+------+------+------+------+------+------+------+ |100|6.24 | 15.32 | 23.81 | 82.84 | 6.52 | 16.12 | 25.83 | 82.92 | +---+-----+------+------+------+------+------+------+------+

Table 100: cubic-dctcp Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+-----+-----+------+------+------+------+------+ | |0 |10 |20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+=====+======+======+======+======+======+ | |cubic| | | | dctcp-sce | | | | +---+-----+-----+-----+------+------+------+------+------+ |1 |92.73|87.54|87.60| 126.28 | 50.28 | 49.77 | 49.69 | 115.82 | +---+-----+-----+-----+------+------+------+------+------+ |5 |14.77|24.31|34.63| 89.26 | 9.54 | 20.04 | 29.91 | 93.48 | +---+-----+-----+-----+------+------+------+------+------+ |10 |8.46 |18.88|28.82| 85.49 | 6.87 | 16.46 | 26.73 | 88.22 | +---+-----+-----+-----+------+------+------+------+------+ |50 |6.61 |17.05|24.37| 82.43 | 4.19 | 13.11 | 22.72 | 82.28 | +---+-----+-----+-----+------+------+------+------+------+ |100|6.41 |15.44|23.87| 82.35 | 3.66 | 12.84 | 22.70 | 82.27 | +---+-----+-----+-----+------+------+------+------+------+

Table 101: cubic-dctcp-sce Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 60] Internet-Draft sceonetwotests July 2019

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |reno | | | | reno | | | | +---+-----+------+------+------+------+------+------+------+ |1 |68.39| 66.58 | 73.72 | 142.08 | 68.61 | 66.66 | 73.74 | 142.94 | +---+-----+------+------+------+------+------+------+------+ |5 |15.73| 25.66 | 38.41 | 92.20 | 14.62 | 25.71 | 37.82 | 92.07 | +---+-----+------+------+------+------+------+------+------+ |10 |8.48 | 20.48 | 29.68 | 86.62 | 8.86 | 20.37 | 29.82 | 86.93 | +---+-----+------+------+------+------+------+------+------+ |50 |6.58 | 15.60 | 23.71 | 82.41 | 6.54 | 15.64 | 23.77 | 82.36 | +---+-----+------+------+------+------+------+------+------+ |100|6.32 | 14.10 | 23.44 | 82.17 | 6.36 | 14.13 | 23.13 | 81.99 | +---+-----+------+------+------+------+------+------+------+

Table 102: reno-reno Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+-----+------+------+------+------+------+------+ | |0 |10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+======+======+======+======+======+======+ | |reno | | | | reno-sce | | | | +---+-----+-----+------+------+------+------+------+------+ |1 |68.77|66.59| 73.78| 137.04 | 50.65 | 50.32 | 50.23 | 116.09 | +---+-----+-----+------+------+------+------+------+------+ |5 |16.12|24.59| 37.04| 90.48 | 9.41 | 19.69 | 30.14 | 91.04 | +---+-----+-----+------+------+------+------+------+------+ |10 |8.46 |19.67| 31.33| 86.72 | 6.76 | 16.53 | 26.33 | 85.41 | +---+-----+-----+------+------+------+------+------+------+ |50 |6.59 |15.65| 23.98| 82.46 | 4.43 | 13.54 | 23.21 | 82.73 | +---+-----+-----+------+------+------+------+------+------+ |100|6.35 |14.57| 23.37| 82.16 | 4.11 | 13.62 | 23.45 | 82.85 | +---+-----+-----+------+------+------+------+------+------+

Table 103: reno-reno-sce Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 61] Internet-Draft sceonetwotests July 2019

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |reno | | | | dctcp | | | | +---+-----+------+------+------+------+------+------+------+ |1 |68.04| 66.19 | 73.70 | 140.21 | 96.81 | 96.80 | 96.58 | 143.33 | +---+-----+------+------+------+------+------+------+------+ |5 |16.18| 26.99 | 38.88 | 92.89 | 18.99 | 24.82 | 35.27 | 93.97 | +---+-----+------+------+------+------+------+------+------+ |10 |8.59 | 19.99 | 30.15 | 86.79 | 8.27 | 17.92 | 27.99 | 88.08 | +---+-----+------+------+------+------+------+------+------+ |50 |6.38 | 15.57 | 23.94 | 82.44 | 6.78 | 16.46 | 25.77 | 83.82 | +---+-----+------+------+------+------+------+------+------+ |100|6.24 | 14.72 | 23.45 | 82.17 | 6.30 | 16.11 | 24.98 | 82.96 | +---+-----+------+------+------+------+------+------+------+

Table 104: reno-dctcp Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+-----+-----+------+------+------+------+------+ | |0 |10 |20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+=====+======+======+======+======+======+ | |reno | | | | dctcp-sce | | | | +---+-----+-----+-----+------+------+------+------+------+ |1 |69.06|67.33|65.57| 133.89 | 50.17 | 49.67 | 49.59 | 115.33 | +---+-----+-----+-----+------+------+------+------+------+ |5 |16.04|24.23|36.62| 89.94 | 9.57 | 19.90 | 30.12 | 93.42 | +---+-----+-----+-----+------+------+------+------+------+ |10 |8.51 |19.93|29.67| 85.60 | 6.65 | 16.53 | 26.99 | 87.11 | +---+-----+-----+-----+------+------+------+------+------+ |50 |6.47 |16.89|24.02| 82.35 | 4.19 | 13.04 | 22.72 | 82.35 | +---+-----+-----+-----+------+------+------+------+------+ |100|6.35 |14.57|23.39| 81.90 | 3.74 | 12.59 | 22.53 | 82.02 | +---+-----+-----+-----+------+------+------+------+------+

Table 105: reno-dctcp-sce Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 62] Internet-Draft sceonetwotests July 2019

+---+------+-----+-----+------+------+-----+-----+------+ | | 0 |10 |20 | 80 | 0 |10 |20 | 80 | +===+======+=====+=====+======+======+=====+=====+======+ | | reno-sce | | | | reno-sce | | | | +---+------+-----+-----+------+------+-----+-----+------+ |1 | 50.10 |50.27|50.22| 117.12 | 50.62 |50.44|50.23| 116.67 | +---+------+-----+-----+------+------+-----+-----+------+ |5 | 9.54 |19.29|29.77| 88.86 | 9.63 |19.25|29.76| 89.43 | +---+------+-----+-----+------+------+-----+-----+------+ |10 | 6.89 |16.55|26.16| 85.44 | 6.62 |16.47|26.14| 85.52 | +---+------+-----+-----+------+------+-----+-----+------+ |50 | 4.50 |13.31|22.99| 82.65 | 4.47 |13.28|22.93| 82.60 | +---+------+-----+-----+------+------+-----+-----+------+ |100| 4.25 |13.89|23.70| 83.01 | 4.14 |13.74|23.59| 82.88 | +---+------+-----+-----+------+------+-----+-----+------+

Table 106: reno-sce-reno-sce Mean TCP RTT (ms); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+-----+-----+------+------+------+------+------+ | | 0 |10 |20 | 80 | 0 | 10 | 20 | 80 | +===+======+=====+=====+======+======+======+======+======+ | | reno-sce | | | | dctcp| | | | +---+------+-----+-----+------+------+------+------+------+ |1 | 49.93 |50.27|50.22| 119.15 | 97.30| 97.00 | 96.39 | 135.73 | +---+------+-----+-----+------+------+------+------+------+ |5 | 9.45 |19.45|30.20| 92.32 | 19.13| 23.40 | 33.39 | 91.37 | +---+------+-----+-----+------+------+------+------+------+ |10 | 6.71 |16.62|26.26| 85.57 | 8.16 | 18.21 | 28.41 | 87.40 | +---+------+-----+-----+------+------+------+------+------+ |50 | 4.42 |13.69|23.24| 82.83 | 6.75 | 16.30 | 26.05 | 84.00 | +---+------+-----+-----+------+------+------+------+------+ |100| 4.07 |13.79|23.61| 82.79 | 6.36 | 16.43 | 26.21 | 83.42 | +---+------+-----+-----+------+------+------+------+------+

Table 107: reno-sce-dctcp Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 63] Internet-Draft sceonetwotests July 2019

+---+------+-----+-----+------+------+-----+-----+------+ | | 0 |10 |20 | 80 | 0 |10 |20 | 80 | +===+======+=====+=====+======+======+=====+=====+======+ | | reno-sce | | | | dctcp-sce | | | | +---+------+-----+-----+------+------+-----+-----+------+ |1 | 50.11 |50.26|50.22| 115.48| 50.26 |49.68|49.61| 114.34 | +---+------+-----+-----+------+------+-----+-----+------+ |5 | 9.52 |19.16|29.74| 88.60 | 9.61 |19.32|29.60| 91.03 | +---+------+-----+-----+------+------+-----+-----+------+ |10 | 6.49 |16.48|25.83| 84.54 | 6.77 |16.38|26.33| 87.03 | +---+------+-----+-----+------+------+-----+-----+------+ |50 | 4.41 |13.17|22.98| 82.60 | 4.12 |13.08|22.99| 82.38 | +---+------+-----+-----+------+------+-----+-----+------+ |100| 4.16 |13.48|23.17| 82.59 | 3.78 |12.83|22.57| 82.04 | +---+------+-----+-----+------+------+-----+-----+------+

Table 108: reno-sce-dctcp-sce Mean TCP RTT (ms); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+-----+------+------+------+------+------+------+------+ | |0 | 10 | 20 | 80 | 0 | 10 | 20 | 80 | +===+=====+======+======+======+======+======+======+======+ | |dctcp| | | | dctcp | | | | +---+-----+------+------+------+------+------+------+------+ |1 |97.66| 96.66 | 96.60 | 151.58 | 97.03 | 96.71 | 96.65 | 150.04 | +---+-----+------+------+------+------+------+------+------+ |5 |18.76| 25.73 | 35.30 | 93.14 | 19.22 | 25.75 | 35.17 | 93.24 | +---+-----+------+------+------+------+------+------+------+ |10 |8.15 | 18.04 | 27.99 | 87.51 | 8.36 | 17.57 | 27.57 | 87.54 | +---+-----+------+------+------+------+------+------+------+ |50 |6.62 | 16.44 | 25.70 | 83.54 | 6.56 | 16.35 | 25.91 | 84.12 | +---+-----+------+------+------+------+------+------+------+ |100|6.46 | 16.25 | 26.01 | 82.89 | 6.56 | 16.07 | 25.86 | 82.93 | +---+-----+------+------+------+------+------+------+------+

Table 109: dctcp-dctcp Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Heist, et al. Expires 4 January 2020 [Page 64] Internet-Draft sceonetwotests July 2019

+---+-----+-----+-----+------+------+------+------+------+ | |0 |10 |20 | 80 | 0 | 10 | 20 | 80 | +===+=====+=====+=====+======+======+======+======+======+ | |dctcp| | | | dctcp-sce | | | | +---+-----+-----+-----+------+------+------+------+------+ |1 |97.80|96.84|96.75| 137.02 | 50.27 | 49.76 | 49.67 | 114.01 | +---+-----+-----+-----+------+------+------+------+------+ |5 |18.89|23.68|33.54| 91.13 | 9.53 | 19.80 | 29.74 | 93.86 | +---+-----+-----+-----+------+------+------+------+------+ |10 |8.09 |17.86|28.37| 85.94 | 6.63 | 16.42 | 26.54 | 87.48 | +---+-----+-----+-----+------+------+------+------+------+ |50 |6.74 |16.51|25.50| 83.04 | 4.06 | 13.08 | 22.68 | 82.42 | +---+-----+-----+-----+------+------+------+------+------+ |100|6.44 |16.27|26.02| 83.05 | 3.74 | 12.88 | 22.68 | 82.08 | +---+-----+-----+-----+------+------+------+------+------+

Table 110: dctcp-dctcp-sce Mean TCP RTT (ms); Columns: netem bi- directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

+---+------+-----+-----+------+------+-----+-----+------+ | | 0 |10 |20 |80 | 0 |10 |20 | 80 | +===+======+=====+=====+======+======+=====+=====+======+ | | dctcp-sce | | | | dctcp-sce | | | | +---+------+-----+-----+------+------+-----+-----+------+ |1 | 50.89 |49.74|49.64|116.00| 50.28 |49.72|49.68| 113.76 | +---+------+-----+-----+------+------+-----+-----+------+ |5 | 9.44 |19.41|30.11|90.46 | 9.53 |19.19|29.78| 91.12 | +---+------+-----+-----+------+------+-----+-----+------+ |10 | 6.66 |16.46|26.39|86.01 | 6.77 |16.44|26.44| 86.28 | +---+------+-----+-----+------+------+-----+-----+------+ |50 | 4.14 |12.85|22.68|82.30 | 4.19 |13.06|22.55| 82.25 | +---+------+-----+-----+------+------+-----+-----+------+ |100| 3.78 |12.57|22.34|81.86 | 3.80 |12.56|22.26| 81.96 | +---+------+-----+-----+------+------+-----+-----+------+

Table 111: dctcp-sce-dctcp-sce Mean TCP RTT (ms); Columns: netem bi-directional Delay (ms); Rows: Cake-limited Bandwidth (Mbit)

Authors’ Addresses

Peter G. Heist Redacted 463 11 Liberec 30 Czech Republic

Email: [email protected]

Heist, et al. Expires 4 January 2020 [Page 65] Internet-Draft sceonetwotests July 2019

Rodney W. Grimes Redacted Portland, OR 97217 United States

Email: [email protected]

Jonathan Morton Kokkonranta 21 FI-31520 Pitkajarvi Finland

Phone: +358 44 927 2377 Email: [email protected]

Heist, et al. Expires 4 January 2020 [Page 66] Network Working Group J. Henry Internet-Draft T. Szigeti Intended status: Informational Cisco Expires: October 15, 2020 L. Contreras Telefonica April 13, 2020

Diffserv to QCI Mapping draft-henry-tsvwg-diffserv-to-qci-04

Abstract

As communication devices become more hybrid, smart devices include more media-rich communication applications, and the boundaries between telecommunication and other applications becomes less clear. Simultaneously, as the end-devices become more mobile, application traffic transits more often between enterprise networks, the Internet, and cellular telecommunication networks, sometimes using simultaneously more than one path and network type. In this context, it is crucial that quality of service be aligned between these different environments. However, this is not always the case by default, and cellular communication networks use a different QoS nomenclature from the Internet and enterprise networks. This document specifies a set of 3rd Generation Partnership Project (3GPP) Quality of Service (QoS) Class Identifiers (QCI) and 5G QoS Identifiers (5QI) to Differentiated Services Code Point (DSCP) mappings, to reconcile the marking recommendations offered by the 3GPP with the recommendations offered by the IETF, so as to maintain a consistent QoS treatment between cellular networks and the Internet. This mapping can be used by enterprises or implementers expecting traffic to flow through both types of network, and wishing to align the QoS treatment applied to one network under their control with the QoS treatment applied to the other network.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any

Henry, et al. Expires October 15, 2020 [Page 1] Internet-Draft DIFFSERV-QCI April 2020

time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on October 15, 2020.

Copyright Notice

Table of Contents

1. Introduction ...... 3 1.1. Related Work ...... 4 1.2. Applicability Statement ...... 5 1.3. Document Organization ...... 5 1.4. Requirements language ...... 6 1.5. Terminology Used in this Document ...... 6 2. Service Comparison and Default Interoperation of Diffserv and 3GPP LTE and 5G ...... 7 2.1. Diffserv Domain Boundaries ...... 7 2.2. QCI and Bearer Model in 3GPP ...... 8 2.3. QCI Definition and Logic ...... 9 2.3.1. Conversational ...... 9 2.3.2. Streaming ...... 10 2.3.3. Interactive ...... 10 2.3.4. Background ...... 10 2.4. QCI implementations ...... 13 2.5. 5QI and flow-based QoS Model in 3GPP 5G ...... 13 2.6. GSMA IPX Guidelines Interpretation and Conflicts . . . . 17 3. P-GW Device Marking and Mapping Capability Recommendations . 18 4. DSCP to QCI or 5QI Mapping Recommendations ...... 19 4.1. Control Traffic ...... 19 4.1.1. Network Control Protocols ...... 19 4.1.2. Operations, Administration, and Maintenance (OAM) . . 20 4.2. User Traffic ...... 20 4.2.1. Telephony ...... 21 4.2.2. Signaling ...... 21

Henry, et al. Expires October 15, 2020 [Page 2] Internet-Draft DIFFSERV-QCI April 2020

4.2.3. Multimedia Conferencing ...... 22 4.2.4. Real-Time Interactive ...... 22 4.2.5. Multimedia Streaming ...... 23 4.2.6. Broadcast Video ...... 23 4.2.7. Low-Latency Data ...... 24 4.2.8. High-Throughput Data ...... 25 4.2.9. Standard ...... 25 4.2.10. Low-Priority Data ...... 26 4.3. Summary of Recommendations for DSCP-to-QCI Mapping . . . 26 5. QCI and 5QI to DSCP Mapping Recommendations ...... 28 5.1. QCI, 5QI and Diffserv Logic Reconciliation ...... 28 5.2. Voice [1] ...... 31 5.3. IMS Signaling [5] ...... 31 5.4. Voice-related QCIs and 5QIs [65, 66, 69] ...... 31 5.5. Video QCIs and 5QIs [67, 2, 4, 71, 72, 73, 74, 76] . . . 32 5.6. Live streaming and interactive gaming [7] ...... 34 5.7. Low latency eMBB and AR/VR [80] ...... 34 5.8. V2X messaging [75,3,9] ...... 35 5.9. Automation and Transport [82, 83, 84, 85, 86] ...... 35 5.10. Non-mission-critical data [6,8,9] ...... 36 5.11. Mission-critical data [70] ...... 37 5.12. Summary of Recommendations for QCI or 5QI to DSCP Mapping 37 6. IANA Considerations ...... 40 7. Specific Security Considerations ...... 40 8. Security Recommendations for General QoS ...... 40 9. References ...... 41 9.1. Normative References ...... 41 9.2. Informative References ...... 42 Authors’ Addresses ...... 43

1. Introduction

3GPP has become the preferred set of standards to define cellular communication principles and protocols. With the augmented capabilities of smartphones, cellular networks increasingly carry non-communication traffic and interconnect with the Internet and Enterprise IP networks. The access networks defined by the 3GPP present several design challenges for ensuring end-to-end quality of service when these networks interconnect with the Internet or to enterprise networks. Some of these challenges relate to the nature of the cellular network itself, being centrally controlled, collision-free and primarily designed around subscription level and associated services, while other challenges relate to the fact that the 3GPP standards are not administered by the same standards body as Internet protocols. While 3GPP has developed tools to enable QoS over cellular networks, little guidance exists on how to maintain consistency of QoS treatment between cellular networks and the Internet, or IP-based Enterprise networks. As such, enterprises and

Henry, et al. Expires October 15, 2020 [Page 3] Internet-Draft DIFFSERV-QCI April 2020

other operators managing traffic flowing through both 3GPP and Internet Protocol links do not always know how to translate 3GPP QoS identifiers into Internet Protocol QoS identifiers and vice versa.The purpose of this document is to provide such guidance.

1.1. Related Work

Several RFCs outline Diffserv QoS recommendations over IP networks, including:

[RFC2474] specifies the Diffserv Codepoint Field. This RFC also details Class Selectors, as well as the Default Forwarding (DF) treatment. [RFC2475] defines a Diffserv architecture [RFC3246] specifies the Expedited Forwarding (EF) Per-Hop Behavior (PHB) [RFC2597] specifies the Assured Forwarding (AF) PHB. [RFC3662] specifies a Lower Effort Per-Domain Behavior (PDB) [RFC4594] presents Configuration Guidelines for Diffserv Service Classes [RFC5127] presents the Aggregation of Diffserv Service Classes [RFC5865] specifies a DSCP for Capacity Admitted Traffic [RFC8622] presents the Lower-Effort Per-Hop Behavior (LE-PHB) for Diffserv

Note: [RFC4594] is intended to be viewed as a framework for supporting Diffserv in any network, regardless of the underlying data-link or physical layer protocols. Its principles could apply to IP traffic carried over cellular DataLink and Physical Layer mediums. Additionally, the principles of [RFC4594] apply to any traffic entering the Internet, regardless of its original source location. Thus, [RFC4594] describes different types of traffic expected in IP networks and provides guidance as to what DSCP marking(s) should be associated with each traffic type. As such, this document draws heavily on [RFC4594] , as well as [RFC5127], and [RFC8100].

In turn, the relevant standard for cellular LTE QoS is 3GPP [TS 23.107], which defines more than 1600 General Packet Radio Service (GPRS) QoS profiles across multiple classes and associated attributes. As this quantity is large and source of potential complexity, the 3GPP Technical Specification Group Services and System Aspects, defining the Policy Charging Control Architecture, leverages a subset of QoS profiles used as QoS Class Identifiers (QCI). For 5G communications, [TS 23.501] defines 5G QoS Identifiers. This document draws on these specifications, which are being progressively updated; the current version of which (at the time of writing) are 3GPP [TS 23.203] v16.2.0 and 3GPP [TS 23.501] v16.3.0.

Henry, et al. Expires October 15, 2020 [Page 4] Internet-Draft DIFFSERV-QCI April 2020

1.2. Applicability Statement

This document is applicable to the use of Differentiated Services that interconnect with 3GPP LTE or 5G cellular networks (referred to as cellular, throughout this document, for simplicity). These guidelines are applicable whether cellular network endpoints are IP- enabled, in which case these guidelines can apply end-to-end, starting from the endpoint operating system, or whether cellular network endpoints are either not IP-enabled, or do not enable QoS, in which case these guidelines apply at the interconnection point between the cellular access network and the Internet or IP network. Such interconnection point can commonly occur at the infrastructure Radio Unit (eNodeB), within the infrastructure core network (CN), or at the edge of the core network toward the Internet or an Enterprise IP network, for example within the Packet Data Network Gateway (P-GW).

1.3. Document Organization

This document is organized as follows:

o Section 2 introduces the QoS logic marking applicable to each domain. We introduce the general logic of Diffserv and the notion of domain boundary. We then examine the 3GPP QoS logic, detailing the concept of bearer, QCI and 5QIs, and showing how QCIs and 5QIs are implemented and used.

o Section 3 provides general recommendations for QoS support at the 3GPP / Diffserv domains boundaries.

o Section 4 proposes a Diffserv to QCI translation scheme, so as to suggest DSCP values that can be directly translated into QCIs or 5QIs values, when traffic moves into a 3GPP domain where QCIs or 5QIs must be used.

o Section 5 proposes a reverse mapping, from QCI to Diffserv. As many QCIs intents do not match existing DSCP values, new DSCP values are proposed wherever needed.

o Section 6 underlines the resulting IANA requirements for this mapping.

o Section 7 and Section 8 examine the security consequences of these new mapping schemes.

Henry, et al. Expires October 15, 2020 [Page 5] Internet-Draft DIFFSERV-QCI April 2020

1.4. Requirements language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

1.5. Terminology Used in this Document

Key terminology used in this document includes:

EPS Bearer: a path that user traffic (IP flows) uses between the UE and the PGW.

GGSN: Gateway GPRS Support Node, responsible for the internetworking between the GPRS network and external networks. PGW performs the GGSN functionalities in EPC.

IP BS Manager: Internet Protocol Bearer Service Manager, a function that manages the IP bearer services. Part of this function can include translation of QoS parameters between EPS and external networks.

UE: User Equipment, the end-device.

EPS Session: a PDN connection, comprised of one or more IP flows, that a UE established and maintains to the EPS.

SAE: System Architecture Evolution.

RAN: Radio access network, the radio segment of the LTE network EPS.

EPC: Evolved Packet Core, the core segment of the LTE network EPS.

EPS: Evolved Packet System, the LTE network, comprised of the RANs and EPC.

HSS: Home Subscriber Server, the database that contains user-related and subscriber-related information.

LUS: Live Uplink Streaming, a video flow (often real-time) sent from a source to a sink.

SGW: Serving Gateway, the point of interconnection between the RAN and the EPC.

Henry, et al. Expires October 15, 2020 [Page 6] Internet-Draft DIFFSERV-QCI April 2020

PGW: Packet Data Network Gateway, point of interconnection between the EPC and external IP networks.

MME: Mobility Management Entity: software function that handles the signaling related to mobility and security for the access network.

PCEF: Policy and Charging Enforcement Function, provides user traffic handling and QoS within the PGW.

PCRF: Policy and Charging Rules Function, a functional entity that provides policy, bandwidth and charging functions for each EPS user.

2. Service Comparison and Default Interoperation of Diffserv and 3GPP LTE and 5G

2.1. Diffserv Domain Boundaries

It is important to recognize that 3GPP standards allow support for principles of [RFC2475]. The user equipment (UE) application function may have no active QoS support, or may support Diffserv or IntServ functions [TS 23.207] v15 5.2.2. When Diffserv is supported, an Internet Protocol Bearer Service Manager (IP BS Manager) function integrated to the UE can translate Diffserv parameters into LTE QoS parameters (e.g. QCI). As such, the UE IP BS Manager function may act as a Diffserv domain boundary (as defined in [RFC2475]) between a Diffserv domain present within the UE networking stack and the LTE Radio Access Network.

Additionally, the P-GW interconnects the UE data plane to the external networks. The P-GW is the element that implements Gateway GPRS (General Packet Radio Service) Support Node (GGSN) functionalities in Evolved Packet Core (EPS) networks. The GGSN includes an IP BS manager function that acts as a Diffserv Edge function, and can translate Diffserv parameters to 3GPP QoS parameters (e.g. QCI or 5G NSA 5QI) and vice versa. In SA 5G, the user plane and control plane are separated, and the P-GW for the user plane (PGW-U) joins the Service Gateway (SGW-U) into the User Plane Function (UPF).

As such, 3GPP standards allow the existence of a Diffserv domain within the UE and outside of the EPS boundaries. The Diffserv domain is not considered within the EPS, where QCIs or 5QIs are used to define and transport QoS parameters.

Henry, et al. Expires October 15, 2020 [Page 7] Internet-Draft DIFFSERV-QCI April 2020

2.2. QCI and Bearer Model in 3GPP

It is important to note that LTE (4G) and 5G standards are an evolution of UMTS standards (2G, 3G) developed in the 1990s. As such, these standards recognize [RFC2475] (1998), but not [RFC4594] (2006). EPS networks rely on the notion of bearers. A bearer is a conduit between the UE and the P-GW, and LTE supports two types of bearers:

o GBR: Guaranteed Bit Rate bearers. These bearers allocate network resources associated to a GBR value associated to the bearer. These resources stay allocated (reserved) for the duration of the existence of the GBR bearer and the flow it carries.

o Non-GBR bearers: also called default bearers, non-GBR are bearers for which network resources are not permanently allocated during the existence of the bearer and the flow it carries. As such, one or more non-GBR bearer may share the same set of temporal resources.

Each EPS bearer is identified by a name and number, and is associated with specific QoS parameters of various types:

1. QoS Class Identifiers (QCI). A QCI is a scalar associated to a bearer, and is used to define the type of traffic and service expected in the bearer. [TS 23.107] v15 defines 4 basic classes: conversational, streaming, interactive and background. These classes are defined more in details in Section 2.3. Each class includes multiple types of traffic, each associated with sets of attributes, thus permitting the definition of more than 1600 different QoS profiles. [TS 23.203] v16 6.1.7.2 reduces the associated complexity by characterizing traffic based on up to 6 attributes, resulting in 26 types of traffic and their associated expected service requirements through the use of 26 scalars (QCI). Each QCI is defined in the relation to the following six performance characteristics:

2. Resource Type (GBR or Non-GBR).

3. Priority: a scalar used as a tie breaker if two packets compete for a given network resource. A lower value indicates a higher priority.

4. Packet Delay Budget: marks the upper bound for the time that a packet may be delayed between the UE and the PCRF (Policy and Charging Rules Function) or the PCEF function (Policy and Charging Enforcement Function) residing inside the P-GW. PCEF supports offline and online charging while PCRF is real-time.

Henry, et al. Expires October 15, 2020 [Page 8] Internet-Draft DIFFSERV-QCI April 2020

Either component, being in charge of policing and charging, can determine resource reservation actions and policies.

5. Packet Error Loss Rate, defines an upper bound for a rate of non- congestion related packet losses. The purpose of the PELR is to allow for appropriate link layer protocol configurations when needed.

6. Maximum Burst Size (only for some GBR QCIs), defines the amount of data which the Radio Access Network (RAN) is expected to deliver within the part of the Packet Delay Budget allocated to the link between the UE and the radio base station. If more data is transmitted from the application, the Packet Delay Budget may be exceeded.

7. Data rate Averaging Window (only for some GBR QCIs), defines the ’sliding window’ duration over which the GBR and MBR are calculated.

Although [TS 23.203] v16 6.1.7.2 associates each QCI with up to 6 characteristics, it is clear that these characteristics are constrained by bandwidth allocation, in particular on the radio link that are associated with three commonly used parameters:

1. Maximum Bit Rate (MBR), only valid for GBR bearers, defines the maximum sustained traffic rate that the bearer can support.

2. Guaranteed Bit Rate (GBR), only valid for GBR bearers, defines the minimum traffic rate reserved for the bearer.

3. Aggregate MBR (AMBR), defines the total amount of bit rate available for a group of non-GBR bearers. AMBR is often used to provide differentiated service levels to different types of customers.

2.3. QCI Definition and Logic

[TS 23.107] v15 6.3 defines four possible traffic classes. These four general classes are used as the foundation from which QCI categories are defined in [TS 23.203]. The categorization is made around the notion of sensitivity to delay.

2.3.1. Conversational

The conversational class is intended to carry real-time traffic flows. The expectation of such class is a live conversation between two humans or a group. Examples of such flows include [TS 23.107] v15 6.3.1 telephony speech, but also VoIP and video conferencing.

Henry, et al. Expires October 15, 2020 [Page 9] Internet-Draft DIFFSERV-QCI April 2020

Video conference would be seen as a different class from telephony in the Diffserv model. However, 3GPP positions them in the same general class, as all of them include live conversations. Sensitivity to delay is high because of the real-time nature of the flows. The time relation between the stream entities have to be preserved (to maintain the same experience for all flows and all parties involved in the conversation).

2.3.2. Streaming

The streaming class is intended for flows where the user is watching real time video, or listening to real-time audio (or both). The real-time data flow is always aiming at a live (human) destination. It is important to note that the Streaming class is intended to be both a real-time flow and a one-way transport. Two-way real-time traffic belongs to the conversational class, and non-real-time flows belong to the interactive or the background classes. The delay sensitivity is lower than that of Conversational flows, because it is expected that the receiving end includes a time alignment function (e.g. buffering). As the flow is unidirectional, variations in delay do not conversely affect the user experience as long as the variation is within the alignment function boundaries.

2.3.3. Interactive

The interactive class is intended for flows where a machine or human is requesting data from a remote equipment (e.g. a server). Examples of human interaction with the remote equipment are: web browsing, data base retrieval, server access. Examples of machines interaction with remote equipment are: polling for measurement records and automatic data base enquiries (tele-machines). Delay sensitivity is average, and is based on round trip time (overall time between emission of the request and reception of the response).

2.3.4. Background

The background class applies to flows where the equipment is sending or receiving data files without direct user interaction (e.g. emails, SMS, database transfers etc.) As such, delay sensitivity is low. Background is described as delivery-time insensitive.

Based upon the above principles, [TS 23.203] has defined several QCIs. [TS 23.203] Release 16 6.1.7-A defines 26 QCIs:

Henry, et al. Expires October 15, 2020 [Page 10] Internet-Draft DIFFSERV-QCI April 2020

+----+------+------+------+------+------+ | 1 | GBR | 2 | 100 ms | 10.E-2 | Conversational Voice | | | | | | | | | 2 | GBR | 4 | 150 ms | 10.E-3 | Conversational Video | | | | | | | (Live Streaming) | | | | | | | | | 3 | GBR | 3 | 50 ms | 10.E-3 | Real Time Gaming, | | | | | | | V2X messages, | | | | | | | Electricity | | | | | | | distribution (medium | | | | | | | voltage) Process | | | | | | | automation | | | | | | | (monitoring) | | | | | | | | | 4 | GBR | 5 | 300 ms | 10.E-6 | Non-Conversational | | | | | | | Video (Buffered | | | | | | | Streaming) | | | | | | | | | 65 | GBR | 0.7 | 75 ms | 10.E-2 | Mission Critical | | | | | | | user plane Push To | | | | | | | Talk voice (e.g., | | | | | | | MCPTT) | | | | | | | | | 66 | GBR | 2 | 100 ms | 10.E-2 | Non-Mission-Critical | | | | | | | user plane Push To | | | | | | | Talk voice | | | | | | | | | 67 | GBR | 1.5 | 100 ms | 10.E-3 | Mission Critical | | | | | | | Video user plane | | | | | | | | | 75 | GBR | 2.5 | 50 ms | 10.E-2 | V2X messages | | | | | | | | | 71 | GBR | 5.6 | 150 ms | 10.E-6 | "Live" Uplink | | | | | | | Streaming | | | | | | | | | 72 | GBR | 5.6 | 300 ms | 10.E-4 | "Live" Uplink | | | | | | | Streaming | | | | | | | | | 73 | GBR | 5.6 | 300 ms | 10.E-8 | "Live" Uplink | | | | | | | Streaming | | | | | | | | | 74 | GBR | 5.6 | 500 ms | 10.E-8 | "Live" Uplink | | | | | | | Streaming | | | | | | | | | 76 | GBR | 5.6 | 500 ms | 10.E-4 | "Live" Uplink | | | | | | | Streaming | | | | | | | | | 5 | Non-GBR | 1 | 100 ms | 10.E-6 | IMS Signalling |

Henry, et al. Expires October 15, 2020 [Page 11] Internet-Draft DIFFSERV-QCI April 2020

| | | | | | | | 6 | Non-GBR | 6 | 300 ms | 10.E-6 | Video (Buffered | | | | | | | Streaming) TCP-based | | | | | | | (e.g. www, email, | | | | | | | chat, ftp, p2p file | | | | | | | sharing, progressive | | | | | | | video) | | | | | | | | | 7 | Non-GBR | 7 | 100 ms | 10.E-3 | Voice, Video (live | | | | | | | streaming), | | | | | | | interactive gaming | | | | | | | | | 8 | Non-GBR | 8 | 300 ms | 10.E-6 | Video (buffered | | | | | | | streaming) TCP-based | | | | | | | (e.g. www, email, | | | | | | | chat, ftp, p2p file | | | | | | | sharing, progressive | | | | | | | video) | | | | | | | | | 9 | Non-GBR | 9 | 300 ms | 10.E-6 | Same as 8 | | | | | | | | | 69 | Non-GBR | 0.5 | 60 ms | 10.E-6 | Mission Critical | | | | | | | delay sensitive | | | | | | | signalling (e.g., | | | | | | | MC-PTT signalling, | | | | | | | MC Video signalling) | | | | | | | | | 70 | Non-GBR | 5.5 | 200 ms | 10.E-6 | Mission Critical | | | | | | | Data (e.g. example | | | | | | | services are the | | | | | | | same as QCI 6/8/9) | | | | | | | | | 79 | Non-GBR | 6.5 | 50 ms | 10.E-2 | V2X messages | | | | | | | | | 80 | Non-GBR | 6.8 | 10 ms | 10.E-2 | Low latency eMMB | | | | | | | applications | | | | | | | (TCP/UDP-based); | | | | | | | augmented reality | | | | | | | | | 82 | GBR | 1.9 | 10 ms | 10.E-6 | Discrete automation | | | | | | | (small packets) | | | | | | | | | 83 | GBR | 2.2 | 10 ms | 10.E-4 | Discrete automation | | | | | | | (large packets) | | | | | | | | | 84 | GBR | 2.4 | 30 ms | 10.E-5 | Intelligent | | | | | | | Transport Systems | | | | | | | |

Henry, et al. Expires October 15, 2020 [Page 12] Internet-Draft DIFFSERV-QCI April 2020

| 85 | GBR | 2.1 | 5 ms | 10.E-5 | Electricity | | | | | | | Distribution - High | | | | | | | Voltage | +----+------+------+------+------+------+

Several QCIs cover the same application types. For example, QCIs 6, 8 and 9 all apply to buffered streaming video and web applications. However, LTE context distinguishes several types of customers and environments. As such, QCI 6 can be used for the prioritization of non-real-time data (i.e. most typically TCP-based services/ applications) of MPS (multimedia priority services) subscribers, when the network supports MPS. QCI 8 can be used for a dedicated "premium bearer" (e.g. associated with premium content) for any subscriber or subscriber group, while QCI 9 can be used for the default bearer for non-privileged subscribers.

2.4. QCI implementations

[TS 23.203] v16 defines multiple QCIs. However, a UE or a EPS does not need to implement all supported QCIs, even when all matching types of traffic are expected between the UE and the network. In practical implementations, it is common for an EPS to implement one GBR bearer where at least QCI 1 is directed (and optionally other GBR QCIs), and another default bearer where all other traffic to and from the same UE is directed. The QCI associated to that second bearer may depend on the subscriber category. As such, the QCI listed in Section 2.3 are indicative of performance and traffic type classifications, and are not strict in their implementation mandate.

2.5. 5QI and flow-based QoS Model in 3GPP 5G

While 4G LTE QoS is enforced at the EPS bearer level, 5G QoS focuses on the transported flows. A QoS Flow ID (QFI) identifies a given QoS Flow. In the User Plane, the traffic with a given QFI within a PDU session is treated in the same way. The 5G QoS Identifier (5QI) is used in 3GPP to identify a specific QoS forwarding behavior for a 5G QoS Flow (similar to the QCI value for LTE, with the difference that 5QI applies to a flow, carried at some point in a bearer, while QCI applies to a bearer within which certain types of flows are expected). As such, the 5QI defines packet loss rate, packet delay budget etc. In the 5G system, the entity named Session Management Function (SMF) manages the QoS information. The SMF provides QFI information to the Radio Access Network (RAN) for mapping the various QoS flows to access network resources (i.e., data radio bearers). The RAN performs packet marking in the uplink on a per QoS Flow basis, with a marking value determined by the QFI and a treatment matching the asscoiated 5QI. The SMF also instructs the User Plane Function (UPF) for classification, bandwidth enforcement and marking

Henry, et al. Expires October 15, 2020 [Page 13] Internet-Draft DIFFSERV-QCI April 2020

of the user plane traffic in downlink. Such packet marking information includes the QFI and the transport level packet marking value (i.e., the value of the DSCP field in the outer IP header). In [TS 23.501], 3GPP provides the 5G QoS characteristics associated with the 5QIs, and specifies the packet forwarding treatment that a QoS Flow receives end-to-end, from the UE up to the UPF (and back). The characteristics considered are:

o Resource type, i.e., if the flow requires resources to be allocated for Guaranteed Bandwidth Rate (GBR), delay critical GBR (DCGBR), or non-GBR.

o Default priority level

o Packet delay budget (PDB), including the PDB consumed in the 5G core network

o Packet Error Rate (PER)

o Averaging window (in milliseconds), applicable for GBR and delay- critical GBR

o Default maximum data burst volume (in bytes), applicable for delay-critical GBR only

The following table shows a simplified version from the standardized [TS 23.501] 5QI to QoS characteristics mapping.

+----+------+------+------+------+------+------+------+ | 5Q | Resou | Prior | Pack | Packe | Defau | Defau | Example | | I | rce | ity | et D | t | lt | lt | Services | | | Type | Level | elay | Error | Max | Avg W | | | | | | Budg | Rate | Burst | indow | | | | | | et | | | | | +----+------+------+------+------+------+------+------+ | 1 | GBR | 20 | 100 | 10.E- | N/A | 2000 | Conversationa | | | | | ms | 2 | | | l voice | | | | | | | | | | | 2 | GBR | 40 | 150 | 10.E- | N/A | 2000 | Conversationa | | | | | ms | 3 | | | l video (live | | | | | | | | | streaming) | | | | | | | | | | | 3 | GBR | 30 | 50 | 10.E- | N/A | 2000 | Real time | | | | | ms | 3 | | | gaming, V2X | | | | | | | | | messages, | | | | | | | | | medium | | | | | | | | | voltage | | | | | | | | | electricity |

Henry, et al. Expires October 15, 2020 [Page 14] Internet-Draft DIFFSERV-QCI April 2020

| | | | | | | | dist. | | | | | | | | | | | 4 | GBR | 50 | 300 | 10.E- | N/A | 2000 | non-conversat | | | | | ms | 6 | | | ional video | | | | | | | | | (buffered | | | | | | | | | streaming) | | | | | | | | | | | 65 | GBR | 7 | 75 | 10.E- | N/A | 2000 | Mission | | | | | ms | 2 | | | critical user | | | | | | | | | plane push- | | | | | | | | | to-talk voice | | | | | | | | | (e.g. MCPTT) | | | | | | | | | | | 66 | GBR | 20 | 100 | 10.E- | N/A | 2000 | Non-mission | | | | | ms | 3 | | | critical user | | | | | | | | | plane push- | | | | | | | | | to-talk voice | | | | | | | | | | | 67 | GBR | 15 | 100 | 10.E- | N/A | 2000 | Mission | | | | | ms | 3 | | | critical user | | | | | | | | | plane video | | | | | | | | | | | 71 | GBR | 56 | 150 | 10.E- | N/A | 2000 | "Live" uplink | | | | | ms | 6 | | | streaming | | | | | | | | | | | 72 | GBR | 56 | 300 | 10.E- | N/A | 2000 | "Live" uplink | | | | | ms | 4 | | | streaming | | | | | | | | | | | 73 | GBR | 56 | 300 | 10.E- | N/A | 2000 | "Live" uplink | | | | | ms | 8 | | | streaming | | | | | | | | | | | 74 | GBR | 56 | 500 | 10.E- | N/A | 2000 | "Live" uplink | | | | | ms | 8 | | | streaming | | | | | | | | | | | 76 | GBR | 56 | 500 | 10.E- | N/A | 2000 | "Live" uplink | | | | | ms | 4 | | | streaming | | | | | | | | | | | 5 | non- | 10 | 100 | 10.E- | N/A | N/A | IMS signaling | | | GBR | | ms | 6 | | | | | | | | | | | | | | 6 | non- | 60 | 300 | 10.E- | N/A | N/A | Video | | | GBR | | ms | 6 | | | (Buffered | | | | | | | | | Streaming) | | | | | | | | | TCP-based | | | | | | | | | (e.g. www, | | | | | | | | | email, chat, | | | | | | | | | etc.) | | | | | | | | | |

Henry, et al. Expires October 15, 2020 [Page 15] Internet-Draft DIFFSERV-QCI April 2020

| 7 | non- | 70 | 100 | 10.E- | N/A | N/A | Voice, Video | | | GBR | | ms | 3 | | | (live | | | | | | | | | streaming), | | | | | | | | | interactive | | | | | | | | | gaming | | | | | | | | | | | 8 | non- | 80 | 300 | 10.E- | N/A | N/A | Video | | | GBR | | ms | 6 | | | (Buffered | | | | | | | | | Streaming) | | | | | | | | | TCP-based | | | | | | | | | (e.g. www, | | | | | | | | | email, chat, | | | | | | | | | etc.) | | | | | | | | | | | 9 | non- | 90 | 300 | 10.E- | N/A | N/A | Same as 8 | | | GBR | | ms | 6 | | | | | | | | | | | | | | 69 | non- | 5 | 60 | 10.E- | N/A | N/A | Mission | | | GBR | | ms | 6 | | | Critical | | | | | | | | | delay | | | | | | | | | sensitive | | | | | | | | | signalling | | | | | | | | | (e.g., MC- | | | | | | | | | PMC) | | | | | | | | | | | 70 | non- | 55 | 200 | 10.E- | N/A | N/A | Mission | | | GBR | | ms | 6 | | | critical data | | | | | | | | | (e.g. same | | | | | | | | | examples as | | | | | | | | | QCI/5QI 6,7,8 | | | | | | | | | | | 79 | non- | 65 | 50 | 10.E- | N/A | N/A | V2X messages | | | GBR | | ms | 2 | | | | | | | | | | | | | | 80 | non- | 68 | 10 | 10.E- | N/A | N/A | Low latency | | | GBR | | ms | 6 | | | eMMB | | | | | | | | | applications | | | | | | | | | (TCP/UDP- | | | | | | | | | based); | | | | | | | | | augmented | | | | | | | | | reality | | | | | | | | | | | 82 | DCGBR | 19 | 10 | 10.E- | 255 B | 2000 | Discrete | | | | | ms | 4 | | ms | automation | | | | | | | | | | | 83 | DCGBR | 22 | 10 | 10.E- | 1354 | 2000 | Discrete | | | | | ms | 4 | B | ms | automation | | | | | | | | | |

Henry, et al. Expires October 15, 2020 [Page 16] Internet-Draft DIFFSERV-QCI April 2020

| 84 | DCGBR | 24 | 30 | 10.E- | 1354 | 2000 | Intelligent | | | | | ms | 5 | B | ms | Transport | | | | | | | | | Systems | | | | | | | | | | | 85 | DCGBR | 21 | 5 ms | 10.E- | 255 B | 2000 | Electricity | | | | | | 5 | | ms | distribution, | | | | | | | | | High voltage, | | | | | | | | | V2X | | | | | | | | | | | 86 | DCGBR | 18 | 5 ms | 10.E- | 1354 | 2000 | V2X, | | | | | | 4 | B | ms | collision | | | | | | | | | avoidance, | | | | | | | | | platooning, | | | | | | | | | self driving | +----+------+------+------+------+------+------+------+

Although the focus of 5QI and that of QCI is different, it should be noted that the traffic examples provided by each QCI match the traffic intent for a 5QI with matching number. The 5QI default priority level is a tenfold expression of the QCI priority level (and this document will refer to the QCI priority levels for simplicity) As such, any given QCI or 5QI can be equivalised to the same DSCP value. In turn, an application and its given DSCP value can be expressed either in a QCI or a 5QI (provided that both exist for the assooiated traffic or application).

2.6. GSMA IPX Guidelines Interpretation and Conflicts

3GPP standards do not define or recommend any specific mapping between each QCI or 5QI and Diffserv, and leaves that mapping choice to the operator of the Edge domain boundary (e.g. UE software stack developer, P-GW operator). However, 3GPP defines that "for the IP based backbone, Differentiated Services defined by IETF shall be used" ([TS 23.107] v15 6.4.7).

The GSM Association (GSMA) has published an Inter-Service Provider IP Backbone Guideline reference document [ir.34] that provides technical guidance to participating service providers for connecting IP based networks and services to achieve roaming and inter-working services. The document built upon [RFC3246] and [RFC2597], and upon the initial definition of 4 service classes in [TS 23.107] v15 to recommend a mapping to EF for conversational traffic, to AF41 for Streaming traffic, to AF31, AF21 and AF11 for different traffic in the Interactive class, and to BE for background traffic.

These GSMA Guidelines were developed without reference to existing IETF specifications for various services, referenced in Section 1.1. Additionally, the same recommendations remained while new traffic

Henry, et al. Expires October 15, 2020 [Page 17] Internet-Draft DIFFSERV-QCI April 2020

types under each 3GPP general class were added. As such, the GSMA recommendations yield to several inconsistencies with [RFC4594], including:

o Recommending EF for real-time (conversational) video, for which [RFC4594] recommends AF41.

o Recommending AF31 for DNS traffic, for which [RFC4594] recommends the standard service class (DF)

o Recommending AF31 for all types of signaling traffic, thus losing the ability to differentiate between the various types of signaling flows, as recommended in[RFC4594] section 5.1.

o Recommending AF21 for WAP browsing and WEB browsing, for which [RFC4594] recommends the High Throughput data class

o Recommending AF11 for remote connection protocols, such as telnet or SSH, for which [RFC4594] recommends the OAM class.

o Recommending DF for file transfers, for which [RFC4594] recommends the High Throughput Data class.

o Recommending DF for email exchanges, for which [RFC4594] recommends the High Throughput Data class.

o Recommending DF for MMS exchanged over SMTP, for which [RFC4594] recommends the High Throughput Data class.

The document [ir.34] aso does not provide guidance for QCIs other than 1 to 9, leaving the case of the 12 other QCIs unaddressed.

Thus, document [ir.34] conflicts with the overall Diffserv traffic- conditioning service plan, both in the services specified and the code points specified for them. As such, these two plans cannot be normalized. Rather, as discussed in [RFC2474] Section 2, the two domains (GSMA and other IP networks) are different Differentiated Services Domains separated by a Differentiated Services Boundary. At that boundary, code points from one domain are translated to code points for the other, and maybe to Default (zero) if there is no corresponding service to translate to.

3. P-GW Device Marking and Mapping Capability Recommendations

This document assumes and RECOMMENDS that all P-GWs (as the interconnects between cellular and other IP networks) and all other interconnection points between cellular and other IP networks support the ability to:

Henry, et al. Expires October 15, 2020 [Page 18] Internet-Draft DIFFSERV-QCI April 2020

o mark DSCP, per Diffserv standards

o mark QCI, per the [TS 23.203] standard, or 5QI, as per the [TS 23.501] standard

o support fully-configurable mappings between DSCP and QCI or 5QI

o process DSCP markings set by cellular endpoint devices

This document further assumes and RECOMMENDS that all cellular endpoint devices (UE) support the ability to:

o mark DSCP, per Diffserv standards

o mark QCI, per the [TS 23.203] standard, OR 5QI, per the [TS 23.501] standard

o support fully-configurable mappings between DSCP (set by applications in software) and QCI or 5QI (set by the operating system and/or the LTE infrastructure)

Having made the assumptions and recommendations above, it bears mentioning that while the mappings presented in this document are RECOMMENDED to replace the current common default practices (as discussed in Section 2.3 and Section 2.4), these mapping recommendations are not expected to fit every last deployment model, and as such MAY be overridden by network administrators, as needed.

4. DSCP to QCI or 5QI Mapping Recommendations

4.1. Control Traffic

4.1.1. Network Control Protocols

The Network Control service class is used for transmitting packets between network devices (e.g., routers) that require control (routing) information to be exchanged between nodes within the administrative domain, as well as across a peering point between different administrative domains.

[RFC4594] Section 3.2 recommends that Network Control Traffic be marked CS6 DSCP. Additionally, as stated in [RFC4594] Section 3.1: "CS7 DSCP value SHOULD be reserved for future use, potentially for future routing or control protocols."

Network Control service is not directly called by any specific QCI or 5QI description, because 3GPP network control does not operate over UE data channels. It should be noted that encapsulated routing

Henry, et al. Expires October 15, 2020 [Page 19] Internet-Draft DIFFSERV-QCI April 2020

protocols for encapsulated or overlay networks (e.g., VPN, Network Virtualization Overlays, etc.) are not Network Control Traffic for any physical network at the cellular space; hence, they SHOULD NOT be marked with CS6 in the first place, and are not expected to be forwarded to the cellular data plane.

However, when such network control traffic is forwarded, it is expected to receive a high priority and level of service. As such, packets marked to CS7 DSCP are RECOMMENDED to be mapped to QCI 82, thus benefiting from a dedicated bearer with low packet error loss rate (10.E-4) and low budget delay (10 ms). Similarly, it is RECOMMENDED to map Network Control Traffic marked CS6 to QCI/5QI 82, thereby admitting it to the Discrete Automation (GBR) category with a relative priority level of 1.9/19.

4.1.2. Operations, Administration, and Maintenance (OAM)

The OAM (Operations, Administration, and Maintenance) service class is recommended for OAM&P (Operations, Administration, and Maintenance and Provisioning). The OAM service class can include network management protocols, such as SNMP, Secure Shell (SSH), TFTP, Syslog, etc., as well as network services, such as NTP, DNS, DHCP, etc.

[RFC4594] Section 3.3, recommends that OAM traffic be marked CS2 DSCP.

Applications using this service class require a low packet loss but are relatively not sensitive to delay. This service class is configured to provide good packet delivery for intermittent flows. As such, packets marked to CS2 are RECOMMENDED to be mapped to QCI/5QI 9, thus admitting it to the non-GBR Buffered video traffic, with a relative priority of 9/90.

4.2. User Traffic

User traffic is defined as packet flows between different users or subscribers. It is the traffic that is sent to or from end-terminals and that supports a very wide variety of applications and services [RFC4594] Section 4.

Network administrators can categorize their applications according to the type of behavior that they require and MAY choose to support all or a subset of the defined service classes.

Henry, et al. Expires October 15, 2020 [Page 20] Internet-Draft DIFFSERV-QCI April 2020

4.2.1. Telephony

The Telephony service class is recommended for applications that require real-time, very low delay, very low jitter, and very low packet loss for relatively constant-rate traffic sources (inelastic traffic sources). This service class SHOULD be used for IP telephony service. The fundamental service offered to traffic in the Telephony service class is minimum jitter, delay, and packet loss service up to a specified upper bound. [RFC4594] Section 4.1 recommends that Telephony traffic be marked EF DSCP.

3GPP [TS 23.203] describes two QCIs adapted to Voice traffic: QCI 1 (GBR) and QCI 7 (non-GBR). The same logic is found in [TS 23.501] for the same 5QIs. However, Telephony traffic as intended in [RFC4594] supposes resource allocation control. Telephony SHOULD be configured to receive guaranteed forwarding resources so that all packets are forwarded quickly. The Telephony service class SHOULD be configured to use Priority Queuing system. QCI 7 does not match these conditions. As such, packets marked to EF are RECOMMENDED to be mapped to QCI/5QI 1, thus admitting it to the GBR Conversational Voice category, with a relative priority of 2/20.

4.2.2. Signaling

The Signaling service class is recommended for delay-sensitive client-server (e.g., traditional telephony) and peer-to-peer application signaling. Telephony signaling includes signaling between 1) IP phone and soft-switch, 2) soft-client and soft-switch, and 3) media gateway and soft-switch as well as peer-to-peer using various protocols. This service class is intended to be used for control of sessions and applications. [RFC4594] Section 4.2 recommends that Signaling traffic be marked CS5 DSCP.

While Signaling is recommended to receive a superior level of service relative to the default class (e.g., relative to QCI 7), it does not require the highest level of service (i.e., GBR and very high priority). As such, it is RECOMMENDED to map Signaling traffic marked CS5 DSCP to QCI/5QI 4, thereby admitting it to the GBR Non- conversational video category, with a relative priority level of 5/50.

Note: Signaling traffic for native Voice dialer applications should be exchanged over a control channel, and is not expected to be forwarded in the data-plane. However, Signaling for non-native (OTT) applications may be carried in the data-plane. In this case, Signaling traffic is control-plane traffic from the perspective of the voice/video telephony overlay-infrastructure. As such, Signaling

Henry, et al. Expires October 15, 2020 [Page 21] Internet-Draft DIFFSERV-QCI April 2020

should be treated with preferential servicing versus other data-plane flows.

4.2.3. Multimedia Conferencing

The Multimedia Conferencing service class is recommended for applications that require real-time service for rate-adaptive traffic. [RFC4594] Section 4.3 recommends Multimedia Conferencing traffic be marked AF4x (that is, AF41, AF42, and AF43, according to the rules defined in [RFC2475]. The Diffserv model allows for three values to allow for different relative priorities of flows of the same nature.

The primary media type typically carried within the Multimedia Conferencing service class marked AF41 is video intended to be a component of a real-time exchange; as such, it is RECOMMENDED to map AF41 into the Conversational Video (Live Streaming) category, with a GBR. Specifically, it is RECOMMENDED to map AF41 to QCI/5QI 2, thereby admitting AF41 into the GBR Conversational Video, with a relative priority of 4/40.

AF42 is typically reserved for video intended to be a component of real-time exchange, but which criticality is less than traffic carried with a marking of AF41. As such, it is RECOMMENDED to map AF42 into the Conversational Video (Live Streaming) category, with a GBR, but a lower priority than QCI/5QI 2. Specifically, it is RECOMMENDED to map AF42 to QCI/5QI 4, thereby admitting AF42 into the GBR Conversational Video, with a relative priority of 5/50.

Traffic marked AF43 is typically used for real-time video exchange of lower criticality. As such, it is RECOMMENDED to map AF43 into the Conversational Video (Live Streaming) category, but without a GBR. Specifically, it is RECOMMENDED to map AF43 to QCI/5QI 7, thereby admitting AF437 into the non-GBR Voice, Video and Interactive gaming, with a relative priority of 7/70.

4.2.4. Real-Time Interactive

The Real-Time Interactive service class is recommended for applications that require low loss and jitter and very low delay for variable-rate inelastic traffic sources. Such applications may include inelastic video-conferencing applications, but may also include gaming applications (as pointed out in [RFC4594] Sections 2.1 through 2.3 and Section 4.4. [RFC4594] Section 4.4 recommends Real- Time Interactive traffic be marked CS4 DSCP.

The primary media type typically carried within the Real-Time Interactive service class is video; as such, it is RECOMMENDED to map

Henry, et al. Expires October 15, 2020 [Page 22] Internet-Draft DIFFSERV-QCI April 2020

this class into a low latency Category. Specifically, it is RECOMMENDED to map CS4 to QCI 80, thereby admitting Real-Time Interactive traffic into the non-GBR category Low Latency eMBB (enhanced Mobile Broadband) applications with a relative priority of 6.8. In cases where GBR is required, for example because a single bearer is allocated for all non-GBR traffic, using a GBR equivalent is also acceptable. In this case, it is RECOMMENDED to map CS4 to QCI/5QI 3, thereby admitting Real-Time Interactive traffic into the GBR category Real-time gaming, with a relative priority of 3/30.

4.2.5. Multimedia Streaming

The Multimedia Streaming service class is recommended for applications that require near-real-time packet forwarding of variable-rate elastic traffic sources. Typically, these flows are unidirectional. [RFC4594] Section 4.5 recommends Multimedia Streaming traffic be marked AF3x (that is, AF31, AF32, and AF33, according to the rules defined in [RFC2475].

The primary media type typically carried within the Multimedia Streaming service class is video; as such, it is RECOMMENDED to map this class into a Video Category. Specifically, it is RECOMMENDED to map AF31 to QCI/5QI 4, thereby admitting AF31 into the GBR Non Conversational Video category, with a relative priority of 5/50.

Flows marked with AF32 are expected to be of the same nature as flows marked with AF32, but with a lower criticality. As such, these flows may not require a dedicated bearer with GBR. Therefore, it is RECOMMENDED to map AF32 to QCI/5QI 6, thereby admitting AF32 traffic into the non-GBR category Video (Buffered Streaming) with a relative priority of 6/60.

Flows marked with AF33 are expected to be of the same nature as flows marked with AF31 and AF32, but with the lowest criticality. As such, it is RECOMMENDED to map AF33 to QCI/5QI 8, thereby admitting AF33 traffic into the non-GBR category Video (Buffered Streaming) with a relative priority of 8/80.

4.2.6. Broadcast Video

The Broadcast Video service class is recommended for applications that require near-real-time packet forwarding with very low packet loss of constant rate and variable-rate inelastic traffic sources. Typically, these flows are unidirectional. [RFC4594] Section 4.6 recommends Broadcast Video traffic be marked CS3 DSCP.

As directly implied by the name, the primary media type typically carried within the Broadcast Video service class is video; as such,

Henry, et al. Expires October 15, 2020 [Page 23] Internet-Draft DIFFSERV-QCI April 2020

it is RECOMMENDED to map this class into a Video Category. Specifically, it is RECOMMENDED to map CS3 to QCI/5QI 4, thereby admitting Multimedia Streaming into the GBR Non Conversational Video category, with a relative priority of 5/50. In cases where GBR availability is constrained, using a non-GBR equivalent is also acceptable. In this case, it is RECOMMENDED to map CS3 to QCI/5QI 6, thereby admitting Real-Time Interactive traffic into the non-GBR category Video with a relative priority of 6/60.

4.2.7. Low-Latency Data

The Low-Latency Data service class is recommended for elastic and time-sensitive data applications, often of a transactional nature, where a user is waiting for a response via the network in order to continue with a task at hand. As such, these flows are considered foreground traffic, with delays or drops to such traffic directly impacting user productivity. [RFC4594] Section 4.7 recommends Low- Latency Data be marked AF2x (that is, AF21, AF22, and AF23, according to the rules defined in [RFC2475].

The primary media type typically carried within the Low-Latency Data service class is data; as such, it is RECOMMENDED to map this class into a data Category. Specifically, it is RECOMMENDED to map AF21 to QCI/5QI 70, thereby admitting AF21 into the non-GBR Mission Critical Data category, with a relative priority of 5.5/55.

Flows marked with AF22 are expected to be of the same nature as flows marked with AF21, but with a lower criticality. Therefore, it is RECOMMENDED to map AF22 to QCI/5QI 6, thereby admitting AF22 traffic into the non-GBR category Video and TCP-based traffic, with a relative priority of 6/60.

Flows marked with AF23 are expected to be of the same nature as flows marked with AF21 and AF22, but with the lowest criticality. As such, it is RECOMMENDED to map AF23 to QCI/5QI 8, thereby admitting AF23 traffic into the non-GBR category Video and TCP-based traffic, with a relative priority of 8/80.

It should be noted that a consequence of such classification is that AF22 is mapped to the same QCI and 5QI as CS3, and AF23 is mapped to the same QCI and 5QI as AF33. However, this overlap is unavoidable, as some QCIs and 5QIs express intents that are expressed in the Diffserv domain through distinct marking values, grouped in the 3GPP domain under the same general category.

Henry, et al. Expires October 15, 2020 [Page 24] Internet-Draft DIFFSERV-QCI April 2020

4.2.8. High-Throughput Data

The High-Throughput Data service class is recommended for elastic applications that require timely packet forwarding of variable-rate traffic sources and, more specifically, is configured to provide efficient, yet constrained (when necessary) throughput for TCP longer-lived flows. These flows are typically not user interactive.

According to [RFC4594] Section 4.8 it can be assumed that this class will consume any available bandwidth and that packets traversing congested links may experience higher queuing delays or packet loss. It is also assumed that this traffic is elastic and responds dynamically to packet loss. [RFC4594] Section 4.8 recommends High- Throughput Data be marked AF1x (that is, AF11, AF12, and AF13, according to the rules defined in [RFC2475].

The primary media type typically carried within the High-Throughput Data service class is data; as such, it is RECOMMENDED to map this class into a data Category. Specifically, it is RECOMMENDED to map AF11 to QCI/5QI 6, thereby admitting AF11 into the non-GBR Video and TCP-based traffic category, with a relative priority of 6/60.

Flows marked with AF12 are expected to be of the same nature as flows marked with AF11, but with a lower criticality. Therefore, it is RECOMMENDED to map AF12 to QCI/5QI 8, thereby admitting AF12 traffic into the non-GBR category Video and TCP-based traffic, with a relative priority of 8/80.

Flows marked with AF13 are expected to be of the same nature as flows marked with AF11 and AF12, but with the lowest criticality. As such, it is RECOMMENDED to map AF13 to QCI/5QI 9, thereby admitting AF13 traffic into the non-GBR category Video and TCP-based traffic, with a relative priority of 9/90.

It should be noted that a consequence of such classification is that AF11 is mapped to the same QCI as CS3 and AF22, AF12 is mapped to the same QCI and 5QI as Af33 and AF23, and AF13 is mapped to the same QCI and 5QI as CS2. However, this overlap is unavoidable, as some QCIs and 5QIs express intents that are expressed in the Diffserv domain through distinct marking values, grouped in the 3GPP domain under the same general category.

4.2.9. Standard

The Standard service class is recommended for traffic that has not been classified into one of the other supported forwarding service classes in the Diffserv network domain. This service class provides the Internet’s "best-effort" forwarding behavior. [RFC4594]

Henry, et al. Expires October 15, 2020 [Page 25] Internet-Draft DIFFSERV-QCI April 2020

Section 4.9 states that the "Standard service class MUST use the Default Forwarding (DF) PHB".

The Standard service class loosely corresponds to the default non-GBR bearer practice in 3GPP. Therefore, it is RECOMMENDED to map Standard service class traffic marked DF DSCP to QCI/5QI 9, thereby admitting it to the low priority Video and TCP-based traffic category, with a relative priority of 9/90.

4.2.10. Low-Priority Data

The Low-Priority Data service class serves applications that the user is willing to accept without service assurances. This service class is specified in [RFC3662] and [RFC8622]. [RFC3662] and [RFC4594] both recommend Low-Priority Data be marked CS1 DSCP. [RFC8622] updates these recommendations and suggests the LE (000001) marking. As such, this document aligns with this recommendation and notes that CS1 marking has become ambiguous.

The Low-Priority Data service class does not have equivalent in the 3GPP domain, where all service is controlled and allocated differentially. As such, there is no clear QCI or 5QI that could be labelled low priority below the best effort category. As such, it is RECOMMENDED to map Low-Priority Data traffic marked CS1 DSCP and LE DSCP to QCI/5QI 9, thereby admitting it to the low priority Video and TCP-based traffic category, with a relative priority of 9/90.

4.3. Summary of Recommendations for DSCP-to-QCI Mapping

The table below summarizes the [RFC4594] DSCP marking recommendations mapped to 3GPP:

Henry, et al. Expires October 15, 2020 [Page 26] Internet-Draft DIFFSERV-QCI April 2020

+------+------+------+------+ | DSCP | Recommended | Resource Type | Priority Level | | | QCI/5QI | | (QCI/5QI) | +------+------+------+------+ | CS7 | 82 | GBR | 1.9 / 19 | | | | | | | CS6 | 82 | GBR | 1.9 / 19 | | | | | | | EF | 1 | GBR | 2 / 20 | | | | | | | CS5 | 4 | GBR | 5 / 50 | | | | | | | AF43 | 7 | non-GBR | 7 / 70 | | | | | | | AF42 | 4 | GBR | 5 / 50 | | | | | | | AF41 | 2 | GBR | 4 / 40 | | | | | | | CS4 | 80 3 | non-BGR GBR | 6.8 / 68, 3 / 30 | | | | | | | AF33 | 8 | non-GBR | 8 / 80 | | | | | | | AF32 | 6 | non-GBR | 6 / 60 | | | | | | | AF31 | 4 | GBR | 5 / 50 | | | | | | | CS3 | 85 | GBR | 2.1 / 21 | | | | | | | AF23 | 8 | Non-GBR | 8 / 80 | | | | | | | AF22 | 6 | Non-GBR | 6 / 60 | | | | | | | AF21 | 70 | Non-GBR | 5.5 / 55 | | | | | | | CS2 | 9 | Non-GBR | 9 / 90 | | | | | | | AF13 | 9 | Non-GBR | 9 / 90 | | | | | | | AF12 | 8 | Non-GBR | 8 / 80 | | | | | | | AF11 | 6 | Non-GBR | 6 / 60 | | | | | | | CS0 | 9 | Non-GBR | 9 / 90 | | | | | | | CS1 | 9 | Non-GBR | 6.8 / 68 | | | | | | | LE | 9 | Non-GBR | 6.8 / 68 | +------+------+------+------+

Henry, et al. Expires October 15, 2020 [Page 27] Internet-Draft DIFFSERV-QCI April 2020

5. QCI and 5QI to DSCP Mapping Recommendations

Traffic travelling from the 3GPP domain toward the Internet or the enterprise domain may already display DSCP marking, if the UE is capable of marking DSCP along with, or without, upstream QCI bearer or 5QI marking, as detailed in Section 2.1.

When Diffserv marking is present in the flows originating from the UE and transiting through the CN (Core Network), and if Diffserv marking are not altered or removed on the path toward the Diffserv domain, then the network can be considered as end-to-end Diffserv compliant. In this case, it is RECOMMENDED that the entity providing the translation from 3GPP to Diffserv ignores the QCI or 5QI value and simply forwards unchanged the Diffserv values expressed by the UE in its various flows.

This general recommendation is not expected to fit every last deployment model, and as such Diffserv marking MAY be overridden by network administrators, as needed, before the flows are forwarded to the Internet, the enterprise network or the Diffserv domain in general. Additionally, within a given Diffserv domain, it is generally NOT RECOMMENDED to pass through DSCP markings from unauthenticated, unidentified or unauthorized devices, as these are typically considered untrusted sources, as detailed in Section 7. Such risk is limited within the 3GPP domain where no upstream traffic is admitted without prior authentication of the UE. However, this risk exists when UE traffic is forwarded to an enterprise domain to which the UE does not belong.

In cases where the UE is unable to apply Diffserv marking, or if these markings are modified or removed within the 3GPP domain, such that these markings may not represent the intent expressed by the UE, and in cases where the QCI is available to represent the flow intent, the recommendations in this section apply. These recommendations MAY apply to the boundary between the 3GPP and the Diffserv model, and MAY also apply to the Diffserv domain, when a given applicaiton traffic flows through both the 3GPP and the Diffserv domains (e.g. multiple paths) and when the enteprise administrator wishes to ensure that the same QoS intent is applied for both paths.

5.1. QCI, 5QI and Diffserv Logic Reconciliation

The QCIs and 5QIs are defined as relative priorities for traffic flows which are described by combinations of 6 or more parameters, as expressed in Section 2.2. As such, QCIs and 5QIs also represent flows in terms of multi-dimensional needs, not just in terms of relative priorities. This multi-dimensional logic is different from the Diffserv logic, where each traffic class is represented as a

Henry, et al. Expires October 15, 2020 [Page 28] Internet-Draft DIFFSERV-QCI April 2020

combination of needs relative to delay, jitter and loss. This characterization around three parameters allows for the construction of a fairly hierarchical traffic categorization infrastructure, where traffic with high sensitivity to delay and jitter also typically has high sensitivity to loss.

By contrast, the 3GPP QCI and 5QI structure presents multiple points where dimensions cross one another with different or opposing vectors. For example, IMS signaling (QCI or 5QI 5) is defined with very high priority (1/10), low loss tolerance (10-6), but is non-GBR and belongs to the signaling category. By contrast, Conversational voice (QCI or 5QI 1) has lower priority (2/20) than IMS signaling, higher loss tolerance (10-2), yet benefits from a GBR. Fitting both QCIs or 5QIs 5 and 1 in a hierarchical model is challenging.

At the same time, QCIs and 5QIs represent needs that can apply to different applications of various criticality but sending flows of the same nature. For example, QCIs or 5QIs 6, 8 and 9 all include voice traffic, video traffic, but also email or FTP. What distinguish these QCIs/5QIs is the criticality of the associated traffic. Diffserv does not envisions voice and FTP as possibly belonging to the same class. As the same time, QCIs or 5QIs 2 and 9 include real-time voice traffic. Diffserv does not allow a type of traffic with stated sensitivity to loss, delay and jitter to be split into categories at both end of the priority spectrum.

As such, it is not expected that QCIs and 5QIs can be mapped to the Diffserv model strictly and hierarchically. Instead, a better approach is to observe the various QCI and 5QI categories, and analyze their intent. This process allows for the grouping of several QCIs or 5QIs into hierarchical groups, that can then be translated into ensembles coherent with the Diffserv logic. This approach, in turn, allows for incorporation of new QCIs and 5QIs as the 3GPP model continues to evolve.

It should be noted, however, that such approach results in partial incompatibility. Some QCIs or 5QIs represent an intent that is simply not present in the Diffserv model. In that case, attempting to artificially stitch the QCI/5QI to an existing Diffserv traffic class and marking would be dangerous. QCI or 5QI traffic forwarded to the Diffserv domain would be mixed with Diffserv traffic that would represent a very different intent.

As such, the result of this classification is that some QCIs and 5QIs call for new Diffserv traffic classes and markings. This consequence is preferable to mixing traffic of different natures into the same pre-existing category.

Henry, et al. Expires October 15, 2020 [Page 29] Internet-Draft DIFFSERV-QCI April 2020

Each QCI is represented with 6 parameters and each 5QI with 7 parameters, including an Example Services value. This parameter is representative of the QCI or 5QI intent. Although [TS 23.203] and [TS 23.501] summarize each QCI or 5QI intent, these standards contain only summaries of more complex classifications expressed in other 3GPP standards. It is often necessary to refer to these other standards to obtain a more complete description of each QCI/5QI and the multiple type of flows that each QCI or 5QI represents.

For the purpose of this document, the QCI or 5QI intent is the primary classification driver, along with the priority level. The secondary elements, such as priority, delay budget and loss tolerance allow for better refinement of the relative classifications of the QCIs and 5QIs. The resource types (GBR, DElay-critical GBR, non-GBR) provide additional visibility into the intent.

Although 26 QCIs are listed in [TS 23.203] and 27 5QIs in [TS 23.501], representing two (GBR, non-GBR) or three resource types (GBR, non-GBR, Delay-Critical GBR) respectively, 21 and 22 priority values, 9 delay budget values, and 7 loss tolerance values, examining the intent in fact surfaces 9 traffic families:

1. Voice QCI/5QI [1] (dialer / conversational voice) is its own group

2. Voice signaling [5] (IMS) is its own group

3. Voice related (other voice applications, including PTT) [65, 66, 69]

4. Video (conversational or not, mission critical or not) [67, 2, 4, 71, 72, 73, 74, 76]

5. Live streaming / interactive gaming is its own group [7]

6. Low latency eMBB, AR/VR is its own group [80]

7. V2X messaging [75, 3, 9]

8. Automation and Transport [82, 83, 84, 85, 86]

9. Non-mission-critical data [6, 8, 9]

10. Mission-critical data is its own group [70]

Henry, et al. Expires October 15, 2020 [Page 30] Internet-Draft DIFFSERV-QCI April 2020

5.2. Voice [1]

Several QCIs or 5QIs are intended to carry voice traffic. However, QCI/5QI 1 stands apart from the others. Its category is Conversational Voice, but this QCI/5QI is intended to represent the VoLTE voice bearer, for dialer and emergency services. QCI/5QI 1 uses a GBR, and has a priority level of 2/20. Its packet delay budget is 100 ms (from UE to P-GW) with a packet error loss of at most 10.E-2. As the GBR is allocated by the infrastructure, QCI/5QI 1 is both admitted and allocated dedicated resources. As such, QCI/5QI 1 maps in intent and function to [RFC5865], Admitted Voice, and is RECOMMENDED for mapping to DSCP 44.

5.3. IMS Signaling [5]

QCI/5QI 5 is intended for Signaling. This category does not represent signaling for VoLTE, as such signaling is not conducted over the UE data channels. Instead, QCI/5QI 5 is intended for IMS services. IP Multimedia System (IMS) is a framework for delivering multimedia services over IP networks. These services include real- time and video applications, and their signaling is recommended to be carried, whenever possible, using IETF protocols such as SIP. Being of signaling nature, QCI/5QI 5 is non-GBR. However, being critical to enabling IMS real-time applications, QCI/5QI 5 has a high priority of 1/10. Its packet delay budget is 100 ms, but packet error loss rate very low, at less than 10.E-6. Overall, QCI/5QI 5 maps rather well to the intent of [RFC4594] signaling for real time applications, and as such is RECOMMENDED to map to [RFC4594] Signaling, CS5.

5.4. Voice-related QCIs and 5QIs [65, 66, 69]

Several QCIs/5QIs display the commonality of targeting voice (non- VoLTE) traffic:

o QCI/5QI 65 is GBR, mission critical PTT voice, priority 0.7/7

o QCI/5QI 66 is GBR, non-mission critical PTT voice, priority 2/20

o QCI/5QI 69 is non-GBR, mission-critical PTT signaling, priority 0.5/5

These QCIs/5QIs are Voice in nature, and naturally fit into a proximity marking model with DSCP 46 and 44.

Additionally, lower priority marks higher precedence intent in QCI and 5QI. However, there is no model in [RFC4594] that distinguishes 3 classes of voice traffic. Therefore, new markings are unavoidable. As such, there is a need to group these markings in the Voice

Henry, et al. Expires October 15, 2020 [Page 31] Internet-Draft DIFFSERV-QCI April 2020

category (101 xxx), and to order 69, 65 and 66 with different markings to reflect their different priority levels.

Among these three QCIs/5QIs, 69 is non-GBR, intended for mission- critical PTT signaling, with the highest priority of the three, at 0.5/5. 69 is intended for signaling, but is latency sensitive, with a low 60 ms delay budget and a low 10.E-6 loss tolerance. Being of Signaling nature for real time applications, QCI/5QI 69 has proximity of intent with CS5 (Voice signaling, 40), but this marking is already used by QCI/5QI 5. Therefore, it is RECOMMENDED to map QCI/5QI 69 to a new DSCP marking, 41.

Similarly, QCI/5QI 66 is GBR and targeted for non-mission critical PTT voice, with a priority level of 2/20. 66 is Voice in nature, and GBR. However, 66 is intended for non-mission-critical traffic, and has a lower priority than mission-critical Voice, a higher tolerance for delay (100 ms vs 75). As such, 66 cannot fit within [RFC4594] model mapping real-time voice to the class EF (DSCP 46). Here again, a new marking is needed. As such, this QCI/5QI fits in intent and proximity closest to Admitted Voice, but is non-GBR, and therefore non-admitted, guiding a new suggested DSCp marking of 43.

Then, QCI/5QI 65 is GBR, intended for mission critical PTT voice, with a relative low priority index of 0.7/7. QCI/5QI 65 receives GBR and is intended for mission critical traffic. Its priority is higher (0.7 vs 2) than QCI/5QI 66, but a lower priority (0.7/7 vs 0.5/5) than QCI/5QI 69. Additionally, 65 cannot be represented by DSCP 44 (used by QCI/5QI 1), or DSCP 46 (used by non-GBR voice). As such, QCI/5QI 65 fits between QCIs/5QIs 69 66, with a new suggested DSCP marking of 42.

5.5. Video QCIs and 5QIs [67, 2, 4, 71, 72, 73, 74, 76]

Although six different QCIs and 5QIs have example services that include some form of video traffic, eight QCIs and 5QIs are video in nature, 67, 2, 4, 71, 72, 73, 74, and 76.

All eight QCIs/5QIs represent video streams and fit naturally in the AF4x category. However, these QCIs/5QIs do not match [RFC4594] intent for multimedia conferencing, in that they are all admitted (being associated to a GBR). They also do not match the category described by [RFC5865] for capacity-admitted traffic. Therefore, there is not a clear possible mapping for any of these QCIs and 5QIs to an existing AF4x category. In order to avoid mixing admitted and non-admitted video in the same class, it is necessary to associate these QCIs/5QIs to new Diffserv classes.

Henry, et al. Expires October 15, 2020 [Page 32] Internet-Draft DIFFSERV-QCI April 2020

In particular, QCI/5QI 67 is GBR, intended for mission-critical video user plane. This QCI/5QI is video in nature, and matches traffic that is rate-adaptive, and real time. 67 priority is high (1.5/15), with a tolerant delay budget (100ms) and rather low loss tolerance (10.E-3). 67 is GBR.

As such, it is RECOMMENDED to map QCI/5QI 67 against the DSCP value closest to AF4x video with lowest discard eligibility (AF41), namely DSCP 33.

Similarly, QCI/5QI 2 is intended for conversational video (live streaming). 2 is also video in nature and associated to a GBR, however its priority is lower than 67 (4/40 vs 1.5/15). Additionally, its delay budget is also larger (150 ms vs 100 ms). Its packet error loss is also 10.E-3. As such, 2 fits well within a video queue, with a larger drop probability than 67. Therefore, it is RECOMMENDED to map QCI/5QI 2 to the video category with a Diffserv marking of 35.

QCIs/5QIs 71, 72, 73, 74 and 76 are intended for "Live" Uplink Streaming (LUS) services, where an end-user with a radio connection (for example a reporter or a drone) streams live video feed into the network or to a second party ([TS 26.939]). This traffic is GBR. However, [TS 26.939] defines LUS and also differentiates GBR from MBR and TBR. At the time of the admission, the infrastructure can offer a Guaranteed Bit Rate, which should match the bare minimum rate expected by the application (and its codec). Because of the burstiness nature of video, the Maximum Bit Rate (MBR) available to the trannsmission should be much higher than the GBR. In fact, the Target Bit Rate (TBR), which is the prefered service operation point for that application, is likely close to the MBR. Thus, the application will receive a treatment between the GBR and the TBR. This allocated bit rate will directly translate in video quality changes, where an available bit rate close to the GBR will result in a lower Mean Opinion Score than a bit rate close to the TBR. As the application detects the contraints on the available bit rate, it may adapt by changing its codec and compression scheme accordingly. Flows with higher compression will have higher delay tolerance and budget (as a single packet burst represents a larger segment of the video flow) but lower loss tolerance (as each lost packet represents a larger segment of the video flow). As such, 71, 72, 73, 74 and 76 express intents similar to QCI/5QI 2, with additional constraints on the directionality of the flow (upstream only) and the bit rate applied by the infrastructure. These constraints are orthogonal to the intent of the flow. As such, it is RECOMMENDED to map QCIs/5QIs 71, 72, 73, 74 and 76 to the same DSCP value as QCI/5QI 2, and thus to the video category with a Diffserv marking of 35.

Henry, et al. Expires October 15, 2020 [Page 33] Internet-Draft DIFFSERV-QCI April 2020

QCI/5QI 4 is intended for non-conversational video (buffered streaming), with a priority of 5/50. 4 is also video in nature. Although it is buffered, it is admitted, being associated to a GBR. QCI/5QI 4 as a lower priority than QCIs/5QIs 67 and 2, and a larger delay budget (300 ms vs 150/100). However, its packet loss tolerance is low (10.E-6). This combination makes it eligible for a video category, but with a higher drop probability than 67 and 2. Therefore, it is RECOMMENDED to map QCI/5QI 4 to DSCP 37.

5.6. Live streaming and interactive gaming [7]

QCI/5QI 7 is non-GBR and intended for live streaming voice or video interactive gaming. Its priority is 7/70. It is the only QCI/5QI targeting this particular traffic mix. In the Diffserv model, voice and video are different categories, and are also different from interactive gaming (real time interactive). In the 3GPP model, live streaming video and mission-critical video are defined in other queues with high priority (e.g. QCI or 5QI 2 for video Live streaming, with a priority of 2/20, or QCI/5QI 67 for mission- critical video, with a priority of 1.5/15). By comparison, QCI/5QI 7 priority is relatively low (7/70), with a 100 ms budget delay and a comparatively rather high loss tolerance (10.E-3).

As such, 7 fits well with bursty (e.g. video) and possibly rate adaptive flows, with possible drop probability. It is also non- admitted (non-GBR), and as such, fits close to [RFC4594] intent for multimedia conferencing, with high discard eligibility. Therefore, it is RECOMMENDED to map QCI/5QI 7 to the existing Diffserv category AF43.

5.7. Low latency eMBB and AR/VR [80]

QCI/5QI 80 is intended for low latency eMBB (enhanced Mobile Broadband) applications, such as Augmented Reality of Virtual Reality (AR/VR). 80 priority is 6.8/68, with a low packet delay budget of 10 ms, and a packet error loss rate of at most 10.E-6. 80 is non-GBR, yet intended for real time applications. Traffic in the AR/VR category typically does not react dynamically to losses, requires bandwidth and a low and predictable delay.

As such, QCI/5QI 80 matches closely the specifications for CS4. Therefore, it is RECOMMENDED to map QCI/5QI 80 to the existing category CS4.

Henry, et al. Expires October 15, 2020 [Page 34] Internet-Draft DIFFSERV-QCI April 2020

5.8. V2X messaging [75,3,9]

Three QCIs/5QIs are intended specifically to carry Vehicle to Anything (V2X) traffic, 75, 3, and 79. All 3 QCIs/5QIs are data in nature, and fit naturally into the AF2x category. However, two of these (75 and 3) are admitted (GBR), and therefore do not fit in the current Diffserv model. 79 is non-admitted, but matches none of the AF2X categories in [RFC4594].

In particular, QCI/5QI 75 is GBR, with a rather high priority (2.5/25), a low delay budget (50 ms), but tolerance to losses (10E- 2). Being low latency data in nature, 75 fits well in the AF2X category. However, being admitted, it fits none of the existing markings. Being the highest traffic (in priority) in this low latency data family, 75 is recommended to be mapped to a new category, as close as possible to the AF2X class, and with a low drop probability. As such, it is RECOMMENDED to map QCI/5QI 75 to DSCP 17.

Similarly, QCI/5QI 3 is intended for V2X messages, but can also be used for Real time gaming, or Utility traffic (medium voltage distribution) or process automation monitoring. QCI/5QI 3 priority is 3/30. 3 is data in nature, but GBR. Its delay budget is low (50 ms), but with some tolerance to loss (10E-3).

QCI/5QI 3 is of the same type as QCI/5QI 75, but with a lower priority. Therefore, 3 should be mapped to a category close to the category to which 75 is mapped, but with a higher drop probability. As such, it is RECOMMENDED to map QCI/5QI 3 to DSCP 19.

Additionally, QCI/5QI 79 is also intended for V2X messages. 79 is similar in nature to 75 and 3, but is non-critical (non-GBR). Its priority is also lower (6.5/65). Its budget delay is similar to that of 75 and 3 (50 ms), and its packet error loss rate is similar to that of 75 (10.E-2).

79 partially matches AF2X, but is not elastic, and therefore cannot fit exactly in [RFC4594] model. As such, it is recommended to a mapping similar to QCI/5QIs 75 and 3, with a higher drop probability. Therefore, it is RECOMMENDED to map QCI/5QI 79 to DSCP 21.

5.9. Automation and Transport [82, 83, 84, 85, 86]

QCI/5QI 84 is intended for intelligent transport systems. As such, its intent is close to the V2X messaging category. QCI 84 is also admitted (GBR in [TS 23.203] and Delay-Critical GBR in [TS 23.501]). However, 84 is intended for traffic with a smaller packet delay budget (30 ms vs 50 ms for QCI/5QI 75) and a smaller packet error

Henry, et al. Expires October 15, 2020 [Page 35] Internet-Draft DIFFSERV-QCI April 2020

loss maximum rate (10.E-6 vs 10.E-2 for QCI/5QI 75). As such, 84 should be mapped against a category above that of 75 or 3. Being admitted, 84 does not map easily into an existing category. As such, it is RECOMMENDED to map QCI/5QI 84 to DSCP category 31.

5QI 86 is also intended for intelligent transport systems, and fits in the same general category as 84. 86 is also admitted (Delay- Critical GBR), with a higher priority (18) than 84 but similar burst rate (1354 bytes). 5QI 86 therefore fits into a category close to that of 84. As such, it is RECOMMENDED to map 5QI 86 to DSCP captegory 29.

QCI/5QI 85 is intended for electricity distribution (high voltage) communication. As such, it is close in intent to QCI/5QI 3. 85 is also GBR. However, 85 priority is lower than that of QCI/5QI 3 (2.1/21 vs 3/30). 85 has also a very low packet delay budget (5 ms vs 50 ms for QCI/5QI 3) and low packet error loss rate (10.E-6 vs 10.E-3 for QCI/5QI 3). As such, 84 should be mapped to a category higher than that of QCI/5QI 3,with a very low drop probability. As such, it is RECOMMENDED to map QCI/5QI 85 to DSCP category 23.

QCIs/5QIs 82 and 83 are both intended for discrete automation control traffic. 82 represents traffic with a higher priority (1.9/19) than traffic matched to 83 (priority 2.2/22). 82 also expects smaller data bursts (255 bytes) than 83 (1358 bytes). However, both QCIs are admitted (GBR), with the same low packet delay budget (10 ms) and packet error loss maximum rate (10.E-4).

As such, 82 and 83 fit in the same general category, with a higher drop probability assigned to 83. They also fit the general intent category of automation traffic types, with a priority higher than that of other M2M traffic types (e.g. V2X messages). As such, they fit well into the AF3X category. However, being both admitted (GBR), they do not easily map to any existing AF3X category, and require new categories.

As such, it is RECOMMENDED to map QCI/5QI 82 to DSCP category 25. Similarly, it is RECOMMENDED to map QCI/5QI 83 to DSCP category 79.

5.10. Non-mission-critical data [6,8,9]

QCIs/5QIs 6, 8 and 8 are intended for non-GBR, Video or TCP data traffic. All 3 QCIs/5QIs are data in nature, non-mission critical, relative low priority and therefore fit naturally into the AF1x category. The inclusion in these QCIs/5QIs’ intent of buffered video is an imperfect fit for AF1X. However, the intent of these QCIs/5QIs is to match buffered, and non-mission critical traffic. As such, they match the intent of AF1X, even if the Diffserv model would not

Henry, et al. Expires October 15, 2020 [Page 36] Internet-Draft DIFFSERV-QCI April 2020

associate buffered video to non-mission critical, buffered and low priority traffic.

The intent of all three QCIs/5QIs is similar. The difference lies in their priority and criticality.

QCI/5QI 6 has priority 6/60, a packet delay budget of 300 ms, and a packet error loss rate of at most 10.E-6. QCI/5QI 8 has a priority 8/80, a packet delay budget of 300 ms, and a packet error loss rate of at most 10.E-6. QCI/5QI 9 has priority 9/90, and also a packet delay budget of 300 ms and a packet error loss rate of at most 10.E-6. As these three QCIs/5QIs represent the same intent and are only different in their priority level, using discard eligibility to differentiate them is logical. As such, it is RECOMMENDED to map QCI/5QI 6 to category AF11. Similarly, it is RECOMMENDED to map QCI/5QI 8 to AF12. And logically, it is RECOMMENDED to map QCI/5QI 9 to AF13.

5.11. Mission-critical data [70]

QCI/5QI 70 is non-GBR, intended for mission critical data, with a priority of 5.5/55, a packet delay budget of 200 ms and a packet error loss rate tolerance of at most 10.E-6. The traffic types intended for 70 are the same as for QCIs/5QIs 6,8,9 categories, namely buffered streaming video and TCP-based traffic, such as www, email, chat, FTP, P2P and other file sharing applications. However, 70 is specifically intended for applications that are mission critical. For this reason, 70 priority is higher than 6, 8 or 9 priorities (5.5/55 vs 6/60, 8/80 and 9/90 respectively). Therefore, 70 fits well in the AF2x family, while 6,8,9 are in AF1x. As 70 displays intermediate differentiated treatment, if also fits well with an intermediate discard eligibility. As such, it is RECOMMENDED to map QCI/5QI 70 to DSCP 20 (AF22).

5.12. Summary of Recommendations for QCI or 5QI to DSCP Mapping

The table below summarizes the 3GPP QCI and 5QI to [RFC4594] DSCP marking recommendations:

+------+------+------+------+------+ | QCI/5Q | Resource | Priority | Example Services | Recommended | | I | Type | Level | | DSCP (PHB) | +------+------+------+------+------+ | 1 | GBR | 2 | Conversational Voice | 44 (VA) | | | | | | | | 2 | GBR | 4 | Conversational Video | 35 (N.A.) | | | | | (Live Streaming) | | | | | | | |

Henry, et al. Expires October 15, 2020 [Page 37] Internet-Draft DIFFSERV-QCI April 2020

| 3 | GBR | 3 | Real Time Gaming, | 19 (N.A.) | | | | | V2X messages, | | | | | | Electricity | | | | | | distribution (medium | | | | | | voltage) Process | | | | | | automation | | | | | | (monitoring) | | | | | | | | | 4 | GBR | 5 | Non-Conversational | 37 (N.A.) | | | | | Video (Buffered | | | | | | Streaming) | | | | | | | | | 65 | GBR | 0.7 | Mission Critical | 42 (N.A.) | | | | | user plane Push To | | | | | | Talk voice (e.g., | | | | | | MCPTT) | | | | | | | | | 66 | GBR | 2 | Non-Mission-Critical | 43 (N.A.) | | | | | user plane Push To | | | | | | Talk voice | | | | | | | | | 67 | GBR | 1.5 | Mission Critical | 33 (N.A.) | | | | | Video user plane | | | | | | | | | 75 | GBR | 2.5 | V2X messages | 17 (N.A.) | | | | | | | | 71 | GBR | 5.6 | Live uplink | 35 (N.A.) | | | | | streaming | | | | | | | | | 72 | GBR | 5.6 | Live uplink | 35 (N.A.) | | | | | streaming | | | | | | | | | 73 | GBR | 5.6 | Live uplink | 35 (N.A.) | | | | | streaming | | | | | | | | | 74 | GBR | 5.6 | Live uplink | 35 (N.A.) | | | | | streaming | | | | | | | | | 76 | GBR | 5.6 | Live uplink | 35 (N.A.) | | | | | streaming | | | | | | | | | 82 | GBR | 1.9 | Discrete automation | 25 (N.A.) | | | | | (small packets) | | | | | | | | | 83 | GBR | 2.2 | Discrete automation | 27 (N.A.) | | | | | (large packets) | | | | | | | | | 84 | GBR | 2.4 | Intelligent | 31 (N.A.) |

Henry, et al. Expires October 15, 2020 [Page 38] Internet-Draft DIFFSERV-QCI April 2020

| | | | Transport Systems | | | | | | | | | 85 | GBR | 2.1 | Electricity | 23 (N.A.) | | | | | Distribution - High | | | | | | Voltage | | | | | | | | | 86 | GBR | 1.8 | Intelligent | 29 (N.A.) | | | | | Transport Systems | | | | | | | | | 5 | Non-GBR | 1 | IMS Signalling | 40 (CS5) | | | | | | | | 6 | Non-GBR | 6 | Video (Buffered | 10 (AF11) | | | | | Streaming) TCP-based | | | | | | (e.g. www, email, | | | | | | chat, ftp, p2p file | | | | | | sharing, progressive | | | | | | video) | | | | | | | | | 7 | Non-GBR | 7 | Voice, Video (live | 38 (AF43) | | | | | streaming), | | | | | | interactive gaming | | | | | | | | | 8 | Non-GBR | 8 | Video (buffered | 12 (AF12) | | | | | streaming) TCP-based | | | | | | (e.g. www, email, | | | | | | chat, ftp, p2p file | | | | | | sharing, progressive | | | | | | video) | | | | | | | | | 9 | Non-GBR | 9 | Same as 8 | 14 (AF13) | | | | | | | | 69 | Non-GBR | 0.5 | Mission Critical | 41 (N.A.) | | | | | delay sensitive | | | | | | signalling (e.g., | | | | | | MC-PTT signalling, | | | | | | MC Video signalling) | | | | | | | | | 70 | Non-GBR | 5.5 | Mission Critical | 20 (AF22) | | | | | Data (e.g. example | | | | | | services are the | | | | | | same as QCI 6/8/9) | | | | | | | | | 79 | Non-GBR | 6.5 | V2X messages | 21 (N.A.) | | | | | | | | 80 | Non-GBR | 6.8 | Low latency eMMB | 32 (CS4) | | | | | applications | | | | | | (TCP/UDP-based); | | | | | | augmented reality | |

Henry, et al. Expires October 15, 2020 [Page 39] Internet-Draft DIFFSERV-QCI April 2020

+------+------+------+------+------+

6. IANA Considerations

This document has no IANA actions. Although this document suggests the use of codepoints in the Pool 1 of the codespace defined in [RFC2474], no exclusive attribution is requested. The recommended utilisation of seven codepoints in Pool 2 and six codepoints in pool 3 is also intended as a recommendation for experimental or Local Use, as defined in [RFC2474].

7. Specific Security Considerations

The recommendations in this document concern widely deployed wired and wireless network functionality, and, for that reason, do not present additional security concerns that do not already exist in these networks.

8. Security Recommendations for General QoS

It may be possible for a wired or wireless device (which could be either a host or a network device) to mark packets (or map packet markings) in a manner that interferes with or degrades existing QoS policies. Such marking or mapping may be done intentionally or unintentionally by developers and/or users and/or administrators of such devices.

To illustrate: A gaming application designed to run on a smartphone may request that all its packets be marked DSCP EF. Although the 3GPP infrastructure may only allocate a non-GBR default QCI (e.g. QCI 9) for this traffic, the translation point into the Internet domain may consider the DSCP marking instead of the allocated QCI, and forward this traffic with a marking of EF. This traffic may then interfere with QoS policies intended to provide priority services for business voice applications.

To mitigate such scenarios, it is RECOMMENDED to implement general QoS security measures, including:

o Setting a traffic conditioning policy reflective of business objectives and policy, such that traffic from authorized users and/or applications and/or endpoints will be accepted by the network; otherwise, packet markings will be "bleached" (i.e., re- marked to DSCP DF). Additionally, Section 5 made it clear that it is generally NOT RECOMMENDED to pass through DSCP markings from unauthorized, unidentified and/or unauthenticated devices, as these are typically considered untrusted sources. This is especially relevant for Internet of Things (IoT) deployments,

Henry, et al. Expires October 15, 2020 [Page 40] Internet-Draft DIFFSERV-QCI April 2020

where tens of billions of devices with little or no security capabilities are being connected to LTE and IP networks, leaving them vulnerable to be utilized as agents for DDoS attacks. These attacks can be amplified with preferential QoS treatments, should the packet markings of such devices be trusted.

o Policing EF marked packet flows, as detailed in [RFC2474] Section 7 and [RFC3246] Section 3.

Finally, it should be noted that the recommendations put forward in this document are not intended to address all attack vectors leveraging QoS marking abuse. Mechanisms that may further help mitigate security risks of both wired and wireless networks deploying QoS include strong device- and/or user-authentication, access- control, rate-limiting, control-plane policing, encryption, and other techniques; however, the implementation recommendations for such mechanisms are beyond the scope of this document to address in detail. Suffice it to say that the security of the devices and networks implementing QoS, including QoS mapping between wired and wireless networks, merits consideration in actual deployments.

9. References

9.1. Normative References

[RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, "Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers", RFC 2474, DOI 10.17487/RFC2474, December 1998, .

[RFC2597] Heinanen, J., Baker, F., Weiss, W., and J. Wroclawski, "Assured Forwarding PHB Group", RFC 2597, DOI 10.17487/RFC2597, June 1999, .

[RFC3246] Davie, B., Charny, A., Bennet, J., Benson, K., Le Boudec, J., Courtney, W., Davari, S., Firoiu, V., and D. Stiliadis, "An Expedited Forwarding PHB (Per-Hop Behavior)", RFC 3246, DOI 10.17487/RFC3246, March 2002, .

[RFC5865] Baker, F., Polk, J., and M. Dolly, "A Differentiated Services Code Point (DSCP) for Capacity-Admitted Traffic", RFC 5865, DOI 10.17487/RFC5865, May 2010, .

Henry, et al. Expires October 15, 2020 [Page 41] Internet-Draft DIFFSERV-QCI April 2020

9.2. Informative References

[ir.34] 3gpp, "guidelines for ipx provider networks - gsma", August 2018, .

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, .

[RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., and W. Weiss, "An Architecture for Differentiated Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, .

[RFC3662] Bless, R., Nichols, K., and K. Wehrle, "A Lower Effort Per-Domain Behavior (PDB) for Differentiated Services", RFC 3662, DOI 10.17487/RFC3662, December 2003, .

[RFC4594] Babiarz, J., Chan, K., and F. Baker, "Configuration Guidelines for DiffServ Service Classes", RFC 4594, DOI 10.17487/RFC4594, August 2006, .

[RFC5127] Chan, K., Babiarz, J., and F. Baker, "Aggregation of Diffserv Service Classes", RFC 5127, DOI 10.17487/RFC5127, February 2008, .

[RFC8100] Geib, R., Ed. and D. Black, "Diffserv-Interconnection Classes and Practice", RFC 8100, DOI 10.17487/RFC8100, March 2017, .

[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, .

[RFC8622] Bless, R., "A Lower-Effort Per-Hop Behavior (LE PHB) for Differentiated Services", RFC 8622, DOI 10.17487/RFC8622, June 2019, .

[TS23.107] 3gpp, "quality of service (qos) concept and architecture v15.0", June 2018, .

Henry, et al. Expires October 15, 2020 [Page 42] Internet-Draft DIFFSERV-QCI April 2020

[TS23.203] 3gpp, "policy and charging control architecture v16.0", December 2019, .

[TS23.207] 3gpp, "end-to-end quality of service (qos) concept and architecture v15.0", June 2018, .

[TS23.501] 3gpp, "system architecture for the 5G System (5GS) v15.0", December 2019, .

[TS26.939] 3gpp, "guidelines on the framework for live uplink streaming (FLUS) v15.0", September 2019, .

Authors’ Addresses

Jerome Henry Cisco

Email: [email protected]

Tim Szigeti Cisco

Email: [email protected]

Luis Miguel Contreras Murillo Telefonica

Email: [email protected]

Henry, et al. Expires October 15, 2020 [Page 43] Transport Area working group (tsvwg) K. De Schepper Internet-Draft Nokia Bell Labs Intended status: Experimental B. Briscoe, Ed. Expires: January 7, 2022 Independent G. White CableLabs July 6, 2021

DualQ Coupled AQMs for Low Latency, Low Loss and Scalable Throughput (L4S) draft-ietf-tsvwg-aqm-dualq-coupled-16

Abstract

The Low Latency Low Loss Scalable Throughput (L4S) architecture allows data flows over the public Internet to achieve consistent low queuing latency, generally zero congestion loss and scaling of per- flow throughput without the scaling problems of standard TCP Reno- friendly congestion controls. To achieve this, L4S data flows have to use one of the family of ’Scalable’ congestion controls (TCP Prague and Data Center TCP are examples) and a form of Explicit Congestion Notification (ECN) with modified behaviour. However, until now, Scalable congestion controls did not co-exist with existing Reno/Cubic traffic --- Scalable controls are so aggressive that ’Classic’ (e.g. Reno-friendly) algorithms sharing an ECN-capable queue would drive themselves to a small capacity share. Therefore, until now, L4S controls could only be deployed where a clean-slate environment could be arranged, such as in private data centres (hence the name DCTCP). This specification defines ‘DualQ Coupled Active Queue Management (AQM)’, which enables Scalable congestion controls that comply with the Prague L4S requirements to co-exist safely with Classic Internet traffic.

Analytical study and implementation testing of the Coupled AQM have shown that Scalable and Classic flows competing under similar conditions run at roughly the same rate. It achieves this indirectly, without having to inspect transport layer flow identifiers. When tested in a residential broadband setting, DCTCP also achieves sub-millisecond average queuing delay and zero congestion loss under a wide range of mixes of DCTCP and ‘Classic’ broadband Internet traffic, without compromising the performance of the Classic traffic. The solution has low complexity and requires no configuration for the public Internet.

De Schepper, et al. Expires January 7, 2022 [Page 1] Internet-Draft DualQ Coupled AQMs July 2021

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

This Internet-Draft will expire on January 7, 2022.

Copyright Notice

Table of Contents

1. Introduction ...... 3 1.1. Outline of the Problem ...... 3 1.2. Scope ...... 6 1.3. Terminology ...... 7 1.4. Features ...... 9 2. DualQ Coupled AQM ...... 10 2.1. Coupled AQM ...... 11 2.2. Dual Queue ...... 12 2.3. Traffic Classification ...... 12 2.4. Overall DualQ Coupled AQM Structure ...... 13 2.5. Normative Requirements for a DualQ Coupled AQM . . . . . 16 2.5.1. Functional Requirements ...... 16 2.5.1.1. Requirements in Unexpected Cases ...... 18 2.5.2. Management Requirements ...... 19

De Schepper, et al. Expires January 7, 2022 [Page 2] Internet-Draft DualQ Coupled AQMs July 2021

2.5.2.1. Configuration ...... 19 2.5.2.2. Monitoring ...... 20 2.5.2.3. Anomaly Detection ...... 21 2.5.2.4. Deployment, Coexistence and Scaling ...... 21 3. IANA Considerations (to be removed by RFC Editor) ...... 22 4. Security Considerations ...... 22 4.1. Overload Handling ...... 22 4.1.1. Avoiding Classic Starvation: Sacrifice L4S Throughput or Delay? ...... 22 4.1.2. Congestion Signal Saturation: Introduce L4S Drop or Delay? ...... 24 4.1.3. Protecting against Unresponsive ECN-Capable Traffic . 25 5. Acknowledgements ...... 25 6. Contributors ...... 25 7. References ...... 26 7.1. Normative References ...... 26 7.2. Informative References ...... 26 Appendix A. Example DualQ Coupled PI2 Algorithm ...... 32 A.1. Pass #1: Core Concepts ...... 32 A.2. Pass #2: Overload Details ...... 42 Appendix B. Example DualQ Coupled Curvy RED Algorithm . . . . . 46 B.1. Curvy RED in Pseudocode ...... 46 B.2. Efficient Implementation of Curvy RED ...... 52 Appendix C. Choice of Coupling Factor, k ...... 54 C.1. RTT-Dependence ...... 54 C.2. Guidance on Controlling Throughput Equivalence . . . . . 55 Authors’ Addresses ...... 56

1. Introduction

This document specifies a framework for DualQ Coupled AQMs, which is the network part of the L4S architecture [I-D.ietf-tsvwg-l4s-arch]. L4S enables both very low queuing latency (sub-millisecond on average) and high throughput at the same time, for ad hoc numbers of capacity-seeking applications all sharing the same capacity.

1.1. Outline of the Problem

Latency is becoming the critical performance factor for many (most?) applications on the public Internet, e.g. interactive Web, Web services, voice, conversational video, interactive video, interactive remote presence, instant messaging, online gaming, remote desktop, cloud-based applications, and video-assisted remote control of machinery and industrial processes. In the developed world, further increases in access network bit-rate offer diminishing returns, whereas latency is still a multi-faceted problem. In the last decade or so, much has been done to reduce propagation time by placing

De Schepper, et al. Expires January 7, 2022 [Page 3] Internet-Draft DualQ Coupled AQMs July 2021

caches or servers closer to users. However, queuing remains a major intermittent component of latency.

Traditionally very low latency has only been available for a few selected low rate applications, that confine their sending rate within a specially carved-off portion of capacity, which is prioritized over other traffic, e.g. Diffserv EF [RFC3246]. Up to now it has not been possible to allow any number of low latency, high throughput applications to seek to fully utilize available capacity, because the capacity-seeking process itself causes too much queuing delay.

To reduce this queuing delay caused by the capacity seeking process, changes either to the network alone or to end-systems alone are in progress. L4S involves a recognition that both approaches are yielding diminishing returns:

o Recent state-of-the-art active queue management (AQM) in the network, e.g. FQ-CoDel [RFC8290], PIE [RFC8033], Adaptive RED [ARED01] ) has reduced queuing delay for all traffic, not just a select few applications. However, no matter how good the AQM, the capacity-seeking (sawtoothing) rate of TCP-like congestion controls represents a lower limit that will either cause queuing delay to vary or cause the link to be under-utilized. These AQMs are tuned to allow a typical capacity-seeking Reno-friendly flow to induce an average queue that roughly doubles the base RTT, adding 5-15 ms of queuing on average (cf. 500 microseconds with L4S for the same mix of long-running and web traffic). However, for many applications low delay is not useful unless it is consistently low. With these AQMs, 99th percentile queuing delay is 20-30 ms (cf. 2 ms with the same traffic over L4S).

o Similarly, recent research into using e2e congestion control without needing an AQM in the network (e.g.BBR [BBRv1], [I-D.cardwell-iccrg-bbr-congestion-control]) seems to have hit a similar lower limit to queuing delay of about 20ms on average (and any additional BBRv1 flow adds another 20ms of queuing) but there are also regular 25ms delay spikes due to bandwidth probes and 60ms spikes due to flow-starts.

L4S learns from the experience of Data Center TCP [RFC8257], which shows the power of complementary changes both in the network and on end-systems. DCTCP teaches us that two small but radical changes to congestion control are needed to cut the two major outstanding causes of queuing delay variability:

1. Far smaller rate variations (sawteeth) than Reno-friendly congestion controls;

De Schepper, et al. Expires January 7, 2022 [Page 4] Internet-Draft DualQ Coupled AQMs July 2021

2. A shift of smoothing and hence smoothing delay from network to sender.

Without the former, a ’Classic’ (e.g. Reno-friendly) flow’s round trip time (RTT) varies between roughly 1 and 2 times the base RTT between the machines in question. Without the latter a ’Classic’ flow’s response to changing events is delayed by a worst-case (transcontinental) RTT, which could be hundreds of times the actual smoothing delay needed for the RTT of typical traffic from localized CDNs.

These changes are the two main features of the family of so-called ’Scalable’ congestion controls (which includes DCTCP). Both these changes only reduce delay in combination with a complementary change in the network and they are both only feasible with ECN, not drop, for the signalling:

1. The smaller sawteeth allow an extremely shallow ECN packet- marking threshold in the queue.

2. And no smoothing in the network means that every fluctuation of the queue is signalled immediately.

Without ECN, either of these would lead to very high loss levels. But, with ECN, the resulting high marking levels are just signals, not impairments.

However, until now, Scalable congestion controls (like DCTCP) did not co-exist well in a shared ECN-capable queue with existing ECN-capable TCP Reno [RFC5681] or Cubic [RFC8312] congestion controls --- Scalable controls are so aggressive that these ’Classic’ algorithms would drive themselves to a small capacity share. Therefore, until now, L4S controls could only be deployed where a clean-slate environment could be arranged, such as in private data centres (hence the name DCTCP).

This document specifies a ‘DualQ Coupled AQM’ extension that solves the problem of coexistence between Scalable and Classic flows, without having to inspect flow identifiers. It is not like flow- queuing approaches [RFC8290] that classify packets by flow identifier into separate queues in order to isolate sparse flows from the higher latency in the queues assigned to heavier flows. If a flow needs both low delay and high throughput, having a queue to itself does not isolate it from the harm it causes to itself. In contrast, DualQ Coupled AQMs addresses the root cause of the latency problem --- they are an enabler for the smooth low latency scalable behaviour of Scalable congestion controls, so that every packet in every flow can

De Schepper, et al. Expires January 7, 2022 [Page 5] Internet-Draft DualQ Coupled AQMs July 2021

enjoy very low latency, then there is no need to isolate each flow into a separate queue.

1.2. Scope

L4S involves complementary changes in the network and on end-systems:

Network: A DualQ Coupled AQM (defined in the present document) or a modification to flow-queue AQMs (described in section 4.2.b of [I-D.ietf-tsvwg-l4s-arch]);

End-system: A Scalable congestion control (defined in section 4 of [I-D.ietf-tsvwg-ecn-l4s-id]).

Packet identifier: The network and end-system parts of L4S can be deployed incrementally, because they both identify L4S packets using the experimentally assigned explicit congestion notification (ECN) codepoints in the IP header: ECT(1) and CE [RFC8311] [I-D.ietf-tsvwg-ecn-l4s-id].

Data Center TCP (DCTCP [RFC8257]) is an example of a Scalable congestion control for controlled environments that has been deployed for some time in Linux, Windows and FreeBSD operating systems. During the progress of this document through the IETF a number of other Scalable congestion controls were implemented, e.g. TCP Prague [I-D.briscoe-iccrg-prague-congestion-control] [PragueLinux], BBRv2 [BBRv2], QUIC Prague and the L4S variant of SCREAM for real- time media [RFC8298].

The focus of this specification is to enable deployment of the network part of the L4S service. Then, without any management intervention, applications can exploit this new network capability as their operating systems migrate to Scalable congestion controls, which can then evolve _while_ their benefits are being enjoyed by everyone on the Internet.

The DualQ Coupled AQM framework can incorporate any AQM designed for a single queue that generates a statistical or deterministic mark/ drop probability driven by the queue dynamics. Pseudocode examples of two different DualQ Coupled AQMs are given in the appendices. In many cases the framework simplifies the basic control algorithm, and requires little extra processing. Therefore it is believed the Coupled AQM would be applicable and easy to deploy in all types of buffers; buffers in cost-reduced mass-market residential equipment; buffers in end-system stacks; buffers in carrier-scale equipment including remote access servers, routers, firewalls and Ethernet switches; buffers in network interface cards, buffers in virtualized network appliances, hypervisors, and so on.

De Schepper, et al. Expires January 7, 2022 [Page 6] Internet-Draft DualQ Coupled AQMs July 2021

For the public Internet, nearly all the benefit will typically be achieved by deploying the Coupled AQM into either end of the access link between a ’site’ and the Internet, which is invariably the bottleneck (see section 6.4 of[I-D.ietf-tsvwg-l4s-arch] about deployment, which also defines the term ’site’ to mean a home, an office, a campus or mobile user equipment).

Latency is not the only concern of L4S:

o The ’Low Loss" part of the name denotes that L4S generally achieves zero congestion loss (which would otherwise cause retransmission delays), due to its use of ECN.

o The "Scalable throughput" part of the name denotes that the per- flow throughput of Scalable congestion controls should scale indefinitely, avoiding the imminent scaling problems with ’TCP- Friendly’ congestion control algorithms [RFC3649].

The former is clearly in scope of this AQM document. However, the latter is an outcome of the end-system behaviour, and therefore outside the scope of this AQM document, even though the AQM is an enabler.

The overall L4S architecture [I-D.ietf-tsvwg-l4s-arch] gives more detail, including on wider deployment aspects such as backwards compatibility of Scalable congestion controls in bottlenecks where a DualQ Coupled AQM has not been deployed. The supporting papers [DualPI2Linux], [PI2] and [DCttH15] give the full rationale for the AQM’s design, both discursively and in more precise mathematical form, as well as the results of performance evaluations.

1.3. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] when, and only when, they appear in all capitals, as shown here.

The DualQ Coupled AQM uses two queues for two services. Each of the following terms identifies both the service and the queue that provides the service:

Classic service/queue: The Classic service is intended for all the congestion control behaviours that co-exist with Reno [RFC5681] (e.g. Reno itself, Cubic [RFC8312], TFRC [RFC5348]).

Low-Latency, Low-Loss Scalable throughput (L4S) service/queue: The ’L4S’ service is intended for traffic from scalable congestion

De Schepper, et al. Expires January 7, 2022 [Page 7] Internet-Draft DualQ Coupled AQMs July 2021

control algorithms, such as TCP Prague [I-D.briscoe-iccrg-prague-congestion-control], which was derived from Data Center TCP [RFC8257]. The L4S service is for more general traffic than just TCP Prague--it allows the set of congestion controls with similar scaling properties to Prague to evolve, such as the examples listed earlier (Relentless, SCReAM, etc.).

Classic Congestion Control: A congestion control behaviour that can co-exist with standard TCP Reno [RFC5681] without causing significantly negative impact on its flow rate [RFC5033]. With Classic congestion controls, such as Reno or Cubic, because flow rate has scaled since TCP congestion control was first designed in 1988, it now takes hundreds of round trips (and growing) to recover after a congestion signal (whether a loss or an ECN mark) as shown in the examples in section 5.1 of [I-D.ietf-tsvwg-l4s-arch] and in [RFC3649]. Therefore control of queuing and utilization becomes very slack, and the slightest disturbances (e.g. from new flows starting) prevent a high rate from being attained.

Scalable Congestion Control: A congestion control where the average time from one congestion signal to the next (the recovery time) remains invariant as the flow rate scales, all other factors being equal. This maintains the same degree of control over queueing and utilization whatever the flow rate, as well as ensuring that high throughput is robust to disturbances. For instance, DCTCP averages 2 congestion signals per round-trip whatever the flow rate, as do other recently developed scalable congestion controls, e.g. Relentless TCP [Mathis09], TCP Prague [I-D.briscoe-iccrg-prague-congestion-control], [PragueLinux], BBRv2 [BBRv2] and the L4S variant of SCREAM for real-time media [SCReAM], [RFC8298]). For the public Internet a Scalable transport has to comply with the requirements in Section 4 of [I-D.ietf-tsvwg-ecn-l4s-id] (aka. the ’Prague L4S requirements’).

C: Abbreviation for Classic, e.g. when used as a subscript.

L: Abbreviation for L4S, e.g. when used as a subscript.

The terms Classic or L4S can also qualify other nouns, such as ’codepoint’, ’identifier’, ’classification’, ’packet’, ’flow’. For example: an L4S packet means a packet with an L4S identifier sent from an L4S congestion control.

Both Classic and L4S services can cope with a proportion of unresponsive or less-responsive traffic as well, but in the L4S case its rate has to be smooth enough or low enough not to build a

De Schepper, et al. Expires January 7, 2022 [Page 8] Internet-Draft DualQ Coupled AQMs July 2021

queue (e.g. DNS, VoIP, game sync datagrams, etc). The DualQ Coupled AQM behaviour is defined to be similar to a single FIFO queue with respect to unresponsive and overload traffic.

Reno-friendly: The subset of Classic traffic that is friendly to the standard Reno congestion control defined for TCP in [RFC5681]. Reno-friendly is used in place of ’TCP-friendly’, given the latter has become imprecise, because the TCP protocol is now used with so many different congestion control behaviours, and Reno is used in non-TCP transports such as QUIC.

Classic ECN: The original Explicit Congestion Notification (ECN) protocol [RFC3168], which requires ECN signals to be treated the same as drops, both when generated in the network and when responded to by the sender.

For L4S, the names used for the four codepoints of the 2-bit IP- ECN field are unchanged from those defined in [RFC3168]: Not ECT, ECT(0), ECT(1) and CE, where ECT stands for ECN-Capable Transport and CE stands for Congestion Experienced. A packet marked with the CE codepoint is termed ’ECN-marked’ or sometimes just ’marked’ where the context makes ECN obvious.

1.4. Features

The AQM couples marking and/or dropping from the Classic queue to the L4S queue in such a way that a flow will get roughly the same throughput whichever it uses. Therefore both queues can feed into the full capacity of a link and no rates need to be configured for the queues. The L4S queue enables Scalable congestion controls like DCTCP or TCP Prague to give very low and predictably low latency, without compromising the performance of competing ’Classic’ Internet traffic.

Thousands of tests have been conducted in a typical fixed residential broadband setting. Experiments used a range of base round trip delays up to 100ms and link rates up to 200 Mb/s between the data centre and home network, with varying amounts of background traffic in both queues. For every L4S packet, the AQM kept the average queuing delay below 1ms (or 2 packets where serialization delay exceeded 1ms on slower links), with 99th percentile no worse than 2ms. No losses at all were introduced by the L4S AQM. Details of the extensive experiments are available [DualPI2Linux], [PI2], [DCttH15].

Subjective testing was also conducted by multiple people all simultaneously using very demanding high bandwidth low latency applications over a single shared access link [L4Sdemo16]. In one

De Schepper, et al. Expires January 7, 2022 [Page 9] Internet-Draft DualQ Coupled AQMs July 2021

application, each user could use finger gestures to pan or zoom their own high definition (HD) sub-window of a larger video scene generated on the fly in ’the cloud’ from a football match. Another user wearing VR goggles was remotely receiving a feed from a 360-degree camera in a racing car, again with the sub-window in their field of vision generated on the fly in ’the cloud’ dependent on their head movements. Even though other users were also downloading large amounts of L4S and Classic data, playing a gaming benchmark and watchings videos over the same 40Mb/s downstream broadband link, latency was so low that the football picture appeared to stick to the user’s finger on the touch pad and the experience fed from the remote camera did not noticeably lag head movements. All the L4S data (even including the downloads) achieved the same very low latency. With an alternative AQM, the video noticeably lagged behind the finger gestures and head movements.

Unlike Diffserv Expedited Forwarding, the L4S queue does not have to be limited to a small proportion of the link capacity in order to achieve low delay. The L4S queue can be filled with a heavy load of capacity-seeking flows (TCP Prague etc.) and still achieve low delay. The L4S queue does not rely on the presence of other traffic in the Classic queue that can be ’overtaken’. It gives low latency to L4S traffic whether or not there is Classic traffic, and the latency of Classic traffic does not suffer when a proportion of the traffic is L4S.

The two queues are only necessary because:

o the large variations (sawteeth) of Classic flows need roughly a base RTT of queuing delay to ensure full utilization

o Scalable flows do not need a queue to keep utilization high, but they cannot keep latency predictably low if they are mixed with Classic traffic,

The L4S queue has latency priority within sub-round trip timescales, but over longer periods the coupling from the Classic to the L4S AQM (explained below) ensures that it does not have bandwidth priority over the Classic queue.

2. DualQ Coupled AQM

There are two main aspects to the approach:

o The Coupled AQM that addresses throughput equivalence between Classic (e.g. Reno, Cubic) flows and L4S flows (that satisfy the Prague L4S requirements).

De Schepper, et al. Expires January 7, 2022 [Page 10] Internet-Draft DualQ Coupled AQMs July 2021

o The Dual Queue structure that provides latency separation for L4S flows to isolate them from the typically large Classic queue.

2.1. Coupled AQM

In the 1990s, the ‘TCP formula’ was derived for the relationship between the steady-state congestion window, cwnd, and the drop probability, p of standard Reno congestion control [RFC5681] . To a first order approximation, the steady-state cwnd of Reno is inversely proportional to the square root of p.

The design focuses on Reno as the worst case, because if it does no harm to Reno, it will not harm Cubic or any traffic designed to be friendly to Reno. TCP Cubic implements a Reno-compatibility mode, which is relevant for typical RTTs under 20ms as long as the throughput of a single flow is less than about 700Mb/s. In such cases it can be assumed that Cubic traffic behaves similarly to Reno (but with a slightly different constant of proportionality). The term ’Classic’ will be used for the collection of Reno-friendly traffic including Cubic and potentially other experimental congestion controls intended not to significantly impact the flow rate of Reno.

A supporting paper [PI2] includes the derivation of the equivalent rate equation for DCTCP, for which cwnd is inversely proportional to p (not the square root), where in this case p is the ECN marking probability. DCTCP is not the only congestion control that behaves like this, so the term ’Scalable’ will be used for all similar congestion control behaviours (see examples in Section 1.2). The term ’L4S’ is used for traffic driven by a Scalable congestion control that also complies with the additional ’Prague L4S’ requirements [I-D.ietf-tsvwg-ecn-l4s-id].

For safe co-existence, under stationary conditions, a Scalable flow has to run at roughly the same rate as a Reno TCP flow (all other factors being equal). So the drop or marking probability for Classic traffic, p_C has to be distinct from the marking probability for L4S traffic, p_L. The original ECN specification [RFC3168] required these probabilities to be the same, but [RFC8311] updates RFC 3168 to enable experiments in which these probabilities are different.

Also, to remain stable, Classic sources need the network to smooth p_C so it changes relatively slowly. It is hard for a network node to know the RTTs of all the flows, so a Classic AQM adds a _worst- case_ RTT of smoothing delay (about 100-200 ms). In contrast, L4S shifts responsibility for smoothing ECN feedback to the sender, which only delays its response by its _own_ RTT, as well as allowing a more immediate response if necessary.

De Schepper, et al. Expires January 7, 2022 [Page 11] Internet-Draft DualQ Coupled AQMs July 2021

The Coupled AQM achieves safe coexistence by making the Classic drop probability p_C proportional to the square of the coupled L4S probability p_CL. p_CL is an input to the instantaneous L4S marking probability p_L but it changes as slowly as p_C. This makes the Reno flow rate roughly equal the DCTCP flow rate, because the squaring of p_CL counterbalances the square root of p_C in the ’TCP formula’ of Classic Reno congestion control.

Stating this as a formula, the relation between Classic drop probability, p_C, and the coupled L4S probability p_CL needs to take the form:

p_C = ( p_CL / k )^2 (1)

where k is the constant of proportionality, which is termed the coupling factor.

2.2. Dual Queue

Classic traffic needs to build a large queue to prevent under- utilization. Therefore a separate queue is provided for L4S traffic, and it is scheduled with priority over the Classic queue. Priority is conditional to prevent starvation of Classic traffic.

Nonetheless, coupled marking ensures that giving priority to L4S traffic still leaves the right amount of spare scheduling time for Classic flows to each get equivalent throughput to DCTCP flows (all other factors such as RTT being equal).

2.3. Traffic Classification

Both the Coupled AQM and DualQ mechanisms need an identifier to distinguish L4S (L) and Classic (C) packets. Then the coupling algorithm can achieve coexistence without having to inspect flow identifiers, because it can apply the appropriate marking or dropping probability to all flows of each type. A separate specification [I-D.ietf-tsvwg-ecn-l4s-id] requires the network to treat the ECT(1) and CE codepoints of the ECN field as this identifier. An additional process document has proved necessary to make the ECT(1) codepoint available for experimentation [RFC8311].

For policy reasons, an operator might choose to steer certain packets (e.g. from certain flows or with certain addresses) out of the L queue, even though they identify themselves as L4S by their ECN codepoints. In such cases, [I-D.ietf-tsvwg-ecn-l4s-id] says that the device "MUST NOT alter the end-to-end L4S ECN identifier", so that it is preserved end-to-end. The aim is that each operator can choose how it treats L4S traffic locally, but an individual operator does

De Schepper, et al. Expires January 7, 2022 [Page 12] Internet-Draft DualQ Coupled AQMs July 2021

not alter the identification of L4S packets, which would prevent other operators downstream from making their own choices on how to treat L4S traffic.

In addition, an operator could use other identifiers to classify certain additional packet types into the L queue that it deems will not risk harm to the L4S service. For instance addresses of specific applications or hosts (see [I-D.ietf-tsvwg-ecn-l4s-id]), specific Diffserv codepoints such as EF (Expedited Forwarding) and Voice-Admit service classes (see [I-D.briscoe-tsvwg-l4s-diffserv]), the Non- Queue-Building (NQB) per-hop behaviour [I-D.ietf-tsvwg-nqb] or certain protocols (e.g. ARP, DNS). Note that the mechanism only reads these identifiers. [I-D.ietf-tsvwg-ecn-l4s-id] says it "MUST NOT alter these non-ECN identifiers". Thus, the L queue is not solely an L4S queue, it can be consider more generally as a low latency queue.

2.4. Overall DualQ Coupled AQM Structure

Figure 1 shows the overall structure that any DualQ Coupled AQM is likely to have. This schematic is intended to aid understanding of the current designs of DualQ Coupled AQMs. However, it is not intended to preclude other innovative ways of satisfying the normative requirements in Section 2.5 that minimally define a DualQ Coupled AQM.

The classifier on the left separates incoming traffic between the two queues (L and C). Each queue has its own AQM that determines the likelihood of marking or dropping (p_L and p_C). It has been proved [PI2] that it is preferable to control load with a linear controller, then square the output before applying it as a drop probability to Reno-friendly traffic (because Reno congestion control decreases its load proportional to the square-root of the increase in drop). So, the AQM for Classic traffic needs to be implemented in two stages: i) a base stage that outputs an internal probability p’ (pronounced p-prime); and ii) a squaring stage that outputs p_C, where

p_C = (p’)^2. (2)

Substituting for p_C in Eqn (1) gives:

p’ = p_CL / k

So the slow-moving input to ECN marking in the L queue (the coupled L4S probability) is:

p_CL = k*p’. (3)

De Schepper, et al. Expires January 7, 2022 [Page 13] Internet-Draft DualQ Coupled AQMs July 2021

The actual ECN marking probability p_L that is applied to the L queue needs to track the immediate L queue delay under L-only congestion conditions, as well as track p_CL under coupled congestion conditions. So the L queue uses a native AQM that calculates a probability p’_L as a function of the instantaneous L queue delay. And, given the L queue has conditional priority over the C queue, whenever the L queue grows, the AQM ought to apply marking probability p’_L, but p_L ought not to fall below p_CL. This suggests:

p_L = max(p’_L, p_CL), (4)

which has also been found to work very well in practice.

The two transformations of p’ in equations (2) and (3) implement the required coupling given in equation (1) earlier.

The constant of proportionality or coupling factor, k, in equation (1) determines the ratio between the congestion probabilities (loss or marking) experienced by L4S and Classic traffic. Thus k indirectly determines the ratio between L4S and Classic flow rates, because flows (assuming they are responsive) adjust their rate in response to congestion probability. Appendix C.2 gives guidance on the choice of k and its effect on relative flow rates.

De Schepper, et al. Expires January 7, 2022 [Page 14] Internet-Draft DualQ Coupled AQMs July 2021

______| | ,------. L4S queue | |===>| ECN | ,’| ______|_| |marker|\ <’ | | ‘------’\\ //‘’ v ^ p_L \\ // ,------. | \\ // |Native |p’_L | \\,. // | L4S |--->(MAX) < | ___ ,------.// | AQM | ^ p_CL ‘\|.’Cond-‘. | IP-ECN |/ ‘------’ | / itional \ ==>|Classifier| ,------. (k*p’) [ priority]==> | |\ | Base | | \scheduler/ ‘------’\\ | AQM |---->: ,’|‘-.___.-’ \\ | |p’ | <’ | \\ ‘------’ (p’^2) //‘’ \\ ^ | // \\,. | v p_C // < | ______.------.// ‘\| | | | Drop |/ Classic |queue |===>|/mark | __|______| ‘------’

Legend: ===> traffic flow; ---> control dependency.

Figure 1: DualQ Coupled AQM Schematic

After the AQMs have applied their dropping or marking, the scheduler forwards their packets to the link. Even though the scheduler gives priority to the L queue, it is not as strong as the coupling from the C queue. This is because, as the C queue grows, the base AQM applies more congestion signals to L traffic (as well as C). As L flows reduce their rate in response, they use less than the scheduling share for L traffic. So, because the scheduler is work preserving, it schedules any C traffic in the gaps.

Giving priority to the L queue has the benefit of very low L queue delay, because the L queue is kept empty whenever L traffic is controlled by the coupling. Also there only has to be a coupling in one direction - from Classic to L4S. Priority has to be conditional in some way to prevent the C queue starving under overload conditions (see Section 4.1). With normal responsive traffic simple strict priority would work, but it would make new Classic traffic wait until its queue activated the coupling and L4S flows had in turn reduced their rate enough to drain the L queue so that Classic traffic could be scheduled. Giving a small weight or limited waiting time for C traffic improves response times for short Classic messages, such as

De Schepper, et al. Expires January 7, 2022 [Page 15] Internet-Draft DualQ Coupled AQMs July 2021

DNS requests and improves Classic flow startup because immediate capacity is available.

Example DualQ Coupled AQM algorithms called DualPI2 and Curvy RED are given in Appendix A and Appendix B. Either example AQM can be used to couple packet marking and dropping across a dual Q.

DualPI2 uses a Proportional-Integral (PI) controller as the Base AQM. Indeed, this Base AQM with just the squared output and no L4S queue can be used as a drop-in replacement for PIE [RFC8033], in which case it is just called PI2 [PI2]. PI2 is a principled simplification of PIE that is both more responsive and more stable in the face of dynamically varying load.

Curvy RED is derived from RED [RFC2309], but its configuration parameters are insensitive to link rate and it requires less operations per packet. However, DualPI2 is more responsive and stable over a wider range of RTTs than Curvy RED. As a consequence, at the time of writing, DualPI2 has attracted more development and evaluation attention than Curvy RED, leaving the Curvy RED design incomplete and not so fully evaluated.

Both AQMs regulate their queue in units of time rather than bytes. As already explained, this ensures configuration can be invariant for different drain rates. With AQMs in a dualQ structure this is particularly important because the drain rate of each queue can vary rapidly as flows for the two queues arrive and depart, even if the combined link rate is constant.

It would be possible to control the queues with other alternative AQMs, as long as the normative requirements (those expressed in capitals) in Section 2.5 are observed.

2.5. Normative Requirements for a DualQ Coupled AQM

The following requirements are intended to capture only the essential aspects of a DualQ Coupled AQM. They are intended to be independent of the particular AQMs used for each queue.

2.5.1. Functional Requirements

A Dual Queue Coupled AQM implementation MUST comply with the prerequisite L4S behaviours for any L4S network node (not just a DualQ) as specified in section 5 of [I-D.ietf-tsvwg-ecn-l4s-id]. These primarily concern classification and remarking as briefly summarized in Section 2.3 earlier. But there is also a subsection (5.5) giving guidance on reducing the burstiness of the link technology underlying any L4S AQM.

De Schepper, et al. Expires January 7, 2022 [Page 16] Internet-Draft DualQ Coupled AQMs July 2021

A Dual Queue Coupled AQM implementation MUST utilize two queues, each with an AQM algorithm. The two queues can be part of a larger queuing hierarchy [I-D.briscoe-tsvwg-l4s-diffserv].

The AQM algorithm for the low latency (L) queue MUST be able to apply ECN marking to ECN-capable packets.

The scheduler draining the two queues MUST give L4S packets priority over Classic, although priority MUST be bounded in order not to starve Classic traffic. The scheduler SHOULD be work-conserving.

[I-D.ietf-tsvwg-ecn-l4s-id] defines the meaning of an ECN marking on L4S traffic, relative to drop of Classic traffic. In order to ensure coexistence of Classic and Scalable L4S traffic, it says, "The likelihood that an AQM drops a Not-ECT Classic packet (p_C) MUST be roughly proportional to the square of the likelihood that it would have marked it if it had been an L4S packet (p_L)." The term ’likelihood’ is used to allow for marking and dropping to be either probabilistic or deterministic.

For the current specification, this translates into the following requirement. A DualQ Coupled AQM MUST apply ECN marking to traffic in the L queue that is no lower than that derived from the likelihood of drop (or ECN marking) in the Classic queue using Eqn. (1).

The constant of proportionality, k, in Eqn (1) determines the relative flow rates of Classic and L4S flows when the AQM concerned is the bottleneck (all other factors being equal). [I-D.ietf-tsvwg-ecn-l4s-id] says, "The constant of proportionality (k) does not have to be standardised for interoperability, but a value of 2 is RECOMMENDED."

Assuming Scalable congestion controls for the Internet will be as aggressive as DCTCP, this will ensure their congestion window will be roughly the same as that of a standards track TCP Reno congestion control (Reno) [RFC5681] and other Reno-friendly controls, such as TCP Cubic in its Reno-compatibility mode.

The choice of k is a matter of operator policy, and operators MAY choose a different value using Table 1 and the guidelines in Appendix C.2.

If multiple customers or users share capacity at a bottleneck (e.g. in the Internet access link of a campus network), the operator’s choice of k will determine capacity sharing between the flows of different customers. However, on the public Internet, access network operators typically isolate customers from each other with some form of layer-2 multiplexing (OFDM(A) in DOCSIS3.1, CDMA in

De Schepper, et al. Expires January 7, 2022 [Page 17] Internet-Draft DualQ Coupled AQMs July 2021

3G, SC-FDMA in LTE) or L3 scheduling (WRR in DSL), rather than relying on host congestion controls to share capacity between customers [RFC0970]. In such cases, the choice of k will solely affect relative flow rates within each customer’s access capacity, not between customers. Also, k will not affect relative flow rates at any times when all flows are Classic or all flows are L4S, and it will not affect the relative throughput of small flows.

2.5.1.1. Requirements in Unexpected Cases

The flexibility to allow operator-specific classifiers (Section 2.3) leads to the need to specify what the AQM in each queue ought to do with packets that do not carry the ECN field expected for that queue. It is expected that the AQM in each queue will inspect the ECN field to determine what sort of congestion notification to signal, then it will decide whether to apply congestion notification to this particular packet, as follows:

o If a packet that does not carry an ECT(1) or CE codepoint is classified into the L queue:

* if the packet is ECT(0), the L AQM SHOULD apply CE-marking using a probability appropriate to Classic congestion control and appropriate to the target delay in the L queue

* if the packet is Not-ECT, the appropriate action depends on whether some other function is protecting the L queue from misbehaving flows (e.g. per-flow queue protection [I-D.briscoe-docsis-q-protection] or latency policing):

+ If separate queue protection is provided, the L AQM SHOULD ignore the packet and forward it unchanged, meaning it should not calculate whether to apply congestion notification and it should neither drop nor CE-mark the packet (for instance, the operator might classify EF traffic that is unresponsive to drop into the L queue, alongside responsive L4S-ECN traffic)

+ if separate queue protection is not provided, the L AQM SHOULD apply drop using a drop probability appropriate to Classic congestion control and appropriate to the target delay in the L queue

o If a packet that carries an ECT(1) codepoint is classified into the C queue:

De Schepper, et al. Expires January 7, 2022 [Page 18] Internet-Draft DualQ Coupled AQMs July 2021

* the C AQM SHOULD apply CE-marking using the coupled AQM probability p_CL (= k*p’).

The above requirements are worded as "SHOULDs", because operator- specific classifiers are for flexibility, by definition. Therefore, alternative actions might be appropriate in the operator’s specific circumstances. An example would be where the operator knows that certain legacy traffic marked with one codepoint actually has a congestion response associated with another codepoint.

If the DualQ Coupled AQM has detected overload, it MUST begin using Classic drop, and continue until the overload episode has subsided. Switching to drop if ECN marking is persistently high is required by Section 7 of [RFC3168] and Section 4.2.1 of [RFC7567].

2.5.2. Management Requirements

2.5.2.1. Configuration

By default, a DualQ Coupled AQM SHOULD NOT need any configuration for use at a bottleneck on the public Internet [RFC7567]. The following parameters MAY be operator-configurable, e.g. to tune for non- Internet settings:

o Optional packet classifier(s) to use in addition to the ECN field (see Section 2.3);

o Expected typical RTT, which can be used to determine the queuing delay of the Classic AQM at its operating point, in order to prevent typical lone flows from under-utilizing capacity. For example:

* for the PI2 algorithm (Appendix A) the queuing delay target is dependent on the typical RTT;

* for the Curvy RED algorithm (Appendix B) the queuing delay at the desired operating point of the curvy ramp is configured to encompass a typical RTT;

* if another Classic AQM was used, it would be likely to need an operating point for the queue based on the typical RTT, and if so it SHOULD be expressed in units of time.

An operating point that is manually calculated might be directly configurable instead, e.g. for links with large numbers of flows where under-utilization by a single flow would be unlikely.

De Schepper, et al. Expires January 7, 2022 [Page 19] Internet-Draft DualQ Coupled AQMs July 2021

o Expected maximum RTT, which can be used to set the stability parameter(s) of the Classic AQM. For example:

* for the PI2 algorithm (Appendix A), the gain parameters of the PI algorithm depend on the maximum RTT.

* for the Curvy RED algorithm (Appendix B) the smoothing parameter is chosen to filter out transients in the queue within a maximum RTT.

Stability parameter(s) that are manually calculated assuming a maximum RTT might be directly configurable instead.

o Coupling factor, k (see Appendix C.2);

o A limit to the conditional priority of L4S. This is scheduler- dependent, but it SHOULD be expressed as a relation between the max delay of a C packet and an L packet. For example:

* for a WRR scheduler a weight ratio between L and C of w:1 means that the maximum delay to a C packet is w times that of an L packet.

* for a time-shifted FIFO (TS-FIFO) scheduler (see Section 4.1.1) a time-shift of tshift means that the maximum delay to a C packet is tshift greater than that of an L packet. tshift could be expressed as a multiple of the typical RTT rather than as an absolute delay.

o The maximum Classic ECN marking probability, p_Cmax, before switching over to drop.

2.5.2.2. Monitoring

An experimental DualQ Coupled AQM SHOULD allow the operator to monitor each of the following operational statistics on demand, per queue and per configurable sample interval, for performance monitoring and perhaps also for accounting in some cases:

o Bits forwarded, from which utilization can be calculated;

o Total packets in the three categories: arrived, presented to the AQM, and forwarded. The difference between the first two will measure any non-AQM tail discard. The difference between the last two will measure proactive AQM discard;

De Schepper, et al. Expires January 7, 2022 [Page 20] Internet-Draft DualQ Coupled AQMs July 2021

o ECN packets marked, non-ECN packets dropped, ECN packets dropped, which can be combined with the three total packet counts above to calculate marking and dropping probabilities;

o Queue delay (not including serialization delay of the head packet or medium acquisition delay) - see further notes below.

Unlike the other statistics, queue delay cannot be captured in a simple accumulating counter. Therefore the type of queue delay statistics produced (mean, percentiles, etc.) will depend on implementation constraints. To facilitate comparative evaluation of different implementations and approaches, an implementation SHOULD allow mean and 99th percentile queue delay to be derived (per queue per sample interval). A relatively simple way to do this would be to store a coarse-grained histogram of queue delay. This could be done with a small number of bins with configurable edges that represent contiguous ranges of queue delay. Then, over a sample interval, each bin would accumulate a count of the number of packets that had fallen within each range. The maximum queue delay per queue per interval MAY also be recorded.

2.5.2.3. Anomaly Detection

An experimental DualQ Coupled AQM SHOULD asynchronously report the following data about anomalous conditions:

o Start-time and duration of overload state.

A hysteresis mechanism SHOULD be used to prevent flapping in and out of overload causing an event storm. For instance, exit from overload state could trigger one report, but also latch a timer. Then, during that time, if the AQM enters and exits overload state any number of times, the duration in overload state is accumulated but no new report is generated until the first time the AQM is out of overload once the timer has expired.

2.5.2.4. Deployment, Coexistence and Scaling

[RFC5706] suggests that deployment, coexistence and scaling should also be covered as management requirements. The raison d’etre of the DualQ Coupled AQM is to enable deployment and coexistence of Scalable congestion controls - as incremental replacements for today’s Reno- friendly controls that do not scale with bandwidth-delay product. Therefore there is no need to repeat these motivating issues here given they are already explained in the Introduction and detailed in the L4S architecture [I-D.ietf-tsvwg-l4s-arch].

De Schepper, et al. Expires January 7, 2022 [Page 21] Internet-Draft DualQ Coupled AQMs July 2021

The descriptions of specific DualQ Coupled AQM algorithms in the appendices cover scaling of their configuration parameters, e.g. with respect to RTT and sampling frequency.

3. IANA Considerations (to be removed by RFC Editor)

This specification contains no IANA considerations.

4. Security Considerations

4.1. Overload Handling

Where the interests of users or flows might conflict, it could be necessary to police traffic to isolate any harm to the performance of individual flows. However it is hard to avoid unintended side- effects with policing, and in a trusted environment policing is not necessary. Therefore per-flow policing (e.g. [I-D.briscoe-docsis-q-protection]) needs to be separable from a basic AQM, as an option under policy control.

However, a basic DualQ AQM does at least need to handle overload. A useful objective would be for the overload behaviour of the DualQ AQM to be at least no worse than a single queue AQM. However, a tradeoff needs to be made between complexity and the risk of either traffic class harming the other. In each of the following three subsections, an overload issue specific to the DualQ is described, followed by proposed solution(s).

Under overload the higher priority L4S service will have to sacrifice some aspect of its performance. Alternative solutions are provided below that each relax a different factor: e.g. throughput, delay, drop. These choices need to be made either by the developer or by operator policy, rather than by the IETF.

4.1.1. Avoiding Classic Starvation: Sacrifice L4S Throughput or Delay?

Priority of L4S is required to be conditional to avoid total starvation of Classic by heavy L4S traffic. This raises the question of whether to sacrifice L4S throughput or L4S delay (or some other policy) to mitigate starvation of Classic:

Sacrifice L4S throughput: By using weighted round robin as the conditional priority scheduler, the L4S service can sacrifice some throughput during overload. This can either be thought of as guaranteeing a minimum throughput service for Classic traffic, or as guaranteeing a maximum delay for a packet at the head of the Classic queue.

De Schepper, et al. Expires January 7, 2022 [Page 22] Internet-Draft DualQ Coupled AQMs July 2021

The scheduling weight of the Classic queue should be small (e.g. 1/16). Then, in most traffic scenarios the scheduler will not interfere and it will not need to - the coupling mechanism and the end-systems will share out the capacity across both queues as if it were a single pool. However, because the congestion coupling only applies in one direction (from C to L), if L4S traffic is over-aggressive or unresponsive, the scheduler weight for Classic traffic will at least be large enough to ensure it does not starve.

In cases where the ratio of L4S to Classic flows (e.g. 19:1) is greater than the ratio of their scheduler weights (e.g. 15:1), the L4S flows will get less than an equal share of the capacity, but only slightly. For instance, with the example numbers given, each L4S flow will get (15/16)/19 = 4.9% when ideally each would get 1/20=5%. In the rather specific case of an unresponsive flow taking up just less than the capacity set aside for L4S (e.g. 14/16 in the above example), using WRR could significantly reduce the capacity left for any responsive L4S flows.

The scheduling weight of the Classic queue should not be too small, otherwise a C packet at the head of the queue could be excessively delayed by a continually busy L queue. For instance if the Classic weight is 1/16, the maximum that a Classic packet at the head of the queue can be delayed by L traffic is the serialization delay of 15 MTU-sized packets.

Sacrifice L4S Delay: To control milder overload of responsive traffic, particularly when close to the maximum congestion signal, the operator could choose to control overload of the Classic queue by allowing some delay to ’leak’ across to the L4S queue. The scheduler can be made to behave like a single First-In First-Out (FIFO) queue with different service times by implementing a very simple conditional priority scheduler that could be called a "time-shifted FIFO" (see the Modifier Earliest Deadline First (MEDF) scheduler of [MEDF]). This scheduler adds tshift to the queue delay of the next L4S packet, before comparing it with the queue delay of the next Classic packet, then it selects the packet with the greater adjusted queue delay. Under regular conditions, this time-shifted FIFO scheduler behaves just like a strict priority scheduler. But under moderate or high overload it prevents starvation of the Classic queue, because the time-shift (tshift) defines the maximum extra queuing delay of Classic packets relative to L4S.

The example implementations in Appendix A and Appendix B could both be implemented with either policy.

De Schepper, et al. Expires January 7, 2022 [Page 23] Internet-Draft DualQ Coupled AQMs July 2021

4.1.2. Congestion Signal Saturation: Introduce L4S Drop or Delay?

To keep the throughput of both L4S and Classic flows roughly equal over the full load range, a different control strategy needs to be defined above the point where one AQM first saturates to a probability of 100% leaving no room to push back the load any harder. If k>1, L4S will saturate first, even though saturation could be caused by unresponsive traffic in either queue.

The term ’unresponsive’ includes cases where a flow becomes temporarily unresponsive, for instance, a real-time flow that takes a while to adapt its rate in response to congestion, or a standard Reno flow that is normally responsive, but above a certain congestion level it will not be able to reduce its congestion window below the allowed minimum of 2 segments [RFC5681], effectively becoming unresponsive. (Note that L4S traffic ought to remain responsive below a window of 2 segments (see [I-D.ietf-tsvwg-ecn-l4s-id]).

Saturation raises the question of whether to relieve congestion by introducing some drop into the L4S queue or by allowing delay to grow in both queues (which could eventually lead to tail drop too):

Drop on Saturation: Saturation can be avoided by setting a maximum threshold for L4S ECN marking (assuming k>1) before saturation starts to make the flow rates of the different traffic types diverge. Above that the drop probability of Classic traffic is applied to all packets of all traffic types. Then experiments have shown that queueing delay can be kept at the target in any overload situation, including with unresponsive traffic, and no further measures are required [DualQ-Test].

Delay on Saturation: When L4S marking saturates, instead of switching to drop, the drop and marking probabilities could be capped. Beyond that, delay will grow either solely in the queue with unresponsive traffic (if WRR is used), or in both queues (if time-shifted FIFO is used). In either case, the higher delay ought to control temporary high congestion. If the overload is more persistent, eventually the combined DualQ will overflow and tail drop will control congestion.

The example implementation in Appendix A solely applies the "drop on saturation" policy. The DOCSIS specification of a DualQ Coupled AQM [DOCSIS3.1] also implements the ’drop on saturation’ policy with a very shallow L buffer. However, the addition of DOCSIS per-flow Queue Protection [I-D.briscoe-docsis-q-protection] turns this into ’delay on saturation’ by redirecting some packets of the flow(s) most responsible for L queue overload into the C queue, which has a higher delay target. If overload continues, this again becomes ’drop on

De Schepper, et al. Expires January 7, 2022 [Page 24] Internet-Draft DualQ Coupled AQMs July 2021

saturation’ as the level of drop in the C queue rises to maintain the target delay of the C queue.

4.1.3. Protecting against Unresponsive ECN-Capable Traffic

Unresponsive traffic has a greater advantage if it is also ECN- capable. The advantage is undetectable at normal low levels of drop/ marking, but it becomes significant with the higher levels of drop/ marking typical during overload. This is an issue whether the ECN- capable traffic is L4S or Classic.

This raises the question of whether and when to switch off ECN marking and use solely drop instead, as required by both Section 7 of [RFC3168] and Section 4.2.1 of [RFC7567].

Experiments with the DualPI2 AQM (Appendix A) have shown that introducing ’drop on saturation’ at 100% L4S marking addresses this problem with unresponsive ECN as well as addressing the saturation problem. It leaves only a small range of congestion levels where unresponsive traffic gains any advantage from using the ECN capability, and the advantage is hardly detectable [DualQ-Test].

5. Acknowledgements

Thanks to Anil Agarwal, Sowmini Varadhan’s, Gabi Bracha, Nicolas Kuhn, Greg Skinner, Tom Henderson and David Pullen for detailed review comments particularly of the appendices and suggestions on how to make the explanations clearer. Thanks also to Tom Henderson for insights on the choice of schedulers and queue delay measurement techniques.

The early contributions of Koen De Schepper, Bob Briscoe, Olga Bondarenko and Inton Tsang were part-funded by the European Community under its Seventh Framework Programme through the Reducing Internet Transport Latency (RITE) project (ICT-317700). Bob Briscoe’s contribution was also part-funded by the Comcast Innovation Fund and the Research Council of Norway through the TimeIn project. The views expressed here are solely those of the authors.

6. Contributors

The following contributed implementations and evaluations that validated and helped to improve this specification:

Olga Albisser of Simula Research Lab, Norway (Olga Bondarenko during early drafts) implemented the prototype DualPI2 AQM for Linux with Koen De Schepper and conducted

De Schepper, et al. Expires January 7, 2022 [Page 25] Internet-Draft DualQ Coupled AQMs July 2021

extensive evaluations as well as implementing the live performance visualization GUI [L4Sdemo16].

Olivier Tilmans of Nokia Bell Labs, Belgium prepared and maintains the Linux implementation of DualPI2 for upstreaming.

Shravya K.S. wrote a model for the ns-3 simulator based on the -01 version of this Internet-Draft. Based on this initial work, Tom Henderson updated that earlier model and created a model for the DualQ variant specified as part of the Low Latency DOCSIS specification, as well as conducting extensive evaluations.

Ing Jyh (Inton) Tsang of Nokia, Belgium built the End-to-End Data Centre to the Home broadband testbed on which DualQ Coupled AQM implementations were tested.

7. References

7.1. Normative References

[I-D.ietf-tsvwg-ecn-l4s-id] Schepper, K. D. and B. Briscoe, "Explicit Congestion Notification (ECN) Protocol for Ultra-Low Queuing Delay (L4S)", draft-ietf-tsvwg-ecn-l4s-id-14 (work in progress), March 2021.

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, .

[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition of Explicit Congestion Notification (ECN) to IP", RFC 3168, DOI 10.17487/RFC3168, September 2001, .

[RFC8311] Black, D., "Relaxing Restrictions on Explicit Congestion Notification (ECN) Experimentation", RFC 8311, DOI 10.17487/RFC8311, January 2018, .

7.2. Informative References

De Schepper, et al. Expires January 7, 2022 [Page 26] Internet-Draft DualQ Coupled AQMs July 2021

[Alizadeh-stability] Alizadeh, M., Javanmard, A., and B. Prabhakar, "Analysis of DCTCP: Stability, Convergence, and Fairness", ACM SIGMETRICS 2011 , June 2011, .

[AQMmetrics] Kwon, M. and S. Fahmy, "A Comparison of Load-based and Queue- based Active Queue Management Algorithms", Proc. Int’l Soc. for Optical Engineering (SPIE) 4866:35--46 DOI: 10.1117/12.473021, 2002, .

[ARED01] Floyd, S., Gummadi, R., and S. Shenker, "Adaptive RED: An Algorithm for Increasing the Robustness of RED’s Active Queue Management", ACIRI Technical Report , August 2001, .

[BBRv1] Cardwell, N., Cheng, Y., Hassas Yeganeh, S., and V. Jacobson, "BBR Congestion Control", Internet Draft draft- cardwell-iccrg-bbr-congestion-control-00, July 2017, .

[BBRv2] Cardwell, N., "BRTCP BBR v2 Alpha/Preview Release", github repository; Linux congestion control module, .

[CCcensus19] Mishra, A., Sun, X., Jain, A., Pande, S., Joshi, R., and B. Leong, "The Great Internet TCP Congestion Control Census", Proc. ACM on Measurement and Analysis of Computing Systems 3(3), December 2019, .

[CoDel] Nichols, K. and V. Jacobson, "Controlling Queue Delay", ACM Queue 10(5), May 2012, .

[CRED_Insights] Briscoe, B., "Insights from Curvy RED (Random Early Detection)", BT Technical Report TR-TUB8-2015-003 arXiv:1904.07339 [cs.NI], July 2015, .

De Schepper, et al. Expires January 7, 2022 [Page 27] Internet-Draft DualQ Coupled AQMs July 2021

[DCttH15] De Schepper, K., Bondarenko, O., Briscoe, B., and I. Tsang, "‘Data Centre to the Home’: Ultra-Low Latency for All", RITE project Technical Report , 2015, .

[DOCSIS3.1] CableLabs, "MAC and Upper Layer Protocols Interface (MULPI) Specification, CM-SP-MULPIv3.1", Data-Over-Cable Service Interface Specifications DOCSIS(R) 3.1 Version i17 or later, January 2019, .

[DualPI2Linux] Albisser, O., De Schepper, K., Briscoe, B., Tilmans, O., and H. Steen, "DUALPI2 - Low Latency, Low Loss and Scalable (L4S) AQM", Proc. Linux Netdev 0x13 , March 2019, .

[DualQ-Test] Steen, H., "Destruction Testing: Ultra-Low Delay using Dual Queue Coupled Active Queue Management", Masters Thesis, Dept of Informatics, Uni Oslo , May 2017.

[I-D.briscoe-docsis-q-protection] Briscoe, B. and G. White, "Queue Protection to Preserve Low Latency", draft-briscoe-docsis-q-protection-00 (work in progress), July 2019.

[I-D.briscoe-iccrg-prague-congestion-control] Schepper, K. D., Tilmans, O., and B. Briscoe, "Prague Congestion Control", draft-briscoe-iccrg-prague- congestion-control-00 (work in progress), March 2021.

[I-D.briscoe-tsvwg-l4s-diffserv] Briscoe, B., "Interactions between Low Latency, Low Loss, Scalable Throughput (L4S) and Differentiated Services", draft-briscoe-tsvwg-l4s-diffserv-02 (work in progress), November 2018.

[I-D.cardwell-iccrg-bbr-congestion-control] Cardwell, N., Cheng, Y., Yeganeh, S. H., and V. Jacobson, "BBR Congestion Control", draft-cardwell-iccrg-bbr- congestion-control-00 (work in progress), July 2017.

De Schepper, et al. Expires January 7, 2022 [Page 28] Internet-Draft DualQ Coupled AQMs July 2021

[I-D.ietf-tsvwg-l4s-arch] Briscoe, B., Schepper, K. D., Bagnulo, M., and G. White, "Low Latency, Low Loss, Scalable Throughput (L4S) Internet Service: Architecture", draft-ietf-tsvwg-l4s-arch-08 (work in progress), November 2020.

[I-D.ietf-tsvwg-nqb] White, G. and T. Fossati, "A Non-Queue-Building Per-Hop Behavior (NQB PHB) for Differentiated Services", draft- ietf-tsvwg-nqb-05 (work in progress), March 2021.

[L4Sdemo16] Bondarenko, O., De Schepper, K., Tsang, I., and B. Briscoe, "Ultra-Low Delay for All: Live Experience, Live Analysis", Proc. MMSYS’16 pp33:1--33:4, May 2016, .

[Labovitz10] Labovitz, C., Iekel-Johnson, S., McPherson, D., Oberheide, J., and F. Jahanian, "Internet Inter-Domain Traffic", Proc ACM SIGCOMM; ACM CCR 40(4):75--86, August 2010, .

[LLD] White, G., Sundaresan, K., and B. Briscoe, "Low Latency DOCSIS: Technology Overview", CableLabs White Paper , February 2019, .

[Mathis09] Mathis, M., "Relentless Congestion Control", PFLDNeT’09 , May 2009, .

[MEDF] Menth, M., Schmid, M., Heiss, H., and T. Reim, "MEDF - a simple scheduling algorithm for two real-time transport service classes with application in the UTRAN", Proc. IEEE Conference on Computer Communications (INFOCOM’03) Vol.2 pp.1116-1122, March 2003.

[PI2] De Schepper, K., Bondarenko, O., Briscoe, B., and I. Tsang, "PI2: A Linearized AQM for both Classic and Scalable TCP", ACM CoNEXT’16 , December 2016, .

De Schepper, et al. Expires January 7, 2022 [Page 29] Internet-Draft DualQ Coupled AQMs July 2021

[PI2param] Briscoe, B., "PI2 Parameters", Technical Report TR-BB- 2021-001 arXiv:2107.01003 [cs.NI], July 2021, .

[PragueLinux] Briscoe, B., De Schepper, K., Albisser, O., Misund, J., Tilmans, O., Kuehlewind, M., and A. Ahmed, "Implementing the ‘TCP Prague’ Requirements for Low Latency Low Loss Scalable Throughput (L4S)", Proc. Linux Netdev 0x13 , March 2019, .

[RFC0970] Nagle, J., "On Packet Switches With Infinite Storage", RFC 970, DOI 10.17487/RFC0970, December 1985, .

[RFC2309] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, S., Wroclawski, J., and L. Zhang, "Recommendations on Queue Management and Congestion Avoidance in the Internet", RFC 2309, DOI 10.17487/RFC2309, April 1998, .

[RFC3649] Floyd, S., "HighSpeed TCP for Large Congestion Windows", RFC 3649, DOI 10.17487/RFC3649, December 2003, .

[RFC5033] Floyd, S. and M. Allman, "Specifying New Congestion Control Algorithms", BCP 133, RFC 5033, DOI 10.17487/RFC5033, August 2007, .

[RFC5348] Floyd, S., Handley, M., Padhye, J., and J. Widmer, "TCP Friendly Rate Control (TFRC): Protocol Specification", RFC 5348, DOI 10.17487/RFC5348, September 2008, .

[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, .

De Schepper, et al. Expires January 7, 2022 [Page 30] Internet-Draft DualQ Coupled AQMs July 2021

[RFC5706] Harrington, D., "Guidelines for Considering Operations and Management of New Protocols and Protocol Extensions", RFC 5706, DOI 10.17487/RFC5706, November 2009, .

[RFC7567] Baker, F., Ed. and G. Fairhurst, Ed., "IETF Recommendations Regarding Active Queue Management", BCP 197, RFC 7567, DOI 10.17487/RFC7567, July 2015, .

[RFC8033] Pan, R., Natarajan, P., Baker, F., and G. White, "Proportional Integral Controller Enhanced (PIE): A Lightweight Control Scheme to Address the Bufferbloat Problem", RFC 8033, DOI 10.17487/RFC8033, February 2017, .

[RFC8034] White, G. and R. Pan, "Active Queue Management (AQM) Based on Proportional Integral Controller Enhanced PIE) for Data-Over-Cable Service Interface Specifications (DOCSIS) Cable Modems", RFC 8034, DOI 10.17487/RFC8034, February 2017, .

[RFC8257] Bensley, S., Thaler, D., Balasubramanian, P., Eggert, L., and G. Judd, "Data Center TCP (DCTCP): TCP Congestion Control for Data Centers", RFC 8257, DOI 10.17487/RFC8257, October 2017, .

[RFC8290] Hoeiland-Joergensen, T., McKenney, P., Taht, D., Gettys, J., and E. Dumazet, "The Flow Queue CoDel Packet Scheduler and Active Queue Management Algorithm", RFC 8290, DOI 10.17487/RFC8290, January 2018, .

[RFC8298] Johansson, I. and Z. Sarker, "Self-Clocked Rate Adaptation for Multimedia", RFC 8298, DOI 10.17487/RFC8298, December 2017, .

[RFC8312] Rhee, I., Xu, L., Ha, S., Zimmermann, A., Eggert, L., and R. Scheffenegger, "CUBIC for Fast Long-Distance Networks", RFC 8312, DOI 10.17487/RFC8312, February 2018, .

[SCReAM] Johansson, I., "SCReAM", github repository; , .

De Schepper, et al. Expires January 7, 2022 [Page 31] Internet-Draft DualQ Coupled AQMs July 2021

[SigQ-Dyn] Briscoe, B., "Rapid Signalling of Queue Dynamics", Technical Report TR-BB-2017-001 arXiv:1904.07044 [cs.NI], September 2017, .

Appendix A. Example DualQ Coupled PI2 Algorithm

As a first concrete example, the pseudocode below gives the DualPI2 algorithm. DualPI2 follows the structure of the DualQ Coupled AQM framework in Figure 1. A simple ramp function (configured in units of queuing time) with unsmoothed ECN marking is used for the Native L4S AQM. The ramp can also be configured as a step function. The PI2 algorithm [PI2] is used for the Classic AQM. PI2 is an improved variant of the PIE AQM [RFC8033].

The pseudocode will be introduced in two passes. The first pass explains the core concepts, deferring handling of overload to the second pass. To aid comparison, line numbers are kept in step between the two passes by using letter suffixes where the longer code needs extra lines.

All variables are assumed to be floating point in their basic units (size in bytes, time in seconds, rates in bytes/second, alpha and beta in Hz, and probabilities from 0 to 1. Constants expressed in k (kilo), M (mega), G (giga), u (micro), m (milli) , %, ... are assumed to be converted to their appropriate multiple or fraction to represent the basic units. A real implementation that wants to use integer values needs to handle appropriate scaling factors and allow accordingly appropriate resolution of its integer types (including temporary internal values during calculations).

A full open source implementation for Linux is available at: https://github.com/L4STeam/sch_dualpi2_upstream and explained in [DualPI2Linux]. The specification of the DualQ Coupled AQM for DOCSIS cable modems and CMTSs is available in [DOCSIS3.1] and explained in [LLD].

A.1. Pass #1: Core Concepts

The pseudocode manipulates three main structures of variables: the packet (pkt), the L4S queue (lq) and the Classic queue (cq). The pseudocode consists of the following six functions:

o The initialization function dualpi2_params_init(...) (Figure 2) that sets parameter defaults (the API for setting non-default values is omitted for brevity)

o The enqueue function dualpi2_enqueue(lq, cq, pkt) (Figure 3)

De Schepper, et al. Expires January 7, 2022 [Page 32] Internet-Draft DualQ Coupled AQMs July 2021

o The dequeue function dualpi2_dequeue(lq, cq, pkt) (Figure 4)

o The recurrence function recur(q, likelihood) for de-randomized ECN marking (shown at the end of Figure 4).

o The L4S AQM function laqm(qdelay) (Figure 5) used to calculate the ECN-marking probability for the L4S queue

o The base AQM function that implements the PI algorithm dualpi2_update(lq, cq) (Figure 6) used to regularly update the base probability (p’), which is squared for the Classic AQM as well as being coupled across to the L4S queue.

It also uses the following functions that are not shown in full here:

o scheduler(), which selects between the head packets of the two queues; the choice of scheduler technology is discussed later;

o cq.len() or lq.len() returns the current length (aka. backlog) of the relevant queue in bytes;

o cq.time() or lq.time() returns the current queuing delay (aka. sojourn time or service time) of the relevant queue in units of time (see Note a);

o mark(pkt) and drop(pkt) for ECN-marking and dropping a packet;

In experiments so far (building on experiments with PIE) on broadband access links ranging from 4 Mb/s to 200 Mb/s with base RTTs from 5 ms to 100 ms, DualPI2 achieves good results with the default parameters in Figure 2. The parameters are categorised by whether they relate to the Base PI2 AQM, the L4S AQM or the framework coupling them together. Constants and variables derived from these parameters are also included at the end of each category. Each parameter is explained as it is encountered in the walk-through of the pseudocode below, and the rationale for the chosen defaults are given so that sensible values can be used in scenarios other than the regular public Internet.

De Schepper, et al. Expires January 7, 2022 [Page 33] Internet-Draft DualQ Coupled AQMs July 2021

1: dualpi2_params_init(...) { % Set input parameter defaults 2: % DualQ Coupled framework parameters 5: limit = MAX_LINK_RATE * 250 ms % Dual buffer size 3: k = 2 % Coupling factor 4: % NOT SHOWN % scheduler-dependent weight or equival’t parameter 6: 7: % PI2 Classic AQM parameters 8: % Typical RTT, RTT_typ = 34 ms 9: target = 15 ms % Queue delay target = RTT_typ * 0.22 * 2 10: RTT_max = 100 ms % Worst case RTT expected 11: % PI2 constants derived from above PI2 parameters 12: p_Cmax = min(1/k^2, 1) % Max Classic drop/mark prob 13: Tupdate = min(target, RTT_max/3) % PI sampling interval 14: alpha = 0.1 * Tupdate / RTT_max^2 % PI integral gain in Hz 15: beta = 0.3 / RTT_max % PI proportional gain in Hz 16: 17: % L4S ramp AQM parameters 18: minTh = 800 us % L4S min marking threshold in time units 19: range = 400 us % Range of L4S ramp in time units 20: Th_len = 2 * MTU % Min L4S marking threshold in bytes 21: % L4S constants incl. those derived from other parameters 22: p_Lmax = 1 % Max L4S marking prob 23: floor = Th_len / MIN_LINK_RATE 24: if (minTh < floor) { 25: % Shift ramp so minTh >= serialization time of 2 MTU 26: minTh = floor 27: } 28: maxTh = minTh+range % L4S max marking threshold in time units 29: }

Figure 2: Example Header Pseudocode for DualQ Coupled PI2 AQM

The overall goal of the code is to maintain the base probability (p’, p-prime as in Section 2.4), which is an internal variable from which the marking and dropping probabilities for L4S and Classic traffic (p_L and p_C) are derived, with p_L in turn being derived from p_CL. The probabilities p_CL and p_C are derived in lines 4 and 5 of the dualpi2_update() function (Figure 6) then used in the dualpi2_dequeue() function where p_L is also derived from p_CL at line 6 (Figure 4). The code walk-through below builds up to explaining that part of the code eventually, but it starts from packet arrival.

De Schepper, et al. Expires January 7, 2022 [Page 34] Internet-Draft DualQ Coupled AQMs July 2021

1: dualpi2_enqueue(lq, cq, pkt) { % Test limit and classify lq or cq 2: if ( lq.len() + cq.len() + MTU > limit) 3: drop(pkt) % drop packet if buffer is full 4: timestamp(pkt) % attach arrival time to packet 5: % Packet classifier 6: if ( ecn(pkt) modulo 2 == 1 ) % ECN bits = ECT(1) or CE 7: lq.enqueue(pkt) 8: else % ECN bits = not-ECT or ECT(0) 9: cq.enqueue(pkt) 10: }

Figure 3: Example Enqueue Pseudocode for DualQ Coupled PI2 AQM

1: dualpi2_dequeue(lq, cq, pkt) { % Couples L4S & Classic queues 2: while ( lq.len() + cq.len() > 0 ) { 3: if ( scheduler() == lq ) { 4: lq.dequeue(pkt) % Scheduler chooses lq 5: p’_L = laqm(lq.time()) % Native L4S AQM 6: p_L = max(p’_L, p_CL) % Combining function 7: if ( recur(lq, p_L) ) % Linear marking 8: mark(pkt) 9: } else { 10: cq.dequeue(pkt) % Scheduler chooses cq 11: if ( recur(cq, p_C) ) { % probability p_C = p’^2 12: if ( ecn(pkt) == 0 ) { % if ECN field = not-ECT 13: drop(pkt) % squared drop 14: continue % continue to the top of the while loop 15: } 16: mark(pkt) % squared mark 17: } 18: } 19: return(pkt) % return the packet and stop 20: } 21: return(NULL) % no packet to dequeue 22: }

23: recur(q, likelihood) { % Returns TRUE with a certain likelihood 24: q.count += likelihood 25: if (q.count > 1) { 26: q.count -= 1 27: return TRUE 28: } 29: return FALSE 30: }

Figure 4: Example Dequeue Pseudocode for DualQ Coupled PI2 AQM

De Schepper, et al. Expires January 7, 2022 [Page 35] Internet-Draft DualQ Coupled AQMs July 2021

When packets arrive, first a common queue limit is checked as shown in line 2 of the enqueuing pseudocode in Figure 3. This assumes a shared buffer for the two queues (Note b discusses the merits of separate buffers). In order to avoid any bias against larger packets, 1 MTU of space is always allowed and the limit is deliberately tested before enqueue.

If limit is not exceeded, the packet is timestamped in line 4. This assumes that queue delay is measured using the sojourn time technique (see Note a for alternatives).

At lines 5-9, the packet is classified and enqueued to the Classic or L4S queue dependent on the least significant bit of the ECN field in the IP header (line 6). Packets with a codepoint having an LSB of 0 (Not-ECT and ECT(0)) will be enqueued in the Classic queue. Otherwise, ECT(1) and CE packets will be enqueued in the L4S queue. Optional additional packet classification flexibility is omitted for brevity (see [I-D.ietf-tsvwg-ecn-l4s-id]).

The dequeue pseudocode (Figure 4) is repeatedly called whenever the lower layer is ready to forward a packet. It schedules one packet for dequeuing (or zero if the queue is empty) then returns control to the caller, so that it does not block while that packet is being forwarded. While making this dequeue decision, it also makes the necessary AQM decisions on dropping or marking. The alternative of applying the AQMs at enqueue would shift some processing from the critical time when each packet is dequeued. However, it would also add a whole queue of delay to the control signals, making the control loop sloppier (for a typical RTT it would double the Classic queue’s feedback delay).

All the dequeue code is contained within a large while loop so that if it decides to drop a packet, it will continue until it selects a packet to schedule. Line 3 of the dequeue pseudocode is where the scheduler chooses between the L4S queue (lq) and the Classic queue (cq). Detailed implementation of the scheduler is not shown (see discussion later).

o If an L4S packet is scheduled, in lines 7 and 8 the packet is ECN- marked with likelihood p_L. The recur() function at the end of Figure 4 is used, which is preferred over random marking because it avoids delay due to randomization when interpreting congestion signals, but it still desynchronizes the saw-teeth of the flows. Line 6 calculates p_L as the maximum of the coupled L4S probability p_CL and the probability from the native L4S AQM p’_L. This implements the max() function shown in Figure 1 to couple the outputs of the two AQMs together. Of the two probabilities input to p_L in line 6:

De Schepper, et al. Expires January 7, 2022 [Page 36] Internet-Draft DualQ Coupled AQMs July 2021

* p’_L is calculated per packet in line 5 by the laqm() function (see Figure 5),

* Whereas p_CL is maintained by the dualpi2_update() function which runs every Tupdate (Tupdate is set in line 13 of Figure 2).

o If a Classic packet is scheduled, lines 10 to 17 drop or mark the packet with probability p_C.

The Native L4S AQM algorithm (Figure 5) is a ramp function, similar to the RED algorithm, but simplified as follows:

o The extent of the ramp is defined in units of queuing delay, not bytes, so that configuration remains invariant as the queue departure rate varies.

o It uses instantaneous queueing delay, which avoids the complexity of smoothing, but also avoids embedding a worst-case RTT of smoothing delay in the network (see Section 2.1).

o The ramp rises linearly directly from 0 to 1, not to an intermediate value of p’_L as RED would, because there is no need to keep ECN marking probability low.

o Marking does not have to be randomized. Determinism is used instead of randomness; to reduce the delay necessary to smooth out the noise of randomness from the signal.

The ramp function requires two configuration parameters, the minimum threshold (minTh) and the width of the ramp (range), both in units of queuing time), as shown in lines 18 & 19 of the initialization function in Figure 2. The ramp function can be configured as a step (see Note c).

Although the DCTCP paper [Alizadeh-stability] recommends an ECN marking threshold of 0.17*RTT_typ, it also shows that the threshold can be much shallower with hardly any worse under-utilization of the link (because the amplitude of DCTCP’s sawteeth is so small). Based on extensive experiments, for the public Internet the default minimum ECN marking threshold in Figure 2 is considered a good compromise, even though it is significantly smaller fraction of RTT_typ.

A minimum marking threshold parameter (Th_len) in transmission units (default 2 MTU) is also necessary to ensure that the ramp does not trigger excessive marking on slow links. The code in lines 24-27 of the initialization function (Figure 2) converts 2 MTU into time units

De Schepper, et al. Expires January 7, 2022 [Page 37] Internet-Draft DualQ Coupled AQMs July 2021

and shifts the ramp so that the min threshold is no shallower than this floor.

1: laqm(qdelay) { % Returns native L4S AQM probability 2: if (qdelay >= maxTh) 3: return 1 4: else if (qdelay > minTh) 5: return (qdelay - minTh)/range % Divide could use a bit-shift 6: else 7: return 0 8: }

Figure 5: Example Pseudocode for the Native L4S AQM

1: dualpi2_update(lq, cq) { % Update p’ every Tupdate 2: curq = cq.time() % use queuing time of first-in Classic packet 3: p’ = p’ + alpha * (curq - target) + beta * (curq - prevq) 4: p_CL = k * p’ % Coupled L4S prob = base prob * coupling factor 5: p_C = p’^2 % Classic prob = (base prob)^2 6: prevq = curq 7: }

(Clamping p’ within the range [0,1] omitted for clarity - see text)

Figure 6: Example PI-Update Pseudocode for DualQ Coupled PI2 AQM

The coupled marking probability, p_CL depends on the base probability (p’), which is kept up to date by the core PI algorithm in Figure 6 executed every Tupdate.

Note that p’ solely depends on the queuing time in the Classic queue. In line 2, the current queuing delay (curq) is evaluated from how long the head packet was in the Classic queue (cq). The function cq.time() (not shown) subtracts the time stamped at enqueue from the current time (see Note a) and implicitly takes the current queuing delay as 0 if the queue is empty.

The algorithm centres on line 3, which is a classical Proportional- Integral (PI) controller that alters p’ dependent on: a) the error between the current queuing delay (curq) and the target queuing delay, ’target’; and b) the change in queuing delay since the last sample. The name ’PI’ represents the fact that the second factor (how fast the queue is growing) is _P_roportional to load while the first is the _I_ntegral of the load (so it removes any standing queue in excess of the target).

The target parameter can be set based on local knowledge, but the aim is for the default to be a good compromise for anywhere in the

De Schepper, et al. Expires January 7, 2022 [Page 38] Internet-Draft DualQ Coupled AQMs July 2021

intended deployment environment---the public Internet. The target queuing delay is related to the typical base RTT, RTT_typ, by two factors, shown in the comment on line 9 of Figure 2 as target = RTT_typ * 0.22 * 2. These factors ensure that, in a large proportion of cases (say 90%), the sawtooth variations in RTT will fit within the buffer without underutilizing the link. Frankly, these factors are educated guesses, but with the emphasis closer to ’educated’ than to ’guess’ (see [PI2param] for background investigations):

o RTT_typ is taken as 34 ms. This is based on an average CDN latency measured in each country weighted by the number of Internet users in that country to produce an overall weighted average for the Internet [PI2param].

o The factor 0.22 is a geometry factor that characterizes the shape of the sawteeth of prevalent Classic congestion controllers. The geometry factor is the difference between the minimum and the average queue delays of the sawteeth, relative to the base RTT. For instance, the geometry factor of standard Reno is 0.5. According to the census of congestion controllers conducted by Mishra _et al_ in Jul-Oct 2019 [CCcensus19], most Classic TCP traffic uses Cubic. And, according to the analysis in [PI2param], if running over a PI2 AQM, a large proportion of this Cubic traffic would be in its Reno-Friendly mode, which has a geometry factor of 0.21 (Linux implementation). The rest of the Cubic traffic would be in true Cubic mode, which has a geometry factor of 0.32. Without modelling the sawtooth profiles from all the other less prevalent congestion controllers, we estimate a 9:1 weighted average of these two, resulting in an average geometry factor of 0.22.

o The factor 2, is a safety factor that increases the target queue to allow for the distribution of RTT_typ around its mean. Otherwise the target queue would only avoid underutilization for those users below the mean. It also provides a safety margin for the proportion of paths in use that span beyond the distance between a user and their local CDN. Currently no data is available on the variance of queue delay around the mean in each region, so there is plenty of room for this guess to become more educated.

The two ’gain factors’ in line 3 of Figure 6, alpha and beta, respectively weight how strongly each of the two elements (Integral and Proportional) alters p’. They are in units of ’per second of delay’ or Hz, because they transform differences in queueing delay into changes in probability (assuming probability has a value from 0 to 1).

De Schepper, et al. Expires January 7, 2022 [Page 39] Internet-Draft DualQ Coupled AQMs July 2021

alpha and beta determine how much p’ ought to change after each update interval (Tupdate). For smaller Tupdate, p’ should change by the same amount per second, but in finer more frequent steps. So alpha depends on Tupdate (see line 13 of the initialization function in Figure 2). It is best to update p’ as frequently as possible, but Tupdate will probably be constrained by hardware performance. As shown in line 13, the update interval should be frequent enough to update at least once in the time taken for the target queue to drain (’target’) as long as it updates at least three times per maximum RTT. Tupdate defaults to 16 ms in the reference Linux implementation because it has to be rounded to a multiple of 4 ms. For link rates from 4 to 200 Mb/s and a maximum RTT of 100ms, it has been verified through extensive testing that Tupdate=16ms (as also recommended in [RFC8033]) is sufficient.

The choice of alpha and beta also determines the AQM’s stable operating range. The AQM ought to change p’ as fast as possible in response to changes in load without over-compensating and therefore causing oscillations in the queue. Therefore, the values of alpha and beta also depend on the RTT of the expected worst-case flow (RTT_max).

The maximum RTT of a PI controller (RTT_max in line 10 of Figure 2) is not an absolute maximum, but more instability (more queue variability) sets in for long-running flows with an RTT above this value. The propagation delay half way round the planet and back in glass fibre is 200 ms. However, hardly any traffic traverses such extreme paths and, since the significant consolidation of Internet traffic between 2007 and 2009 [Labovitz10], a high and growing proportion of all Internet traffic (roughly two-thirds at the time of writing) has been served from content distribution networks (CDNs) or ’cloud’ services distributed close to end-users. The Internet might change again, but for now, designing for a maximum RTT of 100ms is a good compromise between faster queue control at low RTT and some instability on the occasions when a longer path is necessary.

Recommended derivations of the gain constants alpha and beta can be approximated for Reno over a PI2 AQM as: alpha = 0.1 * Tupdate / RTT_max^2; beta = 0.3 / RTT_max, as shown in lines 14 & 15 of Figure 2. These are derived from the stability analysis in [PI2]. For the default values of Tupdate=16 ms and RTT_max = 100 ms, they result in alpha = 0.16; beta = 3.2 (discrepancies are due to rounding). These defaults have been verified with a wide range of link rates, target delays and a range of traffic models with mixed and similar RTTs, short and long flows, etc.

In corner cases, p’ can overflow the range [0,1] so the resulting value of p’ has to be bounded (omitted from the pseudocode). Then,

De Schepper, et al. Expires January 7, 2022 [Page 40] Internet-Draft DualQ Coupled AQMs July 2021

as already explained, the coupled and Classic probabilities are derived from the new p’ in lines 4 and 5 of Figure 6 as p_CL = k*p’ and p_C = p’^2.

Because the coupled L4S marking probability (p_CL) is factored up by k, the dynamic gain parameters alpha and beta are also inherently factored up by k for the L4S queue. So, the effective gain factor for the L4S queue is k*alpha (with defaults alpha = 0.16 Hz and k=2, effective L4S alpha = 0.32 Hz).

Unlike in PIE [RFC8033], alpha and beta do not need to be tuned every Tupdate dependent on p’. Instead, in PI2, alpha and beta are independent of p’ because the squaring applied to Classic traffic tunes them inherently. This is explained in [PI2], which also explains why this more principled approach removes the need for most of the heuristics that had to be added to PIE.

Nonetheless, an implementer might wish to add selected heuristics to either AQM. For instance the Linux reference DualPI2 implementation includes the following:

o Prior to enqueuing an L4S packet, if the L queue contains <2 packets, the packet is flagged to suppress any native L4S AQM marking at dequeue (which depends on sojourn time);

o Classic and coupled marking or dropping (i.e. based on p_C and p_CL from the PI controller) is only applied to a packet if the respective queue length in bytes is > 2 MTU (prior to enqueuing the packet or after dequeuing it, depending on whether the AQM is configured to be applied at enqueue or dequeue);

o In the WRR scheduler, the ’credit’ indicating which queue should transmit is only changed if there are packets in both queues (i.e. if there is actual resource contention). This means that a properly paced L flow might never be delayed by the WRR. The WRR credit is reset in favour of the L queue when the link is idle.

An implementer might also wish to add other heuristics, e.g. burst protection [RFC8033] or enhanced burst protection [RFC8034].

Notes:

a. The drain rate of the queue can vary if it is scheduled relative to other queues, or to cater for fluctuations in a wireless medium. To auto-adjust to changes in drain rate, the queue needs to be measured in time, not bytes or packets [AQMmetrics], [CoDel]. Queuing delay could be measured directly by storing a per-packet time-stamp as each packet is enqueued, and subtracting

De Schepper, et al. Expires January 7, 2022 [Page 41] Internet-Draft DualQ Coupled AQMs July 2021

this from the system time when the packet is dequeued. If time- stamping is not easy to introduce with certain hardware, queuing delay could be predicted indirectly by dividing the size of the queue by the predicted departure rate, which might be known precisely for some link technologies (see for example [RFC8034]).

b. Line 2 of the dualpi2_enqueue() function (Figure 3) assumes an implementation where lq and cq share common buffer memory. An alternative implementation could use separate buffers for each queue, in which case the arriving packet would have to be classified first to determine which buffer to check for available space. The choice is a trade off; a shared buffer can use less memory whereas separate buffers isolate the L4S queue from tail- drop due to large bursts of Classic traffic (e.g. a Classic Reno TCP during slow-start over a long RTT).

c. There has been some concern that using the step function of DCTCP for the Native L4S AQM requires end-systems to smooth the signal for an unnecessarily large number of round trips to ensure sufficient fidelity. A ramp is no worse than a step in initial experiments with existing DCTCP. Therefore, it is recommended that a ramp is configured in place of a step, which will allow congestion control algorithms to investigate faster smoothing algorithms.

A ramp is more general that a step, because an operator can effectively turn the ramp into a step function, as used by DCTCP, by setting the range to zero. There will not be a divide by zero problem at line 5 of Figure 5 because, if minTh is equal to maxTh, the condition for this ramp calculation cannot arise.

A.2. Pass #2: Overload Details

Figure 7 repeats the dequeue function of Figure 4, but with overload details added. Similarly Figure 8 repeats the core PI algorithm of Figure 6 with overload details added. The initialization, enqueue, L4S AQM and recur functions are unchanged.

In line 10 of the initialization function (Figure 2), the maximum Classic drop probability p_Cmax = min(1/k^2, 1) or 1/4 for the default coupling factor k=2. p_Cmax is the point at which it is deemed that the Classic queue has become persistently overloaded, so it switches to using drop, even for ECN-capable packets. ECT packets that are not dropped can still be ECN-marked.

In practice, 25% has been found to be a good threshold to preserve fairness between ECN capable and non ECN capable traffic. This protects the queues against both temporary overload from responsive

De Schepper, et al. Expires January 7, 2022 [Page 42] Internet-Draft DualQ Coupled AQMs July 2021

flows and more persistent overload from any unresponsive traffic that falsely claims to be responsive to ECN.

When the Classic ECN marking probability reaches the p_Cmax threshold (1/k^2), the marking probability coupled to the L4S queue, p_CL will always be 100% for any k (by equation (1) in Section 2). So, for readability, the constant p_Lmax is defined as 1 in line 22 of the initialization function (Figure 2). This is intended to ensure that the L4S queue starts to introduce dropping once ECN-marking saturates at 100% and can rise no further. The ’Prague L4S’ requirements [I-D.ietf-tsvwg-ecn-l4s-id] state that, when an L4S congestion control detects a drop, it falls back to a response that coexists with ’Classic’ Reno congestion control. So it is correct that, when the L4S queue drops packets, it drops them proportional to p’^2, as if they are Classic packets.

Both these switch-overs are triggered by the tests for overload introduced in lines 4b and 12b of the dequeue function (Figure 7). Lines 8c to 8g drop L4S packets with probability p’^2. Lines 8h to 8i mark the remaining packets with probability p_CL. Given p_Lmax = 1, all remaining packets will be marked because, to have reached the else block at line 8b, p_CL >= 1.

Lines 2c to 2d in the core PI algorithm (Figure 8) deal with overload of the L4S queue when there is no Classic traffic. This is necessary, because the core PI algorithm maintains the appropriate drop probability to regulate overload, but it depends on the length of the Classic queue. If there is no Classic queue the naive PI update function in Figure 6 would drop nothing, even if the L4S queue were overloaded - so tail drop would have to take over (lines 2 and 3 of Figure 3).

Instead, the test at line 2a of the full PI update function in Figure 8 keeps delay on target using drop. If the test at line 2a of Figure 8 finds that the Classic queue is empty, line 2d measures the current queue delay using the L4S queue instead. While the L4S queue is not overloaded, its delay will always be tiny compared to the target Classic queue delay. So p_CL will be driven to zero, and the L4S queue will naturally be governed solely by p’_L from the native L4S AQM (lines 5 and 6 of the dequeue algorithm in Figure 7). But, if unresponsive L4S source(s) cause overload, the DualQ transitions smoothly to L4S marking based on the PI algorithm. If overload increases further, it naturally transitions from marking to dropping by the switch-over mechanism already described.

De Schepper, et al. Expires January 7, 2022 [Page 43] Internet-Draft DualQ Coupled AQMs July 2021

1: dualpi2_dequeue(lq, cq, pkt) { % Couples L4S & Classic queues 2: while ( lq.len() + cq.len() > 0 ) { 3: if ( scheduler() == lq ) { 4a: lq.dequeue(pkt) % L4S scheduled 4b: if ( p_CL < p_Lmax ) { % Check for overload saturation 5: p’_L = laqm(lq.time()) % Native L4S AQM 6: p_L = max(p’_L, p_CL) % Combining function 7: if ( recur(lq, p_L) ) % Linear marking 8a: mark(pkt) 8b: } else { % overload saturation 8c: if ( recur(lq, p_C) ) { % probability p_C = p’^2 8e: drop(pkt) % revert to Classic drop due to overload 8f: continue % continue to the top of the while loop 8g: } 8h: if ( recur(lq, p_CL) ) % probability p_CL = k * p’ 8i: mark(pkt) % linear marking of remaining packets 8j: } 9: } else { 10: cq.dequeue(pkt) % Classic scheduled 11: if ( recur(cq, p_C) ) { % probability p_C = p’^2 12a: if ( (ecn(pkt) == 0) % ECN field = not-ECT 12b: OR (p_C >= p_Cmax) ) { % Overload disables ECN 13: drop(pkt) % squared drop, redo loop 14: continue % continue to the top of the while loop 15: } 16: mark(pkt) % squared mark 17: } 18: } 19: return(pkt) % return the packet and stop 20: } 21: return(NULL) % no packet to dequeue 22: }

Figure 7: Example Dequeue Pseudocode for DualQ Coupled PI2 AQM (Including Overload Code)

De Schepper, et al. Expires January 7, 2022 [Page 44] Internet-Draft DualQ Coupled AQMs July 2021

1: dualpi2_update(lq, cq) { % Update p’ every Tupdate 2a: if ( cq.len() > 0 ) 2b: curq = cq.time() %use queuing time of first-in Classic packet 2c: else % Classic queue empty 2d: curq = lq.time() % use queuing time of first-in L4S packet 3: p’ = p’ + alpha * (curq - target) + beta * (curq - prevq) 4: p_CL = p’ * k % Coupled L4S prob = base prob * coupling factor 5: p_C = p’^2 % Classic prob = (base prob)^2 6: prevq = curq 7: }

Figure 8: Example PI-Update Pseudocode for DualQ Coupled PI2 AQM (Including Overload Code)

The choice of scheduler technology is critical to overload protection (see Section 4.1).

o A well-understood weighted scheduler such as weighted round robin (WRR) is recommended. As long as the scheduler weight for Classic is small (e.g. 1/16), its exact value is unimportant because it does not normally determine capacity shares. The weight is only important to prevent unresponsive L4S traffic starving Classic traffic. This is because capacity sharing between the queues is normally determined by the coupled congestion signal, which overrides the scheduler, by making L4S sources leave roughly equal per-flow capacity available for Classic flows.

o Alternatively, a time-shifted FIFO (TS-FIFO) could be used. It works by selecting the head packet that has waited the longest, biased against the Classic traffic by a time-shift of tshift. To implement time-shifted FIFO, the scheduler() function in line 3 of the dequeue code would simply be implemented as the scheduler() function at the bottom of Figure 10 in Appendix B. For the public Internet a good value for tshift is 50ms. For private networks with smaller diameter, about 4*target would be reasonable. TS- FIFO is a very simple scheduler, but complexity might need to be added to address some deficiencies (which is why it is not recommended over WRR):

* TS-FIFO does not fully isolate latency in the L4S queue from uncontrolled bursts in the Classic queue;

* TS-FIFO is only appropriate if time-stamping of packets is feasible;

* Even if time-stamping is supported, the sojourn time of the head packet is always stale. For instance, if a burst arrives at an empty queue, the sojourn time will only measure the delay

De Schepper, et al. Expires January 7, 2022 [Page 45] Internet-Draft DualQ Coupled AQMs July 2021

of the burst once the burst is over, even though the queue knew about it from the start. At the cost of more operations and more storage, a ’scaled sojourn time’ metric of queue delay can be used, which is the sojourn time of a packet scaled by the ratio of the queue sizes when the packet departed and arrived [SigQ-Dyn].

o A strict priority scheduler would be inappropriate, because it would starve Classic if L4S was overloaded.

Appendix B. Example DualQ Coupled Curvy RED Algorithm

As another example of a DualQ Coupled AQM algorithm, the pseudocode below gives the Curvy RED based algorithm. Although the AQM was designed to be efficient in integer arithmetic, to aid understanding it is first given using floating point arithmetic (Figure 10). Then, one possible optimization for integer arithmetic is given, also in pseudocode (Figure 11). To aid comparison, the line numbers are kept in step between the two by using letter suffixes where the longer code needs extra lines.

B.1. Curvy RED in Pseudocode

The pseudocode manipulates three main structures of variables: the packet (pkt), the L4S queue (lq) and the Classic queue (cq) and consists of the following five functions:

o The initialization function cred_params_init(...) (Figure 2) that sets parameter defaults (the API for setting non-default values is omitted for brevity);

o The dequeue function cred_dequeue(lq, cq, pkt) (Figure 4);

o The scheduling function scheduler(), which selects between the head packets of the two queues.

It also uses the following functions that are either shown elsewhere, or not shown in full here:

o The enqueue function, which is identical to that used for DualPI2, dualpi2_enqueue(lq, cq, pkt) in Figure 3;

o mark(pkt) and drop(pkt) for ECN-marking and dropping a packet;

o cq.len() or lq.len() returns the current length (aka. backlog) of the relevant queue in bytes;

De Schepper, et al. Expires January 7, 2022 [Page 46] Internet-Draft DualQ Coupled AQMs July 2021

o cq.time() or lq.time() returns the current queuing delay (aka. sojourn time or service time) of the relevant queue in units of time (see Note a in Appendix A.1).

Because Curvy RED was evaluated before DualPI2, certain improvements introduced for DualPI2 were not evaluated for Curvy RED. In the pseudocode below, the straightforward improvements have been added on the assumption they will provide similar benefits, but that has not been proven experimentally. They are: i) a conditional priority scheduler instead of strict priority ii) a time-based threshold for the native L4S AQM; iii) ECN support for the Classic AQM. A recent evaluation has proved that a minimum ECN-marking threshold (minTh) greatly improves performance, so this is also included in the pseudocode.

Overload protection has not been added to the Curvy RED pseudocode below so as not to detract from the main features. It would be added in exactly the same way as in Appendix A.2 for the DualPI2 pseudocode. The native L4S AQM uses a step threshold, but a ramp like that described for DualPI2 could be used instead. The scheduler uses the simple TS-FIFO algorithm, but it could be replaced with WRR.

The Curvy RED algorithm has not been maintained or evaluated to the same degree as the DualPI2 algorithm. In initial experiments on broadband access links ranging from 4 Mb/s to 200 Mb/s with base RTTs from 5 ms to 100 ms, Curvy RED achieved good results with the default parameters in Figure 9.

The parameters are categorised by whether they relate to the Classic AQM, the L4S AQM or the framework coupling them together. Constants and variables derived from these parameters are also included at the end of each category. These are the raw input parameters for the algorithm. A configuration front-end could accept more meaningful parameters (e.g. RTT_max and RTT_typ) and convert them into these raw parameters, as has been done for DualPI2 in Appendix A. Where necessary, parameters are explained further in the walk-through of the pseudocode below.

De Schepper, et al. Expires January 7, 2022 [Page 47] Internet-Draft DualQ Coupled AQMs July 2021

1: cred_params_init(...) { % Set input parameter defaults 2: % DualQ Coupled framework parameters 3: limit = MAX_LINK_RATE * 250 ms % Dual buffer size 4: k’ = 1 % Coupling factor as a power of 2 5: tshift = 50 ms % Time shift of TS-FIFO scheduler 6: % Constants derived from Classic AQM parameters 7: k = 2^k’ % Coupling factor from Equation (1) 6: 7: % Classic AQM parameters 8: g_C = 5 % EWMA smoothing parameter as a power of 1/2 9: S_C = -1 % Classic ramp scaling factor as a power of 2 10: minTh = 500 ms % No Classic drop/mark below this queue delay 11: % Constants derived from Classic AQM parameters 12: gamma = 2^(-g_C) % EWMA smoothing parameter 13: range_C = 2^S_C % Range of Classic ramp 14: 15: % L4S AQM parameters 16: T = 1 ms % Queue delay threshold for native L4S AQM 17: % Constants derived from above parameters 18: S_L = S_C - k’ % L4S ramp scaling factor as a power of 2 19: range_L = 2^S_L % Range of L4S ramp 20: }

Figure 9: Example Header Pseudocode for DualQ Coupled Curvy RED AQM

De Schepper, et al. Expires January 7, 2022 [Page 48] Internet-Draft DualQ Coupled AQMs July 2021

1: cred_dequeue(lq, cq, pkt) { % Couples L4S & Classic queues 2: while ( lq.len() + cq.len() > 0 ) { 3: if ( scheduler() == lq ) { 4: lq.dequeue(pkt) % L4S scheduled 5a: p_CL = (Q_C - minTh) / range_L 5b: if ( ( lq.time() > T ) 5c: OR ( p_CL > maxrand(U) ) ) 6: mark(pkt) 7: } else { 8: cq.dequeue(pkt) % Classic scheduled 9a: Q_C = gamma * cq.time() + (1-gamma) * Q_C % Classic Q EWMA 10a: sqrt_p_C = (Q_C - minTh) / range_C 10b: if ( sqrt_p_C > maxrand(2*U) ) { 11: if ( (ecn(pkt) == 0) { % ECN field = not-ECT 12: drop(pkt) % Squared drop, redo loop 13: continue % continue to the top of the while loop 14: } 15: mark(pkt) 16: } 17: } 18: return(pkt) % return the packet and stop here 19: } 20: return(NULL) % no packet to dequeue 21: }

22: maxrand(u) { % return the max of u random numbers 23: maxr=0 24: while (u-- > 0) 25: maxr = max(maxr, rand()) % 0 <= rand() < 1 26: return(maxr) 27: }

28: scheduler() { 29: if ( lq.time() + tshift >= cq.time() ) 30: return lq; 31: else 32: return cq; 33: }

Figure 10: Example Dequeue Pseudocode for DualQ Coupled Curvy RED AQM

The dequeue pseudocode (Figure 10) is repeatedly called whenever the lower layer is ready to forward a packet. It schedules one packet for dequeuing (or zero if the queue is empty) then returns control to the caller, so that it does not block while that packet is being forwarded. While making this dequeue decision, it also makes the necessary AQM decisions on dropping or marking. The alternative of applying the AQMs at enqueue would shift some processing from the

De Schepper, et al. Expires January 7, 2022 [Page 49] Internet-Draft DualQ Coupled AQMs July 2021

critical time when each packet is dequeued. However, it would also add a whole queue of delay to the control signals, making the control loop very sloppy.

The code is written assuming the AQMs are applied on dequeue (Note 1). All the dequeue code is contained within a large while loop so that if it decides to drop a packet, it will continue until it selects a packet to schedule. If both queues are empty, the routine returns NULL at line 20. Line 3 of the dequeue pseudocode is where the conditional priority scheduler chooses between the L4S queue (lq) and the Classic queue (cq). The time-shifted FIFO scheduler is shown at lines 28-33, which would be suitable if simplicity is paramount (see Note 2).

Within each queue, the decision whether to forward, drop or mark is taken as follows (to simplify the explanation, it is assumed that U=1):

L4S: If the test at line 3 determines there is an L4S packet to dequeue, the tests at lines 5b and 5c determine whether to mark it. The first is a simple test of whether the L4S queue delay (lq.time()) is greater than a step threshold T (Note 3). The second test is similar to the random ECN marking in RED, but with the following differences: i) marking depends on queuing time, not bytes, in order to scale for any link rate without being reconfigured; ii) marking of the L4S queue depends on a logical OR of two tests; one against its own queuing time and one against the queuing time of the _other_ (Classic) queue; iii) the tests are against the instantaneous queuing time of the L4S queue, but a smoothed average of the other (Classic) queue; iv) the queue is compared with the maximum of U random numbers (but if U=1, this is the same as the single random number used in RED).

Specifically, in line 5a the coupled marking probability p_CL is set to the amount by which the averaged Classic queueing delay Q_C exceeds the minimum queuing delay threshold (minTh) all divided by the L4S scaling parameter range_L. range_L represents the queuing delay (in seconds) added to minTh at which marking probability would hit 100%. Then in line 5c (if U=1) the result is compared with a uniformly distributed random number between 0 and 1, which ensures that, over range_L, marking probability will linearly increase with queueing time.

Classic: If the scheduler at line 3 chooses to dequeue a Classic packet and jumps to line 7, the test at line 10b determines whether to drop or mark it. But before that, line 9a updates Q_C, which is an exponentially weighted moving average (Note 4) of the queuing time of the Classic queue, where cq.time() is the current

De Schepper, et al. Expires January 7, 2022 [Page 50] Internet-Draft DualQ Coupled AQMs July 2021

instantaneous queueing time of the packet at the head of the Classic queue (zero if empty) and gamma is the EWMA constant (default 1/32, see line 12 of the initialization function).

Lines 10a and 10b implement the Classic AQM. In line 10a the averaged queuing time Q_C is divided by the Classic scaling parameter range_C, in the same way that queuing time was scaled for L4S marking. This scaled queuing time will be squared to compute Classic drop probability so, before it is squared, it is effectively the square root of the drop probability, hence it is given the variable name sqrt_p_C. The squaring is done by comparing it with the maximum out of two random numbers (assuming U=1). Comparing it with the maximum out of two is the same as the logical ‘AND’ of two tests, which ensures drop probability rises with the square of queuing time.

The AQM functions in each queue (lines 5c & 10b) are two cases of a new generalization of RED called Curvy RED, motivated as follows. When the performance of this AQM was compared with FQ-CoDel and PIE, their goal of holding queuing delay to a fixed target seemed misguided [CRED_Insights]. As the number of flows increases, if the AQM does not allow host congestion controllers to increase queuing delay, it has to introduce abnormally high levels of loss. Then loss rather than queuing becomes the dominant cause of delay for short flows, due to timeouts and tail losses.

Curvy RED constrains delay with a softened target that allows some increase in delay as load increases. This is achieved by increasing drop probability on a convex curve relative to queue growth (the square curve in the Classic queue, if U=1). Like RED, the curve hugs the zero axis while the queue is shallow. Then, as load increases, it introduces a growing barrier to higher delay. But, unlike RED, it requires only two parameters, not three. The disadvantage of Curvy RED (compared to a PI controller for example) is that it is not adapted to a wide range of RTTs. Curvy RED can be used as is when the RTT range to be supported is limited, otherwise an adaptation mechanism is required.

From our limited experiments with Curvy RED so far, recommended values of these parameters are: S_C = -1; g_C = 5; T = 5 * MTU at the link rate (about 1ms at 60Mb/s) for the range of base RTTs typical on the public Internet. [CRED_Insights] explains why these parameters are applicable whatever rate link this AQM implementation is deployed on and how the parameters would need to be adjusted for a scenario with a different range of RTTs (e.g. a data centre). The setting of k depends on policy (see Section 2.5 and Appendix C.2 respectively for its recommended setting and guidance on alternatives).

De Schepper, et al. Expires January 7, 2022 [Page 51] Internet-Draft DualQ Coupled AQMs July 2021

There is also a cUrviness parameter, U, which is a small positive integer. It is likely to take the same hard-coded value for all implementations, once experiments have determined a good value. Only U=1 has been used in experiments so far, but results might be even better with U=2 or higher.

Notes:

1. The alternative of applying the AQMs at enqueue would shift some processing from the critical time when each packet is dequeued. However, it would also add a whole queue of delay to the control signals, making the control loop sloppier (for a typical RTT it would double the Classic queue’s feedback delay). On a platform where packet timestamping is feasible, e.g. Linux, it is also easiest to apply the AQMs at dequeue because that is where queuing time is also measured.

2. WRR better isolates the L4S queue from large delay bursts in the Classic queue, but it is slightly less simple than TS-FIFO. If WRR were used, a low default Classic weight (e.g. 1/16) would need to be configured in place of the time shift in line 5 of the initialization function (Figure 9).

3. A step function is shown for simplicity. A ramp function (see Figure 5 and the discussion around it in Appendix A.1) is recommended, because it is more general than a step and has the potential to enable L4S congestion controls to converge more rapidly.

4. An EWMA is only one possible way to filter bursts; other more adaptive smoothing methods could be valid and it might be appropriate to decrease the EWMA faster than it increases, e.g. by using the minimum of the smoothed and instantaneous queue delays, min(Q_C, qc.time()).

B.2. Efficient Implementation of Curvy RED

Although code optimization depends on the platform, the following notes explain where the design of Curvy RED was particularly motivated by efficient implementation.

The Classic AQM at line 10b calls maxrand(2*U), which gives twice as much curviness as the call to maxrand(U) in the marking function at line 5c. This is the trick that implements the square rule in equation (1) (Section 2.1). This is based on the fact that, given a number X from 1 to 6, the probability that two dice throws will both be less than X is the square of the probability that one throw will be less than X. So, when U=1, the L4S marking function is linear and

De Schepper, et al. Expires January 7, 2022 [Page 52] Internet-Draft DualQ Coupled AQMs July 2021

the Classic dropping function is squared. If U=2, L4S would be a square function and Classic would be quartic. And so on.

The maxrand(u) function in lines 16-21 simply generates u random numbers and returns the maximum. Typically, maxrand(u) could be run in parallel out of band. For instance, if U=1, the Classic queue would require the maximum of two random numbers. So, instead of calling maxrand(2*U) in-band, the maximum of every pair of values from a pseudorandom number generator could be generated out-of-band, and held in a buffer ready for the Classic queue to consume.

1: cred_dequeue(lq, cq, pkt) { % Couples L4S & Classic queues 2: while ( lq.len() + cq.len() > 0 ) { 3: if ( scheduler() == lq ) { 4: lq.dequeue(pkt) % L4S scheduled 5: if ((lq.time() > T) OR (Q_C >> (S_L-2) > maxrand(U))) 6: mark(pkt) 7: } else { 8: cq.dequeue(pkt) % Classic scheduled 9: Q_C += (qc.ns() - Q_C) >> g_C % Classic Q EWMA 10: if ( (Q_C >> (S_C-2) ) > maxrand(2*U) ) { 11: if ( (ecn(pkt) == 0) { % ECN field = not-ECT 12: drop(pkt) % Squared drop, redo loop 13: continue % continue to the top of the while loop 14: } 15: mark(pkt) 16: } 17: } 18: return(pkt) % return the packet and stop here 19: } 20: return(NULL) % no packet to dequeue 21: }

Figure 11: Optimised Example Dequeue Pseudocode for Coupled DualQ AQM using Integer Arithmetic

The two ranges, range_L and range_C are expressed as powers of 2 so that division can be implemented as a right bit-shift (>>) in lines 5 and 10 of the integer variant of the pseudocode (Figure 11).

For the integer variant of the pseudocode, an integer version of the rand() function used at line 25 of the maxrand(function) in Figure 10 would be arranged to return an integer in the range 0 <= maxrand() < 2^32 (not shown). This would scale up all the floating point probabilities in the range [0,1] by 2^32.

Queuing delays are also scaled up by 2^32, but in two stages: i) In line 9 queuing time qc.ns() is returned in integer nanoseconds,

De Schepper, et al. Expires January 7, 2022 [Page 53] Internet-Draft DualQ Coupled AQMs July 2021

making the value about 2^30 times larger than when the units were seconds, ii) then in lines 5 and 10 an adjustment of -2 to the right bit-shift multiplies the result by 2^2, to complete the scaling by 2^32.

In line 8 of the initialization function, the EWMA constant gamma is represented as an integer power of 2, g_C, so that in line 9 of the integer code the division needed to weight the moving average can be implemented by a right bit-shift (>> g_C).

Appendix C. Choice of Coupling Factor, k

C.1. RTT-Dependence

Where Classic flows compete for the same capacity, their relative flow rates depend not only on the congestion probability, but also on their end-to-end RTT (= base RTT + queue delay). The rates of competing Reno [RFC5681] flows are roughly inversely proportional to their RTTs. Cubic exhibits similar RTT-dependence when in Reno- compatibility mode, but is less RTT-dependent otherwise.

Until the early experiments with the DualQ Coupled AQM, the importance of the reasonably large Classic queue in mitigating RTT- dependence had not been appreciated. Appendix A.1.6 of [I-D.ietf-tsvwg-ecn-l4s-id] uses numerical examples to explain why bloated buffers had concealed the RTT-dependence of Classic congestion controls before that time. Then it explains why, the more that queuing delays have reduced, the more that RTT-dependence has surfaced as a potential starvation problem for long RTT flows.

Given that congestion control on end-systems is voluntary, there is no reason why it has to be voluntarily RTT-dependent. Therefore [I-D.ietf-tsvwg-ecn-l4s-id] requires L4S congestion controls to be significantly less RTT-dependent than the standard Reno congestion control [RFC5681]. Following this approach means there is no need for network devices to address RTT-dependence, although there would be no harm if they did, which per-flow queuing inherently does.

At the time of writing, the range of approaches to RTT-dependence in L4S congestion controls has not settled. Therefore, the guidance on the choice of the coupling factor in Appendix C.2 is given against DCTCP [RFC8257], which has well-understood RTT-dependence. The guidance is given for various RTT ratios, so that it can be adapted to future circumstances.

De Schepper, et al. Expires January 7, 2022 [Page 54] Internet-Draft DualQ Coupled AQMs July 2021

C.2. Guidance on Controlling Throughput Equivalence

+------+------+------+ | RTT_C / RTT_L | Reno | Cubic | +------+------+------+ | 1 | k’=1 | k’=0 | | 2 | k’=2 | k’=1 | | 3 | k’=2 | k’=2 | | 4 | k’=3 | k’=2 | | 5 | k’=3 | k’=3 | +------+------+------+

Table 1: Value of k’ for which DCTCP throughput is roughly the same as Reno or Cubic, for some example RTT ratios

In the above appendices that give example DualQ Coupled algorithms, to aid efficient implementation, a coupling factor that is an integer power of 2 is always used. k’ is always used to denote the power. k’ is related to the coupling factor k in Equation (1) (Section 2.1) by k=2^k’.

To determine the appropriate coupling factor policy, the operator first has to judge whether it wants DCTCP flows to have roughly equal throughput with Reno or with Cubic (because, even in its Reno- compatibility mode, Cubic is about 1.4 times more aggressive than Reno). Then the operator needs to decide at what ratio of RTTs it wants DCTCP and Classic flows to have roughly equal throughput. For example choosing k’=0 (equivalent to k=1) will make DCTCP throughput roughly the same as Cubic, _if their RTTs are the same_.

However, even if the base RTTs are the same, the actual RTTs are unlikely to be the same, because Classic (Cubic or Reno) traffic needs roughly a typical base round trip of queue to avoid under- utilization and excess drop. Whereas L4S (DCTCP) does not. The operator might still choose this policy if it judges that DCTCP throughput should be rewarded for keeping its own queue short.

On the other hand, the operator will choose one of the higher values for k’, if it wants to slow DCTCP down to roughly the same throughput as Classic flows, to compensate for Classic flows slowing themselves down by causing themselves extra queuing delay.

The values for k’ in the table are derived from the formulae below, which were developed in [DCttH15]:

2^k’ = 1.64 (RTT_reno / RTT_dc) (5) 2^k’ = 1.19 (RTT_cubic / RTT_dc ) (6)

De Schepper, et al. Expires January 7, 2022 [Page 55] Internet-Draft DualQ Coupled AQMs July 2021

For localized traffic from a particular ISP’s data centre, using the measured RTTs, it was calculated that a value of k’=3 (equivalent to k=8) would achieve throughput equivalence, and experiments verified the formula very closely.

For a typical mix of RTTs from local data centres and across the general Internet, a value of k’=1 (equivalent to k=2) is recommended as a good workable compromise.

Authors’ Addresses

Koen De Schepper Nokia Bell Labs Antwerp Belgium

Email: [email protected] URI: https://www.bell-labs.com/usr/koen.de_schepper

Bob Briscoe (editor) Independent UK

Email: [email protected] URI: http://bobbriscoe.net/

Greg White CableLabs Louisville, CO US

Email: [email protected]

De Schepper, et al. Expires January 7, 2022 [Page 56] Internet Engineering Task Force G. Fairhurst Internet-Draft T. Jones Updates: 4821, 4960, 6951, 8085, 8261 (if University of Aberdeen approved) M. Tuexen Intended status: Standards Track I. Ruengeler Expires: 12 December 2020 T. Voelker Muenster University of Applied Sciences 10 June 2020

Packetization Layer Path MTU Discovery for Datagram Transports draft-ietf-tsvwg-datagram-plpmtud-22

Abstract

This document describes a robust method for Path MTU Discovery (PMTUD) for datagram Packetization Layers (PLs). It describes an extension to RFC 1191 and RFC 8201, which specifies ICMP-based Path MTU Discovery for IPv4 and IPv6. The method allows a PL, or a datagram application that uses a PL, to discover whether a network path can support the current size of datagram. This can be used to detect and reduce the message size when a sender encounters a packet black hole (where packets are discarded). The method can probe a network path with progressively larger packets to discover whether the maximum packet size can be increased. This allows a sender to determine an appropriate packet size, providing functionality for datagram transports that is equivalent to the Packetization Layer PMTUD specification for TCP, specified in RFC 4821.

This document updates RFC 4821 to specify the PLPMTUD method for datagram PLs. It also updates RFC 8085 to refer to the method specified in this document instead of the method in RFC 4821 for use with UDP datagrams. Section 7.3 of RFC 4960 recommends an endpoint apply the techniques in RFC 4821 on a per-destination-address basis. RFC 4960, RFC 6951, and RFC 8261 are updated to recommend that SCTP, SCTP encapsulated in UDP and SCTP encapsulated in DTLS use the method specified in this document instead of the method in RFC 4821.

The document also provides implementation notes for incorporating Datagram PMTUD into IETF datagram transports or applications that use datagram transports.

When published, this specification updates RFC 4960, RFC 4821, RFC 8085 and RFC 8261.

Fairhurst, et al. Expires 12 December 2020 [Page 1] Internet-Draft DPLPMTUD June 2020

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

This Internet-Draft will expire on 12 December 2020.

Copyright Notice

This document is subject to BCP 78 and the IETF Trust’s Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.

Table of Contents

1. Introduction ...... 4 1.1. Classical Path MTU Discovery ...... 4 1.2. Packetization Layer Path MTU Discovery ...... 6 1.3. Path MTU Discovery for Datagram Services ...... 7 2. Terminology ...... 8 3. Features Required to Provide Datagram PLPMTUD ...... 11 4. DPLPMTUD Mechanisms ...... 14 4.1. PLPMTU Probe Packets ...... 14 4.2. Confirmation of Probed Packet Size ...... 15 4.3. Black Hole Detection and Reducing the PLPMTU ...... 15 4.4. The Maximum Packet Size (MPS) ...... 17 4.5. Disabling the Effect of PMTUD ...... 18 4.6. Response to PTB Messages ...... 18 4.6.1. Validation of PTB Messages ...... 18 4.6.2. Use of PTB Messages ...... 19

Fairhurst, et al. Expires 12 December 2020 [Page 2] Internet-Draft DPLPMTUD June 2020

5. Datagram Packetization Layer PMTUD ...... 20 5.1. DPLPMTUD Components ...... 21 5.1.1. Timers ...... 21 5.1.2. Constants ...... 22 5.1.3. Variables ...... 23 5.1.4. Overview of DPLPMTUD Phases ...... 24 5.2. State Machine ...... 26 5.3. Search to Increase the PLPMTU ...... 29 5.3.1. Probing for a larger PLPMTU ...... 29 5.3.2. Selection of Probe Sizes ...... 30 5.3.3. Resilience to Inconsistent Path Information . . . . . 30 5.4. Robustness to Inconsistent Paths ...... 31 6. Specification of Protocol-Specific Methods ...... 31 6.1. Application support for DPLPMTUD with UDP or UDP-Lite . . 31 6.1.1. Application Request ...... 32 6.1.2. Application Response ...... 32 6.1.3. Sending Application Probe Packets ...... 32 6.1.4. Initial Connectivity ...... 32 6.1.5. Validating the Path ...... 32 6.1.6. Handling of PTB Messages ...... 32 6.2. DPLPMTUD for SCTP ...... 33 6.2.1. SCTP/IPv4 and SCTP/IPv6 ...... 33 6.2.1.1. Initial Connectivity ...... 33 6.2.1.2. Sending SCTP Probe Packets ...... 33 6.2.1.3. Validating the Path with SCTP ...... 34 6.2.1.4. PTB Message Handling by SCTP ...... 34 6.2.2. DPLPMTUD for SCTP/UDP ...... 34 6.2.2.1. Initial Connectivity ...... 35 6.2.2.2. Sending SCTP/UDP Probe Packets ...... 35 6.2.2.3. Validating the Path with SCTP/UDP ...... 35 6.2.2.4. Handling of PTB Messages by SCTP/UDP ...... 35 6.2.3. DPLPMTUD for SCTP/DTLS ...... 35 6.2.3.1. Initial Connectivity ...... 35 6.2.3.2. Sending SCTP/DTLS Probe Packets ...... 36 6.2.3.3. Validating the Path with SCTP/DTLS ...... 36 6.2.3.4. Handling of PTB Messages by SCTP/DTLS ...... 36 6.3. DPLPMTUD for QUIC ...... 36 7. Acknowledgments ...... 36 8. IANA Considerations ...... 36 9. Security Considerations ...... 37 10. References ...... 38 10.1. Normative References ...... 38 10.2. Informative References ...... 39 Appendix A. Revision Notes ...... 41 Authors’ Addresses ...... 46

Fairhurst, et al. Expires 12 December 2020 [Page 3] Internet-Draft DPLPMTUD June 2020

1. Introduction

The IETF has specified datagram transport using UDP, SCTP, and DCCP, as well as protocols layered on top of these transports (e.g., SCTP/ UDP, DCCP/UDP, QUIC/UDP), and direct datagram transport over the IP network layer. This document describes a robust method for Path MTU Discovery (PMTUD) that can be used with these transport protocols (or the applications that use their transport service) to discover an appropriate size of packet to use across an Internet path.

1.1. Classical Path MTU Discovery

Classical Path Maximum Transmission Unit Discovery (PMTUD) can be used with any transport that is able to process ICMP Packet Too Big (PTB) messages (e.g., [RFC1191] and [RFC8201]). In this document, the term PTB message is applied to both IPv4 ICMP Unreachable messages (type 3) that carry the error Fragmentation Needed (Type 3, Code 4) [RFC0792] and ICMPv6 Packet Too Big messages (Type 2) [RFC4443]. When a sender receives a PTB message, it reduces the effective MTU to the value reported as the Link MTU in the PTB message. A method from time-to-time increases the packet size in attempt to discover an increase in the supported PMTU. The packets sent with a size larger than the current effective PMTU are known as probe packets.

Packets not intended as probe packets are either fragmented to the current effective PMTU, or the attempt to send fails with an error code. Applications can be provided with a primitive to let them read the Maximum Packet Size (MPS), derived from the current effective PMTU.

Classical PMTUD is subject to protocol failures. One failure arises when traffic using a packet size larger than the actual PMTU is black-holed (all datagrams larger than the actual PMTU, are discarded). This could arise when the PTB messages are not delivered back to the sender for some reason (see for example [RFC2923]).

Examples where PTB messages are not delivered include:

* The generation of ICMP messages is usually rate limited. This could result in no PTB messages being generated to the sender (see section 2.4 of [RFC4443])

* ICMP messages can be filtered by middleboxes (including firewalls) [RFC4890]. A firewall could be configured with a policy to block incoming ICMP messages, which would prevent reception of PTB messages to a sending endpoint behind this firewall.

Fairhurst, et al. Expires 12 December 2020 [Page 4] Internet-Draft DPLPMTUD June 2020

* When the router issuing the ICMP message drops a tunneled packet, the resulting ICMP message will be directed to the tunnel ingress. This tunnel endpoint is responsible for forwarding the ICMP message and also processing the quoted packet within the payload field to remove the effect of the tunnel, and return a correctly formatted ICMP message to the sender [I-D.ietf-intarea-tunnels]. Failure to do this prevents the PTB message reaching the original sender.

* Asymmetry in forwarding can result in there being no return route to the original sender, which would prevent an ICMP message being delivered to the sender. This issue can also arise when policy- based routing is used, Equal Cost Multipath (ECMP) routing is used, or a middlebox acts as an application load balancer. An example is where the path towards the server is chosen by ECMP routing depending on bytes in the IP payload. In this case, when a packet sent by the server encounters a problem after the ECMP router, then any resulting ICMP message also needs to be directed by the ECMP router towards the original sender.

* There are additional cases where the next hop destination fails to receive a packet because of its size. This could be due to misconfiguration of the layer 2 path between nodes, for instance the MTU configured in a layer 2 switch, or misconfiguration of the Maximum Receive Unit (MRU). If a packet is dropped by the link, this will not cause a PTB message to be sent to the original sender.

Another failure could result if a node that is not on the network path sends a PTB message that attempts to force a sender to change the effective PMTU [RFC8201]. A sender can protect itself from reacting to such messages by utilizing the quoted packet within a PTB message payload to validate that the received PTB message was generated in response to a packet that had actually originated from the sender. However, there are situations where a sender would be unable to provide this validation. Examples where validation of the PTB message is not possible include:

* When a router issuing the ICMP message implements RFC792 [RFC0792], it is only required to include the first 64 bits of the IP payload of the packet within the quoted payload. There could be insufficient bytes remaining for the sender to interpret the quoted transport information.

Note: The recommendation in RFC1812 [RFC1812] is that IPv4 routers return a quoted packet with as much of the original datagram as possible without the length of the ICMP datagram exceeding 576

Fairhurst, et al. Expires 12 December 2020 [Page 5] Internet-Draft DPLPMTUD June 2020

bytes. IPv6 routers include as much of the invoking packet as possible without the ICMPv6 packet exceeding 1280 bytes [RFC4443].

* The use of tunnels/encryption can reduce the size of the quoted packet returned to the original source address, increasing the risk that there could be insufficient bytes remaining for the sender to interpret the quoted transport information.

* Even when the PTB message includes sufficient bytes of the quoted packet, the network layer could lack sufficient context to validate the message, because validation depends on information about the active transport flows at an endpoint node (e.g., the socket/address pairs being used, and other protocol header information).

* When a packet is encapsulated/tunneled over an encrypted transport, the tunnel/encapsulation ingress might have insufficient context, or computational power, to reconstruct the transport header that would be needed to perform validation.

* When an ICMP message is generated by a router in a network segment that has inserted a header into a packet, the quoted packet could contain additional protocol header information that was not included in the original sent packet, and which the PL sender does not process or may not know how to process. This could disrupt the ability of the sender to validate this PTB message.

* A Network Address Translation (NAT) device that translates a packet header, ought to also translate ICMP messages and update the ICMP quoted packet [RFC5508] in that message. If this is not correctly translated then the sender would not be able to associate the message with the PL that originated the packet, and hence this ICMP message cannot be validated.

1.2. Packetization Layer Path MTU Discovery

The term Packetization Layer (PL) has been introduced to describe the layer that is responsible for placing data blocks into the payload of IP packets and selecting an appropriate MPS. This function is often performed by a transport protocol (e.g., DCCP, RTP, SCTP, QUIC), but can also be performed by other encapsulation methods working above the transport layer.

In contrast to PMTUD, Packetization Layer Path MTU Discovery (PLPMTUD) [RFC4821] introduced a method that does not rely upon reception and validation of PTB messages. It is therefore more robust than Classical PMTUD. This has become the recommended approach for implementing discovery of the PMTU [BCP145].

Fairhurst, et al. Expires 12 December 2020 [Page 6] Internet-Draft DPLPMTUD June 2020

It uses a general strategy where the PL sends probe packets to search for the largest size of unfragmented datagram that can be sent over a network path. Probe packets are sent to explore using a larger packet size. If a probe packet is successfully delivered (as determined by the PL), then the PLPMTU is raised to the size of the successful probe. If a black hole is detected (e.g., where packets of size PLPMTU are consistently not received), the method reduces the PLPMTU.

Datagram PLPMTUD introduces flexibility in implementation. At one extreme, it can be configured to only perform Black Hole Detection and recovery with increased robustness compared to Classical PMTUD. At the other extreme, all PTB processing can be disabled, and PLPMTUD replaces Classical PMTUD.

PLPMTUD can also include additional consistency checks without increasing the risk that data is lost when probing to discover the Path MTU. For example, information available at the PL, or higher layers, enables received PTB messages to be validated before being utilized.

1.3. Path MTU Discovery for Datagram Services

Section 5 of this document presents a set of algorithms for datagram protocols to discover the largest size of unfragmented datagram that can be sent over a network path. The method relies upon features of the PL described in Section 3 and applies to transport protocols operating over IPv4 and IPv6. It does not require cooperation from the lower layers, although it can utilize PTB messages when these received messages are made available to the PL.

The message size guidelines in section 3.2 of the UDP Usage Guidelines [BCP145] state "an application SHOULD either use the Path MTU information provided by the IP layer or implement Path MTU Discovery (PMTUD)", but does not provide a mechanism for discovering the largest size of unfragmented datagram that can be used on a network path. The present document updates RFC 8085 to specify this method in place of PLPMTUD [RFC4821] and provides a mechanism for sharing the discovered largest size as the MPS (see Section 4.4).

Section 10.2 of [RFC4821] recommended a PLPMTUD probing method for the Stream Control Transport Protocol (SCTP). SCTP utilizes probe packets consisting of a minimal sized HEARTBEAT chunk bundled with a PAD chunk as defined in [RFC4820]. However, RFC 4821 did not provide a complete specification. The present document replaces that description by providing a complete specification.

Fairhurst, et al. Expires 12 December 2020 [Page 7] Internet-Draft DPLPMTUD June 2020

The Datagram Congestion Control Protocol (DCCP) [RFC4340] requires implementations to support Classical PMTUD and states that a DCCP sender "MUST maintain the MPS allowed for each active DCCP session". It also defines the current congestion control MPS (CCMPS) supported by a network path. This recommends use of PMTUD, and suggests use of control packets (DCCP-Sync) as path probe packets, because they do not risk application data loss. The method defined in this specification can be used with DCCP.

Section 4 and Section 5 define the protocol mechanisms and specification for Datagram Packetization Layer Path MTU Discovery (DPLPMTUD).

Section 6 specifies the method for datagram transports and provides information to enable the implementation of PLPMTUD with other datagram transports and applications that use datagram transports.

Section 6 also provides updated recommendations for [RFC6951] and [RFC8261].

2. Terminology

The following terminology is defined. Relevant terms are directly copied from [RFC4821], and the definitions in [RFC1122].

Acknowledged PL: A PL that includes a mechanism that can confirm successful delivery of datagrams to the remote PL endpoint (e.g., SCTP). Typically, the PL receiver returns acknowledgments corresponding to the received datagrams, which can be utilised to detect black-holing of packets (c.f., Unacknowledged PL).

Actual PMTU: The Actual PMTU is the PMTU of a network path between a sender PL and a destination PL, which the DPLPMTUD algorithm seeks to determine.

Black Hole: A Black Hole is encountered when a sender is unaware that packets are not being delivered to the destination end point. Two types of Black Hole are relevant to DPLPMTUD:

* Packets encounter a packet Black Hole when packets are not delivered to the destination endpoint (e.g., when the sender transmits packets of a particular size with a previously known

Fairhurst, et al. Expires 12 December 2020 [Page 8] Internet-Draft DPLPMTUD June 2020

effective PMTU and they are discarded by the network).

* An ICMP Black Hole is encountered when the sender is unaware that packets are not delivered to the destination endpoint because PTB messages are not received by the originating PL sender.

Classical Path MTU Discovery: Classical PMTUD is a process described in [RFC1191] and [RFC8201], in which nodes rely on PTB messages to learn the largest size of unfragmented packet that can be used across a network path.

Datagram: A datagram is a transport-layer protocol data unit, transmitted in the payload of an IP packet.

Effective PMTU: The Effective PMTU is the current estimated value for PMTU that is used by a PMTUD. This is equivalent to the PLPMTU derived by PLPMTUD plus the size of any headers added below the PL, including the IP layer headers.

EMTU_S: The Effective MTU for sending (EMTU_S) is defined in [RFC1122] as "the maximum IP datagram size that may be sent, for a particular combination of IP source and destination addresses...".

EMTU_R: The Effective MTU for receiving (EMTU_R) is designated in [RFC1122] as "the largest datagram size that can be reassembled".

Link: A Link is a communication facility or medium over which nodes can communicate at the link layer, i.e., a layer below the IP layer. Examples are Ethernet LANs and Internet (or higher) layer tunnels.

Link MTU: The Link Maximum Transmission Unit (MTU) is the size in bytes of the largest IP packet, including the IP header and payload, that can be transmitted over a link. Note that this could more properly be called the IP MTU, to be consistent with how other standards organizations use the acronym. This includes the IP header, but excludes link layer headers and other framing that is not part of IP or the IP payload. Other standards organizations generally define the link MTU to include the link layer headers. This specification continues the requirement in [RFC4821], that states "All links MUST enforce their MTU: links that might non- deterministically deliver packets that are larger than their rated MTU MUST consistently discard such packets."

MAX_PLPMTU: The MAX_PLPMTU is the largest size of PLPMTU that DPLPMTUD will attempt to use (see the constants defined in Section 5.1.2).

Fairhurst, et al. Expires 12 December 2020 [Page 9] Internet-Draft DPLPMTUD June 2020

MIN_PLPMTU: The MIN_PLPMTU is the smallest size of PLPMTU that DPLPMTUD will attempt to use (see the constants defined in Section 5.1.2).

MPS: The Maximum Packet Size (MPS) is the largest size of application data block that can be sent across a network path by a PL using a single Datagram (see Section 4.4).

MSL: Maximum Segment Lifetime (MSL) The maximum delay a packet is expected to experience across a path, taken as 2 minutes [BCP145].

Packet: A Packet is the IP header(s) and any extension headers/ options plus the IP payload.

Packetization Layer (PL): The PL is a layer of the network stack that places data into packets and performs transport protocol functions. Examples of a PL include: TCP, SCTP, SCTP over UDP, SCTP over DTLS, or QUIC.

Path: The Path is the set of links and routers traversed by a packet between a source node and a destination node by a particular flow.

Path MTU (PMTU): The Path MTU (PMTU) is the minimum of the Link MTU of all the links forming a network path between a source node and a destination node, as used by PMTUD.

PTB: In this document, the term PTB message is applied to both IPv4 ICMP Unreachable messages (type 3) that carry the error Fragmentation Needed (Type 3, Code 4) [RFC0792] and ICMPv6 Packet Too Big messages (Type 2) [RFC4443].

PTB_SIZE: The PTB_SIZE is a value reported in a validated PTB message that indicates next hop link MTU of a router along the path.

PL_PTB_SIZE: The size reported in a validated PTB message, reduced by the size of all headers added by layers below the PL.

PLPMTU: The Packetization Layer PMTU is an estimate of the largest size of PL datagram that can be sent by a path, controled by PLPMTUD.

PLPMTUD: Packetization Layer Path MTU Discovery (PLPMTUD), the method described in this document for datagram PLs, which is an extension to Classical PMTU Discovery.

Probe packet: A probe packet is a datagram sent with a purposely chosen size (typically the current PLPMTU or larger) to detect if

Fairhurst, et al. Expires 12 December 2020 [Page 10] Internet-Draft DPLPMTUD June 2020

packets of this size can be successfully sent end-to-end across the network path.

Unacknowledged PL: A PL that does not itself provide a mechanism to confirm delivery of datagrams to the remote PL endpoint (e.g., UDP), and therefore requires DPLPMTUD to provide a mechanism to detect black-holing of packets (c.f., Acknowledged PL).

3. Features Required to Provide Datagram PLPMTUD

The principles expressed in [RFC4821] apply to the use of the technique with any PL. TCP PLPMTUD has been defined using standard TCP protocol mechanisms. Unlike TCP, a datagram PL requires additional mechanisms and considerations to implement PLPMTUD.

The requirements for datagram PLPMTUD are:

1. Managing the PLPMTU: For datagram PLs, the PLPMTU is managed by DPLPMTUD. A PL MUST NOT send a datagram (other than a probe packet) with a size at the PL that is larger than the current PLPMTU.

2. Probe packets: The network interface below PL is REQUIRED to provide a way to transmit a probe packet that is larger than the PLPMTU. In IPv4, a probe packet MUST be sent with the Don’t Fragment (DF) bit set in the IP header, and without network layer endpoint fragmentation. In IPv6, a probe packet is always sent without source fragmentation (as specified in section 5.4 of [RFC8201]).

3. Reception feedback: The destination PL endpoint is REQUIRED to provide a feedback method that indicates to the DPLPMTUD sender when a probe packet has been received by the destination PL endpoint. Section 6 provides examples of how a PL can provide this acknowledgment of received probe packets.

4. Probe loss recovery: It is RECOMMENDED to use probe packets that do not carry any user data that would require retransmission if lost. Most datagram transports permit this. If a probe packet contains user data requiring retransmission in case of loss, the PL (or layers above) are REQUIRED to arrange any retransmission/ repair of any resulting loss. The PL is REQUIRED to be robust in the case where probe packets are lost due to other reasons (including link transmission error, congestion).

5. PMTU parameters: A DPLPMTUD sender is RECOMMENDED to utilize information about the maximum size of packet that can be transmitted by the sender on the local link (e.g., the local Link

Fairhurst, et al. Expires 12 December 2020 [Page 11] Internet-Draft DPLPMTUD June 2020

MTU). A PL sender MAY utilize similar information about the maximum size of network layer packet that a receiver can accept when this is supplied (note this could be less than EMTU_R). This avoids implementations trying to send probe packets that can not be transferred by the local link. Too high of a value could reduce the efficiency of the search algorithm. Some applications also have a maximum transport protocol data unit (PDU) size, in which case there is no benefit from probing for a size larger than this (unless a transport allows multiplexing multiple applications PDUs into the same datagram).

6. Processing PTB messages: A DPLPMTUD sender MAY optionally utilize PTB messages received from the network layer to help identify when a network path does not support the current size of probe packet. Any received PTB message MUST be validated before it is used to update the PLPMTU discovery information [RFC8201]. This validation confirms that the PTB message was sent in response to a packet originating by the sender, and needs to be performed before the PLPMTU discovery method reacts to the PTB message. A PTB message MUST NOT be used to increase the PLPMTU [RFC8201], but could trigger a probe to test for a larger PLPMTU. A valid PTB_SIZE is converted to a PL_PTB_SIZE before it is to be used in the DPLPMTUD state machine. A PL_PTB_SIZE that is greater than that currently probed SHOULD be ignored. (This PTB message ought to be discarded without further processing, but could be utilized as an input that enables a resilience mode).

7. Probing and congestion control: A PL MAY use a congestion controller to decide when to send a probe packet. If transmission of probe packets is limited by the congestion controller, this could result in transmission of probe packets being delayed or suspended during congestion. When the transmission of probe packets is not controlled by the congestion controller, the interval between probe packets MUST be at least one RTT. Loss of a probe packet SHOULD NOT be treated as an indication of congestion and SHOULD NOT trigger a congestion control reaction [RFC4821], because this could result in unnecessary reduction of the sending rate. An update to the PLPMTU (or MPS) MUST NOT increase the congestion window measured in bytes [RFC4821]. Therefore, an increase in the packet size does not cause an increase in the data rate in bytes per second. A PL that maintains the congestion window in terms of a limit to the number of outstanding fixed size packets SHOULD adapt this limit to compensate for the size of the actual packets. The transmission of probe packets can interact with the operation of a PL that performs burst mitigation or pacing and could need transmission of probe packets to be regulated by these methods.

Fairhurst, et al. Expires 12 December 2020 [Page 12] Internet-Draft DPLPMTUD June 2020

8. Probing and flow control: Flow control at the PL concerns the end-to-end flow of data using the PL service. Flow control SHOULD NOT apply to DPLPMTU when probe packets use a design that does not carry user data to the remote application.

9. Shared PLPMTU state: The PMTU value calculated from the PLPMTU MAY also be stored with the corresponding entry associated with the destination in the IP layer cache, and used by other PL instances. The specification of PLPMTUD [RFC4821] states: "If PLPMTUD updates the MTU for a particular path, all Packetization Layer sessions that share the path representation (as described in Section 5.2 of [RFC4821]) SHOULD be notified to make use of the new MTU". Such methods MUST be robust to the wide variety of underlying network forwarding behaviors. Section 5.2 of [RFC8201] provides guidance on the caching of PMTU information and also the relation to IPv6 flow labels.

In addition, the following principles are stated for design of a DPLPMTUD method:

* A PL MAY be designed to segment data blocks larger than the MPS into multiple datagrams. However, not all datagram PLs support segmentation of data blocks. It is RECOMMENDED that methods avoid forcing an application to use an arbitrary small MPS for transmission while the method is searching for the currently supported PLPMTU. A reduced MPS can adversely impact the performance of an application.

* To assist applications in choosing a suitable data block size, the PL is RECOMMENDED to provide a primitive that returns the MPS derived from the PLPMTU to the higher layer using the PL. The value of the MPS can change following a change in the path, or loss of probe packets.

* Path validation: It is RECOMMENDED that methods are robust to path changes that could have occurred since the path characteristics were last confirmed, and to the possibility of inconsistent path information being received.

* Datagram reordering: A method is REQUIRED to be robust to the possibility that a flow encounters reordering, or the traffic (including probe packets) is divided over more than one network path.

* Datagram delay and duplication: The feedback mechanism is REQUIRED to be robust to the possibility that packets could be significantly delayed or duplicated along a network path.

Fairhurst, et al. Expires 12 December 2020 [Page 13] Internet-Draft DPLPMTUD June 2020

* When to probe: It is RECOMMENDED that methods determine whether the path has changed since it last measured the path. This can help determine when to probe the path again.

4. DPLPMTUD Mechanisms

This section lists the protocol mechanisms used in this specification.

4.1. PLPMTU Probe Packets

The DPLPMTUD method relies upon the PL sender being able to generate probe packets with a specific size. TCP is able to generate these probe packets by choosing to appropriately segment data being sent [RFC4821]. In contrast, a datagram PL that constructs a probe packet has to either request an application to send a data block that is larger than that generated by an application, or to utilize padding functions to extend a datagram beyond the size of the application data block. Protocols that permit exchange of control messages (without an application data block) can generate a probe packet by extending a control message with padding data. The total size of a probe packet includes all headers and padding added to the payload data being sent (e.g., including protocol option fields, security- related fields such as an Authenticated Encryption with Associated Data (AEAD) tag and TLS record layer padding).

A receiver is REQUIRED to be able to distinguish an in-band data block from any added padding. This is needed to ensure that any added padding is not passed on to an application at the receiver.

This results in three possible ways that a sender can create a probe packet:

Probing using padding data: A probe packet that contains only control information together with any padding, which is needed to be inflated to the size of the probe packet. Since these probe packets do not carry an application-supplied data block, they do not typically require retransmission, although they do still consume network capacity and incur endpoint processing.

Probing using application data and padding data: A probe packet that contains a data block supplied by an application that is combined with padding to inflate the length of the datagram to the size of the probe packet.

Probing using application data: A probe packet that contains a data block supplied by an application that matches the size of the

Fairhurst, et al. Expires 12 December 2020 [Page 14] Internet-Draft DPLPMTUD June 2020

probe packet. This method requests the application to issue a data block of the desired probe size.

A PL that uses a probe packet carrying application data and needs protection from the loss of this probe packet could perform transport-layer retransmission/repair of the data block (e.g., by retransmission after loss is detected or by duplicating the data block in a datagram without the padding data). This retransmitted data block might possibly need to be sent using a smaller PLPMTU, which could force the PL to to use a smaller packet size to traverse the end-to-end path. (This could utilize endpoint network-layer fragmentation or a PL that can re-segment the data block into multiple datagrams).

DPLPMTUD MAY choose to use only one of these methods to simplify the implementation.

Probe messages sent by a PL MUST contain enough information to uniquely identify the probe within Maximum Segment Lifetime (e.g., including a unique identifier from the PL or the DPLPMTUD implementation), while being robust to reordering and replay of probe response and PTB messages.

4.2. Confirmation of Probed Packet Size

The PL needs a method to determine (confirm) when probe packets have been successfully received end-to-end across a network path.

Transport protocols can include end-to-end methods that detect and report reception of specific datagrams that they send (e.g., DCCP, SCTP, and QUIC provide keep-alive/heartbeat features). When supported, this mechanism MAY also be used by DPLPMTUD to acknowledge reception of a probe packet.

A PL that does not acknowledge data reception (e.g., UDP and UDP- Lite) is unable itself to detect when the packets that it sends are discarded because their size is greater than the actual PMTU. These PLs need to rely on an application protocol to detect this loss.

Section 6 specifies this function for a set of IETF-specified protocols.

4.3. Black Hole Detection and Reducing the PLPMTU

The description that follows uses the set of constants defined in Section 5.1.2 and variables defined in Section 5.1.3.

Fairhurst, et al. Expires 12 December 2020 [Page 15] Internet-Draft DPLPMTUD June 2020

Black Hole Detection is triggered by an indication that the network path could be unable to support the current PLPMTU size.

There are three indicators that can detect black holes:

* A validated PTB message can be received that indicates a PL_PTB_SIZE less than the current PLPMTU. A DPLPMTUD method MUST NOT rely solely on this method.

* A PL can use the DPLPMTUD probing mechanism to periodically generate probe packets of the size of the current PLPMTU (e.g., using the confirmation timer Section 5.1.1). A timer tracks whether acknowledgments are received. Successive loss of probes is an indication that the current path no longer supports the PLPMTU (e.g., when the number of probe packets sent without receiving an acknowledgment, PROBE_COUNT, becomes greater than MAX_PROBES).

* A PL can utilize an event that indicates the network path no longer sustains the sender’s PLPMTU size. This could use a mechanism implemented within the PL to detect excessive loss of data sent with a specific packet size and then conclude that this excessive loss could be a result of an invalid PLPMTU (as in PLPMTUD for TCP [RFC4821]).

The three methods can result in different transmission patterns for packet probes and are expected to result in different responsiveness following a change in the actual PMTU.

A PL MAY inhibit sending probe packets when no application data has been sent since the previous probe packet. A PL that resumes sending user data MAY continue PLPMTU discovery for each path. This allows it to use an up-to-date PLPMTU. However, this could result in additional packets being sent.

When the method detects the current PLPMTU is not supported, DPLPMTUD sets a lower PLPMTU, and sets a lower MPS. The PL then confirms that the new PLPMTU can be successfully used across the path. A probe packet could need to have a size less than the size of the data block generated by the application.

Fairhurst, et al. Expires 12 December 2020 [Page 16] Internet-Draft DPLPMTUD June 2020

4.4. The Maximum Packet Size (MPS)

The result of probing determines a usable PLPMTU, which is used to set the MPS used by the application. The MPS is smaller than the PLPMTU because it is reduced by the size of PL headers (including the overhead of security-related fields such as an AEAD tag and TLS record layer padding). The relationship between the MPS and the PLPMTUD is illustrated in Figure 1.

any additional headers .--- MPS -----. | | | v v v +------+ | IP | ** | PL | protocol data | +------+

<----- PLPMTU -----> <------PMTU ------>

Figure 1: Relationship between MPS and PLPMTU

A PL is unable to send a packet (other than a probe packet) with a size larger than the current PLPMTU at the network layer. To avoid this, a PL MAY be designed to segment data blocks larger than the MPS into multiple datagrams.

DPLPMTUD seeks to avoid IP fragmentation. An attempt to send a data block larger than the MPS will therefore fail if a PL is unable to segment data. To determine the largest data block that can be sent, a PL SHOULD provide applications with a primitive that returns the MPS, derived from the current PLPMTU.

If DPLPMTUD results in a change to the MPS, the application needs to adapt to the new MPS. A particular case can arise when packets have been sent with a size less than the MPS and the PLPMTU was subsequently reduced. If these packets are lost, the PL MAY segment the data using the new MPS. If a PL is unable to re-segment a previously sent datagram (e.g., [RFC4960]), then the sender either discards the datagram or could perform retransmission using network- layer fragmentation to form multiple IP packets not larger than the PLPMTU. For IPv4, the use of endpoint fragmentation by the sender is preferred over clearing the DF bit in the IPv4 header. Operational experience reveals that IP fragmentation can reduce the reliability of Internet communication [I-D.ietf-intarea-frag-fragile], which may reduce the probability of successful retransmission.

Fairhurst, et al. Expires 12 December 2020 [Page 17] Internet-Draft DPLPMTUD June 2020

4.5. Disabling the Effect of PMTUD

A PL implementing this specification MUST suspend network layer processing of outgoing packets that enforces a PMTU [RFC1191][RFC8201] for each flow utilizing DPLPMTUD, and instead use DPLPMTUD to control the size of packets that are sent by a flow. This removes the need for the network layer to drop or fragment sent packets that have a size greater than the PMTU.

4.6. Response to PTB Messages

This method requires the DPLPMTUD sender to validate any received PTB message before using the PTB information. The response to a PTB message depends on the PL_PTB_SIZE calculated from the PTB_SIZE in the PTB message, the state of the PLPMTUD state machine, and the IP protocol being used.

Section 4.6.1 first describes validation for both IPv4 ICMP Unreachable messages (type 3) and ICMPv6 Packet Too Big messages, both of which are referred to as PTB messages in this document.

4.6.1. Validation of PTB Messages

This section specifies utilization and validation of PTB messages.

* A simple implementation MAY ignore received PTB messages and in this case the PLPMTU is not updated when a PTB message is received.

* A PL that supports PTB messages MUST validate these messages before they are further processed.

A PL that receives a PTB message from a router or middlebox performs ICMP validation (see Section 4 of [RFC8201] and Section 5.2 of [BCP145]). Because DPLPMTUD operates at the PL, the PL needs to check that each received PTB message is received in response to a packet transmitted by the endpoint PL performing DPLPMTUD.

The PL MUST check the protocol information in the quoted packet carried in an ICMP PTB message payload to validate the message originated from the sending node. This validation includes determining that the combination of the IP addresses, the protocol, the source port and destination port match those returned in the quoted packet - this is also necessary for the PTB message to be passed to the corresponding PL.

The validation SHOULD utilize information that it is not simple for an off-path attacker to determine [BCP145]. For example, it could

Fairhurst, et al. Expires 12 December 2020 [Page 18] Internet-Draft DPLPMTUD June 2020

check the value of a protocol header field known only to the two PL endpoints. A datagram application that uses well-known source and destination ports ought to also rely on other information to complete this validation.

These checks are intended to provide protection from packets that originate from a node that is not on the network path. A PTB message that does not complete the validation MUST NOT be further utilized by the DPLPMTUD method, as discussed in the Security Considerations section.

Section 4.6.2 describes this processing of PTB messages.

4.6.2. Use of PTB Messages

PTB messages that have been validated MAY be utilized by the DPLPMTUD algorithm, but MUST NOT be used directly to set the PLPMTU.

Before using the size reported in the PTB message it must first be converted to a PL_PTB_SIZE. The PL_PTB_SIZE is smaller than the PTB_SIZE because it is reduced by headers below the PL including any IP options or extensions added to the PL packet.

A method that utilizes these PTB messages can improve the speed at which the algorithm detects an appropriate PLPMTU by triggering an immediate probe for the PL_PTB_SIZE (resulting in a network-layer packet of size PTB_SIZE), compared to one that relies solely on probing using a timer-based search algorithm.

A set of checks are intended to provide protection from a router that reports an unexpected PTB_SIZE. The PL also needs to check that the indicated PL_PTB_SIZE is less than the size used by probe packets and at least the minimum size accepted.

This section provides a summary of how PTB messages can be utilized. (This uses the set of constants defined in Section 5.1.2). This processing depends on the PL_PTB_SIZE and the current value of a set of variables:

PL_PTB_SIZE < MIN_PLPMTU * Invalid PL_PTB_SIZE see Section 4.6.1.

* PTB message ought to be discarded without further processing (i.e., PLPMTU is not modified).

* The information could be utilized as an input that triggers enabling a resilience mode (see Section 5.3.3).

Fairhurst, et al. Expires 12 December 2020 [Page 19] Internet-Draft DPLPMTUD June 2020

MIN_PLPMTU < PL_PTB_SIZE < BASE_PLPMTU * A robust PL MAY enter an error state (see Section 5.2) for an IPv4 path when the PL_PTB_SIZE reported in the PTB message is larger than or equal to 68 bytes [RFC0791] and when this is less than the BASE_PLPMTU.

* A robust PL MAY enter an error state (see Section 5.2) for an IPv6 path when the PL_PTB_SIZE reported in the PTB message is larger than or equal to 1280 bytes [RFC8200] and when this is less than the BASE_PLPMTU.

BASE_PLPMTU <= PL_PTB_SIZE < PLPMTU * This could be an indication of a black hole. The PLPMTU SHOULD be set to BASE_PLPMTU (the PLPMTU is reduced to the BASE_PLPMTU to avoid unnecessary packet loss when a black hole is encountered).

* The PL ought to start a search to quickly discover the new PLPMTU. The PL_PTB_SIZE reported in the PTB message can be used to initialize a search algorithm.

PLPMTU < PL_PTB_SIZE < PROBED_SIZE * The PLPMTU continues to be valid, but the size of a packet used to search (PROBED_SIZE) was larger than the actual PMTU.

* The PLPMTU is not updated.

* The PL can use the reported PL_PTB_SIZE from the PTB message as the next search point when it resumes the search algorithm.

PL_PTB_SIZE >= PROBED_SIZE * Inconsistent network signal.

* PTB message ought to be discarded without further processing (i.e., PLPMTU is not modified).

* The information could be utilized as an input to trigger enabling a resilience mode.

5. Datagram Packetization Layer PMTUD

This section specifies Datagram PLPMTUD (DPLPMTUD). The method can be introduced at various points (as indicated with * in the figure below) in the IP protocol stack to discover the PLPMTU so that an application can utilize an appropriate MPS for the current network path.

Fairhurst, et al. Expires 12 December 2020 [Page 20] Internet-Draft DPLPMTUD June 2020

DPLPMTUD SHOULD only be performed at one layer between a pair of endpoints. Therefore, an upper PL or application should avoid using DPLPMTUD when this is already enabled in a lower layer. A PL MUST adjust the MPS indicated by DPLPMTUD to account for any additional overhead introduced by the PL.

+------+ | Application* | +-----+------+---+ | | +---+--+ +--+--+ | QUIC*| |SCTP*| +---+--+ +-+-+-+ | | | +---+ +----+ | | | | +-+--+-+ | | UDP | | +---+--+ | | | +------+------+--+ | Network Interface | +------+

Figure 2: Examples where DPLPMTUD can be implemented

The central idea of DPLPMTUD is probing by a sender. Probe packets are sent to find the maximum size of user message that can be completely transferred across the network path from the sender to the destination.

The following sections identify the components needed for implementation, provides an overview of the phases of operation, and specifies the state machine and search algorithm.

5.1. DPLPMTUD Components

This section describes the timers, constants, and variables of DPLPMTUD.

5.1.1. Timers

The method utilizes up to three timers:

PROBE_TIMER: The PROBE_TIMER is configured to expire after a period longer than the maximum time to receive an acknowledgment to a probe packet. This value MUST NOT be smaller than 1 second, and SHOULD be larger than 15 seconds. Guidance on selection of the

Fairhurst, et al. Expires 12 December 2020 [Page 21] Internet-Draft DPLPMTUD June 2020

timer value are provided in Section 3.1.1 of the UDP Usage Guidelines [BCP145].

PMTU_RAISE_TIMER: The PMTU_RAISE_TIMER is configured to the period a sender will continue to use the current PLPMTU, after which it re- enters the Search phase. This timer has a period of 600 seconds, as recommended by PLPMTUD [RFC4821].

DPLPMTUD MAY inhibit sending probe packets when no application data has been sent since the previous probe packet. A PL preferring to use an up-to-date PMTU once user data is sent again, can choose to continue PMTU discovery for each path. However, this will result in sending additional packets.

CONFIRMATION_TIMER: When an acknowledged PL is used, this timer MUST NOT be used. For other PLs, the CONFIRMATION_TIMER is configured to the period a PL sender waits before confirming the current PLPMTU is still supported. This is less than the PMTU_RAISE_TIMER and used to decrease the PLPMTU (e.g., when a black hole is encountered). Confirmation needs to be frequent enough when data is flowing that the sending PL does not black hole extensive amounts of traffic. Guidance on selection of the timer value are provided in Section 3.1.1 of the UDP Usage Guidelines [BCP145].

DPLPMTD specifies various timers, however an implementation could choose to realise these timer functions using a single timer.

5.1.2. Constants

The following constants are defined:

MAX_PROBES: The MAX_PROBES is the maximum value of the PROBE_COUNT counter (see Section 5.1.3). MAX_PROBES represents the limit for the number of consecutive probe attempts of any size. Search algorithms benefit from a MAX_PROBES value greater than 1 because this can provide robustness to isolated packet loss. The default value of MAX_PROBES is 3.

MIN_PLPMTU: The MIN_PLPMTU is the smallest size of PLPMTU that DPLPMTUD will attempt to use. An endpoint could need to be configure the MIN_PLPMTU to provide space for extension headers and other encapsulations at layers below the PL. This value can

Fairhurst, et al. Expires 12 December 2020 [Page 22] Internet-Draft DPLPMTUD June 2020

be interface and path dependent. For IPv6, this size is greater than or equal to the size at the PL that results in an 1280 byte IPv6 packet, as specified in [RFC8200]. For IPv4, this size is greater than or equal to the size at the PL that results in an 68 byte IPv4 packet. Note: An IPv4 router is required to be able to forward a datagram of 68 bytes without further fragmentation. This is the combined size of an IPv4 header and the minimum fragment size of 8 bytes. In addition, receivers are required to be able to reassemble fragmented datagrams at least up to 576 bytes, as stated in section 3.3.3 of [RFC1122].

MAX_PLPMTU: The MAX_PLPMTU is the largest size of PLPMTU. This has to be less than or equal to the maximum size of the PL packet that can be sent on the outgoing interface (constrained by the local interface MTU). When known, this also ought to be less than the maximum size of PL packet that can be received by the remote endpoint (constrained by EMTU_R). It can be limited by the design or configuration of the PL being used. An application, or PL, MAY choose a smaller MAX_PLPMTU when there is no need to send packets larger than a specific size.

BASE_PLPMTU: The BASE_PLPMTU is a configured size expected to work for most paths. The size is equal to or larger than the MIN_PLPMTU and smaller than the MAX_PLPMTU. For most PLs a suitable BASE_PLPMTU will be larger than 1200 bytes. When using IPv4, there is no currently equivalent size specified and a default BASE_PLPMTU of 1200 bytes is RECOMMENDED.

5.1.3. Variables

This method utilizes a set of variables:

PROBED_SIZE: The PROBED_SIZE is the size of the current probe packet as determined at the PL. This is a tentative value for the PLPMTU, which is awaiting confirmation by an acknowledgment.

PROBE_COUNT: The PROBE_COUNT is a count of the number of successive unsuccessful probe packets that have been sent. Each time a probe packet is acknowledged, the value is set to zero. (Some probe loss is expected while searching, therefore loss of a single probe is not an indication of a PMTU problem.)

The figure below illustrates the relationship between the packet size constants and variables at a point of time when the DPLPMTUD algorithm performs path probing to increase the size of the PLPMTU. A probe packet has been sent of size PROBED_SIZE. Once this is acknowledged, the PLPMTU will raise to PROBED_SIZE allowing the

Fairhurst, et al. Expires 12 December 2020 [Page 23] Internet-Draft DPLPMTUD June 2020

DPLPMTUD algorithm to further increase PROBED_SIZE toward sending a probe with the size of the actual PMTU.

Figure 3: Relationships between packet size constants and variables

5.1.4. Overview of DPLPMTUD Phases

This section provides a high-level informative view of the DPLPMTUD method, by describing the movement of the method through several phases of operation. More detail is available in the state machine Section 5.2.

+------+ +------>| Base |------+ Connectivity | +------+ | or BASE_PLPMTU | | | confirmation failed | | v | | Connectivity +------+ | | and BASE_PLPMTU | Error | | | confirmed +------+ | | | Consistent | v | connectivity Black Hole | +------+ | and BASE_PLPMTU detected | | Search |<------+ confirmed | +------+ | ^ | | | | | Raise | | Search | timer | | algorithm | expired | | completed | | | | | v | +------+ +---| Search Complete | +------+

Figure 4: DPLPMTUD Phases

Fairhurst, et al. Expires 12 December 2020 [Page 24] Internet-Draft DPLPMTUD June 2020

Base: The Base Phase confirms connectivity to the remote peer using packets of the BASE_PLPMTU. The confirmation of connectivity is implicit for a connection-oriented PL (where it can be performed in a PL connection handshake). A connectionless PL sends a probe packet and uses acknowledgment of this probe packet to confirm that the remote peer is reachable.

The sender also confirms that BASE_PLPMTU is supported across the network path. This may be achieved using a PL mechanism (e.g., using a handshake packet of size BASE_PLPMTU), or by sending a probe packet of size BASE_PLPMTU and confirming that this is received.

A probe packet of size BASE_PLPMTU can be sent immediately on the initial entry to the Base Phase (following a connectivity check). A PL that does not wish to support a path with a PLPMTU less than BASE_PLPMTU can simplify the phase into a single step by performing the connectivity checks with a probe of the BASE_PLPMTU size.

Once confirmed, DPLPMTUD enters the Search Phase. If the Base Phase fails to confirm the BASE_PLPMTU, DPLPMTUD enters the Error Phase.

Search: The Search Phase utilizes a search algorithm to send probe packets to seek to increase the PLPMTU. The algorithm concludes when it has found a suitable PLPMTU, by entering the Search Complete Phase.

A PL could respond to PTB messages using the PTB to advance or terminate the search, see Section 4.6.

Search Complete: The Search Complete Phase is entered when the PLPMTU is supported across the network path. A PL can use a CONFIRMATION_TIMER to periodically repeat a probe packet for the current PLPMTU size. If the sender is unable to confirm reachability (e.g., if the CONFIRMATION_TIMER expires) or the PL signals a lack of reachability, a black hole has been detected and DPLPMTUD enters the Base phase.

The PMTU_RAISE_TIMER is used to periodically resume the search phase to discover if the PLPMTU can be raised. Black Hole Detection causes the sender to enter the Base Phase.

Error: The Error Phase is entered when there is conflicting or invalid PLPMTU information for the path (e.g., a failure to support the BASE_PLPMTU) that cause DPLPMTUD to be unable to progress and the PLPMTU is lowered.

Fairhurst, et al. Expires 12 December 2020 [Page 25] Internet-Draft DPLPMTUD June 2020

DPLPMTUD remains in the Error Phase until a consistent view of the path can be discovered and it has also been confirmed that the path supports the BASE_PLPMTU (or DPLPMTUD is suspended).

A method that only reduces the PLPMTU to a suitable size would be sufficient to ensure reliable operation, but can be very inefficient when the actual PMTU changes or when the method (for whatever reason) makes a suboptimal choice for the PLPMTU.

A full implementation of DPLPMTUD provides an algorithm enabling the DPLPMTUD sender to increase the PLPMTU following a change in the characteristics of the path, such as when a link is reconfigured with a larger MTU, or when there is a change in the set of links traversed by an end-to-end flow (e.g., after a routing or path fail-over decision).

5.2. State Machine

A state machine for DPLPMTUD is depicted in Figure 5. If multipath or multihoming is supported, a state machine is needed for each path.

Note: Not all changes are shown to simplify the diagram.

Fairhurst, et al. Expires 12 December 2020 [Page 26] Internet-Draft DPLPMTUD June 2020

| | | Start | PL indicates loss | | of connectivity v v +------+ +------+ | DISABLED | | ERROR | +------+ PROBE_TIMER expiry: +------+ | PL indicates PROBE_COUNT = MAX_PROBES or ^ | | connectivity PTB: PL_PTB_SIZE < BASE_PLPMTU | | +------+ +------+ | | | | v | BASE_PLPMTU Probe | +------+ acked | | BASE |------>+ +------+ | ^ | ^ ^ | Black hole detected | | | | Black hole detected | +------+ | | +------+ | | +----+ | | | PROBE_TIMER expiry: | | | PROBE_COUNT < MAX_PROBES | | | | | | PMTU_RAISE_TIMER expiry | | | +------+ | | | | | | | | | v | v +------+ +------+ |SEARCH_COMPLETE| | SEARCHING | +------+ +------+ | ^ ^ | | ^ | | | | | | | | +------+ | | | | MAX_PLPMTU Probe acked or | | | | PROBE_TIMER expiry: PROBE_COUNT = MAX_PROBES or | | +----+ PTB: PL_PTB_SIZE = PLPMTU +----+ CONFIRMATION_TIMER expiry: PROBE_TIMER expiry: PROBE_COUNT < MAX_PROBES or PROBE_COUNT < MAX_PROBES or PLPMTU Probe acked Probe acked or PTB: PLPMTU < PL_PTB_SIZE < PROBED_SIZE

Figure 5: State machine for Datagram PLPMTUD

The following states are defined:

DISABLED: The DISABLED state is the initial state before probing has started. It is also entered from any other state, when the PL indicates loss of connectivity. This state is left once the PL

Fairhurst, et al. Expires 12 December 2020 [Page 27] Internet-Draft DPLPMTUD June 2020

indicates connectivity to the remote PL. When transitioning to the BASE state, a probe packet of size BASE_PLPMTU can be sent immediately.

BASE: The BASE state is used to confirm that the BASE_PLPMTU size is supported by the network path and is designed to allow an application to continue working when there are transient reductions in the actual PMTU. It also seeks to avoid long periods when a sender searching for a larger PLPMTU is unaware that packets are not being delivered due to a packet or ICMP Black Hole.

On entry, the PROBED_SIZE is set to the BASE_PLPMTU size and the PROBE_COUNT is set to zero.

Each time a probe packet is sent, the PROBE_TIMER is started. The state is exited when the probe packet is acknowledged, and the PL sender enters the SEARCHING state.

The state is also left when the PROBE_COUNT reaches MAX_PROBES or a received PTB message is validated. This causes the PL sender to enter the ERROR state.

SEARCHING: The SEARCHING state is the main probing state. This state is entered when probing for the BASE_PLPMTU completes.

Each time a probe packet is acknowledged, the PROBE_COUNT is set to zero, the PLPMTU is set to the PROBED_SIZE and then the PROBED_SIZE is increased using the search algorithm (as described in Section 5.3.

When a probe packet is sent and not acknowledged within the period of the PROBE_TIMER, the PROBE_COUNT is incremented and a new probe packet is transmitted.

The state is exited to enter SEARCH_COMPLETE when the PROBE_COUNT reaches MAX_PROBES, a validated PTB is received that corresponds to the last successfully probed size (PL_PTB_SIZE = PLPMTU), or a probe of size MAX_PLPMTU is acknowledged (PLPMTU = MAX_PLPMTU).

When a black hole is detected in the SEARCHING state, this causes the PL sender to enter the BASE state.

SEARCH_COMPLETE: The SEARCH_COMPLETE state indicates that a search has completed. This is the normal maintenance state, where the PL is not probing to update the PLPMTU. DPLPMTUD remains in this state until either the PMTU_RAISE_TIMER expires or a black hole is detected.

Fairhurst, et al. Expires 12 December 2020 [Page 28] Internet-Draft DPLPMTUD June 2020

When DPLPMTUD uses an unacknowledged PL and is in the SEARCH_COMPLETE state, a CONFIRMATION_TIMER periodically resets the PROBE_COUNT and schedules a probe packet with the size of the PLPMTU. If MAX_PROBES successive PLPMTUD sized probes fail to be acknowledged the method enters the BASE state. When used with an acknowledged PL (e.g., SCTP), DPLPMTUD SHOULD NOT continue to generate PLPMTU probes in this state.

ERROR: The ERROR state represents the case where either the network path is not known to support a PLPMTU of at least the BASE_PLPMTU size or when there is contradictory information about the network path that would otherwise result in excessive variation in the MPS signaled to the higher layer. The state implements a method to mitigate oscillation in the state-event engine. It signals a conservative value of the MPS to the higher layer by the PL. The state is exited when packet probes no longer detect the error. The PL sender then enters the SEARCHING state.

Implementations are permitted to enable endpoint fragmentation if the DPLPMTUD is unable to validate MIN_PLPMTU within PROBE_COUNT probes. If DPLPMTUD is unable to validate MIN_PLPMTU the implementation will transition to the DISABLED state.

Note: MIN_PLPMTU could be identical to BASE_PLPMTU, simplifying the actions in this state.

5.3. Search to Increase the PLPMTU

This section describes the algorithms used by DPLPMTUD to search for a larger PLPMTU.

5.3.1. Probing for a larger PLPMTU

Implementations use a search algorithm across the search range to determine whether a larger PLPMTU can be supported across a network path.

The method discovers the search range by confirming the minimum PLPMTU and then using the probe method to select a PROBED_SIZE less than or equal to MAX_PLPMTU. MAX_PLPMTU is the minimum of the local MTU and EMTU_R (when this is learned from the remote endpoint). The MAX_PLPMTU MAY be reduced by an application that sets a maximum to the size of datagrams it will send.

The PROBE_COUNT is initialized to zero when the first probe with a size greater than or equal to PLPMTUD is sent. Each probe packet successfully sent to the remote peer is confirmed by acknowledgment at the PL, see Section 4.1.

Fairhurst, et al. Expires 12 December 2020 [Page 29] Internet-Draft DPLPMTUD June 2020

Each time a probe packet is sent to the destination, the PROBE_TIMER is started. The timer is canceled when the PL receives acknowledgment that the probe packet has been successfully sent across the path Section 4.1. This confirms that the PROBED_SIZE is supported, and the PROBED_SIZE value is then assigned to the PLPMTU. The search algorithm can continue to send subsequent probe packets of an increasing size.

If the timer expires before a probe packet is acknowledged, the probe has failed to confirm the PROBED_SIZE. Each time the PROBE_TIMER expires, the PROBE_COUNT is incremented, the PROBE_TIMER is reinitialized, and a new probe of the same size or any other size (determined by the search algorithm) can be sent. The maximum number of consecutive failed probes is configured (MAX_PROBES). If the value of the PROBE_COUNT reaches MAX_PROBES, probing will stop, and the PL sender enters the SEARCH_COMPLETE state.

5.3.2. Selection of Probe Sizes

The search algorithm determines a minimum useful gain in PLPMTU. It would not be constructive for a PL sender to attempt to probe for all sizes. This would incur unnecessary load on the path. Implementations SHOULD select the set of probe packet sizes to maximize the gain in PLPMTU from each search step.

Implementations could optimize the search procedure by selecting step sizes from a table of common PMTU sizes. When selecting the appropriate next size to search, an implementer ought to also consider that there can be common sizes of MPS that applications seek to use, and their could be common sizes of MTU used within the network.

5.3.3. Resilience to Inconsistent Path Information

A decision to increase the PLPMTU needs to be resilient to the possibility that information learned about the network path is inconsistent. A path is inconsistent when, for example, probe packets are lost due to other reasons (i.e., not packet size) or due to frequent path changes. Frequent path changes could occur by unexpected "flapping" - where some packets from a flow pass along one path, but other packets follow a different path with different properties.

A PL sender is able to detect inconsistency from the sequence of PLPMTU probes that are acknowledged or the sequence of PTB messages that it receives. When inconsistent path information is detected, a PL sender could use an alternate search mode that clamps the offered

Fairhurst, et al. Expires 12 December 2020 [Page 30] Internet-Draft DPLPMTUD June 2020

MPS to a smaller value for a period of time. This avoids unnecessary loss of packets.

5.4. Robustness to Inconsistent Paths

Some paths could be unable to sustain packets of the BASE_PLPMTU size. The Error State could be implemented to provide rubustness to such paths. This allows fallback to a smaller than desired PLPMTU, rather than suffer connectivity failure. This could utilize methods such as endpoint IP fragmentation to enable the PL sender to communicate using packets smaller than the BASE_PLPMTU.

6. Specification of Protocol-Specific Methods

DPLPMTUD requires protocol-specific details to be specified for each PL that is used.

The first subsection provides guidance on how to implement the DPLPMTUD method as a part of an application using UDP or UDP-Lite. The guidance also applies to other datagram services that do not include a specific transport protocol (such as a tunnel encapsulation). The following subsections describe how DPLPMTUD can be implemented as a part of the transport service, allowing applications using the service to benefit from discovery of the PLPMTU without themselves needing to implement this method when using SCTP and QUIC.

6.1. Application support for DPLPMTUD with UDP or UDP-Lite

The current specifications of UDP [RFC0768] and UDP-Lite [RFC3828] do not define a method in the RFC-series that supports PLPMTUD. In particular, the UDP transport does not provide the transport features needed to implement datagram PLPMTUD.

The DPLPMTUD method can be implemented as a part of an application built directly or indirectly on UDP or UDP-Lite, but relies on higher-layer protocol features to implement the method [BCP145].

Some primitives used by DPLPMTUD might not be available via the Datagram API (e.g., the ability to access the PLPMTU from the IP layer cache, or interpret received PTB messages).

In addition, it is recommended that PMTU discovery is not performed by multiple protocol layers. An application SHOULD avoid using DPLPMTUD when the underlying transport system provides this capability. A common method for managing the PLPMTU has benefits, both in the ability to share state between different processes and opportunities to coordinate probing for different PL instances.

Fairhurst, et al. Expires 12 December 2020 [Page 31] Internet-Draft DPLPMTUD June 2020

6.1.1. Application Request

An application needs an application-layer protocol mechanism (such as a message acknowledgment method) that solicits a response from a destination endpoint. The method SHOULD allow the sender to check the value returned in the response to provide additional protection from off-path insertion of data [BCP145]. Suitable methods include a parameter known only to the two endpoints, such as a session ID or initialized sequence number.

6.1.2. Application Response

An application needs an application-layer protocol mechanism to communicate the response from the destination endpoint. This response could indicate successful reception of the probe across the path, but could also indicate that some (or all packets) have failed to reach the destination.

6.1.3. Sending Application Probe Packets

A probe packet can carry an application data block, but the successful transmission of this data is at risk when used for probing. Some applications might prefer to use a probe packet that does not carry an application data block to avoid disruption to data transfer.

6.1.4. Initial Connectivity

An application that does not have other higher-layer information confirming connectivity with the remote peer SHOULD implement a connectivity mechanism using acknowledged probe packets before entering the BASE state.

6.1.5. Validating the Path

An application that does not have other higher-layer information confirming correct delivery of datagrams SHOULD implement the CONFIRMATION_TIMER to periodically send probe packets while in the SEARCH_COMPLETE state.

6.1.6. Handling of PTB Messages

An application that is able and wishes to receive PTB messages MUST perform ICMP validation as specified in Section 5.2 of [BCP145]. This requires that the application checks each received PTB message to validate that it was is received in response to transmitted traffic and that the reported PL_PTB_SIZE is less than the current probed size (see Section 4.6.2). A validated PTB message MAY be used

Fairhurst, et al. Expires 12 December 2020 [Page 32] Internet-Draft DPLPMTUD June 2020

as input to the DPLPMTUD algorithm, but MUST NOT be used directly to set the PLPMTU.

6.2. DPLPMTUD for SCTP

Section 10.2 of [RFC4821] specified a recommended PLPMTUD probing method for SCTP and Section 7.3 of [RFC4960] recommended an endpoint apply the techniques in RFC4821 on a per-destination-address basis. The specification for DPLPMTUD continues the practice of using the PL to discover the PMTU, but updates, RFC4960 with a recommendation to use the method specified in this document: The RECOMMENDED method for generating probes is to add a chunk consisting only of padding to an SCTP message. The PAD chunk defined in [RFC4820] SHOULD be attached to a minimum length HEARTBEAT (HB) chunk to build a probe packet. This enables probing without affecting the transfer of user messages and without being limited by congestion control or flow control. This is preferred to using DATA chunks (with padding as required) as path probes.

Section 6.9 of [RFC4960] describes dividing the user messages into data chunks sent by the PL when using SCTP. This notes that once an SCTP message has been sent, it cannot be re-segmented. [RFC4960] describes the method to retransmit data chunks when the MPS has reduced, and the use of IP fragmentation for this case. This is unchanged by this document.

6.2.1. SCTP/IPv4 and SCTP/IPv6

6.2.1.1. Initial Connectivity

The base protocol is specified in [RFC4960]. This provides an acknowledged PL. A sender can therefore enter the BASE state as soon as connectivity has been confirmed.

6.2.1.2. Sending SCTP Probe Packets

Probe packets consist of an SCTP common header followed by a HEARTBEAT chunk and a PAD chunk. The PAD chunk is used to control the length of the probe packet. The HEARTBEAT chunk is used to trigger the sending of a HEARTBEAT ACK chunk. The reception of the HEARTBEAT ACK chunk acknowledges reception of a successful probe. A successful probe updates the association and path counters, but an unsuccessful probe is discounted (assumed to be a result of choosing too large a PLPMTU).

The SCTP sender needs to be able to determine the total size of a probe packet. The HEARTBEAT chunk could carry a Heartbeat Information parameter that includes, besides the information

Fairhurst, et al. Expires 12 December 2020 [Page 33] Internet-Draft DPLPMTUD June 2020

suggested in [RFC4960], the probe size to help an implementation associate a HEARTBEAT-ACK with the size of probe that was sent. The sender could also use other methods, such as sending a nonce and verifying the information returned also contains the corresponding nonce. The length of the PAD chunk is computed by reducing the probing size by the size of the SCTP common header and the HEARTBEAT chunk. The payload of the PAD chunk contains arbitrary data. When transmitted at the IP layer, the PMTU size also includes the IPv4 or IPv6 header(s).

Probing can start directly after the PL handshake, this can be done before data is sent. Assuming this behavior (i.e., the PMTU is smaller than or equal to the interface MTU), this process will take several round trip time periods, dependent on the number of DPLPMTUD probes sent. The Heartbeat timer can be used to implement the PROBE_TIMER.

6.2.1.3. Validating the Path with SCTP

Since SCTP provides an acknowledged PL, a sender MUST NOT implement the CONFIRMATION_TIMER while in the SEARCH_COMPLETE state.

6.2.1.4. PTB Message Handling by SCTP

Normal ICMP validation MUST be performed as specified in Appendix C of [RFC4960]. This requires that the first 8 bytes of the SCTP common header are quoted in the payload of the PTB message, which can be the case for ICMPv4 and is normally the case for ICMPv6.

When a PTB message has been validated, the PL_PTB_SIZE calculated from the PTB_SIZE reported in the PTB message SHOULD be used with the DPLPMTUD algorithm, providing that the reported PL_PTB_SIZE is less than the current probe size (see Section 4.6).

6.2.2. DPLPMTUD for SCTP/UDP

The UDP encapsulation of SCTP is specified in [RFC6951].

This specification updates the reference to RFC 4821 in section 5.6 of RFC 6951 to refer to XXXTHISRFCXXX. RFC 6951 is updated by addition of the following sentence at the end of section 5.6: "The RECOMMENDED method for determining the MTU of the path is specified in XXXTHISRFCXXX".

XXX RFC EDITOR - please replace XXXTHISRFCXXX when published XXX

Fairhurst, et al. Expires 12 December 2020 [Page 34] Internet-Draft DPLPMTUD June 2020

6.2.2.1. Initial Connectivity

A sender can enter the BASE state as soon as SCTP connectivity has been confirmed.

6.2.2.2. Sending SCTP/UDP Probe Packets

Packet probing can be performed as specified in Section 6.2.1.2. The size of the probe packet includes the 8 bytes of UDP Header. This has to be considered when filling the probe packet with the PAD chunk.

6.2.2.3. Validating the Path with SCTP/UDP

SCTP provides an acknowledged PL, therefore a sender does not implement the CONFIRMATION_TIMER while in the SEARCH_COMPLETE state.

6.2.2.4. Handling of PTB Messages by SCTP/UDP

ICMP validation MUST be performed for PTB messages as specified in Appendix C of [RFC4960]. This requires that the first 8 bytes of the SCTP common header are contained in the PTB message, which can be the case for ICMPv4 (but note the UDP header also consumes a part of the quoted packet header) and is normally the case for ICMPv6. When the validation is completed, the PL_PTB_SIZE calculated from the PTB_SIZE in the PTB message SHOULD be used with the DPLPMTUD providing that the reported PL_PTB_SIZE is less than the current probe size.

6.2.3. DPLPMTUD for SCTP/DTLS

The Datagram Transport Layer Security (DTLS) encapsulation of SCTP is specified in [RFC8261]. This is used for data channels in WebRTC implementations. This specification updates the reference to RFC 4821 in section 5 of RFC 8261 to refer to XXXTHISRFCXXX.

XXX RFC EDITOR - please replace XXXTHISRFCXXX when published XXX

6.2.3.1. Initial Connectivity

A sender can enter the BASE state as soon as SCTP connectivity has been confirmed.

Fairhurst, et al. Expires 12 December 2020 [Page 35] Internet-Draft DPLPMTUD June 2020

6.2.3.2. Sending SCTP/DTLS Probe Packets

Packet probing can be done, as specified in Section 6.2.1.2. The maximum payload is reduced by the size of the DTLS headers, which has to be considered when filling the PAD chunk. The size of the probe packet includes the DTLS PL headers. This has to be considered when filling the probe packet with the PAD chunk.

6.2.3.3. Validating the Path with SCTP/DTLS

Since SCTP provides an acknowledged PL, a sender MUST NOT implement the CONFIRMATION_TIMER while in the SEARCH_COMPLETE state.

6.2.3.4. Handling of PTB Messages by SCTP/DTLS

[RFC4960] does not specify a way to validate SCTP/DTLS ICMP message payload and neither does this document. This can prevent processing of PTB messages at the PL.

6.3. DPLPMTUD for QUIC

QUIC [I-D.ietf-quic-transport] is a UDP-based PL that provides reception feedback. The UDP payload includes a QUIC packet header, a protected payload, and any authentication fields. It supports padding and packet coalescence that can be used to construct probe packets. From the perspective of DPLPMTUD, QUIC can function as an acknowledged PL. [I-D.ietf-quic-transport] describes the method for using DPLPMTUD with QUIC packets.

7. Acknowledgments

This work was partially funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 644334 (NEAT). The views expressed are solely those of the author(s).

Thanks to all that have commented or contributed, the TSVWG and QUIC working groups, and Mathew Calder and Julius Flohr for providing early implementations.

8. IANA Considerations

This memo includes no request to IANA.

If there are no requirements for IANA, the section will be removed during conversion into an RFC by the RFC Editor.

Fairhurst, et al. Expires 12 December 2020 [Page 36] Internet-Draft DPLPMTUD June 2020

9. Security Considerations

The security considerations for the use of UDP and SCTP are provided in the referenced RFCs.

To avoid excessive load, the interval between individual probe packets MUST be at least one RTT, and the interval between rounds of probing is determined by the PMTU_RAISE_TIMER.

A PL sender needs to ensure that the method used to confirm reception of probe packets protects from off-path attackers injecting packets into the path. This protection is provided in IETF-defined protocols (e.g., TCP, SCTP) using a randomly-initialized sequence number. A description of one way to do this when using UDP is provided in section 5.1 of [BCP145]).

There are cases where ICMP Packet Too Big (PTB) messages are not delivered due to policy, configuration or equipment design (see Section 1.1). This method therefore does not rely upon PTB messages being received, but is able to utilize these when they are received by the sender. PTB messages could potentially be used to cause a node to inappropriately reduce the PLPMTU. A node supporting DPLPMTUD MUST therefore appropriately validate the payload of PTB messages to ensure these are received in response to transmitted traffic (i.e., a reported error condition that corresponds to a datagram actually sent by the path layer, see Section 4.6.1).

An on-path attacker able to create a PTB message could forge PTB messages that include a valid quoted IP packet. Such an attack could be used to drive down the PLPMTU. An on-path device could similarly force a reduction of the PLPMTU by implementing a policy that drops packets larger than a configured size. There are two ways this method can be mitigated against such attacks: First, by ensuring that a PL sender never reduces the PLPMTU below the base size, solely in response to receiving a PTB message. This is achieved by first entering the BASE state when such a message is received. Second, the design does not require processing of PTB messages, a PL sender could therefore suspend processing of PTB messages (e.g., in a robustness mode after detecting that subsequent probes actually confirm that a size larger than the PTB_SIZE is supported by a path).

Parsing the quoted packet inside a PTB message can introduce addional per-packet processing at the PL sender. This processing SHOULD be limited to avoid a denial of service attack when arbitrary headers are included. Rate-limiting the processing could result in PTB messages not being received by a PL, however the DPLPMTUD method is robust to such loss.

Fairhurst, et al. Expires 12 December 2020 [Page 37] Internet-Draft DPLPMTUD June 2020

The successful processing of an ICMP message can trigger a probe when the reported PTB size is valid, but this does not directly update the PLPMTU for the path. This prevents a message attempting to black hole data by indicating a size larger than supported by the path.

It is possible that the information about a path is not stable. This could be a result of forwarding across more than one path that has a different actual PMTU or a single path presents a varying PMTU. The design of a PLPMTUD implementation SHOULD consider how to mitigate the effects of varying path information. One possible mitigation is to provide robustness (see Section 5.4) in the method that avoids oscillation in the MPS.

DPLPMTUD methods can introduce padding data to inflate the length of the datagram to the total size required for a probe packet. The total size of a probe packet includes all headers and padding added to the payload data being sent (e.g., including security-related fields such as an AEAD tag and TLS record layer padding). The value of the padding data does not influence the DPLPMTUD search algorithm, and therefore needs to be set consistent with the policy of the PL.

If a PL can make use of cryptographic confidentiality or data- integrity mechanisms, then the design ought to avoid adding anything (e.g., padding) to DPLPMTUD probe packets that is not also protected by those cryptographic mechanisms.

10. References

10.1. Normative References

[BCP145] Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage Guidelines", BCP 145, RFC 8085, March 2017.

[RFC0768] Postel, J., "User Datagram Protocol", STD 6, RFC 768, DOI 10.17487/RFC0768, August 1980, .

[RFC0791] Postel, J., "Internet Protocol", STD 5, RFC 791, DOI 10.17487/RFC0791, September 1981, .

[RFC1191] Mogul, J.C. and S.E. Deering, "Path MTU discovery", RFC 1191, DOI 10.17487/RFC1191, November 1990, .

Fairhurst, et al. Expires 12 December 2020 [Page 38] Internet-Draft DPLPMTUD June 2020

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, .

[RFC3828] Larzon, L-A., Degermark, M., Pink, S., Jonsson, L-E., Ed., and G. Fairhurst, Ed., "The Lightweight User Datagram Protocol (UDP-Lite)", RFC 3828, DOI 10.17487/RFC3828, July 2004, .

[RFC4820] Tuexen, M., Stewart, R., and P. Lei, "Padding Chunk and Parameter for the Stream Control Transmission Protocol (SCTP)", RFC 4820, DOI 10.17487/RFC4820, March 2007, .

[RFC4960] Stewart, R., Ed., "Stream Control Transmission Protocol", RFC 4960, DOI 10.17487/RFC4960, September 2007, .

[RFC6951] Tuexen, M. and R. Stewart, "UDP Encapsulation of Stream Control Transmission Protocol (SCTP) Packets for End-Host to End-Host Communication", RFC 6951, DOI 10.17487/RFC6951, May 2013, .

[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, .

[RFC8200] Deering, S. and R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", STD 86, RFC 8200, DOI 10.17487/RFC8200, July 2017, .

[RFC8201] McCann, J., Deering, S., Mogul, J., and R. Hinden, Ed., "Path MTU Discovery for IP version 6", STD 87, RFC 8201, DOI 10.17487/RFC8201, July 2017, .

[RFC8261] Tuexen, M., Stewart, R., Jesup, R., and S. Loreto, "Datagram Transport Layer Security (DTLS) Encapsulation of SCTP Packets", RFC 8261, DOI 10.17487/RFC8261, November 2017, .

10.2. Informative References

[I-D.ietf-intarea-frag-fragile] Bonica, R., Baker, F., Huston, G., Hinden, R., Troan, O.,

Fairhurst, et al. Expires 12 December 2020 [Page 39] Internet-Draft DPLPMTUD June 2020

and F. Gont, "IP Fragmentation Considered Fragile", Work in Progress, Internet-Draft, draft-ietf-intarea-frag- fragile-17, 30 September 2019, .

[I-D.ietf-intarea-tunnels] Touch, J. and M. Townsley, "IP Tunnels in the Internet Architecture", Work in Progress, Internet-Draft, draft- ietf-intarea-tunnels-10, 12 September 2019, .

[I-D.ietf-quic-transport] Iyengar, J. and M. Thomson, "QUIC: A UDP-Based Multiplexed and Secure Transport", Work in Progress, Internet-Draft, draft-ietf-quic-transport-27, 21 February 2020, .

[RFC0792] Postel, J., "Internet Control Message Protocol", STD 5, RFC 792, DOI 10.17487/RFC0792, September 1981, .

[RFC1122] Braden, R., Ed., "Requirements for Internet Hosts - Communication Layers", STD 3, RFC 1122, DOI 10.17487/RFC1122, October 1989, .

[RFC1812] Baker, F., Ed., "Requirements for IP Version 4 Routers", RFC 1812, DOI 10.17487/RFC1812, June 1995, .

[RFC2923] Lahey, K., "TCP Problems with Path MTU Discovery", RFC 2923, DOI 10.17487/RFC2923, September 2000, .

[RFC4340] Kohler, E., Handley, M., and S. Floyd, "Datagram Congestion Control Protocol (DCCP)", RFC 4340, DOI 10.17487/RFC4340, March 2006, .

[RFC4443] Conta, A., Deering, S., and M. Gupta, Ed., "Internet Control Message Protocol (ICMPv6) for the Internet Protocol Version 6 (IPv6) Specification", STD 89, RFC 4443, DOI 10.17487/RFC4443, March 2006, .

Fairhurst, et al. Expires 12 December 2020 [Page 40] Internet-Draft DPLPMTUD June 2020

[RFC4821] Mathis, M. and J. Heffner, "Packetization Layer Path MTU Discovery", RFC 4821, DOI 10.17487/RFC4821, March 2007, .

[RFC4890] Davies, E. and J. Mohacsi, "Recommendations for Filtering ICMPv6 Messages in Firewalls", RFC 4890, DOI 10.17487/RFC4890, May 2007, .

[RFC5508] Srisuresh, P., Ford, B., Sivakumar, S., and S. Guha, "NAT Behavioral Requirements for ICMP", BCP 148, RFC 5508, DOI 10.17487/RFC5508, April 2009, .

Appendix A. Revision Notes

Note to RFC-Editor: please remove this entire section prior to publication.

Individual draft -00:

* Comments and corrections are welcome directly to the authors or via the IETF TSVWG working group mailing list.

* This update is proposed for WG comments.

Individual draft -01:

* Contains the first representation of the algorithm, showing the states and timers

* This update is proposed for WG comments.

Individual draft -02:

* Contains updated representation of the algorithm, and textual corrections.

* The text describing when to set the effective PMTU has not yet been validated by the authors

* To determine security to off-path-attacks: We need to decide whether a received PTB message SHOULD/MUST be validated? The text on how to handle a PTB message indicating a link MTU larger than the probe has yet not been validated by the authors

* No text currently describes how to handle inconsistent results from arbitrary re-routing along different parallel paths

Fairhurst, et al. Expires 12 December 2020 [Page 41] Internet-Draft DPLPMTUD June 2020

* This update is proposed for WG comments.

Working Group draft -00:

* This draft follows a successful adoption call for TSVWG

* There is still work to complete, please comment on this draft.

Working Group draft -01:

* This draft includes improved introduction.

* The draft is updated to require ICMP validation prior to accepting PTB messages - this to be confirmed by WG

* Section added to discuss Selection of Probe Size - methods to be evaluated and recommendations to be considered

* Section added to align with work proposed in the QUIC WG.

Working Group draft -02:

* The draft was updated based on feedback from the WG, and a detailed review by Magnus Westerlund.

* The document updates RFC 4821.

* Requirements list updated.

* Added more explicit discussion of a simpler black-hole detection mode.

* This draft includes reorganisation of the section on IETF protocols.

* Added more discussion of implementation within an application.

* Added text on flapping paths.

* Replaced ’effective MTU’ with new term PLPMTU.

Working Group draft -03:

* Updated figures

* Added more discussion on blackhole detection

* Added figure describing just blackhole detection

Fairhurst, et al. Expires 12 December 2020 [Page 42] Internet-Draft DPLPMTUD June 2020

* Added figure relating MPS sizes

Working Group draft -04:

* Described phases and named these consistently.

* Corrected transition from confirmation directly to the search phase (Base has been checked).

* Redrawn state diagrams.

* Renamed BASE_MTU to BASE_PMTU (because it is a base for the PMTU).

* Clarified Error state.

* Clarified suspending DPLPMTUD.

* Verified normative text in requirements section.

* Removed duplicate text.

* Changed all text to refer to /packet probe/probe packet/ /validation/verification/ added term /Probe Confirmation/ and clarified BlackHole detection.

Working Group draft -05:

* Updated security considerations.

* Feedback after speaking with Joe Touch helped improve UDP-Options description.

Working Group draft -06:

* Updated description of ICMP issues in section 1.1

* Update to description of QUIC.

Working group draft -07:

* Moved description of the PTB processing method from the PTB requirements section.

* Clarified what is performed in the PTB validation check.

* Updated security consideration to explain PTB security without needing to read the rest of the document.

Fairhurst, et al. Expires 12 December 2020 [Page 43] Internet-Draft DPLPMTUD June 2020

* Reformatted state machine diagram

Working group draft -08:

* Moved to rfcxml v3+

* Rendered diagrams to svg in html version.

* Removed Appendix A. Event-driven state changes.

* Removed section on DPLPMTUD with UDP Options.

* Shortened the description of phases.

Working group draft -09:

* Remove final mention of UDP Options

* Add Initial Connectivity sections to each PL

* Add to disable outgoing pmtu enforcement of packets

Working group draft -10:

* Address comments from Lars Eggert

* Reinforce that PROBE_COUNT is successive attempts to probe for any size

* Redefine MAX_PROBES to 3

* Address PTB_SIZE of 0 or less that MIN_PLPMTU

Working group draft -11:

* Restore a sentence removed in previous rev

* De-acronymise QUIC

* Address some nits

Working group draft -12:

* Add TSVWG, QUIC and implementers to acknowledgments

* Shorten a diagram line.

* Address nits from Julius and Wes.

Fairhurst, et al. Expires 12 December 2020 [Page 44] Internet-Draft DPLPMTUD June 2020

* Be clearer when talking about IP layer caches

Working group draft -13, -14:

* Updated after WGLC.

Working group draft -15:

* Updated after AD evaluation and prepared for IETF-LC.

Working group draft -16:

* Updated text after SECDIR review.

Working group draft -17:

* Updated text after GENART and IETF-LC.

* Renamed BASE_MTU to BASE_PLPMTU, and MIN and MAX PMTU to PLPMTU (because these are about a base for the PLPMTU), and ensured consistent separation of PMTU and PLPMTU.

* Adopted US-style English throughout.

Working group draft -18:

* Updated text and address nits from OPSDIR, ART and IESG reviews.

* Order PTB processing based on PL_PTB_SIZE

Working group draft -19:

* Updated text and address nits based on comments from Tim Chown and Murray S. Kucherawy.

Working group draft -20:

* Address nits and comments from IESG

* Refer to BCP 145 rather than RFC 8085 in most places.

* Update probing method text for SCTP and QUIC.

Working group draft -21:

* Update QUIC text for skipping into BASE state.

Working group draft -22:

Fairhurst, et al. Expires 12 December 2020 [Page 45] Internet-Draft DPLPMTUD June 2020

* Add a section reference to MPS

* Clarify MIN_PLPMTU text

* Remove most QUIC text

* Make QUIC reference informative.

Authors’ Addresses

Godred Fairhurst University of Aberdeen School of Engineering Fraser Noble Building Aberdeen AB24 3UE United Kingdom

Email: [email protected]

Tom Jones University of Aberdeen School of Engineering Fraser Noble Building Aberdeen AB24 3UE United Kingdom

Email: [email protected]

Michael Tuexen Muenster University of Applied Sciences Stegerwaldstrasse 39 48565 Steinfurt Germany

Email: [email protected]

Irene Ruengeler Muenster University of Applied Sciences Stegerwaldstrasse 39 48565 Steinfurt Germany

Email: [email protected]

Fairhurst, et al. Expires 12 December 2020 [Page 46] Internet-Draft DPLPMTUD June 2020

Timo Voelker Muenster University of Applied Sciences Stegerwaldstrasse 39 48565 Steinfurt Germany

Email: [email protected]

Fairhurst, et al. Expires 12 December 2020 [Page 47] Transport Area Working Group B. Briscoe Internet-Draft Independent Updates: 3819 (if approved) J. Kaippallimalil Intended status: Best Current Practice Futurewei Expires: November 26, 2021 May 25, 2021

Guidelines for Adding Congestion Notification to Protocols that Encapsulate IP draft-ietf-tsvwg-ecn-encap-guidelines-16

Abstract

The purpose of this document is to guide the design of congestion notification in any lower layer or tunnelling protocol that encapsulates IP. The aim is for explicit congestion signals to propagate consistently from lower layer protocols into IP. Then the IP internetwork layer can act as a portability layer to carry congestion notification from non-IP-aware congested nodes up to the transport layer (L4). Following these guidelines should assure interworking among IP layer and lower layer congestion notification mechanisms, whether specified by the IETF or other standards bodies. This document updates the advice to subnetwork designers about ECN in RFC 3819.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

This Internet-Draft will expire on November 26, 2021.

Copyright Notice

Briscoe & KaippallimalilExpires November 26, 2021 [Page 1] Internet-Draft ECN Encapsulation Guidelines May 2021

Table of Contents

1. Introduction ...... 3 1.1. Update to RFC 3819 ...... 5 1.2. Scope ...... 5 2. Terminology ...... 7 3. Modes of Operation ...... 9 3.1. Feed-Forward-and-Up Mode ...... 9 3.2. Feed-Up-and-Forward Mode ...... 11 3.3. Feed-Backward Mode ...... 12 3.4. Null Mode ...... 14 4. Feed-Forward-and-Up Mode: Guidelines for Adding Congestion Notification ...... 14 4.1. IP-in-IP Tunnels with Shim Headers ...... 15 4.2. Wire Protocol Design: Indication of ECN Support . . . . . 16 4.3. Encapsulation Guidelines ...... 18 4.4. Decapsulation Guidelines ...... 20 4.5. Sequences of Similar Tunnels or Subnets ...... 22 4.6. Reframing and Congestion Markings ...... 22 5. Feed-Up-and-Forward Mode: Guidelines for Adding Congestion Notification ...... 23 6. Feed-Backward Mode: Guidelines for Adding Congestion Notification ...... 24 7. IANA Considerations ...... 25 8. Security Considerations ...... 25 9. Conclusions ...... 26 10. Acknowledgements ...... 27 11. Contributors ...... 27 12. Comments Solicited ...... 27 13. References ...... 27 13.1. Normative References ...... 27 13.2. Informative References ...... 28 Appendix A. Changes in This Version (to be removed by RFC Editor) ...... 33 Authors’ Addresses ...... 38

Briscoe & KaippallimalilExpires November 26, 2021 [Page 2] Internet-Draft ECN Encapsulation Guidelines May 2021

1. Introduction

The benefits of Explicit Congestion Notification (ECN) described in [RFC8087] and summarized below can only be fully realized if support for ECN is added to the relevant subnetwork technology, as well as to IP. When a lower layer buffer drops a packet obviously it does not just drop at that layer; the packet disappears from all layers. In contrast, when active queue management (AQM) at a lower layer marks a packet with ECN, the marking needs to be explicitly propagated up the layers. The same is true if AQM marks the outer header of a packet that encapsulates inner tunnelled headers. Forwarding ECN is not as straightforward as other headers because it has to be assumed ECN may be only partially deployed. If a lower layer header that contains ECN congestion indications is stripped off by a subnet egress that is not ECN-aware, or if the ultimate receiver or sender is not ECN- aware, congestion needs to be indicated by dropping a packet, not marking it.

The purpose of this document is to guide the addition of congestion notification to any subnet technology or tunnelling protocol, so that lower layer AQM algorithms can signal congestion explicitly and it will propagate consistently into encapsulated (higher layer) headers, otherwise the signals will not reach their ultimate destination.

ECN is defined in the IP header (v4 and v6) [RFC3168] to allow a resource to notify the onset of queue build-up without having to drop packets, by explicitly marking a proportion of packets with the congestion experienced (CE) codepoint.

Given a suitable marking scheme, ECN removes nearly all congestion loss and it cuts delays for two main reasons:

o It avoids the delay when recovering from congestion losses, which particularly benefits small flows or real-time flows, making their delivery time predictably short [RFC2884];

o As ECN is used more widely by end-systems, it will gradually remove the need to configure a degree of delay into buffers before they start to notify congestion (the cause of bufferbloat). This is because drop involves a trade-off between sending a timely signal and trying to avoid impairment, whereas ECN is solely a signal not an impairment, so there is no harm triggering it earlier.

Some lower layer technologies (e.g. MPLS, Ethernet) are used to form subnetworks with IP-aware nodes only at the edges. These networks are often sized so that it is rare for interior queues to overflow. However, until recently this was more due to the inability of TCP to

Briscoe & KaippallimalilExpires November 26, 2021 [Page 3] Internet-Draft ECN Encapsulation Guidelines May 2021

saturate the links. For many years, fixes such as window scaling [RFC7323] proved hard to deploy. And the Reno variant of TCP has remained in widespread use despite its inability to scale to high flow rates. However, now that modern operating systems are finally capable of saturating interior links, even the buffers of well- provisioned interior switches will need to signal episodes of queuing.

Propagation of ECN is defined for MPLS [RFC5129], and is being defined for TRILL [RFC7780], [I-D.ietf-trill-ecn-support], but it remains to be defined for a number of other subnetwork technologies.

Similarly, ECN propagation is yet to be defined for many tunnelling protocols. [RFC6040] defines how ECN should be propagated for IP-in- IPv4 [RFC2003], IP-in-IPv6 [RFC2473] and IPsec [RFC4301] tunnels, but there are numerous other tunnelling protocols with a shim and/or a layer 2 header between two IP headers (v4 or v6). Some address ECN propagation between the IP headers, but many do not. This document gives guidance on how to address ECN propagation for future tunnelling protocols, and a companion standards track specification [I-D.ietf-tsvwg-rfc6040update-shim] updates those existing IP-shim- (L2)-IP protocols that are under IETF change control and still widely used.

Incremental deployment is the most delicate aspect when adding support for ECN. The original ECN protocol in IP [RFC3168] was carefully designed so that a congested buffer would not mark a packet (rather than drop it) unless both source and destination hosts were ECN-capable. Otherwise its congestion markings would never be detected and congestion would just build up further. However, to support congestion marking below the IP layer or within tunnels, it is not sufficient to only check that the two layer 4 transport endpoints support ECN; correct operation also depends on the decapsulator at each subnet or tunnel egress faithfully propagating congestion notifications to the higher layer. Otherwise, a legacy decapsulator might silently fail to propagate any ECN signals from the outer to the forwarded header. Then the lost signals would never be detected and again congestion would build up further. The guidelines given later require protocol designers to carefully consider incremental deployment, and suggest various safe approaches for different circumstances.

Of course, the IETF does not have standards authority over every link layer protocol. So this document gives guidelines for designing propagation of congestion notification across the interface between IP and protocols that may encapsulate IP (i.e. that can be layered beneath IP). Each lower layer technology will exhibit different issues and compromises, so the IETF or the relevant standards body

Briscoe & KaippallimalilExpires November 26, 2021 [Page 4] Internet-Draft ECN Encapsulation Guidelines May 2021

must be free to define the specifics of each lower layer congestion notification scheme. Nonetheless, if the guidelines are followed, congestion notification should interwork between different technologies, using IP in its role as a ’portability layer’.

Therefore, the capitalized terms ’SHOULD’ or ’SHOULD NOT’ are often used in preference to ’MUST’ or ’MUST NOT’, because it is difficult to know the compromises that will be necessary in each protocol design. If a particular protocol design chooses not to follow a ’SHOULD (NOT)’ given in the advice below, it MUST include a sound justification.

It has not been possible to give common guidelines for all lower layer technologies, because they do not all fit a common pattern. Instead they have been divided into a few distinct modes of operation: feed-forward-and-upward; feed-upward-and-forward; feed- backward; and null mode. These modes are described in Section 3, then in the subsequent sections separate guidelines are given for each mode.

1.1. Update to RFC 3819

This document updates the brief advice to subnetwork designers about ECN in [RFC3819], by replacing the last two paragraphs of Section 13 with the following sentence:

By following the guidelines in [this document], subnetwork designers can enable a layer-2 protocol to participate in congestion control without dropping packets via propagation of explicit congestion notification (ECN [RFC3168]) to receivers.

and adding [this document] as an informative reference. {RFC Editor: Please replace both instances of [this document] above with the number of the present RFC when published.}

1.2. Scope

This document only concerns wire protocol processing of explicit notification of congestion. It makes no changes or recommendations concerning algorithms for congestion marking or for congestion response, because algorithm issues should be independent of the layer the algorithm operates in.

The default ECN semantics are described in [RFC3168] and updated by [RFC8311]. Also the guidelines for AQM designers [RFC7567] clarify the semantics of both drop and ECN signals from AQM algorithms. [RFC4774] is the appropriate best current practice specification of how algorithms with alternative semantics for the ECN field can be

Briscoe & KaippallimalilExpires November 26, 2021 [Page 5] Internet-Draft ECN Encapsulation Guidelines May 2021

partitioned from Internet traffic that uses the default ECN semantics. There are two main examples for how alternative ECN semantics have been defined in practice:

o RFC 4774 suggests using the ECN field in combination with a Diffserv codepoint such as in PCN [RFC6660], Voice over 3G [UTRAN] or Voice over LTE (VoLTE) [LTE-RA];

o RFC 8311 suggests using the ECT(1) codepoint of the ECN field to indicate alternative semantics such as for the experimental Low Latency Low Loss Scalable throughput (L4S) service [I-D.ietf-tsvwg-ecn-l4s-id]).

The aim is that the default rules for encapsulating and decapsulating the ECN field are sufficiently generic that tunnels and subnets will encapsulate and decapsulate packets without regard to how algorithms elsewhere are setting or interpreting the semantics of the ECN field. [RFC6040] updates RFC 4774 to allow alternative encapsulation and decapsulation behaviours to be defined for alternative ECN semantics. However it reinforces the same point - that it is far preferable to try to fit within the common ECN encapsulation and decapsulation behaviours, because expecting all lower layer technologies and tunnels to be updated is likely to be completely impractical.

Alternative semantics for the ECN field can be defined to depend on the traffic class indicated by the DSCP. Therefore correct propagation of congestion signals could depend on correct propagation of the DSCP between the layers and along the path. For instance, if the meaning of the ECN field depends on the DSCP (as in PCN or VoLTE) and if the outer DSCP is stripped on descapsulation, as in the pipe model of [RFC2983], the special semantics of the ECN field would be lost. Similarly, if the DSCP is changed at the boundary between Diffserv domains, the special ECN semantics would also be lost. This is an important implication of the localized scope of most Diffserv arrangements. In this document, correct propagation of traffic class information is assumed, while what ’correct’ means and how it is achieved is covered elsewhere (e.g. RFC 2983) and is outside the scope of the present document.

The guidelines in this document do ensure that common encapsulation and decapsulation rules are sufficiently generic to cover cases where ECT(1) is used instead of ECT(0) to identify alternative ECN semantics (as in L4S [I-D.ietf-tsvwg-ecn-l4s-id]) and where ECN marking algorithms use ECT(1) to encode 3 severity levels into the ECN field (e.g. PCN [RFC6660]) rather than the default of 2. All these different semantics for the ECN field work because it has been possible to define common default decapsulation rules that allow for all cases.

Briscoe & KaippallimalilExpires November 26, 2021 [Page 6] Internet-Draft ECN Encapsulation Guidelines May 2021

Note that the guidelines in this document do not necessarily require the subnet wire protocol to be changed to add support for congestion notification. For instance, the Feed-Up-and-Forward Mode (Section 3.2) and the Null Mode (Section 3.4) do not. Another way to add congestion notification without consuming header space in the subnet protocol might be to use a parallel control plane protocol.

This document focuses on the congestion notification interface between IP and lower layer or tunnel protocols that can encapsulate IP, where the term ’IP’ includes v4 or v6, unicast, multicast or anycast. However, it is likely that the guidelines will also be useful when a lower layer protocol or tunnel encapsulates itself, e.g. Ethernet MAC in MAC ([IEEE802.1Q]; previously 802.1ah) or when it encapsulates other protocols. In the feed-backward mode, propagation of congestion signals for multicast and anycast packets is out-of-scope (because the complexity would make it unlikely to be attempted).

2. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

Further terminology used within this document:

Protocol data unit (PDU): Information that is delivered as a unit among peer entities of a layered network consisting of protocol control information (typically a header) and possibly user data (payload) of that layer. The scope of this document includes layer 2 and layer 3 networks, where the PDU is respectively termed a frame or a packet (or a cell in ATM). PDU is a general term for any of these. This definition also includes a payload with a shim header lying somewhere between layer 2 and 3.

Transport: The end-to-end transmission control function, conventionally considered at layer-4 in the OSI reference model. Given the audience for this document will often use the word transport to mean low level bit carriage, whenever the term is used it will be qualified, e.g. ’L4 transport’.

Encapsulator: The link or tunnel endpoint function that adds an outer header to a PDU (also termed the ’link ingress’, the ’subnet ingress’, the ’ingress tunnel endpoint’ or just the ’ingress’ where the context is clear).

Briscoe & KaippallimalilExpires November 26, 2021 [Page 7] Internet-Draft ECN Encapsulation Guidelines May 2021

Decapsulator: The link or tunnel endpoint function that removes an outer header from a PDU (also termed the ’link egress’, the ’subnet egress’, the ’egress tunnel endpoint’ or just the ’egress’ where the context is clear).

Incoming header: The header of an arriving PDU before encapsulation.

Outer header: The header added to encapsulate a PDU.

Inner header: The header encapsulated by the outer header.

Outgoing header: The header forwarded by the decapsulator.

CE: Congestion Experienced [RFC3168]

ECT: ECN-Capable (L4) Transport [RFC3168]

Not-ECT: Not ECN-Capable (L4) Transport [RFC3168]

Load Regulator: For each flow of PDUs, the transport function that is capable of controlling the data rate. Typically located at the data source, but in-path nodes can regulate load in some congestion control arrangements (e.g. admission control, policing nodes or transport circuit-breakers [RFC8084]). Note the term "a function capable of controlling the load" deliberately includes a transport that does not actually control the load responsively but ideally it ought to (e.g. a sending application without congestion control that uses UDP).

ECN-PDU: A PDU at the IP layer or below with a capacity to signal congestion that is part of a congestion control feedback loop within which all the nodes necessary to propagate the signal back to the Load Regulator are capable of doing that propagation. An IP packet with a non-zero ECN field implies that the endpoints are ECN-capable, so this would be an ECN-PDU. However, ECN-PDU is intended to be a general term for a PDU at lower layers, as well as at the IP layer.

Not-ECN-PDU: A PDU at the IP layer or below that is part of a congestion control feedback-loop within which at least one node necessary to propagate any explicit congestion notification signals back to the Load Regulator is not capable of doing that propagation.

Briscoe & KaippallimalilExpires November 26, 2021 [Page 8] Internet-Draft ECN Encapsulation Guidelines May 2021

3. Modes of Operation

This section sets down the different modes by which congestion information is passed between the lower layer and the higher one. It acts as a reference framework for the following sections, which give normative guidelines for designers of explicit congestion notification protocols, taking each mode in turn:

Feed-Forward-and-Up: Nodes feed forward congestion notification towards the egress within the lower layer then up and along the layers towards the end-to-end destination at the transport layer. The following local optimisation is possible:

Feed-Up-and-Forward: A lower layer switch feeds-up congestion notification directly into the higher layer (e.g. into the ECN field in the IP header), irrespective of whether the node is at the egress of a subnet.

Feed-Backward: Nodes feed back congestion signals towards the ingress of the lower layer and (optionally) attempt to control congestion within their own layer.

Null: Nodes cannot experience congestion at the lower layer except at ingress nodes (which are IP-aware or equivalently higher-layer- aware).

3.1. Feed-Forward-and-Up Mode

Like IP and MPLS, many subnet technologies are based on self- contained protocol data units (PDUs) or frames sent unreliably. They provide no feedback channel at the subnetwork layer, instead relying on higher layers (e.g. TCP) to feed back loss signals.

In these cases, ECN may best be supported by standardising explicit notification of congestion into the lower layer protocol that carries the data forwards. Then a specification is needed for how the egress of the lower layer subnet propagates this explicit signal into the forwarded upper layer (IP) header. This signal continues forwards until it finally reaches the destination transport (at L4). Then typically the destination will feed this congestion notification back to the source transport using an end-to-end protocol (e.g. TCP). This is the arrangement that has already been used to add ECN to IP- in-IP tunnels [RFC6040], IP-in-MPLS and MPLS-in-MPLS [RFC5129].

This mode is illustrated in Figure 1. Along the middle of the figure, layers 2, 3 and 4 of the protocol stack are shown, and one packet is shown along the bottom as it progresses across the network from source to destination, crossing two subnets connected by a

Briscoe & KaippallimalilExpires November 26, 2021 [Page 9] Internet-Draft ECN Encapsulation Guidelines May 2021

router, and crossing two switches on the path across each subnet. Congestion at the output of the first switch (shown as *) leads to a congestion marking in the L2 header (shown as C in the illustration of the packet). The chevrons show the progress of the resulting congestion indication. It is propagated from link to link across the subnet in the L2 header, then when the router removes the marked L2 header, it propagates the marking up into the L3 (IP) header. The router forwards the marked L3 header into subnet 2, and when it adds a new L2 header it copies the L3 marking into the L2 header as well, as shown by the ’C’s in both layers (assuming the technology of subnet 2 also supports explicit congestion marking).

Note that there is no implication that each ’C’ marking is encoded the same; a different encoding might be used for the ’C’ marking in each protocol.

Finally, for completeness, we show the L3 marking arriving at the destination, where the host transport protocol (e.g. TCP) feeds it back to the source in the L4 acknowledgement (the ’C’ at L4 in the packet at the top of the diagram).

_ _ _ /______| | |C| ACK Packet (V) \ |_|_|_| +---+ layer: 2 3 4 header +---+ | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4 | | +---+ | ^ | | | ...... Packet U. . | >>|>>> Packet U >>>>>>>>>>>>|>^ |L3 | | +---+ +---+ | ^ | +---+ +---+ | | | | | *|>>>>>|>>>|>>>>>|>^ | | | | | | |L2 |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| source subnet A router subnet B dest ______| | | | | | | | |C| | | |C| | | |C|C| Data______\ |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| Packet (U) / layer: 4 3 2A 4 3 2A 4 3 4 3 2B header

Figure 1: Feed-Forward-and-Up Mode

Of course, modern networks are rarely as simple as this text-book example, often involving multiple nested layers. For example, a 3GPP mobile network may have two IP-in-IP (GTP [GTPv1]) tunnels in series and an MPLS backhaul between the base station and the first router. Nonetheless, the example illustrates the general idea of feeding congestion notification forward then upward whenever a header is removed at the egress of a subnet.

Briscoe & KaippallimalilExpires November 26, 2021 [Page 10] Internet-Draft ECN Encapsulation Guidelines May 2021

Note that the FECN (forward ECN ) bit in Frame Relay [Buck00] and the explicit forward congestion indication (EFCI [ITU-T.I.371]) bit in ATM user data cells follow a feed-forward pattern. However, in ATM, this arrangement is only part of a feed-forward-and-backward pattern at the lower layer, not feed-forward-and-up out of the lower layer-- the intention was never to interface to IP ECN at the subnet egress. To our knowledge, Frame Relay FECN is solely used to detect where more capacity should be provisioned.

3.2. Feed-Up-and-Forward Mode

Ethernet is particularly difficult to extend incrementally to support explicit congestion notification. One way to support ECN in such cases has been to use so called ’layer-3 switches’. These are Ethernet switches that dig into the Ethernet payload to find an IP header and manipulate or act on certain IP fields (specifically Diffserv & ECN). For instance, in Data Center TCP [RFC8257], layer-3 switches are configured to mark the ECN field of the IP header within the Ethernet payload when their output buffer becomes congested. With respect to switching, a layer-3 switch acts solely on the addresses in the Ethernet header; it does not use IP addresses, and it does not decrement the TTL field in the IP header.

_ _ _ /______| | |C| ACK packet (V) \ |_|_|_| +---+ layer: 2 3 4 header +---+ | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet V <<<<<<<<<<<<<|<< |L4 | | +---+ | ^ | | | . . . >>>> Packet U >>>|>>>|>>> Packet U >>>>>>>>>>>>|>^ |L3 | | +--^+ +---+ | | +---+ +---+ | | | | | *| | | | | | | | | | |L2 |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| source subnet E router subnet F dest ______| | | | | | | |C| | | | |C| | | |C|C| data______\ |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (U) / layer: 4 3 2 4 3 2 4 3 4 3 2 header

Figure 2: Feed-Up-and-Forward Mode

By comparing Figure 2 with Figure 1, it can be seen that subnet E (perhaps a subnet of layer-3 Ethernet switches) works in feed-up-and- forward mode by notifying congestion directly into L3 at the point of congestion, even though the congested switch does not otherwise act at L3. In this example, the technology in subnet F (e.g. MPLS) does

Briscoe & KaippallimalilExpires November 26, 2021 [Page 11] Internet-Draft ECN Encapsulation Guidelines May 2021

support ECN natively, so when the router adds the layer-2 header it copies the ECN marking from L3 to L2 as well.

3.3. Feed-Backward Mode

In some layer 2 technologies, explicit congestion notification has been defined for use internally within the subnet with its own feedback and load regulation, but typically the interface with IP for ECN has not been defined.

For instance, for the available bit-rate (ABR) service in ATM, the relative rate mechanism was one of the more popular mechanisms for managing traffic, tending to supersede earlier designs. In this approach ATM switches send special resource management (RM) cells in both the forward and backward directions to control the ingress rate of user data into a virtual circuit. If a switch buffer is approaching congestion or is congested it sends an RM cell back towards the ingress with respectively the No Increase (NI) or Congestion Indication (CI) bit set in its message type field [ATM-TM-ABR]. The ingress then holds or decreases its sending bit- rate accordingly.

Briscoe & KaippallimalilExpires November 26, 2021 [Page 12] Internet-Draft ECN Encapsulation Guidelines May 2021

_ _ _ /______| | |C| ACK packet (X) \ |_|_|_| +---+ layer: 2 3 4 header +---+ | <|<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Packet X <<<<<<<<<<<<<|<< |L4 | | +---+ | ^ | | | | *|>>> Packet W >>>>>>>>>>>>|>^ |L3 | | +---+ +---+ | | +---+ +---+ | | | | | | | | | <|<<<<<|<<<|<(V)<|<<<| | |L2 | | . . | . |Packet U | . . | . | . . | . | . . | .*| . . | |L2 |___|_____|___|_____|___|_____|___|_____|___|_____|___|_____|___| source subnet G router subnet H dest ______later | | | | | | | | | | | | | | | | |C| | data______\ |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (W) / 4 3 2 4 3 2 4 3 4 3 2 _ /__ |C| Feedback control \ |_| cell/frame (V) 2 ______earlier | | | | | | | | | | | | | | | | | | | data______\ |__|_|_|_| |__|_|_|_| |__|_|_| |__|_|_|_| packet (U) / layer: 4 3 2 4 3 2 4 3 4 3 2 header

Figure 3: Feed-Backward Mode

ATM’s feed-backward approach does not fit well when layered beneath IP’s feed-forward approach--unless the initial data source is the same node as the ATM ingress. Figure 3 shows the feed-backward approach being used in subnet H. If the final switch on the path is congested (*), it does not feed-forward any congestion indications on packet (U). Instead it sends a control cell (V) back to the router at the ATM ingress.

However, the backward feedback does not reach the original data source directly because IP does not support backward feedback (and subnet G is independent of subnet H). Instead, the router in the middle throttles down its sending rate but the original data sources don’t reduce their rates. The resulting rate mismatch causes the middle router’s buffer at layer 3 to back up until it becomes congested, which it signals forwards on later data packets at layer 3 (e.g. packet W). Note that the forward signal from the middle router is not triggered directly by the backward signal. Rather, it is triggered by congestion resulting from the middle router’s mismatched rate response to the backward signal.

Briscoe & KaippallimalilExpires November 26, 2021 [Page 13] Internet-Draft ECN Encapsulation Guidelines May 2021

In response to this later forward signalling, end-to-end feedback at layer-4 finally completes the tortuous path of congestion indications back to the origin data source, as before.

Quantized congestion notification (QCN [IEEE802.1Q]) would suffer from similar problems if extended to multiple subnets. However, from the start QCN was clearly characterized as solely applicable to a single subnet (see Section 6).

3.4. Null Mode

Often link and physical layer resources are ’non-blocking’ by design. In these cases congestion notification may be implemented but it does not need to be deployed at the lower layer; ECN in IP would be sufficient.

A degenerate example is a point-to-point Ethernet link. Excess loading of the link merely causes the queue from the higher layer to back up, while the lower layer remains immune to congestion. Even a whole meshed subnetwork can be made immune to interior congestion by limiting ingress capacity and sufficient sizing of interior links, e.g. a non-blocking fat-tree network [Leiserson85]. An alternative to fat links near the root is numerous thin links with multi-path routing to ensure even worst-case patterns of load cannot congest any link, e.g. a Clos network [Clos53].

4. Feed-Forward-and-Up Mode: Guidelines for Adding Congestion Notification

Feed-forward-and-up is the mode already used for signalling ECN up the layers through MPLS into IP [RFC5129] and through IP-in-IP tunnels [RFC6040], whether encapsulating with IPv4 [RFC2003], IPv6 [RFC2473] or IPsec [RFC4301]. These RFCs take a consistent approach and the following guidelines are designed to ensure this consistency continues as ECN support is added to other protocols that encapsulate IP. The guidelines are also designed to ensure compliance with the more general best current practice for the design of alternate ECN schemes given in [RFC4774] and extended by [RFC8311].

The rest of this section is structured as follows:

o Section 4.1 addresses the most straightforward cases, where [RFC6040] can be applied directly to add ECN to tunnels that are effectively IP-in-IP tunnels, but with shim header(s) between the IP headers.

o The subsequent sections give guidelines for adding ECN to a subnet technology that uses feed-forward-and-up mode like IP, but it is

Briscoe & KaippallimalilExpires November 26, 2021 [Page 14] Internet-Draft ECN Encapsulation Guidelines May 2021

not so similar to IP that [RFC6040] rules can be applied directly. Specifically:

* Sections 4.2, 4.3 and 4.4 respectively address how to add ECN support to the wire protocol and to the encapsulators and decapsulators at the ingress and egress of the subnet.

* Section 4.5 deals with the special, but common, case of sequences of tunnels or subnets that all use the same technology

* Section 4.6 deals with the question of reframing when IP packets do not map 1:1 into lower layer frames.

4.1. IP-in-IP Tunnels with Shim Headers

A common pattern for many tunnelling protocols is to encapsulate an inner IP header with shim header(s) then an outer IP header. A shim header is defined as one that is not sufficient alone to forward the packet as an outer header. Another common pattern is for a shim to encapsulate a layer 2 (L2) header, which in turn encapsulates (or might encapsulate) an IP header. [I-D.ietf-tsvwg-rfc6040update-shim] clarifies that RFC 6040 is just as applicable when there are shim(s) and possibly a L2 header between two IP headers.

However, it is not always feasible or necessary to propagate ECN between IP headers when separated by a shim. For instance, it might be too costly to dig to arbitrary depths to find an inner IP header, there may be little or no congestion within the tunnel by design (see null mode in Section 3.4 above), or a legacy implementation might not support ECN. In cases where a tunnel does not support ECN, it is important that the ingress does not copy the ECN field from an inner IP header to an outer. Therefore section 4 of [I-D.ietf-tsvwg-rfc6040update-shim] requires network operators to configure the ingress of a tunnel that does not support ECN so that it zeros the ECN field in the outer IP header.

Nonetheless, in many cases it is feasible to propagate the ECN field between IP headers separated by shim header(s) and/or a L2 header. Particularly in the typical case when the outer IP header and the shim(s) are added (or removed) as part of the same procedure. Even if the shim(s) encapsulate a L2 header, it is often possible to find an inner IP header within the L2 PDU and propagate ECN between that and the outer IP header. This can be thought of as a special case of the feed-up-and-forward mode (Section 3.2), so the guidelines for this mode apply (Section 5).

Briscoe & KaippallimalilExpires November 26, 2021 [Page 15] Internet-Draft ECN Encapsulation Guidelines May 2021

Numerous shim protocols have been defined for IP tunnelling. More recent ones e.g. Geneve [RFC8926] and Generic UDP Encapsulation (GUE) [I-D.ietf-intarea-gue] cite and follow RFC 6040. And some earlier ones, e.g. CAPWAP [RFC5415] and LISP [RFC6830], cite RFC 3168, which is compatible with RFC 6040.

However, as Section 9.3 of RFC 3168 pointed out, ECN support needs to be defined for many earlier shim-based tunnelling protocols, e.g. L2TPv2 [RFC2661], L2TPv3 [RFC3931], GRE [RFC2784], PPTP [RFC2637], GTP [GTPv1], [GTPv1-U], [GTPv2-C] and Teredo [RFC4380] as well as some recent ones, e.g. VXLAN [RFC7348], NVGRE [RFC7637] and NSH [RFC8300].

All these IP-based encapsulations can be updated in one shot by simple reference to RFC 6040. However, it would not be appropriate to update all these protocols from within the present guidance document. Instead a companion specification [I-D.ietf-tsvwg-rfc6040update-shim] has been prepared that has the appropriate standards track status to update standards track protocols. For those that are not under IETF change control [I-D.ietf-tsvwg-rfc6040update-shim] can only recommend that the relevant body updates them.

4.2. Wire Protocol Design: Indication of ECN Support

This section is intended to guide the redesign of any lower layer protocol that encapsulate IP to add native ECN support at the lower layer. It reflects the approaches used in [RFC6040] and in [RFC5129]. Therefore IP-in-IP tunnels or IP-in-MPLS or MPLS-in-MPLS encapsulations that already comply with [RFC6040] or [RFC5129] will already satisfy this guidance.

A lower layer (or subnet) congestion notification system:

1. SHOULD NOT apply explicit congestion notifications to PDUs that are destined for legacy layer-4 transport implementations that will not understand ECN, and

2. SHOULD NOT apply explicit congestion notifications to PDUs if the egress of the subnet might not propagate congestion notifications onward into the higher layer.

We use the term ECN-PDUs for a PDU on a feedback loop that will propagate congestion notification properly because it meets both the above criteria. And a Not-ECN-PDU is a PDU on a feedback loop that does not meet at least one of the criteria, and will therefore not propagate congestion notification properly. A

Briscoe & KaippallimalilExpires November 26, 2021 [Page 16] Internet-Draft ECN Encapsulation Guidelines May 2021

corollary of the above is that a lower layer congestion notification protocol:

3. SHOULD be able to distinguish ECN-PDUs from Not-ECN-PDUs.

Note that there is no need for all interior nodes within a subnet to be able to mark congestion explicitly. A mix of ECN and drop signals from different nodes is fine. However, if _any_ interior nodes might generate ECN markings, guideline 2 above says that all relevant egress node(s) SHOULD be able to propagate those markings up to the higher layer.

In IP, if the ECN field in each PDU is cleared to the Not-ECT (not ECN-capable transport) codepoint, it indicates that the L4 transport will not understand congestion markings. A congested buffer must not mark these Not-ECT PDUs, and therefore drops them instead.

The mechanism a lower layer uses to distinguish the ECN-capability of PDUs need not mimic that of IP. The above guidelines merely say that the lower layer system, as a whole, should achieve the same outcome. For instance, ECN-capable feedback loops might use PDUs that are identified by a particular set of labels or tags. Alternatively, logical link protocols that use flow state might determine whether a PDU can be congestion marked by checking for ECN-support in the flow state. Other protocols might depend on out-of-band control signals.

The per-domain checking of ECN support in MPLS [RFC5129] is a good example of a way to avoid sending congestion markings to L4 transports that will not understand them, without using any header space in the subnet protocol.

In MPLS, header space is extremely limited, therefore RFC5129 does not provide a field in the MPLS header to indicate whether the PDU is an ECN-PDU or a Not-ECN-PDU. Instead, interior nodes in a domain are allowed to set explicit congestion indications without checking whether the PDU is destined for a L4 transport that will understand them. Nonetheless, this is made safe by requiring that the network operator upgrades all decapsulating edges of a whole domain at once, as soon as even one switch within the domain is configured to mark rather than drop during congestion. Therefore, any edge node that might decapsulate a packet will be capable of checking whether the higher layer transport is ECN-capable. When decapsulating a CE- marked packet, if the decapsulator discovers that the higher layer (inner header) indicates the transport is not ECN-capable, it drops the packet--effectively on behalf of the earlier congested node (see Decapsulation Guideline 1 in Section 4.4).

Briscoe & KaippallimalilExpires November 26, 2021 [Page 17] Internet-Draft ECN Encapsulation Guidelines May 2021

It was only appropriate to define such an incremental deployment strategy because MPLS is targeted solely at professional operators, who can be expected to ensure that a whole subnetwork is consistently configured. This strategy might not be appropriate for other link technologies targeted at zero-configuration deployment or deployment by the general public (e.g. Ethernet). For such ’plug-and-play’ environments it will be necessary to invent a failsafe approach that ensures congestion markings will never fall into black holes, no matter how inconsistently a system is put together. Alternatively, congestion notification relying on correct system configuration could be confined to flavours of Ethernet intended only for professional network operators, such as Provider Backbone Bridges (PBB [IEEE802.1Q]; previously 802.1ah).

ECN support in TRILL [I-D.ietf-trill-ecn-support] provides a good example of how to add ECN to a lower layer protocol without relying on careful and consistent operator configuration. TRILL provides an extension header word with space for flags of different categories depending on whether logic to understand the extension is critical. The congestion experienced marking has been defined as a ’critical ingress-to-egress’ flag. So if a transit RBridge sets this flag and an egress RBridge does not have any logic to process it, it will drop it; which is the desired default action anyway. Therefore TRILL RBridges can be updated with support for ECN in no particular order and, at the egress of the TRILL campus, congestion notification will be propagated to IP as ECN whenever ECN logic has been implemented, or as drop otherwise.

QCN [IEEE802.1Q] is not intended to extend beyond a single subnet, or to interoperate with ECN. Nonetheless, the way QCN indicates to lower layer devices that the end-points will not understand QCN provides another example that a lower layer protocol designer might be able to mimic for their scenario. An operator can define certain Priority Code Points (PCPs [IEEE802.1Q]; previously 802.1p) to indicate non-QCN frames and an ingress bridge is required to map arriving not-QCN-capable IP packets to one of these non-QCN PCPs.

4.3. Encapsulation Guidelines

This section is intended to guide the redesign of any node that encapsulates IP with a lower layer header when adding native ECN support to the lower layer protocol. It reflects the approaches used in [RFC6040] and in [RFC5129]. Therefore IP-in-IP tunnels or IP-in- MPLS or MPLS-in-MPLS encapsulations that already comply with [RFC6040] or [RFC5129] will already satisfy this guidance.

1. Egress Capability Check: A subnet ingress needs to be sure that the corresponding egress of a subnet will propagate any

Briscoe & KaippallimalilExpires November 26, 2021 [Page 18] Internet-Draft ECN Encapsulation Guidelines May 2021

congestion notification added to the outer header across the subnet. This is necessary in addition to checking that an incoming PDU indicates an ECN-capable (L4) transport. Examples of how this guarantee might be provided include:

* by configuration (e.g. if any label switches in a domain support ECN marking, [RFC5129] requires all egress nodes to have been configured to propagate ECN)

* by the ingress explicitly checking that the egress propagates ECN (e.g. an early attempt to add ECN support to TRILL used IS-IS to check path capabilities before adding ECN extension flags to each frame [RFC7780]).

* by inherent design of the protocol (e.g. by encoding ECN marking on the outer header in such a way that a legacy egress that does not understand ECN will consider the PDU corrupt or invalid and discard it, thus at least propagating a form of congestion signal).

2. Egress Fails Capability Check: If the ingress cannot guarantee that the egress will propagate congestion notification, the ingress SHOULD disable ECN at the lower layer when it forwards the PDU. An example of how the ingress might disable ECN at the lower layer would be by setting the outer header of the PDU to identify it as a Not-ECN-PDU, assuming the subnet technology supports such a concept.

3. Standard Congestion Monitoring Baseline: Once the ingress to a subnet has established that the egress will correctly propagate ECN, on encapsulation it SHOULD encode the same level of congestion in outer headers as is arriving in incoming headers. For example it might copy any incoming congestion notification into the outer header of the lower layer protocol.

This ensures that bulk congestion monitoring of outer headers (e.g. by a network management node monitoring ECN in passing frames) will measure congestion accumulated along the whole upstream path - since the Load Regulator not just since the ingress of the subnet. A node that is not the Load Regulator SHOULD NOT re-initialize the level of CE markings in the outer to zero.

It would still also be possible to measure congestion introduced across one subnet (or tunnel) by subtracting the level of CE markings on inner headers from that on outer headers (see Appendix C of [RFC6040]). For example:

Briscoe & KaippallimalilExpires November 26, 2021 [Page 19] Internet-Draft ECN Encapsulation Guidelines May 2021

* If this guideline has been followed and if the level of CE markings is 0.4% on the outer and 0.1% on the inner, 0.4% congestion has been introduced across all the networks since the load regulator, and 0.3% (= 0.4% - 0.1%) has been introduced since the ingress to the current subnet (or tunnel);

* Without this guideline, if the subnet ingress had reinitialized the outer congestion level to zero, the outer and inner would measure 0.1% and 0.3%. It would still be possible to infer that the congestion introduced since the Load Regulator was 0.4% (= 0.1% + 0.3%). But only if the monitoring system somehow knows whether the subnet ingress reinitialized the congestion level.

As long as subnet and tunnel technologies use the standard congestion monitoring baseline in this guideline, monitoring systems will know to use the former approach, rather than having to "somehow know" which approach to use.

4.4. Decapsulation Guidelines

This section is intended to guide the redesign of any node that decapsulates IP from within a lower layer header when adding native ECN support to the lower layer protocol. It reflects the approaches used in [RFC6040] and in [RFC5129]. Therefore IP-in-IP tunnels or IP-in-MPLS or MPLS-in-MPLS encapsulations that already comply with [RFC6040] or [RFC5129] will already satisfy this guidance.

A subnet egress SHOULD NOT simply copy congestion notification from outer headers to the forwarded header. It SHOULD calculate the outgoing congestion notification field from the inner and outer headers using the following guidelines. If there is any conflict, rules earlier in the list take precedence over rules later in the list:

1. If the arriving inner header is a Not-ECN-PDU it implies the L4 transport will not understand explicit congestion markings. Then:

* If the outer header carries an explicit congestion marking, drop is the only indication of congestion that the L4 transport will understand. If the congestion marking is the most severe possible, the packet MUST be dropped. However, if congestion can be marked with multiple levels of severity and the packet’s marking is not the most severe, this requirement can be relaxed to: the packet SHOULD be dropped.

Briscoe & KaippallimalilExpires November 26, 2021 [Page 20] Internet-Draft ECN Encapsulation Guidelines May 2021

* If the outer is an ECN-PDU that carries no indication of congestion or a Not-ECN-PDU the PDU SHOULD be forwarded, but still as a Not-ECN-PDU.

2. If the outer header does not support explicit congestion notification (a Not-ECN-PDU), but the inner header does (an ECN- PDU), the inner header SHOULD be forwarded unchanged.

3. In some lower layer protocols congestion may be signalled as a numerical level, such as in the control frames of quantized congestion notification (QCN [IEEE802.1Q]). If such a multi-bit encoding encapsulates an ECN-capable IP data packet, a function will be needed to convert the quantized congestion level into the frequency of congestion markings in outgoing IP packets.

4. Congestion indications might be encoded by a severity level. For instance increasing levels of congestion might be encoded by numerically increasing indications, e.g. pre-congestion notification (PCN) can be encoded in each PDU at three severity levels in IP or MPLS [RFC6660] and the default encapsulation and decapsulation rules [RFC6040] are compatible with this interpretation of the ECN field.

If the arriving inner header is an ECN-PDU, where the inner and outer headers carry indications of congestion of different severity, the more severe indication SHOULD be forwarded in preference to the less severe.

5. The inner and outer headers might carry a combination of congestion notification fields that should not be possible given any currently used protocol transitions. For instance, if Encapsulation Guideline 3 in Section 4.3 had been followed, it should not be possible to have a less severe indication of congestion in the outer than in the inner. It MAY be appropriate to log unexpected combinations of headers and possibly raise an alarm.

If a safe outgoing codepoint can be defined for such a PDU, the PDU SHOULD be forwarded rather than dropped. Some implementers discard PDUs with currently unused combinations of headers just in case they represent an attack. However, an approach using alarms and policy-mediated drop is preferable to hard-coded drop, so that operators can keep track of possible attacks but currently unused combinations are not precluded from future use through new standards actions.

Briscoe & KaippallimalilExpires November 26, 2021 [Page 21] Internet-Draft ECN Encapsulation Guidelines May 2021

4.5. Sequences of Similar Tunnels or Subnets

In some deployments, particularly in 3GPP networks, an IP packet may traverse two or more IP-in-IP tunnels in sequence that all use identical technology (e.g. GTP).

In such cases, it would be sufficient for every encapsulation and decapsulation in the chain to comply with RFC 6040. Alternatively, as an optimisation, a node that decapsulates a packet and immediately re-encapsulates it for the next tunnel MAY copy the incoming outer ECN field directly to the outgoing outer and the incoming inner ECN field directly to the outgoing inner. Then the overall behavior across the sequence of tunnel segments would still be consistent with RFC 6040.

Appendix C of RFC6040 describes how a tunnel egress can monitor how much congestion has been introduced within a tunnel. A network operator might want to monitor how much congestion had been introduced within a whole sequence of tunnels. Using the technique in Appendix C of RFC6040 at the final egress, the operator could monitor the whole sequence of tunnels, but only if the above optimisation were used consistently along the sequence of tunnels, in order to make it appear as a single tunnel. Therefore, tunnel endpoint implementations SHOULD allow the operator to configure whether this optimisation is enabled.

When ECN support is added to a subnet technology, consideration SHOULD be given to a similar optimisation between subnets in sequence if they all use the same technology.

4.6. Reframing and Congestion Markings

The guidance in this section is worded in terms of framing boundaries, but it applies equally whether the protocol data units are frames, cells or packets.

Where an AQM marks the ECN field of IP packets as they queue into a layer-2 link, there will be no problem with framing boundaries, because the ECN markings would be applied directly to IP packets. The guidance in this section is only applicable where an ECN capability is being added to a layer-2 protocol so that layer-2 frames can be ECN-marked by an AQM at layer-2. This would only be necessary where AQM will be applied at pure layer-2 nodes (without IP-awareness).

When layer-2 frame headers are stripped off and IP PDUs with different boundaries are forwarded, the provisions in RFC7141 for handling congestion indications when splitting or merging packets

Briscoe & KaippallimalilExpires November 26, 2021 [Page 22] Internet-Draft ECN Encapsulation Guidelines May 2021

apply (see Section 2.4 of [RFC7141]. Those provisions include: "The general rule to follow is that the number of octets in packets with congestion indications SHOULD be equivalent before and after merging or splitting." See RFC 7141 for the complete provisions and related discussion, including an exception to that general rule.

As also recommended in RFC 7141, the mechanism for propagating congestion indications SHOULD ensure that any new incoming congestion indication is propagated immediately, and not held awaiting possible arrival of further congestion indications sufficient to indicate congestion for all of the octets of an outgoing IP PDU.

5. Feed-Up-and-Forward Mode: Guidelines for Adding Congestion Notification

The guidance in this section is applicable, for example, when IP packets:

o are encapsulated in Ethernet headers, which have no support for ECN;

o are forwarded by the eNode-B (base station) of a 3GPP radio access network, which is required to apply ECN marking during congestion, [LTE-RA], [UTRAN], but the Packet Data Convergence Protocol (PDCP) that encapsulates the IP header over the radio access has no support for ECN.

This guidance also generalizes to encapsulation by other subnet technologies with no native support for explicit congestion notification at the lower layer, but with support for finding and processing an IP header. It is unlikely to be applicable or necessary for IP-in-IP encapsulation, where feed-forward-and-up mode based on [RFC6040] would be more appropriate.

Marking the IP header while switching at layer-2 (by using a layer-3 switch) or while forwarding in a radio access network seems to represent a layering violation. However, it can be considered as a benign optimisation if the guidelines below are followed. Feed-up- and-forward is certainly not a general alternative to implementing feed-forward congestion notification in the lower layer, because:

o IPv4 and IPv6 are not the only layer-3 protocols that might be encapsulated by lower layer protocols

o Link-layer encryption might be in use, making the layer-2 payload inaccessible

Briscoe & KaippallimalilExpires November 26, 2021 [Page 23] Internet-Draft ECN Encapsulation Guidelines May 2021

o Many Ethernet switches do not have ’layer-3 switch’ capabilities so they cannot read or modify an IP payload

o It might be costly to find an IP header (v4 or v6) when it may be encapsulated by more than one lower layer header, e.g. Ethernet MAC in MAC ([IEEE802.1Q]; previously 802.1ah).

Nonetheless, configuring lower layer equipment to look for an ECN field in an encapsulated IP header is a useful optimisation. If the implementation follows the guidelines below, this optimisation does not have to be confined to a controlled environment such as within a data centre; it could usefully be applied on any network--even if the operator is not sure whether the above issues will never apply:

1. If a native lower-layer congestion notification mechanism exists for a subnet technology, it is safe to mix feed-up-and-forward with feed-forward-and-up on other switches in the same subnet. However, it will generally be more efficient to use the native mechanism.

2. The depth of the search for an IP header SHOULD be limited. If an IP header is not found soon enough, or an unrecognized or unreadable header is encountered, the switch SHOULD resort to an alternative means of signalling congestion (e.g. drop, or the native lower layer mechanism if available).

3. It is sufficient to use the first IP header found in the stack; the egress of the relevant tunnel can propagate congestion notification upwards to any more deeply encapsulated IP headers later.

6. Feed-Backward Mode: Guidelines for Adding Congestion Notification

It can be seen from Section 3.3 that congestion notification in a subnet using feed-backward mode has generally not been designed to be directly coupled with IP layer congestion notification. The subnet attempts to minimize congestion internally, and if the incoming load at the ingress exceeds the capacity somewhere through the subnet, the layer 3 buffer into the ingress backs up. Thus, a feed-backward mode subnet is in some sense similar to a null mode subnet, in that there is no need for any direct interaction between the subnet and higher layer congestion notification. Therefore no detailed protocol design guidelines are appropriate. Nonetheless, a more general guideline is appropriate:

A subnetwork technology intended to eventually interface to IP SHOULD NOT be designed using only the feed-backward mode, which is certainly best for a stand-alone subnet, but would need to be

Briscoe & KaippallimalilExpires November 26, 2021 [Page 24] Internet-Draft ECN Encapsulation Guidelines May 2021

modified to work efficiently as part of the wider Internet, because IP uses feed-forward-and-up mode.

The feed-backward approach at least works beneath IP, where the term ’works’ is used only in a narrow functional sense because feed- backward can result in very inefficient and sluggish congestion control--except if it is confined to the subnet directly connected to the original data source, when it is faster than feed-forward. It would be valid to design a protocol that could work in feed-backward mode for paths that only cross one subnet, and in feed-forward-and-up mode for paths that cross subnets.

In the early days of TCP/IP, a similar feed-backward approach was tried for explicit congestion signalling, using source-quench (SQ) ICMP control packets. However, SQ fell out of favour and is now formally deprecated [RFC6633]. The main problem was that it is hard for a data source to tell the difference between a spoofed SQ message and a quench request from a genuine buffer on the path. It is also hard for a lower layer buffer to address an SQ message to the original source port number, which may be buried within many layers of headers, and possibly encrypted.

QCN (also known as backward congestion notification, BCN; see Sections 30--33 of [IEEE802.1Q]; previously known as 802.1Qau) uses a feed-backward mode structurally similar to ATM’s relative rate mechanism. However, QCN confines its applicability to scenarios such as some data centres where all endpoints are directly attached by the same Ethernet technology. If a QCN subnet were later connected into a wider IP-based internetwork (e.g. when attempting to interconnect multiple data centres) it would suffer the inefficiency shown in Figure 3.

7. IANA Considerations

This memo includes no request to IANA.

8. Security Considerations

If a lower layer wire protocol is redesigned to include explicit congestion signalling in-band in the protocol header, care SHOULD be take to ensure that the field used is specified as mutable during transit. Otherwise interior nodes signalling congestion would invalidate any authentication protocol applied to the lower layer header--by altering a header field that had been assumed as immutable.

The redesign of protocols that encapsulate IP in order to propagate congestion signals between layers raises potential signal integrity

Briscoe & KaippallimalilExpires November 26, 2021 [Page 25] Internet-Draft ECN Encapsulation Guidelines May 2021

concerns. Experimental or proposed approaches exist for assuring the end-to-end integrity of in-band congestion signals, e.g.:

o Congestion exposure (ConEx ) for networks to audit that their congestion signals are not being suppressed by other networks or by receivers, and for networks to police that senders are responding sufficiently to the signals, irrespective of the L4 transport protocol used [RFC7713].

o A test for a sender to detect whether a network or the receiver is suppressing congestion signals (for example see 2nd para of Section 20.2 of [RFC3168]).

Given these end-to-end approaches are already being specified, it would make little sense to attempt to design hop-by-hop congestion signal integrity into a new lower layer protocol, because end-to-end integrity inherently achieves hop-by-hop integrity.

Section 6 gives vulnerability to spoofing as one of the reasons for deprecating feed-backward mode.

9. Conclusions

Following the guidance in this document enables ECN support to be extended to numerous protocols that encapsulate IP (v4 & v6) in a consistent way, so that IP continues to fulfil its role as an end-to- end interoperability layer. This includes:

o A wide range of tunnelling protocols including those with various forms of shim header between two IP headers, possibly also separated by a L2 header;

o A wide range of subnet technologies, particularly those that work in the same ’feed-forward-and-up’ mode that is used to support ECN in IP and MPLS.

Guidelines have been defined for supporting propagation of ECN between Ethernet and IP on so-called Layer-3 Ethernet switches, using a ’feed-up-and-forward’ mode. This approach could enable other subnet technologies to pass ECN signals into the IP layer, even if they do not support ECN natively.

Finally, attempting to add ECN to a subnet technology in feed- backward mode is deprecated except in special cases, due to its likely sluggish response to congestion.

Briscoe & KaippallimalilExpires November 26, 2021 [Page 26] Internet-Draft ECN Encapsulation Guidelines May 2021

10. Acknowledgements

Thanks to Gorry Fairhurst and David Black for extensive reviews. Thanks also to the following reviewers: Joe Touch, Andrew McGregor, Richard Scheffenegger, Ingemar Johansson, Piers O’Hanlon, Donald Eastlake, Jonathan Morton and Michael Welzl, who pointed out that lower layer congestion notification signals may have different semantics to those in IP. Thanks are also due to the tsvwg chairs, TSV ADs and IETF liaison people such as Eric Gray, Dan Romascanu and Gonzalo Camarillo for helping with the liaisons with the IEEE and 3GPP. And thanks to Georg Mayer and particularly to Erik Guttman for the extensive search and categorisation of any 3GPP specifications that cite ECN specifications.

Bob Briscoe was part-funded by the European Community under its Seventh Framework Programme through the Trilogy project (ICT-216372) for initial drafts and through the Reducing Internet Transport Latency (RITE) project (ICT-317700) subsequently. The views expressed here are solely those of the authors.

11. Contributors

Pat Thaler Broadcom Corporation (retired) CA USA

Pat was a co-author of this draft, but retired before its publication.

12. Comments Solicited

Comments and questions are encouraged and very welcome. They can be addressed to the IETF Transport Area working group mailing list , and/or to the authors.

13. References

13.1. Normative References

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, .

Briscoe & KaippallimalilExpires November 26, 2021 [Page 27] Internet-Draft ECN Encapsulation Guidelines May 2021

[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition of Explicit Congestion Notification (ECN) to IP", RFC 3168, DOI 10.17487/RFC3168, September 2001, .

[RFC3819] Karn, P., Ed., Bormann, C., Fairhurst, G., Grossman, D., Ludwig, R., Mahdavi, J., Montenegro, G., Touch, J., and L. Wood, "Advice for Internet Subnetwork Designers", BCP 89, RFC 3819, DOI 10.17487/RFC3819, July 2004, .

[RFC4774] Floyd, S., "Specifying Alternate Semantics for the Explicit Congestion Notification (ECN) Field", BCP 124, RFC 4774, DOI 10.17487/RFC4774, November 2006, .

[RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion Marking in MPLS", RFC 5129, DOI 10.17487/RFC5129, January 2008, .

[RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion Notification", RFC 6040, DOI 10.17487/RFC6040, November 2010, .

[RFC7141] Briscoe, B. and J. Manner, "Byte and Packet Congestion Notification", BCP 41, RFC 7141, DOI 10.17487/RFC7141, February 2014, .

13.2. Informative References

[ATM-TM-ABR] Cisco, "Understanding the Available Bit Rate (ABR) Service Category for ATM VCs", Design Technote 10415, June 2005.

[Buck00] Buckwalter, J., "Frame Relay: Technology and Practice", Pub. Addison Wesley ISBN-13: 978-0201485240, 2000.

[Clos53] Clos, C., "A Study of Non-Blocking Switching Networks", Bell Systems Technical Journal 32(2):406--424, March 1953.

[GTPv1] 3GPP, "GPRS Tunnelling Protocol (GTP) across the Gn and Gp interface", Technical Specification TS 29.060.

[GTPv1-U] 3GPP, "General Packet Radio System (GPRS) Tunnelling Protocol User Plane (GTPv1-U)", Technical Specification TS 29.281.

Briscoe & KaippallimalilExpires November 26, 2021 [Page 28] Internet-Draft ECN Encapsulation Guidelines May 2021

[GTPv2-C] 3GPP, "Evolved General Packet Radio Service (GPRS) Tunnelling Protocol for Control plane (GTPv2-C)", Technical Specification TS 29.274.

[I-D.ietf-intarea-gue] Herbert, T., Yong, L., and O. Zia, "Generic UDP Encapsulation", draft-ietf-intarea-gue-09 (work in progress), October 2019.

[I-D.ietf-trill-ecn-support] Eastlake, D. E. and B. Briscoe, "TRILL (TRansparent Interconnection of Lots of Links): ECN (Explicit Congestion Notification) Support", draft-ietf-trill-ecn- support-07 (work in progress), February 2018.

[I-D.ietf-tsvwg-rfc6040update-shim] Briscoe, B., "Propagating Explicit Congestion Notification Across IP Tunnel Headers Separated by a Shim", draft-ietf- tsvwg-rfc6040update-shim-13 (work in progress), March 2021.

[IEEE802.1Q] IEEE, "IEEE Standard for Local and Metropolitan Area Networks--Virtual Bridged Local Area Networks--Amendment 6: Provider Backbone Bridges", IEEE Std 802.1Q-2018, July 2018, .

[ITU-T.I.371] ITU-T, "Traffic Control and Congestion Control in B-ISDN", ITU-T Rec. I.371 (03/04), March 2004, .

[Leiserson85] Leiserson, C., "Fat-trees: universal networks for hardware-efficient supercomputing", IEEE Transactions on Computers 34(10):892-901, October 1985.

[LTE-RA] 3GPP, "Evolved Universal Terrestrial Radio Access (E-UTRA) and Evolved Universal Terrestrial Radio Access Network (E-UTRAN); Overall description; Stage 2", Technical Specification TS 36.300.

Briscoe & KaippallimalilExpires November 26, 2021 [Page 29] Internet-Draft ECN Encapsulation Guidelines May 2021

[RFC2003] Perkins, C., "IP Encapsulation within IP", RFC 2003, DOI 10.17487/RFC2003, October 1996, .

[RFC2473] Conta, A. and S. Deering, "Generic Packet Tunneling in IPv6 Specification", RFC 2473, DOI 10.17487/RFC2473, December 1998, .

[RFC2637] Hamzeh, K., Pall, G., Verthein, W., Taarud, J., Little, W., and G. Zorn, "Point-to-Point Tunneling Protocol (PPTP)", RFC 2637, DOI 10.17487/RFC2637, July 1999, .

[RFC2661] Townsley, W., Valencia, A., Rubens, A., Pall, G., Zorn, G., and B. Palter, "Layer Two Tunneling Protocol "L2TP"", RFC 2661, DOI 10.17487/RFC2661, August 1999, .

[RFC2784] Farinacci, D., Li, T., Hanks, S., Meyer, D., and P. Traina, "Generic Routing Encapsulation (GRE)", RFC 2784, DOI 10.17487/RFC2784, March 2000, .

[RFC2884] Hadi Salim, J. and U. Ahmed, "Performance Evaluation of Explicit Congestion Notification (ECN) in IP Networks", RFC 2884, DOI 10.17487/RFC2884, July 2000, .

[RFC2983] Black, D., "Differentiated Services and Tunnels", RFC 2983, DOI 10.17487/RFC2983, October 2000, .

[RFC3931] Lau, J., Ed., Townsley, M., Ed., and I. Goyret, Ed., "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", RFC 3931, DOI 10.17487/RFC3931, March 2005, .

[RFC4301] Kent, S. and K. Seo, "Security Architecture for the Internet Protocol", RFC 4301, DOI 10.17487/RFC4301, December 2005, .

[RFC4380] Huitema, C., "Teredo: Tunneling IPv6 over UDP through Network Address Translations (NATs)", RFC 4380, DOI 10.17487/RFC4380, February 2006, .

Briscoe & KaippallimalilExpires November 26, 2021 [Page 30] Internet-Draft ECN Encapsulation Guidelines May 2021

[RFC5415] Calhoun, P., Ed., Montemurro, M., Ed., and D. Stanley, Ed., "Control And Provisioning of Wireless Access Points (CAPWAP) Protocol Specification", RFC 5415, DOI 10.17487/RFC5415, March 2009, .

[RFC6633] Gont, F., "Deprecation of ICMP Source Quench Messages", RFC 6633, DOI 10.17487/RFC6633, May 2012, .

[RFC6660] Briscoe, B., Moncaster, T., and M. Menth, "Encoding Three Pre-Congestion Notification (PCN) States in the IP Header Using a Single Diffserv Codepoint (DSCP)", RFC 6660, DOI 10.17487/RFC6660, July 2012, .

[RFC6830] Farinacci, D., Fuller, V., Meyer, D., and D. Lewis, "The Locator/ID Separation Protocol (LISP)", RFC 6830, DOI 10.17487/RFC6830, January 2013, .

[RFC7323] Borman, D., Braden, B., Jacobson, V., and R. Scheffenegger, Ed., "TCP Extensions for High Performance", RFC 7323, DOI 10.17487/RFC7323, September 2014, .

[RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, L., Sridhar, T., Bursell, M., and C. Wright, "Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014, .

[RFC7567] Baker, F., Ed. and G. Fairhurst, Ed., "IETF Recommendations Regarding Active Queue Management", BCP 197, RFC 7567, DOI 10.17487/RFC7567, July 2015, .

[RFC7637] Garg, P., Ed. and Y. Wang, Ed., "NVGRE: Network Virtualization Using Generic Routing Encapsulation", RFC 7637, DOI 10.17487/RFC7637, September 2015, .

[RFC7713] Mathis, M. and B. Briscoe, "Congestion Exposure (ConEx) Concepts, Abstract Mechanism, and Requirements", RFC 7713, DOI 10.17487/RFC7713, December 2015, .

Briscoe & KaippallimalilExpires November 26, 2021 [Page 31] Internet-Draft ECN Encapsulation Guidelines May 2021

[RFC7780] Eastlake 3rd, D., Zhang, M., Perlman, R., Banerjee, A., Ghanwani, A., and S. Gupta, "Transparent Interconnection of Lots of Links (TRILL): Clarifications, Corrections, and Updates", RFC 7780, DOI 10.17487/RFC7780, February 2016, .

[RFC8084] Fairhurst, G., "Network Transport Circuit Breakers", BCP 208, RFC 8084, DOI 10.17487/RFC8084, March 2017, .

[RFC8087] Fairhurst, G. and M. Welzl, "The Benefits of Using Explicit Congestion Notification (ECN)", RFC 8087, DOI 10.17487/RFC8087, March 2017, .

[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, .

[RFC8257] Bensley, S., Thaler, D., Balasubramanian, P., Eggert, L., and G. Judd, "Data Center TCP (DCTCP): TCP Congestion Control for Data Centers", RFC 8257, DOI 10.17487/RFC8257, October 2017, .

[RFC8300] Quinn, P., Ed., Elzur, U., Ed., and C. Pignataro, Ed., "Network Service Header (NSH)", RFC 8300, DOI 10.17487/RFC8300, January 2018, .

[RFC8311] Black, D., "Relaxing Restrictions on Explicit Congestion Notification (ECN) Experimentation", RFC 8311, DOI 10.17487/RFC8311, January 2018, .

[RFC8926] Gross, J., Ed., Ganga, I., Ed., and T. Sridhar, Ed., "Geneve: Generic Network Virtualization Encapsulation", RFC 8926, DOI 10.17487/RFC8926, November 2020, .

[UTRAN] 3GPP, "UTRAN Overall Description", Technical Specification TS 25.401.

Briscoe & KaippallimalilExpires November 26, 2021 [Page 32] Internet-Draft ECN Encapsulation Guidelines May 2021

Appendix A. Changes in This Version (to be removed by RFC Editor)

From ietf-12 to ietf-13

* Following 3rd tsvwg WGLC:

+ Formalized update to RFC 3819 in its own subsection (1.1) and referred to it in the abstract

+ Scope: Clarified that the specification of alternative ECN semantics using ECT(1) was not in RFC 4774, but rather in RFC 8311, and that the problem with using a DSCP to indicate alternative semantics has issues at domain boundaries as well as tunnels.

+ Terminology: tighted up definitions of ECN-PDU and Not-ECN- PDU, and removed definition of Congestion Baseline, given it was only used once.

+ Mentioned QCN where feed-backward is first introduced (S.3), referring forward to where it is discussed more deeply (S.4).

+ Clarified that IS-IS solution to adding ECN support to TRILL was not pursued

+ Completely rewrote the rationale for the guideline about a Standard Congestion Monitoring Baseline, to focus on standardization of the otherwise unknown scenario used, rather than the relative usefulness of the info in each approach

+ Explained the re-framing problem better and added fragmentation as another possible cause of the problem

+ Acknowledged new reviewers

+ Updated references, replaced citations of 802.1Qau and 802.1ah with rolled up 802.1Q, and added citations of Fat trees and Clos Networks

+ Numerous other editorial improvements

From ietf-11 to ietf-12

* Updated references

From ietf-10 to ietf-11

Briscoe & KaippallimalilExpires November 26, 2021 [Page 33] Internet-Draft ECN Encapsulation Guidelines May 2021

* Removed short section (was 3) ’Guidelines for All Cases’ because it was out of scope, being covered by RFC 4774. Expanded the Scope section (1.2) to explain all this. Explained that the default encap/decap rules already support certain alternative semantics, particularly all three of the alternative semantics for ECT(1): equivalent to ECT(0) , higher severity than ECT(0), and unmarked but implying different marking semantics from ECT(0).

* Clarified why the QCN example was being given even though not about increment deployment of ECN

* Pointed to the spoofing issue with feed-backward mode from the Security Considerations section, to aid security review.

* Removed any ambiguity in the word ’transport’ throughout

From ietf-09 to ietf-10

* Updated section 5.1 on "IP-in-IP tunnels with Shim Headers" to be consistent with updates to draft-ietf-tsvwg-rfc6040update- shim.

* Removed reference to the ECN nonce, which has been made historic by RFC 8311

* Removed "Open Issues" Appendix, given all have been addressed.

From ietf-08 to ietf-09

* Updated para in Intro that listed all the IP-in-IP tunnelling protocols, to instead refer to draft-ietf-tsvwg-rfc6040update- shim

* Updated section 5.1 on "IP-in-IP tunnels with Shim Headers" to summarize guidance that has evolved as rfc6040update-shim has developed.

From ietf-07 to ietf-08: Refreshed to avoid expiry. Updated references.

From ietf-06 to ietf-07:

* Added the people involved in liaisons to the acknowledgements.

From ietf-05 to ietf-06:

Briscoe & KaippallimalilExpires November 26, 2021 [Page 34] Internet-Draft ECN Encapsulation Guidelines May 2021

* Introduction: Added GUE and Geneve as examples of tightly coupled shims between IP headers that cite RFC 6040. And added VXLAN to list of those that do not.

* Replaced normative text about tightly coupled shims between IP headers, with reference to new draft-ietf-tsvwg-rfc6040update- shim

* Wire Protocol Design: Indication of ECN Support: Added TRILL as an example of a well-design protocol that does not need an indication of ECN support in the wire protocol.

* Encapsulation Guidelines: In the case of a Not-ECN-PDU with a CE outer, replaced SHOULD be dropped, with explanations of when SHOULD or MUST are appropriate.

* Feed-Up-and-Forward Mode: Explained examples more carefully, referred to PDCP and cited UTRAN spec as well as E-UTRAN.

* Updated references.

* Marked open issues as resolved, but did not delete Open Issues Appendix (yet).

From ietf-04 to ietf-05:

* Explained why tightly coupled shim headers only "SHOULD" comply with RFC 6040, not "MUST".

* Updated references

From ietf-03 to ietf-04:

* Addressed Richard Scheffenegger’s review comments: primarily editorial corrections, and addition of examples for clarity.

From ietf-02 to ietf-03:

* Updated references, ad cited RFC4774.

From ietf-01 to ietf-02:

* Added Section for guidelines that are applicable in all cases.

* Updated references.

From ietf-00 to ietf-01: Updated references.

Briscoe & KaippallimalilExpires November 26, 2021 [Page 35] Internet-Draft ECN Encapsulation Guidelines May 2021

From briscoe-04 to ietf-00: Changed filename following tsvwg adoption.

From briscoe-03 to 04:

* Re-arranged the introduction to describe the purpose of the document first before introducing ECN in more depth. And clarified the introduction throughout.

* Added applicability to 3GPP TS 36.300.

From briscoe-02 to 03:

* Scope section:

+ Added dependence on correct propagation of traffic class information

+ For the feed-backward mode, deemed multicast and anycast out of scope

* Ensured all guidelines referring to subnet technologies also refer to tunnels and vice versa by adding applicability sentences at the start of sections 4.1, 4.2, 4.3, 4.4, 4.6 and 5.

* Added Security Considerations on ensuring congestion signal fields are classed as immutable and on using end-to-end congestion signal integrity technologies rather than hop-by- hop.

From briscoe-01 to 02:

* Added authors: JK & PT

* Added

+ Section 4.1 "IP-in-IP Tunnels with Tightly Coupled Shim Headers"

+ Section 4.5 "Sequences of Similar Tunnels or Subnets"

+ roadmap at the start of Section 4, given the subsections have become quite fragmented.

+ Section 9 "Conclusions"

Briscoe & KaippallimalilExpires November 26, 2021 [Page 36] Internet-Draft ECN Encapsulation Guidelines May 2021

* Clarified why transports are starting to be able to saturate interior links

* Under Section 1.1, addressed the question of alternative signal semantics and included multicast & anycast.

* Under Section 3.1, included a 3GPP example.

* Section 4.2. "Wire Protocol Design":

+ Altered guideline 2. to make it clear that it only applies to the immediate subnet egress, not later ones

+ Added a reminder that it is only necessary to check that ECN propagates at the egress, not whether interior nodes mark ECN

+ Added example of how QCN uses 802.1p to indicate support for QCN.

* Added references to Appendix C of RFC6040, about monitoring the amount of congestion signals introduced within a tunnel

* Appendix A: Added more issues to be addressed, including plan to produce a standards track update to IP-in-IP tunnel protocols.

* Updated acks and references

From briscoe-00 to 01:

* Intended status: BCP (was Informational) & updates 3819 added.

* Briefer Introduction: Introductory para justifying benefits of ECN. Moved all but a brief enumeration of modes of operation to their own new section (from both Intro & Scope). Introduced incr. deployment as most tricky part.

* Tightened & added to terminology section

* Structured with Modes of Operation, then Guidelines section for each mode.

* Tightened up guideline text to remove vagueness / passive voice / ambiguity and highlight main guidelines as numbered items.

* Added Outstanding Document Issues Appendix

Briscoe & KaippallimalilExpires November 26, 2021 [Page 37] Internet-Draft ECN Encapsulation Guidelines May 2021

* Updated references

Authors’ Addresses

Bob Briscoe Independent UK

EMail: [email protected] URI: http://bobbriscoe.net/

John Kaippallimalil Futurewei 5700 Tennyson Parkway, Suite 600 Plano, Texas 75024 USA

EMail: [email protected]

Briscoe & KaippallimalilExpires November 26, 2021 [Page 38] Transport Services (tsv) K. De Schepper Internet-Draft Nokia Bell Labs Intended status: Experimental B. Briscoe, Ed. Expires: January 27, 2022 Independent July 26, 2021

Explicit Congestion Notification (ECN) Protocol for Very Low Queuing Delay (L4S) draft-ietf-tsvwg-ecn-l4s-id-19

Abstract

This specification defines the protocol to be used for a new network service called low latency, low loss and scalable throughput (L4S). L4S uses an Explicit Congestion Notification (ECN) scheme at the IP layer that is similar to the original (or ’Classic’) ECN approach, except as specified within. L4S uses ’scalable’ congestion control, which induces much more frequent control signals from the network and it responds to them with much more fine-grained adjustments, so that very low (typically sub-millisecond on average) and consistently low queuing delay becomes possible for L4S traffic without compromising link utilization. Thus even capacity-seeking (TCP-like) traffic can have high bandwidth and very low delay at the same time, even during periods of high traffic load.

The L4S identifier defined in this document distinguishes L4S from ’Classic’ (e.g. TCP-Reno-friendly) traffic. It gives an incremental migration path so that suitably modified network bottlenecks can distinguish and isolate existing traffic that still follows the Classic behaviour, to prevent it degrading the low queuing delay and low loss of L4S traffic. This specification defines the rules that L4S transports and network elements need to follow with the intention that L4S flows neither harm each other’s performance nor that of Classic traffic. Examples of new active queue management (AQM) marking algorithms and examples of new transports (whether TCP-like or real-time) are specified separately.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

De Schepper & Briscoe Expires January 27, 2022 [Page 1] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

This Internet-Draft will expire on January 27, 2022.

Copyright Notice

Table of Contents

1. Introduction ...... 3 1.1. Latency, Loss and Scaling Problems ...... 5 1.2. Terminology ...... 7 1.3. Scope ...... 8 2. Choice of L4S Packet Identifier: Requirements ...... 9 3. L4S Packet Identification ...... 10 4. Transport Layer Behaviour (the ’Prague Requirements’) . . . . 11 4.1. Codepoint Setting ...... 11 4.2. Prerequisite Transport Feedback ...... 11 4.3. Prerequisite Congestion Response ...... 12 4.4. Filtering or Smoothing of ECN Feedback ...... 15 5. Network Node Behaviour ...... 15 5.1. Classification and Re-Marking Behaviour ...... 15 5.2. The Strength of L4S CE Marking Relative to Drop . . . . . 16 5.3. Exception for L4S Packet Identification by Network Nodes with Transport-Layer Awareness ...... 18 5.4. Interaction of the L4S Identifier with other Identifiers 18 5.4.1. DualQ Examples of Other Identifiers Complementing L4S Identifiers ...... 18 5.4.1.1. Inclusion of Additional Traffic with L4S . . . . 18 5.4.1.2. Exclusion of Traffic From L4S Treatment . . . . . 20 5.4.1.3. Generalized Combination of L4S and Other Identifiers ...... 21 5.4.2. Per-Flow Queuing Examples of Other Identifiers

De Schepper & Briscoe Expires January 27, 2022 [Page 2] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

Complementing L4S Identifiers ...... 22 5.5. Limiting Packet Bursts from Links Supporting L4S AQMs . . 22 6. Behaviour of Tunnels and Encapsulations ...... 23 6.1. No Change to ECN Tunnels and Encapsulations in General . 23 6.2. VPN Behaviour to Avoid Limitations of Anti-Replay . . . . 24 7. L4S Experiments ...... 25 7.1. Open Questions ...... 25 7.2. Open Issues ...... 26 7.3. Future Potential ...... 27 8. IANA Considerations ...... 27 9. Security Considerations ...... 28 10. Acknowledgements ...... 28 11. References ...... 29 11.1. Normative References ...... 29 11.2. Informative References ...... 29 Appendix A. The ’Prague L4S Requirements’ ...... 38 A.1. Requirements for Scalable Transport Protocols ...... 38 A.1.1. Use of L4S Packet Identifier ...... 38 A.1.2. Accurate ECN Feedback ...... 39 A.1.3. Capable of Replacement by Classic Congestion Control 39 A.1.4. Fall back to Classic Congestion Control on Packet Loss ...... 39 A.1.5. Coexistence with Classic Congestion Control at Classic ECN bottlenecks ...... 40 A.1.6. Reduce RTT dependence ...... 43 A.1.7. Scaling down to fractional congestion windows . . . . 44 A.1.8. Measuring Reordering Tolerance in Time Units . . . . 45 A.2. Scalable Transport Protocol Optimizations ...... 48 A.2.1. Setting ECT in Control Packets and Retransmissions . 48 A.2.2. Faster than Additive Increase ...... 48 A.2.3. Faster Convergence at Flow Start ...... 49 Appendix B. Compromises in the Choice of L4S Identifier . . . . 49 Appendix C. Potential Competing Uses for the ECT(1) Codepoint . 54 C.1. Integrity of Congestion Feedback ...... 54 C.2. Notification of Less Severe Congestion than CE . . . . . 55 Authors’ Addresses ...... 56

1. Introduction

This specification defines the protocol to be used for a new network service called low latency, low loss and scalable throughput (L4S). L4S uses an Explicit Congestion Notification (ECN) scheme at the IP layer that is similar to the original (or ’Classic’) Explicit Congestion Notification (ECN [RFC3168]). RFC 3168 required an ECN mark to be equivalent to a drop, both when applied in the network and when responded to by a transport. Unlike Classic ECN marking, the network applies L4S marking more immediately and more aggressively than drop, and the transport response to each mark is reduced and

De Schepper & Briscoe Expires January 27, 2022 [Page 3] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

smoothed relative to that for drop. The two changes counterbalance each other so that the throughput of an L4S flow will be roughly the same as a comparable non-L4S flow under the same conditions. Nonetheless, the much more frequent control signals and the finer responses to them result in very low queuing delay without compromising link utilization, and this low delay can be maintained during high load. For instance, queuing delay under heavy and highly varying load with the example DCTCP/DualQ solution cited below on a DSL or Ethernet link is sub-millisecond on average and roughly 1 to 2 milliseconds at the 99th percentile without losing link utilization [DualPI2Linux], [DCttH15]. Note that the inherent queuing delay while waiting to acquire a discontinuous medium such as WiFi has to be minimized in its own right, so it would be additional to the above (see section 6.3 of [I-D.ietf-tsvwg-l4s-arch]).

L4S relies on ’scalable’ congestion controls for these delay properties and for preserving low delay as flow rate scales, hence the name. The congestion control used in Data Center TCP (DCTCP) is an example of a scalable congestion control, but DCTCP is applicable solely to controlled environments like data centres [RFC8257], because it is too aggressive to co-exist with existing TCP-Reno- friendly traffic. The DualQ Coupled AQM, which is defined in a complementary experimental specification [I-D.ietf-tsvwg-aqm-dualq-coupled], is an AQM framework that enables scalable congestion controls derived from DCTCP to co-exist with existing traffic, each getting roughly the same flow rate when they compete under similar conditions. Note that a scalable congestion control is still not safe to deploy on the Internet unless it satisfies the requirements listed in Section 4.

L4S is not only for elastic (TCP-like) traffic - there are scalable congestion controls for real-time media, such as the L4S variant of the SCReAM [RFC8298] real-time media congestion avoidance technique (RMCAT). The factor that distinguishes L4S from Classic traffic is its behaviour in response to congestion. The transport wire protocol, e.g. TCP, QUIC, SCTP, DCCP, RTP/RTCP, is orthogonal (and therefore not suitable for distinguishing L4S from Classic packets).

The L4S identifier defined in this document is the key piece that distinguishes L4S from ’Classic’ (e.g. Reno-friendly) traffic. It gives an incremental migration path so that suitably modified network bottlenecks can distinguish and isolate existing Classic traffic from L4S traffic to prevent the former from degrading the very low delay and loss of the new scalable transports, without harming Classic performance at these bottlenecks. Initial implementation of the separate parts of the system has been motivated by the performance benefits.

De Schepper & Briscoe Expires January 27, 2022 [Page 4] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

1.1. Latency, Loss and Scaling Problems

Latency is becoming the critical performance factor for many (most?) applications on the public Internet, e.g. interactive Web, Web services, voice, conversational video, interactive video, interactive remote presence, instant messaging, online gaming, remote desktop, cloud-based applications, and video-assisted remote control of machinery and industrial processes. In the ’developed’ world, further increases in access network bit-rate offer diminishing returns, whereas latency is still a multi-faceted problem. In the last decade or so, much has been done to reduce propagation time by placing caches or servers closer to users. However, queuing remains a major intermittent component of latency.

The Diffserv architecture provides Expedited Forwarding [RFC3246], so that low latency traffic can jump the queue of other traffic. If growth in high-throughput latency-sensitive applications continues, periods with solely latency-sensitive traffic will become increasingly common on links where traffic aggregation is low. For instance, on the access links dedicated to individual sites (homes, small enterprises or mobile devices). These links also tend to become the path bottleneck under load. During these periods, if all the traffic were marked for the same treatment at these bottlenecks, Diffserv would make no difference. Instead, it becomes imperative to remove the underlying causes of any unnecessary delay.

The bufferbloat project has shown that excessively-large buffering (’bufferbloat’) has been introducing significantly more delay than the underlying propagation time. These delays appear only intermittently--only when a capacity-seeking (e.g. TCP) flow is long enough for the queue to fill the buffer, making every packet in other flows sharing the buffer sit through the queue.

Active queue management (AQM) was originally developed to solve this problem (and others). Unlike Diffserv, which gives low latency to some traffic at the expense of others, AQM controls latency for _all_ traffic in a class. In general, AQM methods introduce an increasing level of discard from the buffer the longer the queue persists above a shallow threshold. This gives sufficient signals to capacity- seeking (aka. greedy) flows to keep the buffer empty for its intended purpose: absorbing bursts. However, RED [RFC2309] and other algorithms from the 1990s were sensitive to their configuration and hard to set correctly. So, this form of AQM was not widely deployed.

More recent state-of-the-art AQM methods, e.g. FQ-CoDel [RFC8290], PIE [RFC8033], Adaptive RED [ARED01], are easier to configure, because they define the queuing threshold in time not bytes, so it is invariant for different link rates. However, no matter how good the

De Schepper & Briscoe Expires January 27, 2022 [Page 5] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

AQM, the sawtoothing sending window of a Classic congestion control will either cause queuing delay to vary or cause the link to be under-utilized. Even with a perfectly tuned AQM, the additional queuing delay will be of the same order as the underlying speed-of- light delay across the network.

If a sender’s own behaviour is introducing queuing delay variation, no AQM in the network can ’un-vary’ the delay without significantly compromising link utilization. Even flow-queuing (e.g. [RFC8290]), which isolates one flow from another, cannot isolate a flow from the delay variations it inflicts on itself. Therefore those applications that need to seek out high bandwidth but also need low latency will have to migrate to scalable congestion control.

Altering host behaviour is not enough on its own though. Even if hosts adopt low latency behaviour (scalable congestion controls), they need to be isolated from the behaviour of existing Classic congestion controls that induce large queue variations. L4S enables that migration by providing latency isolation in the network and distinguishing the two types of packets that need to be isolated: L4S and Classic. L4S isolation can be achieved with a queue per flow (e.g. [RFC8290]) but a DualQ [I-D.ietf-tsvwg-aqm-dualq-coupled] is sufficient, and actually gives better tail latency. Both approaches are addressed in this document.

The DualQ solution was developed to make very low latency available without requiring per-flow queues at every bottleneck. This was because FQ has well-known downsides - not least the need to inspect transport layer headers in the network, which makes it incompatible with privacy approaches such as IPSec VPN tunnels, and incompatible with link layer queue management, where transport layer headers can be hidden, e.g. 5G.

Latency is not the only concern addressed by L4S: It was known when TCP congestion avoidance was first developed that it would not scale to high bandwidth-delay products (footnote 6 of Jacobson and Karels [TCP-CA]). Given regular broadband bit-rates over WAN distances are already [RFC3649] beyond the scaling range of Reno congestion control, ’less unscalable’ Cubic [RFC8312] and Compound [I-D.sridharan-tcpm-ctcp] variants of TCP have been successfully deployed. However, these are now approaching their scaling limits. Unfortunately, fully scalable congestion controls such as DCTCP [RFC8257] outcompete Classic ECN congestion controls sharing the same queue, which is why they have been confined to private data centres or research testbeds.

It turns out that these scalable congestion control algorithms that solve the latency problem can also solve the scalability problem of

De Schepper & Briscoe Expires January 27, 2022 [Page 6] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

Classic congestion controls. The finer sawteeth in the congestion window have low amplitude, so they cause very little queuing delay variation and the average time to recover from one congestion signal to the next (the average duration of each sawtooth) remains invariant, which maintains constant tight control as flow-rate scales. A background paper [DCttH15] gives the full explanation of why the design solves both the latency and the scaling problems, both in plain English and in more precise mathematical form. The explanation is summarised without the maths in the L4S architecture document [I-D.ietf-tsvwg-l4s-arch].

1.2. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. In this document, these words will appear with that interpretation only when in ALL CAPS. Lower case uses of these words are not to be interpreted as carrying RFC-2119 significance.

Classic Congestion Control: A congestion control behaviour that can co-exist with standard Reno [RFC5681] without causing significantly negative impact on its flow rate [RFC5033]. With Classic congestion controls, such as Reno or Cubic, because flow rate has scaled since TCP congestion control was first designed in 1988, it now takes hundreds of round trips (and growing) to recover after a congestion signal (whether a loss or an ECN mark) as shown in the examples in section 5.1 of [I-D.ietf-tsvwg-l4s-arch] and in [RFC3649]. Therefore control of queuing and utilization becomes very slack, and the slightest disturbances (e.g. from new flows starting) prevent a high rate from being attained.

Classic service: The Classic service is intended for all the congestion control behaviours that co-exist with Reno [RFC5681]

De Schepper & Briscoe Expires January 27, 2022 [Page 7] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

(e.g. Reno itself, Cubic [RFC8312], Compound [I-D.sridharan-tcpm-ctcp], TFRC [RFC5348]). The term ’Classic queue’ means a queue providing the Classic service.

Low-Latency, Low-Loss Scalable throughput (L4S) service: The ’L4S’ service is intended for traffic from scalable congestion control algorithms, such as TCP Prague [I-D.briscoe-iccrg-prague-congestion-control], which was derived from DCTCP [RFC8257]. The L4S service is for more general traffic than just TCP Prague--it allows the set of congestion controls with similar scaling properties to Prague to evolve, such as the examples listed above (Relentless, SCReAM). The term ’L4S queue’ means a queue providing the L4S service.

The terms Classic or L4S can also qualify other nouns, such as ’queue’, ’codepoint’, ’identifier’, ’classification’, ’packet’, ’flow’. For example: an L4S packet means a packet with an L4S identifier sent from an L4S congestion control.

Both Classic and L4S services can cope with a proportion of unresponsive or less-responsive traffic as well, but in the L4S case its rate has to be smooth enough or low enough not to build a queue (e.g. DNS, VoIP, game sync datagrams, etc).

Classic ECN: The original Explicit Congestion Notification (ECN) protocol [RFC3168], which requires ECN signals to be treated the same as drops, both when generated in the network and when responded to by the sender. For L4S, the names used for the four codepoints of the 2-bit IP-ECN field are unchanged from those defined in [RFC3168]: Not ECT, ECT(0), ECT(1) and CE, where ECT stands for ECN-Capable Transport and CE stands for Congestion Experienced. A packet marked with the CE codepoint is termed ’ECN-marked’ or sometimes just ’marked’ where the context makes ECN obvious.

1.3. Scope

The new L4S identifier defined in this specification is applicable for IPv4 and IPv6 packets (as for Classic ECN [RFC3168]). It is applicable for the unicast, multicast and anycast forwarding modes.

De Schepper & Briscoe Expires January 27, 2022 [Page 8] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

The L4S identifier is an orthogonal packet classification to the Differentiated Services Code Point (DSCP) [RFC2474]. Section 5.4 explains what this means in practice.

This document is intended for experimental status, so it does not update any standards track RFCs. Therefore it depends on [RFC8311], which is a standards track specification that:

o updates the ECN proposed standard [RFC3168] to allow experimental track RFCs to relax the requirement that an ECN mark must be equivalent to a drop (when the network applies markings and/or when the sender responds to them). For instance, in the ABE experiment [RFC8511] this permits a sender to respond less to ECN marks than to drops;

o changes the status of the experimental ECN nonce [RFC3540] to historic;

o makes consequent updates to the following additional proposed standard RFCs to reflect the above two bullets:

* ECN for RTP [RFC6679];

* the congestion control specifications of various DCCP congestion control identifier (CCID) profiles [RFC4341], [RFC4342], [RFC5622].

This document is about identifiers that are used for interoperation between hosts and networks. So the audience is broad, covering developers of host transports and network AQMs, as well as covering how operators might wish to combine various identifiers, which would require flexibility from equipment developers.

2. Choice of L4S Packet Identifier: Requirements

This subsection briefly records the process that led to the chosen L4S identifier.

The identifier for packets using the Low Latency, Low Loss, Scalable throughput (L4S) service needs to meet the following requirements:

o it SHOULD survive end-to-end between source and destination endpoints: across the boundary between host and network, between interconnected networks, and through middleboxes;

o it SHOULD be visible at the IP layer;

o it SHOULD be common to IPv4 and IPv6 and transport-agnostic;

De Schepper & Briscoe Expires January 27, 2022 [Page 9] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

o it SHOULD be incrementally deployable;

o it SHOULD enable an AQM to classify packets encapsulated by outer IP or lower-layer headers;

o it SHOULD consume minimal extra codepoints;

o it SHOULD be consistent on all the packets of a transport layer flow, so that some packets of a flow are not served by a different queue to others.

Whether the identifier would be recoverable if the experiment failed is a factor that could be taken into account. However, this has not been made a requirement, because that would favour schemes that would be easier to fail, rather than those more likely to succeed.

It is recognised that any choice of identifier is unlikely to satisfy all these requirements, particularly given the limited space left in the IP header. Therefore a compromise will always be necessary, which is why all the above requirements are expressed with the word ’SHOULD’ not ’MUST’.

After extensive assessment of alternative schemes, "ECT(1) and CE codepoints" was chosen as the best compromise. Therefore this scheme is defined in detail in the following sections, while Appendix B records its pros and cons against the above requirements.

3. L4S Packet Identification

The L4S treatment is an experimental track alternative packet marking treatment to the Classic ECN treatment in [RFC3168], which has been updated by [RFC8311] to allow experiments such as the one defined in the present specification. [RFC4774] discusses some of the issues and evaluation criteria when defining alternative ECN semantics. Like Classic ECN, L4S ECN identifies both network and host behaviour: it identifies the marking treatment that network nodes are expected to apply to L4S packets, and it identifies packets that have been sent from hosts that are expected to comply with a broad type of sending behaviour.

For a packet to receive L4S treatment as it is forwarded, the sender sets the ECN field in the IP header to the ECT(1) codepoint. See Section 4 for full transport layer behaviour requirements, including feedback and congestion response.

A network node that implements the L4S service always classifies arriving ECT(1) packets for L4S treatment and by default classifies CE packets for L4S treatment unless the heuristics described in

De Schepper & Briscoe Expires January 27, 2022 [Page 10] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

Section 5.3 are employed. See Section 5 for full network element behaviour requirements, including classification, ECN-marking and interaction of the L4S identifier with other identifiers and per-hop behaviours.

4. Transport Layer Behaviour (the ’Prague Requirements’)

4.1. Codepoint Setting

A sender that wishes a packet to receive L4S treatment as it is forwarded, MUST set the ECN field in the IP header (v4 or v6) to the ECT(1) codepoint.

4.2. Prerequisite Transport Feedback

For a transport protocol to provide scalable congestion control (Section 4.3) it MUST provide feedback of the extent of CE marking on the forward path. When ECN was added to TCP [RFC3168], the feedback method reported no more than one CE mark per round trip. Some transport protocols derived from TCP mimic this behaviour while others report the accurate extent of ECN marking. This means that some transport protocols will need to be updated as a prerequisite for scalable congestion control. The position for a few well-known transport protocols is given below.

TCP: Support for the accurate ECN feedback requirements [RFC7560] (such as that provided by AccECN [I-D.ietf-tcpm-accurate-ecn]) by both ends is a prerequisite for scalable congestion control in TCP. Therefore, the presence of ECT(1) in the IP headers even in one direction of a TCP connection will imply that both ends must be supporting accurate ECN feedback. However, the converse does not apply. So even if both ends support AccECN, either of the two ends can choose not to use a scalable congestion control, whatever the other end’s choice.

SCTP: A suitable ECN feedback mechanism for SCTP could add a chunk to report the number of received CE marks (e.g. [I-D.stewart-tsvwg-sctpecn]), and update the ECN feedback protocol sketched out in Appendix A of the standards track specification of SCTP [RFC4960].

RTP over UDP: A prerequisite for scalable congestion control is for both (all) ends of one media-level hop to signal ECN support [RFC6679] and use the new generic RTCP feedback format of [RFC8888]. The presence of ECT(1) implies that both (all) ends of that media-level hop support ECN. However, the converse does not apply. So each end of a media-level hop can independently choose

De Schepper & Briscoe Expires January 27, 2022 [Page 11] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

not to use a scalable congestion control, even if both ends support ECN.

QUIC: Support for sufficiently fine-grained ECN feedback is provided by the v1 IETF QUIC transport [RFC9000].

DCCP: The ACK vector in DCCP [RFC4340] is already sufficient to report the extent of CE marking as needed by a scalable congestion control.

4.3. Prerequisite Congestion Response

As a condition for a host to send packets with the L4S identifier (ECT(1)), it SHOULD implement a congestion control behaviour that ensures that, in steady state, the average duration between induced ECN marks does not increase as flow rate scales up, all other factors being equal. This is termed a scalable congestion control. This invariant duration ensures that, as flow rate scales, the average period with no feedback information about capacity does not become excessive. It also ensures that queue variations remain small, without having to sacrifice utilization.

With a congestion control that sawtooths to probe capacity, this duration is called the recovery time, because each time the sawtooth yields, on average it take this time to recover to its previous high point. A scalable congestion control does not have to sawtooth, but it has to coexist with scalable congestion controls that do.

For instance, for DCTCP [RFC8257], TCP Prague [I-D.briscoe-iccrg-prague-congestion-control], [PragueLinux] and the L4S variant of SCReAM [RFC8298], the average recovery time is always half a round trip (or half a reference round trip), whatever the flow rate.

As with all transport behaviours, a detailed specification (probably an experimental RFC) is expected for each congestion control, following the guidelines for specifying new congestion control algorithms in [RFC5033]. In addition it is expected to document these L4S-specific matters, specifically the timescale over which the proportionality is averaged, and control of burstiness. The recovery time requirement above is worded as a ’SHOULD’ rather than a ’MUST’ to allow reasonable flexibility for such implementations.

The condition ’all other factors being equal’, allows the recovery time to be different for different round trip times, as long as it does not increase with flow rate for any particular RTT.

De Schepper & Briscoe Expires January 27, 2022 [Page 12] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

Saying that the recovery time remains roughly invariant is equivalent to saying that the number of ECN CE marks per round trip remains invariant as flow rate scales, all other factors being equal. For instance, an average recovery time of half of 1 RTT is equivalent to 2 ECN marks per round trip. For those familiar with steady-state congestion response functions, it is also equivalent to say that the congestion window is inversely proportional to the proportion of bytes in packets marked with the CE codepoint (see section 2 of [PI2]).

In order to coexist safely with other Internet traffic, a scalable congestion control MUST NOT tag its packets with the ECT(1) codepoint unless it complies with the following bulleted requirements:

o A scalable congestion control MUST be capable of being replaced by a Classic congestion control (by application and/or by administrative control). If a Classic congestion control is activated, it will not tag its packets with the ECT(1) codepoint (see Appendix A.1.3 for rationale).

o As well as responding to ECN markings, a scalable congestion control MUST react to packet loss in a way that will coexist safely with Classic congestion controls such as standard Reno [RFC5681], as required by [RFC5033] (see Appendix A.1.4 for rationale).

o In uncontrolled environments, monitoring MUST be implemented to support detection of problems with an ECN-capable AQM at the path bottleneck that appears not to support L4S and might be in a shared queue. Such monitoring SHOULD be applied to live traffic that is using Scalable congestion control. Alternatively, monitoring need not be applied to live traffic, if monitoring has been arranged to cover the paths that live traffic takes through uncontrolled environments.

The detection function SHOULD be capable of making the congestion control adapt its ECN-marking response to coexist safely with Classic congestion controls such as standard Reno [RFC5681], as required by [RFC5033]. Alternatively, if adaptation is not implemented and problems with such an AQM are detected, the scalable congestion control MUST be replaced by a Classic congestion control.

Note that a scalable congestion control is not expected to change to setting ECT(0) while it transiently adapts to coexist with Classic congestion controls.

See Appendix A.1.5 and [I-D.ietf-tsvwg-l4sops] for rationale.

De Schepper & Briscoe Expires January 27, 2022 [Page 13] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

o In the range between the minimum likely RTT and typical RTTs expected in the intended deployment scenario, a scalable congestion control MUST converge towards a rate that is as independent of RTT as is possible without compromising stability or efficiency (see Appendix A.1.6 for rationale).

o A scalable congestion control SHOULD remain responsive to congestion when typical RTTs over the public Internet are significantly smaller because they are no longer inflated by queuing delay. It would be preferable for the minimum window of a scalable congestion control to be lower than 1 segment rather than use the timeout approach described for TCP in S.6.1.2 of [RFC3168] (or an equivalent for other transports). However, a lower minimum is not set as a formal requirement for L4S experiments (see Appendix A.1.7 for rationale).

o A scalable congestion control’s loss detection SHOULD be resilient to reordering over an adaptive time interval that scales with throughput and adapts to reordering (as in [RFC8985]), as opposed to counting only in fixed units of packets (as in the 3 DupACK rule of [RFC5681] and [RFC6675], which is not scalable). As data rates increase (e.g., due to new and/or improved technology), congestion controls that detect loss by counting in units of packets become more likely to incorrectly treat reordering events as congestion-caused loss events (see Appendix A.1.8 for further rationale). This requirement does not apply to congestion controls that are solely used in controlled environments where the network introduces hardly any reordering.

o A scalable congestion control is expected to limit the queue caused by bursts of packets. It would not seem necessary to set the limit any lower than 10% of the minimum RTT expected in a typical deployment (e.g. additional queuing of roughly 250 us for the public Internet). This would be converted to a number of packets under the worst-case assumption that the bottleneck link capacity equals the current flow rate. No normative requirement to limit bursts is given here and, until there is more industry experience from the L4S experiment, it is not even known whether one is needed - it seems to be in an L4S sender’s self-interest to limit bursts.

Each sender in a session can use a scalable congestion control independently of the congestion control used by the receiver(s) when they send data. Therefore there might be ECT(1) packets in one direction and ECT(0) or Not-ECT in the other.

Later (Section 5.4.1.1) this document discusses the conditions for mixing other "’Safe’ Unresponsive Traffic" (e.g. DNS, LDAP, NTP,

De Schepper & Briscoe Expires January 27, 2022 [Page 14] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

voice, game sync packets) with L4S traffic. To be clear, although such traffic can share the same queue as L4S traffic, it is not appropriate for the sender to tag it as ECT(1), except in the (unlikely) case that it satisfies the above conditions.

4.4. Filtering or Smoothing of ECN Feedback

Section 5.2 below specifies that an L4S AQM is expected to signal L4S ECN without filtering or smoothing. This contrasts with a Classic AQM, which filters out variations in the queue before signalling ECN marking or drop. In the L4S architecture [I-D.ietf-tsvwg-l4s-arch], responsibility for smoothing out these variations shifts to the sender’s congestion control.

This shift of responsibility has the advantage that each sender can smooth variations over a timescale proportionate to its own RTT. Whereas, in the Classic approach, the network doesn’t know the RTTs of all the flows, so it has to smooth out variations for a worst-case RTT to ensure stability. For all the typical flows with shorter RTT than the worst-case, this makes congestion control unnecessarily sluggish.

This also gives an L4S sender the choice not to smooth, depending on its context (start-up, congestion avoidance, etc). Therefore, this document places no requirement on an L4S congestion control to smooth out variations in any particular way. Implementers are encouraged to openly publish the approach they take to smoothing, and the results and experience they gain during the L4S experiment.

5. Network Node Behaviour

5.1. Classification and Re-Marking Behaviour

A network node that implements the L4S service:

o MUST classify arriving ECT(1) packets for L4S treatment, unless overridden by another classifier (e.g., see Section 5.4.1.2);

o MUST classify arriving CE packets for L4S treatment as well, unless overridden by a another classifier or unless the exception referred to next applies;

CE packets might have originated as ECT(1) or ECT(0), but the above rule to classify them as if they originated as ECT(1) is the safe choice (see Appendix B for rationale). The exception is where some flow-aware in-network mechanism happens to be available for distinguishing CE packets that originated as ECT(0), as

De Schepper & Briscoe Expires January 27, 2022 [Page 15] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

described in Section 5.3, but there is no implication that such a mechanism is necessary.

An L4S AQM treatment follows similar codepoint transition rules to those in RFC 3168. Specifically, the ECT(1) codepoint MUST NOT be changed to any other codepoint than CE, and CE MUST NOT be changed to any other codepoint. An ECT(1) packet is classified as ECN-capable and, if congestion increases, an L4S AQM algorithm will increasingly mark the ECN field as CE, otherwise forwarding packets unchanged as ECT(1). Necessary conditions for an L4S marking treatment are defined in Section 5.2.

Under persistent overload an L4S marking treatment MUST begin applying drop to L4S traffic until the overload episode has subsided, as recommended for all AQM methods in [RFC7567] (Section 4.2.1), which follows the similar advice in RFC 3168 (Section 7). During overload, it MUST apply the same drop probability to L4S traffic as it would to Classic traffic.

Where an L4S AQM is transport-aware, this requirement could be satisfied by using drop in only the most overloaded individual per- flow AQMs. In a DualQ with flow-aware queue protection (e.g. [I-D.briscoe-docsis-q-protection]), this could be achieved by redirecting packets in those flows contributing most to the overload out of the L4S queue so that they are subjected to drop in the Classic queue.

For backward compatibility in uncontrolled environments, a network node that implements the L4S treatment MUST also implement an AQM treatment for the Classic service as defined in Section 1.2. This Classic AQM treatment need not mark ECT(0) packets, but if it does, see Section 5.2 for the strengths of the markings relative to drop. It MUST classify arriving ECT(0) and Not-ECT packets for treatment by this Classic AQM (for the DualQ Coupled AQM, see the extensive discussion on classification in Sections 2.3 and 2.5.1.1 of [I-D.ietf-tsvwg-aqm-dualq-coupled]).

In case unforeseen problems arise with the L4S experiment, it MUST be possible to configure an L4S implementation to disable the L4S treatment. Once disabled, all packets of all ECN codepoints will receive Classic treatment and ECT(1) packets MUST be treated as if they were {ToDo: Not-ECT / ECT(0) ?}.

5.2. The Strength of L4S CE Marking Relative to Drop

The relative strengths of L4S CE and drop are irrelevant where AQMs are implemented in separate queues per-application-flow, which are then explicitly scheduled (e.g. with an FQ scheduler as in

De Schepper & Briscoe Expires January 27, 2022 [Page 16] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

[RFC8290]). Nonetheless, the relationship between them needs to be defined for the coupling between L4S and Classic congestion signals in a DualQ Coupled AQM [I-D.ietf-tsvwg-aqm-dualq-coupled], as below.

Unless an AQM node schedules application flows explicitly, the likelihood that the AQM drops a Not-ECT Classic packet (p_C) MUST be roughly proportional to the square of the likelihood that it would have marked it if it had been an L4S packet (p_L). That is

p_C ˜= (p_L / k)^2

The constant of proportionality (k) does not have to be standardised for interoperability, but a value of 2 is RECOMMENDED. The term ’likelihood’ is used above to allow for marking and dropping to be either probabilistic or deterministic.

This formula ensures that Scalable and Classic flows will converge to roughly equal congestion windows, for the worst case of Reno congestion control. This is because the congestion windows of Scalable and Classic congestion controls are inversely proportional to p_L and sqrt(p_C) respectively. So squaring p_C in the above formula counterbalances the square root that characterizes Reno- friendly flows.

Note that, contrary to RFC 3168, an AQM implementing the L4S and Classic treatments does not mark an ECT(1) packet under the same conditions that it would have dropped a Not-ECT packet, as allowed by [RFC8311], which updates RFC 3168. However, if it marks ECT(0) packets, it does so under the same conditions that it would have dropped a Not-ECT packet [RFC3168].

Also, L4S CE marking needs to be interpreted as an unsmoothed signal, in contrast to the Classic approach in which AQMs filter out variations before signalling congestion. An L4S AQM SHOULD NOT smooth or filter out variations in the queue before signalling congestion. In the L4S architecture [I-D.ietf-tsvwg-l4s-arch], the sender, not the network, is responsible for smoothing out variations.

This requirement is worded as ’SHOULD NOT’ rather than ’MUST NOT’ to allow for the case where the signals from a Classic smoothed AQM are coupled with those from an unsmoothed L4S AQM. Nonetheless, the spirit of the requirement is for all systems to expect that L4S ECN signalling is unsmoothed and unfiltered, which is important for interoperability.

De Schepper & Briscoe Expires January 27, 2022 [Page 17] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

5.3. Exception for L4S Packet Identification by Network Nodes with Transport-Layer Awareness

To implement the L4S treatment, a network node does not need to identify transport-layer flows. Nonetheless, if an implementer is willing to identify transport-layer flows at a network node, and if the most recent ECT packet in the same flow was ECT(0), the node MAY classify CE packets for Classic ECN [RFC3168] treatment. In all other cases, a network node MUST classify all CE packets for L4S treatment. Examples of such other cases are: i) if no ECT packets have yet been identified in a flow; ii) if it is not desirable for a network node to identify transport-layer flows; or iii) if the most recent ECT packet in a flow was ECT(1).

If an implementer uses flow-awareness to classify CE packets, to determine whether the flow is using ECT(0) or ECT(1) it only uses the most recent ECT packet of a flow (this advice will need to be verified as part of L4S experiments). This is because a sender might switch from sending ECT(1) (L4S) packets to sending ECT(0) (Classic ECN) packets, or back again, in the middle of a transport-layer flow (e.g. it might manually switch its congestion control module mid- connection, or it might be deliberately attempting to confuse the network).

5.4. Interaction of the L4S Identifier with other Identifiers

The examples in this section concern how additional identifiers might complement the L4S identifier to classify packets between class-based queues. Firstly Section 5.4.1 considers two queues, L4S and Classic, as in the Coupled DualQ AQM [I-D.ietf-tsvwg-aqm-dualq-coupled], either alone (Section 5.4.1.1) or within a larger queuing hierarchy (Section 5.4.1.2). Then Section 5.4.2 considers schemes that might combine per-flow 5-tuples with other identifiers.

5.4.1. DualQ Examples of Other Identifiers Complementing L4S Identifiers

5.4.1.1. Inclusion of Additional Traffic with L4S

In a typical case for the public Internet a network element that implements L4S in a shared queue might want to classify some low-rate but unresponsive traffic (e.g. DNS, LDAP, NTP, voice, game sync packets) into the low latency queue to mix with L4S traffic.

In this case it would not be appropriate to call the queue an L4S queue, because it is shared by L4S and non-L4S traffic. Instead it will be called the low latency or L queue. The L queue then offers two different treatments:

De Schepper & Briscoe Expires January 27, 2022 [Page 18] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

o The L4S treatment, which is a combination of the L4S AQM treatment and a priority scheduling treatment;

o The low latency treatment, which is solely the priority scheduling treatment, without ECN-marking by the AQM.

To identify packets for just the scheduling treatment, it would be inappropriate to use the L4S ECT(1) identifier, because such traffic is unresponsive to ECN marking. Examples of relevant non-ECN identifiers are:

o address ranges of specific applications or hosts configured to be, or known to be, safe, e.g. hard-coded IoT devices sending low intensity traffic;

o certain low data-volume applications or protocols (e.g. ARP, DNS);

o specific Diffserv codepoints that indicate traffic with limited burstiness such as the EF (Expedited Forwarding [RFC3246]), Voice- Admit [RFC5865] or proposed NQB (Non-Queue-Building [I-D.ietf-tsvwg-nqb]) service classes or equivalent local-use DSCPs (see [I-D.briscoe-tsvwg-l4s-diffserv]).

In summary, a network element that implements L4S in a shared queue MAY classify additional types of packets into the L queue based on identifiers other than the ECN field, but the types SHOULD be ’safe’ to mix with L4S traffic, where ’safe’ is explained in Section 5.4.1.1.1.

A packet that carries one of these non-ECN identifiers to classify it into the L queue would not be subject to the L4S ECN marking treatment, unless it also carried an ECT(1) or CE codepoint. The specification of an L4S AQM MUST define the behaviour for packets with unexpected combinations of codepoints, e.g. a non-ECN-based classifier for the L queue, but ECT(0) in the ECN field (for examples see section 2.5.1.1 of [I-D.ietf-tsvwg-aqm-dualq-coupled]).

For clarity, non-ECN identifiers, such as the examples itemized above, might be used by some network operators who believe they identify non-L4S traffic that would be safe to mix with L4S traffic. They are not alternative ways for a host to indicate that it is sending L4S packets. Only the ECT(1) ECN codepoint indicates to a network element that a host is sending L4S packets (and CE indicates that it could have originated as ECT(1)). Specifically ECT(1) indicates that the host claims its behaviour satisfies the prerequisite transport requirements in Section 4.

De Schepper & Briscoe Expires January 27, 2022 [Page 19] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

To include additional traffic with L4S, a network element only reads identifiers such as those itemized above. It MUST NOT alter these non-ECN identifiers, so that they survive for any potential use later on the network path.

5.4.1.1.1. ’Safe’ Unresponsive Traffic

The above section requires unresponsive traffic to be ’safe’ to mix with L4S traffic. Ideally this means that the sender never sends any sequence of packets at a rate that exceeds the available capacity of the bottleneck link. However, typically an unresponsive transport does not even know the bottleneck capacity of the path, let alone its available capacity. Nonetheless, an application can be considered safe enough if it paces packets out (not necessarily completely regularly) such that its maximum instantaneous rate from packet to packet stays well below a typical broadband access rate.

This is a vague but useful definition, because many low latency applications of interest, such as DNS, voice, game sync packets, RPC, ACKs, keep-alives, could match this description.

5.4.1.2. Exclusion of Traffic From L4S Treatment

To extend the above example, an operator might want to exclude some traffic from the L4S treatment for a policy reason, e.g. security (traffic from malicious sources) or commercial (e.g. initially the operator may wish to confine the benefits of L4S to business customers).

In this exclusion case, the operator MUST classify on the relevant locally-used identifiers (e.g. source addresses) before classifying the non-matching traffic on the end-to-end L4S ECN identifier.

The operator MUST NOT alter the end-to-end L4S ECN identifier from L4S to Classic, because an operator decision to exclude certain traffic from L4S treatment is local-only. The end-to-end L4S identifier then survives for other operators to use, or indeed, they can apply their own policy, independently based on their own choice of locally-used identifiers. This approach also allows any operator to remove its locally-applied exclusions in future, e.g. if it wishes to widen the benefit of the L4S treatment to all its customers.

An operator that excludes traffic carrying the L4S identifier from L4S treatment MUST NOT treat such traffic as if it carries the ECT(0) codepoint, which could confuse the sender.

De Schepper & Briscoe Expires January 27, 2022 [Page 20] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

5.4.1.3. Generalized Combination of L4S and Other Identifiers

L4S concerns low latency, which it can provide for all traffic without differentiation and without _necessarily_ affecting bandwidth allocation. Diffserv provides for differentiation of both bandwidth and low latency, but its control of latency depends on its control of bandwidth. The two can be combined if a network operator wants to control bandwidth allocation but it also wants to provide low latency - for any amount of traffic within one of these allocations of bandwidth (rather than only providing low latency by limiting bandwidth) [I-D.briscoe-tsvwg-l4s-diffserv].

The DualQ examples so far have been framed in the context of providing the default Best Efforts Per-Hop Behaviour (PHB) using two queues - a Low Latency (L) queue and a Classic (C) Queue. This single DualQ structure is expected to be the most common and useful arrangement. But, more generally, an operator might choose to control bandwidth allocation through a hierarchy of Diffserv PHBs at a node, and to offer one (or more) of these PHBs with a low latency and a Classic variant.

In the first case, if we assume that a network element provides no PHBs except the DualQ, if a packet carries ECT(1) or CE, the network element would classify it for the L4S treatment irrespective of its DSCP. And, if a packet carried (say) the EF DSCP, the network element could classify it into the L queue irrespective of its ECN codepoint. However, where the DualQ is in a hierarchy of other PHBs, the classifier would classify some traffic into other PHBs based on DSCP before classifying between the low latency and Classic queues (based on ECT(1), CE and perhaps also the EF DSCP or other identifiers as in the above example). [I-D.briscoe-tsvwg-l4s-diffserv] gives a number of examples of such arrangements to address various requirements.

[I-D.briscoe-tsvwg-l4s-diffserv] describes how an operator might use L4S to offer low latency for all L4S traffic as well as using Diffserv for bandwidth differentiation. It identifies two main types of approach, which can be combined: the operator might split certain Diffserv PHBs between L4S and a corresponding Classic service. Or it might split the L4S and/or the Classic service into multiple Diffserv PHBs. In either of these cases, a packet would have to be classified on its Diffserv and ECN codepoints.

In summary, there are numerous ways in which the L4S ECN identifier (ECT(1) and CE) could be combined with other identifiers to achieve particular objectives. The following categorization articulates those that are valid, but it is not necessarily exhaustive. Those

De Schepper & Briscoe Expires January 27, 2022 [Page 21] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

tagged ’Recommended-standard-use’ could be set by the sending host or a network. Those tagged ’Local-use’ would only be set by a network:

1. Identifiers Complementing the L4S Identifier

A. Including More Traffic in the L Queue (Could use Recommended-standard-use or Local-use identifiers)

B. Excluding Certain Traffic from the L Queue (Local-use only)

2. Identifiers to place L4S classification in a PHB Hierarchy (Could use Recommended-standard-use or Local-use identifiers)

A. PHBs Before L4S ECN Classification

B. PHBs After L4S ECN Classification

5.4.2. Per-Flow Queuing Examples of Other Identifiers Complementing L4S Identifiers

At a node with per-flow queueing (e.g. FQ-CoDel [RFC8290]), the L4S identifier could complement the Layer-4 flow ID as a further level of flow granularity (i.e. Not-ECT and ECT(0) queued separately from ECT(1) and CE packets). "Risk of reordering Classic CE packets" in Appendix B discusses the resulting ambiguity if packets originally marked ECT(0) are marked CE by an upstream AQM before they arrive at a node that classifies CE as L4S. It argues that the risk of reordering is vanishingly small and the consequence of such a low level of reordering is minimal.

Alternatively, it could be assumed that it is not in a flow’s own interest to mix Classic and L4S identifiers. Then the AQM could use the ECN field to switch itself between a Classic and an L4S AQM behaviour within one per-flow queue. For instance, for ECN-capable packets, the AQM might consist of a simple marking threshold and an L4S ECN identifier might simply select a shallower threshold than a Classic ECN identifier would.

5.5. Limiting Packet Bursts from Links Supporting L4S AQMs

As well as senders needing to limit packet bursts (Section 4.3), links need to limit the degree of burstiness they introduce. In both cases (senders and links) this is a tradeoff, because batch-handling of packets is done for good reason, e.g. processing efficiency or to make efficient use of medium acquisition delay. Some take the attitude that there is no point reducing burst delay at the sender below that introduced by links (or vice versa). However, delay

De Schepper & Briscoe Expires January 27, 2022 [Page 22] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

reduction proceeds by cutting down ’the longest pole in the tent’, which turns the spotlight on the next longest, and so on.

This document does not set any quantified requirements for links to limit burst delay, primarily because link technologies are outside the remit of L4S specifications. Nonetheless, it would not make sense to implement an L4S AQM that feeds into a particular link technology without also reviewing opportunities to reduce any form of burst delay introduced by that link technology. This would at least limit the bursts that the link would otherwise introduce into the onward traffic, which would cause jumpy feedback to the sender as well as potential extra queuing delay downstream. This document does not presume to even give guidance on an appropriate target for such burst delay until there is more industry experience of L4S. However, as suggested in Section 4.3 it would not seem necessary to limit bursts lower than roughly 10% of the minimum base RTT expected in the typical deployment scenario (e.g. 250 us burst duration for links within the public Internet).

6. Behaviour of Tunnels and Encapsulations

6.1. No Change to ECN Tunnels and Encapsulations in General

The L4S identifier is expected to work through and within any tunnel without modification, as long as the tunnel propagates the ECN field in any of the ways that have been defined since the first variant in the year 2001 [RFC3168]. L4S will also work with (but does not rely on) any of the more recent updates to ECN propagation in [RFC4301], [RFC6040] or [I-D.ietf-tsvwg-rfc6040update-shim]. However, it is likely that some tunnels still do not implement ECN propagation at all. In these cases, L4S will work through such tunnels, but within them the outer header of L4S traffic will appear as Classic.

AQMs are typically implemented where an IP-layer buffer feeds into a lower layer, so they are agnostic to link layer encapsulations. Where a bottleneck link is not IP-aware, the L4S identifier is still expected to work within any lower layer encapsulation without modification, as long it propagates the ECN field as defined for the link technology, for example for MPLS [RFC5129] or TRILL [I-D.ietf-trill-ecn-support]. In some of these cases, e.g. layer-3 Ethernet switches, the AQM accesses the IP layer header within the outer encapsulation, so again the L4S identifier is expected to work without modification. Nonetheless, the programme to define ECN for other lower layers is still in progress [I-D.ietf-tsvwg-ecn-encap-guidelines].

De Schepper & Briscoe Expires January 27, 2022 [Page 23] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

6.2. VPN Behaviour to Avoid Limitations of Anti-Replay

If a mix of L4S and Classic packets is sent into the same security association (SA) of a virtual private network (VPN), and if the VPN egress is employing the optional anti-replay feature, it could inappropriately discard Classic packets (or discard the records in Classic packets) by mistaking their greater queuing delay for a replay attack (see [Heist21] for the potential performance impact). This known problem is common to both IPsec [RFC4301] and DTLS [RFC6347] VPNs, given they use similar anti-replay window mechanisms. The mechanism used can only check for replay within its window, so if the window is smaller than the degree of reordering, it can only assume there might be a replay attack and discard all the packets behind the trailing edge of the window. The specifications of IPsec AH [RFC4302] and ESP [RFC4303] suggest that an implementer scales the size of the anti-replay window with interface speed, and the current draft of DTLS 1.3 [I-D.ietf-tls-dtls13] says "The receiver SHOULD pick a window large enough to handle any plausible reordering, which depends on the data rate." However, in practice, the size of a VPN’s anti-replay window is not always scaled appropriately.

If a VPN carrying traffic participating in the L4S experiment experiences inappropriate replay detection, the foremost remedy would be to ensure that the egress is configured to comply with the above window-sizing requirements.

If an implementation of a VPN egress does not support a sufficiently large anti-replay window, e.g. due to hardware limitations, one of the temporary alternatives listed in order of preference below might be feasible instead:

o If the VPN can be configured to classify packets into different SAs indexed by DSCP, apply the appropriate locally defined DSCPs to Classic and L4S packets. The DSCPs could be applied by the network (based on the least significant bit of the ECN field), or by the sending host. Such DSCPs would only need to survive as far as the VPN ingress.

o If the above is not possible and it is necessary to use L4S, either of the following might be appropriate as a last resort:

* disable anti-replay protection at the VPN egress, after considering the security implications (optional anti-replay is mandatory in both IPsec and DTLS);

* configure the tunnel ingress not to propagate ECN to the outer, which would lose the benefits of L4S and Classic ECN over the VPN.

De Schepper & Briscoe Expires January 27, 2022 [Page 24] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

Modification to VPN implementations is outside the present scope, which is why this section has so far focused on reconfiguration. Although this document does not define any requirements for VPN implementations, determining whether there is a need for such requirements could be one aspect of L4S experimentation.

7. L4S Experiments

This section describes open questions that L4S Experiments ought to focus on. This section also documents outstanding open issues that will need to be investigated as part of L4S experimentation, given they could not be fully resolved during the WG phase. It also lists metrics that will need to be monitored during experiments (summarizing text elsewhere in L4S documents) and finally lists some potential future directions that researchers might wish to investigate.

In addition to this section, [I-D.ietf-tsvwg-aqm-dualq-coupled] sets operational and management requirements for experiments with DualQ Coupled AQMs; and General operational and management requirements for experiments with L4S congestion controls are given in Section 4 and Section 5 above, e.g. co-existence and scaling requirements, incremental deployment arrangements.

The specification of each scalable congestion control will need to include protocol-specific requirements for configuration and monitoring performance during experiments. Appendix A of [RFC5706] provides a helpful checklist.

7.1. Open Questions

L4S experiments would be expected to answer the following questions:

o Have all the parts of L4S been deployed, and if so, what proportion of paths support it?

o Does use of L4S over the Internet result in significantly improved user experience?

o Has L4S enabled novel interactive applications?

o Did use of L4S over the Internet result in improvements to the following metrics:

o

* queue delay (mean and 99th percentile) under various loads;

De Schepper & Briscoe Expires January 27, 2022 [Page 25] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

* utilization;

* starvation / fairness;

* scaling range of flow rates and RTTs?

o How much does burstiness in the Internet affect L4S performance, and how much limitation of bustiness was needed and/or was realized - both at senders and at links, especially radio links?

o Was per-flow queue protection typically (un)necessary?

* How well did overload protection or queue protection work?

o How well did L4S flows coexist with Classic flows when sharing a bottleneck?

o

* How frequently did problems arise?

* What caused any coexistence problems, and were any problems due to single-queue Classic ECN AQMs (this assumes single-queue Classic ECN AQMs can be distinguished from FQ ones)?

o How prevalent were problems with the L4S service due to tunnels / encapsulations that do not support ECN decapsulation?

o How easy was it to implement a fully compliant L4S congestion control, over various different transport protocols (TCP. QUIC, RMCAT, etc)?

Monitoring for harm to other traffic, specifically bandwidth starvation or excess queuing delay, will need to be conducted alongside all early L4S experiments. It is hard, if not impossible, for an individual flow to measure its impact on other traffic. So such monitoring will need to be conducted using bespoke monitoring across flows and/or across classes of traffic.

7.2. Open Issues

o What is the best way forward to deal with L4S over single-queue Classic ECN AQM bottlenecks, given current problems with misdetecting L4S AQMs as Classic ECN AQMs?

o Fixing the poor Interaction between current L4S congestion controls and CoDel with only Classic ECN support during flow startup.

De Schepper & Briscoe Expires January 27, 2022 [Page 26] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

7.3. Future Potential

Researchers might find that L4S opens up the following interesting areas for investigation:

o Potential for faster convergence time and tracking of available capacity;

o Potential for improvements to particular link technologies, and cross-layer interactions with them;

o Potential for using virtual queues, e.g. to further reduce latency jitter, or to leave headroom for capacity variation in radio networks;

o Development and specification of reverse path congestion control using L4S building bocks (e.g. AccECN, QUIC);

o Once queuing delay is cut down, what becomes the ’second longest pole in the tent’ (other than the speed of light)?

o Novel alternatives to the existing set of L4S AQMs;

o Novel applications enabled by L4S.

8. IANA Considerations

The 01 codepoint of the ECN Field of the IP header is specified by the present Experimental RFC. The process for an experimental RFC to assign this codepoint in the IP header (v4 and v6) is documented in Proposed Standard [RFC8311], which updates the Proposed Standard [RFC3168].

When the present document is published as an RFC, IANA is asked to update the 01 entry in the registry, "ECN Field (Bits 6-7)" to the following (see https://www.iana.org/assignments/dscp-registry/dscp- registry.xhtml#ecn-field ):

+------+------+------+ | Binary | Keyword | References | +------+------+------+ | 01 | ECT(1) (ECN-Capable | [RFC8311] | | | Transport(1))[1] | [RFC Errata 5399] | | | | [RFCXXXX] | +------+------+------+

[XXXX is the number that the RFC Editor assigns to the present document (this sentence to be removed by the RFC Editor)].

De Schepper & Briscoe Expires January 27, 2022 [Page 27] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

9. Security Considerations

Approaches to assure the integrity of signals using the new identifier are introduced in Appendix C.1. See the security considerations in the L4S architecture [I-D.ietf-tsvwg-l4s-arch] for further discussion of mis-use of the identifier, as well as extensive discussion of policing rate and latency in regard to L4S.

If the anti-replay window of a VPN egress is too small, it will mistake deliberate delay differences as a replay attack, and discard higher delay packets (e.g. Classic) carried within the same security association (SA) as low delay packets (e.g. L4S). Section 6.2 recommends that VPNs used in L4S experiments are configured with a sufficiently large anti-replay window, as required by the relevant specifications. It also discusses other alternatives.

If a user taking part in the L4S experiment sets up a VPN without being aware of the above advice, and if the user allows anyone to send traffic into their VPN, they would open up a DoS vulnerability in which an attacker could induce the VPN’s anti-replay mechanism to discard enough of the user’s Classic (C) traffic (if they are receiving any) to cause a significant rate reduction. While the user is actively downloading C traffic, the attacker sends C traffic into the VPN to fill the remainder of the bottleneck link, then sends intermittent L4S packets to maximize the chance of exceeding the VPN’s replay window. The user can prevent this attack by following the recommendations in Section 6.2.

The recommendation to detect loss in time units prevents the ACK- splitting attacks described in [Savage-TCP].

10. Acknowledgements

Thanks to Richard Scheffenegger, John Leslie, David Taeht, Jonathan Morton, Gorry Fairhurst, Michael Welzl, Mikael Abrahamsson and Andrew McGregor for the discussions that led to this specification. Ing-jyh (Inton) Tsang was a contributor to the early drafts of this document. And thanks to Mikael Abrahamsson, Lloyd Wood, Nicolas Kuhn, Greg White, Tom Henderson, David Black, Gorry Fairhurst, Brian Carpenter, Jake Holland, Rod Grimes, Richard Scheffenegger, Sebastian Moeller, Neal Cardwell, Praveen Balasubramanian, Reza Marandian Hagh, Stuart Cheshire and Vidhi Goel for providing help and reviewing this draft and thanks to Ingemar Johansson for reviewing and providing substantial text. Thanks to Sebastian Moeller for identifying the interaction with VPN anti-replay and to Jonathan Morton for identifying the attack based on this. Particular thanks to tsvwg chairs Gorry Fairhurst, David Black and Wes Eddy for patiently helping this and the other L4S drafts through the IETF process.

De Schepper & Briscoe Expires January 27, 2022 [Page 28] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

Appendix A listing the Prague L4S Requirements is based on text authored by Marcelo Bagnulo Braun that was originally an appendix to [I-D.ietf-tsvwg-l4s-arch]. That text was in turn based on the collective output of the attendees listed in the minutes of a ’bar BoF’ on DCTCP Evolution during IETF-94 [TCPPrague].

The authors’ contributions were part-funded by the European Community under its Seventh Framework Programme through the Reducing Internet Transport Latency (RITE) project (ICT-317700). Bob Briscoe was also funded partly by the Research Council of Norway through the TimeIn project, partly by CableLabs and partly by the Comcast Innovation Fund. The views expressed here are solely those of the authors.

11. References

11.1. Normative References

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, .

[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition of Explicit Congestion Notification (ECN) to IP", RFC 3168, DOI 10.17487/RFC3168, September 2001, .

[RFC4774] Floyd, S., "Specifying Alternate Semantics for the Explicit Congestion Notification (ECN) Field", BCP 124, RFC 4774, DOI 10.17487/RFC4774, November 2006, .

[RFC6679] Westerlund, M., Johansson, I., Perkins, C., O’Hanlon, P., and K. Carlberg, "Explicit Congestion Notification (ECN) for RTP over UDP", RFC 6679, DOI 10.17487/RFC6679, August 2012, .

11.2. Informative References

[A2DTCP] Zhang, T., Wang, J., Huang, J., Huang, Y., Chen, J., and Y. Pan, "Adaptive-Acceleration Data Center TCP", IEEE Transactions on Computers 64(6):1522-1533, June 2015, .

[Ahmed19] Ahmed, A., "Extending TCP for Low Round Trip Delay", Masters Thesis, Uni Oslo , August 2019, .

De Schepper & Briscoe Expires January 27, 2022 [Page 29] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

[Alizadeh-stability] Alizadeh, M., Javanmard, A., and B. Prabhakar, "Analysis of DCTCP: Stability, Convergence, and Fairness", ACM SIGMETRICS 2011 , June 2011.

[ARED01] Floyd, S., Gummadi, R., and S. Shenker, "Adaptive RED: An Algorithm for Increasing the Robustness of RED’s Active Queue Management", ACIRI Technical Report , August 2001, .

[BBRv2] Cardwell, N., "TCP BBR v2 Alpha/Preview Release", github repository; Linux congestion control module, .

[DCttH15] De Schepper, K., Bondarenko, O., Briscoe, B., and I. Tsang, "’Data Centre to the Home’: Ultra-Low Latency for All", RITE Project Technical Report , 2015, .

[DualPI2Linux] Albisser, O., De Schepper, K., Briscoe, B., Tilmans, O., and H. Steen, "DUALPI2 - Low Latency, Low Loss and Scalable (L4S) AQM", Proc. Linux Netdev 0x13 , March 2019, .

[ecn-fallback] Briscoe, B. and A. Ahmed, "TCP Prague Fall-back on Detection of a Classic ECN AQM", bobbriscoe.net Technical Report TR-BB-2019-002, April 2020, .

[Heist21] Heist, P., "Dropped Packets for Tunnels with Replay Protection Enabled", github README, May 2021, .

[I-D.briscoe-docsis-q-protection] Briscoe, B. and G. White, "Queue Protection to Preserve Low Latency", draft-briscoe-docsis-q-protection-00 (work in progress), July 2019.

De Schepper & Briscoe Expires January 27, 2022 [Page 30] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

[I-D.ietf-tcpm-accurate-ecn] Briscoe, B., Kuehlewind, M., and R. Scheffenegger, "More Accurate ECN Feedback in TCP", draft-ietf-tcpm-accurate- ecn-15 (work in progress), July 2021.

[I-D.ietf-tcpm-generalized-ecn] Bagnulo, M. and B. Briscoe, "ECN++: Adding Explicit Congestion Notification (ECN) to TCP Control Packets", draft-ietf-tcpm-generalized-ecn-07 (work in progress), February 2021.

[I-D.ietf-tls-dtls13] Rescorla, E., Tschofenig, H., and N. Modadugu, "The Datagram Transport Layer Security (DTLS) Protocol Version 1.3", draft-ietf-tls-dtls13-43 (work in progress), April 2021.

[I-D.ietf-tsvwg-aqm-dualq-coupled] Schepper, K. D., Briscoe, B., and G. White, "DualQ Coupled AQMs for Low Latency, Low Loss and Scalable Throughput (L4S)", draft-ietf-tsvwg-aqm-dualq-coupled-16 (work in progress), July 2021.

[I-D.ietf-tsvwg-ecn-encap-guidelines] Briscoe, B. and J. Kaippallimalil, "Guidelines for Adding Congestion Notification to Protocols that Encapsulate IP", draft-ietf-tsvwg-ecn-encap-guidelines-16 (work in progress), May 2021.

De Schepper & Briscoe Expires January 27, 2022 [Page 31] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

[I-D.ietf-tsvwg-l4sops] White, G., "Operational Guidance for Deployment of L4S in the Internet", draft-ietf-tsvwg-l4sops-01 (work in progress), July 2021.

[I-D.ietf-tsvwg-nqb] White, G. and T. Fossati, "A Non-Queue-Building Per-Hop Behavior (NQB PHB) for Differentiated Services", draft- ietf-tsvwg-nqb-06 (work in progress), July 2021.

[I-D.sridharan-tcpm-ctcp] Sridharan, M., Tan, K., Bansal, D., and D. Thaler, "Compound TCP: A New TCP Congestion Control for High-Speed and Long Distance Networks", draft-sridharan-tcpm-ctcp-02 (work in progress), November 2008.

[I-D.stewart-tsvwg-sctpecn] Stewart, R. R., Tuexen, M., and X. Dong, "ECN for Stream Control Transmission Protocol (SCTP)", draft-stewart- tsvwg-sctpecn-05 (work in progress), January 2014.

[LinuxPacedChirping] Misund, J. and B. Briscoe, "Paced Chirping - Rethinking TCP start-up", Proc. Linux Netdev 0x13 , March 2019, .

[Mathis09] Mathis, M., "Relentless Congestion Control", PFLDNeT’09 , May 2009, .

[Paced-Chirping] Misund, J., "Rapid Acceleration in TCP Prague", Masters Thesis , May 2018, .

[PI2] De Schepper, K., Bondarenko, O., Tsang, I., and B. Briscoe, "PI^2 : A Linearized AQM for both Classic and Scalable TCP", Proc. ACM CoNEXT 2016 pp.105-119, December 2016, .

De Schepper & Briscoe Expires January 27, 2022 [Page 32] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

[QV] Briscoe, B. and P. Hurtig, "Up to Speed with Queue View", RITE Technical Report D2.3; Appendix C.2, August 2015, .

[RFC3540] Spring, N., Wetherall, D., and D. Ely, "Robust Explicit Congestion Notification (ECN) Signaling with Nonces", RFC 3540, DOI 10.17487/RFC3540, June 2003, .

[RFC3649] Floyd, S., "HighSpeed TCP for Large Congestion Windows", RFC 3649, DOI 10.17487/RFC3649, December 2003, .

[RFC4301] Kent, S. and K. Seo, "Security Architecture for the Internet Protocol", RFC 4301, DOI 10.17487/RFC4301, December 2005, .

De Schepper & Briscoe Expires January 27, 2022 [Page 33] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

[RFC4302] Kent, S., "IP Authentication Header", RFC 4302, DOI 10.17487/RFC4302, December 2005, .

[RFC4303] Kent, S., "IP Encapsulating Security Payload (ESP)", RFC 4303, DOI 10.17487/RFC4303, December 2005, .

[RFC4340] Kohler, E., Handley, M., and S. Floyd, "Datagram Congestion Control Protocol (DCCP)", RFC 4340, DOI 10.17487/RFC4340, March 2006, .

[RFC4341] Floyd, S. and E. Kohler, "Profile for Datagram Congestion Control Protocol (DCCP) Congestion Control ID 2: TCP-like Congestion Control", RFC 4341, DOI 10.17487/RFC4341, March 2006, .

[RFC4342] Floyd, S., Kohler, E., and J. Padhye, "Profile for Datagram Congestion Control Protocol (DCCP) Congestion Control ID 3: TCP-Friendly Rate Control (TFRC)", RFC 4342, DOI 10.17487/RFC4342, March 2006, .

[RFC4960] Stewart, R., Ed., "Stream Control Transmission Protocol", RFC 4960, DOI 10.17487/RFC4960, September 2007, .

[RFC5033] Floyd, S. and M. Allman, "Specifying New Congestion Control Algorithms", BCP 133, RFC 5033, DOI 10.17487/RFC5033, August 2007, .

[RFC5129] Davie, B., Briscoe, B., and J. Tay, "Explicit Congestion Marking in MPLS", RFC 5129, DOI 10.17487/RFC5129, January 2008, .

[RFC5348] Floyd, S., Handley, M., Padhye, J., and J. Widmer, "TCP Friendly Rate Control (TFRC): Protocol Specification", RFC 5348, DOI 10.17487/RFC5348, September 2008, .

[RFC5562] Kuzmanovic, A., Mondal, A., Floyd, S., and K. Ramakrishnan, "Adding Explicit Congestion Notification (ECN) Capability to TCP’s SYN/ACK Packets", RFC 5562, DOI 10.17487/RFC5562, June 2009, .

De Schepper & Briscoe Expires January 27, 2022 [Page 34] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

[RFC5622] Floyd, S. and E. Kohler, "Profile for Datagram Congestion Control Protocol (DCCP) Congestion ID 4: TCP-Friendly Rate Control for Small Packets (TFRC-SP)", RFC 5622, DOI 10.17487/RFC5622, August 2009, .

[RFC5681] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, .

[RFC5706] Harrington, D., "Guidelines for Considering Operations and Management of New Protocols and Protocol Extensions", RFC 5706, DOI 10.17487/RFC5706, November 2009, .

[RFC5865] Baker, F., Polk, J., and M. Dolly, "A Differentiated Services Code Point (DSCP) for Capacity-Admitted Traffic", RFC 5865, DOI 10.17487/RFC5865, May 2010, .

[RFC5925] Touch, J., Mankin, A., and R. Bonica, "The TCP Authentication Option", RFC 5925, DOI 10.17487/RFC5925, June 2010, .

[RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion Notification", RFC 6040, DOI 10.17487/RFC6040, November 2010, .

[RFC6077] Papadimitriou, D., Ed., Welzl, M., Scharf, M., and B. Briscoe, "Open Research Issues in Internet Congestion Control", RFC 6077, DOI 10.17487/RFC6077, February 2011, .

[RFC6347] Rescorla, E. and N. Modadugu, "Datagram Transport Layer Security Version 1.2", RFC 6347, DOI 10.17487/RFC6347, January 2012, .

[RFC6675] Blanton, E., Allman, M., Wang, L., Jarvinen, I., Kojo, M., and Y. Nishida, "A Conservative Loss Recovery Algorithm Based on Selective Acknowledgment (SACK) for TCP", RFC 6675, DOI 10.17487/RFC6675, August 2012, .

De Schepper & Briscoe Expires January 27, 2022 [Page 35] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

[RFC7560] Kuehlewind, M., Ed., Scheffenegger, R., and B. Briscoe, "Problem Statement and Requirements for Increased Accuracy in Explicit Congestion Notification (ECN) Feedback", RFC 7560, DOI 10.17487/RFC7560, August 2015, .

[RFC7567] Baker, F., Ed. and G. Fairhurst, Ed., "IETF Recommendations Regarding Active Queue Management", BCP 197, RFC 7567, DOI 10.17487/RFC7567, July 2015, .

[RFC7713] Mathis, M. and B. Briscoe, "Congestion Exposure (ConEx) Concepts, Abstract Mechanism, and Requirements", RFC 7713, DOI 10.17487/RFC7713, December 2015, .

[RFC8257] Bensley, S., Thaler, D., Balasubramanian, P., Eggert, L., and G. Judd, "Data Center TCP (DCTCP): TCP Congestion Control for Data Centers", RFC 8257, DOI 10.17487/RFC8257, October 2017, .

[RFC8298] Johansson, I. and Z. Sarker, "Self-Clocked Rate Adaptation for Multimedia", RFC 8298, DOI 10.17487/RFC8298, December 2017, .

[RFC8311] Black, D., "Relaxing Restrictions on Explicit Congestion Notification (ECN) Experimentation", RFC 8311, DOI 10.17487/RFC8311, January 2018, .

[RFC8312] Rhee, I., Xu, L., Ha, S., Zimmermann, A., Eggert, L., and R. Scheffenegger, "CUBIC for Fast Long-Distance Networks", RFC 8312, DOI 10.17487/RFC8312, February 2018, .

De Schepper & Briscoe Expires January 27, 2022 [Page 36] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

[RFC8511] Khademi, N., Welzl, M., Armitage, G., and G. Fairhurst, "TCP Alternative Backoff with ECN (ABE)", RFC 8511, DOI 10.17487/RFC8511, December 2018, .

[RFC8888] Sarker, Z., Perkins, C., Singh, V., and M. Ramalho, "RTP Control Protocol (RTCP) Feedback for Congestion Control", RFC 8888, DOI 10.17487/RFC8888, January 2021, .

[RFC8985] Cheng, Y., Cardwell, N., Dukkipati, N., and P. Jha, "The RACK-TLP Loss Detection Algorithm for TCP", RFC 8985, DOI 10.17487/RFC8985, February 2021, .

[RFC9000] Iyengar, J., Ed. and M. Thomson, Ed., "QUIC: A UDP-Based Multiplexed and Secure Transport", RFC 9000, DOI 10.17487/RFC9000, May 2021, .

[Savage-TCP] Savage, S., Cardwell, N., Wetherall, D., and T. Anderson, "TCP Congestion Control with a Misbehaving Receiver", ACM SIGCOMM Computer Communication Review 29(5):71--78, October 1999.

[SCReAM] Johansson, I., "SCReAM", github repository; , .

[sub-mss-prob] Briscoe, B. and K. De Schepper, "Scaling TCP’s Congestion Window for Small Round Trip Times", BT Technical Report TR-TUB8-2015-002, May 2015, .

[TCP-CA] Jacobson, V. and M. Karels, "Congestion Avoidance and Control", Laurence Berkeley Labs Technical Report , November 1988, .

[TCPPrague] Briscoe, B., "Notes: DCTCP evolution ’bar BoF’: Tue 21 Jul 2015, 17:40, Prague", tcpprague mailing list archive , July 2015, .

De Schepper & Briscoe Expires January 27, 2022 [Page 37] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

[VCP] Xia, Y., Subramanian, L., Stoica, I., and S. Kalyanaraman, "One more bit is enough", Proc. SIGCOMM’05, ACM CCR 35(4)37--48, 2005, .

Appendix A. The ’Prague L4S Requirements’

This appendix is informative, not normative. It gives a list of modifications to current scalable congestion controls so that they can be deployed over the public Internet and coexist safely with existing traffic. The list complements the normative requirements in Section 4 that a sender has to comply with before it can set the L4S identifier in packets it sends into the Internet. As well as necessary safety improvements (requirements) this appendix also includes preferable performance improvements (optimizations).

These recommendations have become know as the Prague L4S Requirements, because they were originally identified at an ad hoc meeting during IETF-94 in Prague [TCPPrague]. They were originally called the ’TCP Prague Requirements’, but they are not solely applicable to TCP, so the name and wording has been generalized for all transport protocols, and the name ’TCP Prague’ is now used for a specific implementation of the requirements.

At the time of writing, DCTCP [RFC8257] is the most widely used scalable transport protocol. In its current form, DCTCP is specified to be deployable only in controlled environments. Deploying it in the public Internet would lead to a number of issues, both from the safety and the performance perspective. The modifications and additional mechanisms listed in this section will be necessary for its deployment over the global Internet. Where an example is needed, DCTCP is used as a base, but it is likely that most of these requirements equally apply to other scalable congestion controls, covering adaptive real-time media, etc., not just capacity-seeking behaviours.

A.1. Requirements for Scalable Transport Protocols

A.1.1. Use of L4S Packet Identifier

Description: A scalable congestion control needs to distinguish the packets it sends from those sent by Classic congestion controls (see the precise normative requirement wording in Section 4.1).

Motivation: It needs to be possible for a network node to classify L4S packets without flow state into a queue that applies an L4S ECN marking behaviour and isolates L4S packets from the queuing delay of Classic packets.

De Schepper & Briscoe Expires January 27, 2022 [Page 38] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

A.1.2. Accurate ECN Feedback

Description: The transport protocol for a scalable congestion control needs to provide timely, accurate feedback about the extent of ECN marking experienced by all packets (see the precise normative requirement wording in Section 4.2).

Motivation: Classic congestion controls only need feedback about the existence of a congestion episode within a round trip, not precisely how many packets were marked with ECN or dropped. Therefore, in 2001, when ECN feedback was added to TCP [RFC3168], it could not inform the sender of more than one ECN mark per RTT. Since then, requirements for more accurate ECN feedback in TCP have been defined in [RFC7560] and [I-D.ietf-tcpm-accurate-ecn] specifies a change to the TCP protocol to satisfy these requirements. Most other transport protocols already satisfy this requirement (see Section 4.2).

A.1.3. Capable of Replacement by Classic Congestion Control

Description: It needs to be possible to replace the implementation of a scalable congestion control with a Classic control (see the precise normative requirement wording in Section 4.3).

Motivation: L4S is an experimental protocol, therefore it seems prudent to be able to disable it at source in case of insurmountable problems, perhaps due to some unexpected interaction on a particular sender; over a particular path or network; with a particular receiver or even ultimately an insurmountable problem with the experiment as a whole.

A.1.4. Fall back to Classic Congestion Control on Packet Loss

Description: As well as responding to ECN markings in a scalable way, a scalable congestion control needs to react to packet loss in a way that will coexist safely with a Reno congestion control [RFC5681] (see the precise normative requirement wording in Section 4.3).

Motivation: Part of the safety conditions for deploying a scalable congestion control on the public Internet is to make sure that it behaves properly when it builds a queue at a network bottleneck that has not been upgraded to support L4S. Packet loss can have many causes, but it usually has to be conservatively assumed that it is a sign of congestion. Therefore, on detecting packet loss, a scalable congestion control will need to fall back to Classic congestion control behaviour. If it does not comply with this requirement it could starve Classic traffic.

De Schepper & Briscoe Expires January 27, 2022 [Page 39] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

A scalable congestion control can be used for different types of transport, e.g. for real-time media or for reliable transport like TCP. Therefore, the particular Classic congestion control behaviour to fall back on will need to be dependent on the specific congestion control implementation. In the particular case of DCTCP, the DCTCP specification [RFC8257] states that "It is RECOMMENDED that an implementation deal with loss episodes in the same way as conventional TCP." For safe deployment of a scalable congestion control in the public Internet, the above requirement would need to be defined as a "MUST".

Even though a bottleneck is L4S capable, it might still become overloaded and have to drop packets. In this case, the sender may receive a high proportion of packets marked with the CE bit set and also experience loss. Current DCTCP implementations each react differently to this situation. At least one implementation reacts only to the drop signal (e.g. by halving the CWND) and at least another DCTCP implementation reacts to both signals (e.g. by halving the CWND due to the drop and also further reducing the CWND based on the proportion of marked packet). A third approach for the public Internet has been proposed that adjusts the loss response to result in a halving when combined with the ECN response. We believe that further experimentation is needed to understand what is the best behaviour for the public Internet, which may or not be one of these existing approaches.

A.1.5. Coexistence with Classic Congestion Control at Classic ECN bottlenecks

Description: Monitoring has to be in place so that a non-L4S but ECN- capable AQM can be detected at path bottlenecks. This is in case such an AQM has been implemented in a shared queue, in which case any long-running scalable flow would predominate over any simultaneous long-running Classic flow sharing the queue. The requirement is written so that such a problem could either be resolved in real-time, or via administrative intervention (see the precise normative requirement wording in Section 4.3).

Motivation: Similarly to the requirement in Appendix A.1.4, this requirement is a safety condition to ensure an L4S congestion control coexists well with Classic flows when it builds a queue at a shared network bottleneck that has not been upgraded to support L4S. Nonetheless, if necessary, it is considered reasonable to resolve such problems over management timescales (possibly involving human intervention) because:

De Schepper & Briscoe Expires January 27, 2022 [Page 40] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

o although a Classic flow can considerably reduce its throughput in the face of a competing scalable flow, it still makes progress and does not starve;

o implementations of a Classic ECN AQM in a queue that is intended to be shared are believed to be rare;

o detection of such AQMs is not always clear-cut; so focused out-of- band testing (or even contacting the relevant network operator) would improve certainty.

Therefore, the relevant normative requirement (Section 4.3) is divided into three stages: monitoring, detection and action:

Monitoring: Monitoring involves collection of the measurement data to be analysed. Monitoring is expressed as a ’MUST’ for uncontrolled environments, although the placement of the monitoring function is left open. Whether monitoring has to be applied in real-time is expressed as a ’SHOULD’. This allows for the possibility that the operator of an L4S sender (e.g. a CDN) might prefer to test out-of-band for signs of Classic ECN AQMs, perhaps to avoid continually consuming resources to monitor live traffic.

Detection: Detection involves analysis of the monitored data to detect the likelihood of a Classic ECN AQM. The requirements recommend that detection occurs live in real-time. However, detection is allowed to be deferred (e.g. it might involve further testing targeted at candidate AQMs);

Action: This involves the act of switching the sender to a Classic congestion control. This might occur in real-time within the congestion control for the subsequent duration of a flow, or it might involve administrative action to switch to Classic congestion control for a specific interface or for a certain set of destination addresses.

Instead of the sender taking action itself, the operator of the sender (e.g. a CDN) might prefer to ask the network operator to modify the Classic AQM’s treatment of L4S packets; or to ensure L4S packets bypass the AQM; or to upgrade the AQM to support L4S. Once L4S flows no longer shared the Classic ECN AQM they would obviously no longer detect it, and the requirement to act on it would no longer apply.

The whole set of normative requirements concerning Classic ECN AQMs does not apply in controlled environments, such as private networks or data centre networks. CDN servers placed within an access ISP’s

De Schepper & Briscoe Expires January 27, 2022 [Page 41] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

network can be considered as a single controlled environment, but any onward networks served by the access network, including all the attached customer networks, would be unlikely to fall under the same degree of coordinated control. Monitoring is expressed as a ’MUST’ for these uncontrolled segments of paths (e.g. beyond the access ISP in a home network), because there is a possibility that there might be a shared queue Classic ECN AQM in that segment. Nonetheless, the intent is to only require occasional monitoring of these uncontrolled regions, and not to burden CDN operators if monitoring never uncovers any potential problems, given it is anyway in the CDN’s own interests not to degrade the service of its own customers.

More detailed discussion of all the above options and alternatives can be found in [I-D.ietf-tsvwg-l4sops].

Having said all the above, the approach recommended in the requirements is to monitor, detect and act in real-time on live traffic. A passive monitoring algorithm to detect a Classic ECN AQM at the bottleneck and fall back to Classic congestion control is described in an extensive technical report [ecn-fallback], which also provides a link to Linux source code, and a large online visualization of its evaluation results. Very briefly, the algorithm primarily monitors RTT variation using the same algorithm that maintains the mean deviation of TCP’s smoothed RTT, but it smooths over a duration of the order of a Classic sawtooth. The outcome is also conditioned on other metrics such as the presence of CE marking and congestion avoidance phase having stabilized. The report also identifies further work to improve the approach, for instance improvements with low capacity links and combining the measurements with a cache of what had been learned about a path in previous connections. The report also suggests alternative approaches.

Although using passive measurements within live traffic (as above) can detect a Classic ECN AQM, it is much harder (perhaps impossible) to determine whether or not the AQM is in a shared queue. Nonetheless, this is much easier using active test traffic out-of- band, because two flows can be used. Section 4 of the same report [ecn-fallback] describes a simple technique to detect a Classic ECN AQM and determine whether it is in a shared queue, summarized here.

An L4S-enabled test server could be set up so that, when a test client accesses it, it serves a script that gets the client to open two parallel long-running flows. It could serve one with a Classic congestion control (C, that sets ECT(0)) and one with a scaleable CC (L, that sets ECT(1)).If neither flow induces any ECN marks, it can be presumed the path does not contain a Classic ECN AQM. If either flow induces some ECN marks, the server could measure the relative

De Schepper & Briscoe Expires January 27, 2022 [Page 42] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

flow rates and round trip times of the two flows. Table 1 shows the AQM that can be inferred for various cases.

+------+------+------+ | Rate | RTT | Inferred AQM | +------+------+------+ | L > C | L = C | Classic ECN AQM (FIFO) | | L = C | L = C | Classic ECN AQM (FQ) | | L = C | L < C | FQ-L4S AQM | | L ˜= C | L < C | Coupled DualQ AQM | +------+------+------+

Table 1: Out-of-band testing with two parallel flows. L:=L4S, C:=Classic.

Finally, we motivate the recommendation in Section 4.3 that a scalable congestion control is not expected to change to setting ECT(0) while it adapts its behaviour to coexist with Classic flows. This is because the sender needs to continue to check whether it made the right decision - and switch back if it was wrong, or if a different link becomes the bottleneck:

o If, as recommended, the sender changes only its behaviour but not its codepoint to Classic, its codepoint will still be compatible with either an L4S or a Classic AQM. If the bottleneck does actually support both, it will still classify ECT(1) into the same L4S queue, where the sender can measure that switching to Classic behaviour was wrong, so that it can switch back.

o In contrast, if the sender changes both its behaviour and its codepoint to Classic, even if the bottleneck supports both, it will classify ECT(0) into the Classic queue, reinforcing the sender’s incorrect decision so that it never switches back.

o Also, not changing codepoint avoids the risk of being flipped to a different path by a load balancer or multipath routing that hashes on the whole of the ex-ToS byte (unfortunately still a common pathology).

Note that if a flow is configured to _only_ use a Classic congestion control, it is then entirely appropriate not to use ECT(1).

A.1.6. Reduce RTT dependence

Description: A scalable congestion control needs to reduce RTT bias as much as possible at least over the low to typical range of RTTs that will interact in the intended deployment scenario (see the precise normative requirement wording in Section 4.3).

De Schepper & Briscoe Expires January 27, 2022 [Page 43] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

Motivation: The throughput of Classic congestion controls is known to be inversely proportional to RTT, so one would expect flows over very low RTT paths to nearly starve flows over larger RTTs. However, Classic congestion controls have never allowed a very low RTT path to exist because they induce a large queue. For instance, consider two paths with base RTT 1 ms and 100 ms. If a Classic congestion control induces a 100 ms queue, it turns these RTTs into 101 ms and 200 ms leading to a throughput ratio of about 2:1. Whereas if a scalable congestion control induces only a 1 ms queue, the ratio is 2:101, leading to a throughput ratio of about 50:1.

Therefore, with very small queues, long RTT flows will essentially starve, unless scalable congestion controls comply with this requirement.

The RTT bias in current Classic congestion controls works satisfactorily when the RTT is higher than typical, and L4S does not change that. So, there is no additional requirement for high RTT L4S flows to remove RTT bias - they can but they don’t have to.

A.1.7. Scaling down to fractional congestion windows

Description: A scalable congestion control needs to remain responsive to congestion when typical RTTs over the public Internet are significantly smaller because they are no longer inflated by queuing delay (see the precise normative requirement wording in Section 4.3).

Motivation: As currently specified, the minimum congestion window of ECN-capable TCP (and its derivatives) is expected to be 2 sender maximum segment sizes (SMSS), or 1 SMSS after a retransmission timeout. Once the congestion window reaches this minimum, if there is further ECN-marking, TCP is meant to wait for a retransmission timeout before sending another segment (see section 6.1.2 of [RFC3168]). In practice, most known window-based congestion control algorithms become unresponsive to congestion signals at this point. No matter how much drop or ECN marking, the congestion window no longer reduces. Instead, the sender’s lack of any further congestion response forces the queue to grow, overriding any AQM and increasing queuing delay (making the window large enough to become responsive again).

Most congestion controls for other transport protocols have a similar minimum, albeit when measured in bytes for those that use smaller packets.

L4S mechanisms significantly reduce queueing delay so, over the same path, the RTT becomes lower. Then this problem becomes surprisingly common [sub-mss-prob]. This is because, for the same link capacity,

De Schepper & Briscoe Expires January 27, 2022 [Page 44] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

smaller RTT implies a smaller window. For instance, consider a residential setting with an upstream broadband Internet access of 8 Mb/s, assuming a max segment size of 1500 B. Two upstream flows will each have the minimum window of 2 SMSS if the RTT is 6 ms or less, which is quite common when accessing a nearby data centre. So, any more than two such parallel TCP flows will become unresponsive and increase queuing delay.

Unless scalable congestion controls address this requirement from the start, they will frequently become unresponsive, negating the low latency benefit of L4S, for themselves and for others.

That would seem to imply that scalable congestion controllers ought to be required to be able work with a congestion window less than 1 SMSS. For instance, if an ECN-capable TCP gets an ECN-mark when it is already sitting at a window of 1 SMSS, RFC 3168 requires it to defer sending for a retransmission timeout. A less drastic but more complex mechanism can maintain a congestion window less than 1 SMSS (significantly less if necessary), as described in [Ahmed19]. Other approaches are likely to be feasible.

However, the requirement in Section 4.3 is worded as a "SHOULD" because the existence of a minimum window is not all bad. When competing with an unresponsive flow, a minimum window naturally protects the flow from starvation by at least keeping some data flowing.

By stating the requirement to go lower than 1 SMSS as a "SHOULD", while the requirement in RFC 3168 still stands as well, we shall be able to watch the choices of minimum window evolve in different scalable congestion controllers.

A.1.8. Measuring Reordering Tolerance in Time Units

Description: When detecting loss, a scalable congestion control needs to be tolerant to reordering over an adaptive time interval, which scales with throughput, rather than counting only in fixed units of packets, which does not scale (see the precise normative requirement wording in Section 4.3).

Motivation: A primary purpose of L4S is scalable throughput (it’s in the name). Scalability in all dimensions is, of course, also a goal of all IETF technology. The inverse linear congestion response in Section 4.3 is necessary, but not sufficient, to solve the congestion control scalability problem identified in [RFC3649]. As well as maintaining frequent ECN signals as rate scales, it is also important to ensure that a potentially false perception of loss does not limit throughput scaling.

De Schepper & Briscoe Expires January 27, 2022 [Page 45] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

End-systems cannot know whether a missing packet is due to loss or reordering, except in hindsight - if it appears later. So they can only deem that there has been a loss if a gap in the sequence space has not been filled, either after a certain number of subsequent packets has arrived (e.g. the 3 DupACK rule of standard TCP congestion control [RFC5681]) or after a certain amount of time (e.g. the RACK approach [RFC8985]).

As we attempt to scale packet rate over the years:

o Even if only _some_ sending hosts still deem that loss has occurred by counting reordered packets, _all_ networks will have to keep reducing the time over which they keep packets in order. If some link technologies keep the time within which reordering occurs roughly unchanged, then loss over these links, as perceived by these hosts, will appear to continually rise over the years.

o In contrast, if all senders detect loss in units of time, the time over which the network has to keep packets in order stays roughly invariant.

Therefore hosts have an incentive to detect loss in time units (so as not to fool themselves too often into detecting losses when there are none). And for hosts that are changing their congestion control implementation to L4S, there is no downside to including time-based loss detection code in the change (loss recovery implemented in hardware is an exception, covered later). Therefore requiring L4S hosts to detect loss in time-based units would not be a burden.

If this requirement is not placed on L4S hosts, even though it would be no burden on them to do so, all networks will face unnecessary uncertainty over whether some L4S hosts might be detecting loss by counting packets. Then _all_ link technologies will have to unnecessarily keep reducing the time within which reordering occurs. That is not a problem for some link technologies, but it becomes increasingly challenging for other link technologies to continue to scale, particularly those relying on channel bonding for scaling, such as LTE, 5G and DOCSIS.

Given Internet paths traverse many link technologies, any scaling limit for these more challenging access link technologies would become a scaling limit for the Internet as a whole.

It might be asked how it helps to place this loss detection requirement only on L4S hosts, because networks will still face uncertainty over whether non-L4S flows are detecting loss by counting DupACKs. The answer is that those link technologies for which it is challenging to keep squeezing the reordering time will only need to

De Schepper & Briscoe Expires January 27, 2022 [Page 46] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

do so for non-L4S traffic (which they can do because the L4S identifier is visible at the IP layer). Therefore, they can focus their processing and memory resources into scaling non-L4S (Classic) traffic. Then, the higher the proportion of L4S traffic, the less of a scaling challenge they will have.

To summarize, there is no reason for L4S hosts not to be part of the solution instead of part of the problem.

Requirement ("MUST") or recommendation ("SHOULD")? As explained above, this is a subtle interoperability issue between hosts and networks, which seems to need a "MUST". Unless networks can be certain that all L4S hosts follow the time-based approach, they still have to cater for the worst case - continually squeeze reordering into a smaller and smaller duration - just for hosts that might be using the counting approach. However, it was decided to express this as a recommendation, using "SHOULD". The main justification was that networks can still be fairly certain that L4S hosts will follow this recommendation, because following it offers only gain and no pain.

Details:

The speed of loss recovery is much more significant for short flows than long, therefore a good compromise is to adapt the reordering window; from a small fraction of the RTT at the start of a flow, to a larger fraction of the RTT for flows that continue for many round trips.

This is broadly the approach adopted by TCP RACK (Recent ACKnowledgements) [RFC8985]. However, RACK starts with the 3 DupACK approach, because the RTT estimate is not necessarily stable. As long as the initial window is paced, such initial use of 3 DupACK counting would amount to time-based loss detection and therefore would satisfy the time-based loss detection recommendation of Section 4.3. This is because pacing of the initial window would ensure that 3 DupACKs early in the connection would be spread over a small fraction of the round trip.

As mentioned above, hardware implementations of loss recovery using DupACK counting exist (e.g. some implementations of RoCEv2 for RDMA). For low latency, these implementations can change their congestion control to implement L4S, because the congestion control (as distinct from loss recovery) is implemented in software. But they cannot easily satisfy this loss recovery requirement. However, it is believed they do not need to, because such implementations are believed to solely exist in controlled environments, where the network technology keeps reordering extremely low anyway. This is

De Schepper & Briscoe Expires January 27, 2022 [Page 47] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

why controlled environments with hardly any reordering are excluded from the scope of the normative recommendation in Section 4.3.

Detecting loss in time units also prevents the ACK-splitting attacks described in [Savage-TCP].

A.2. Scalable Transport Protocol Optimizations

A.2.1. Setting ECT in Control Packets and Retransmissions

Description: This item concerns TCP and its derivatives (e.g. SCTP) as well as RTP/RTCP [RFC6679]. The original specification of ECN for TCP precluded the use of ECN on control packets and retransmissions. To improve performance, scalable transport protocols ought to enable ECN at the IP layer in TCP control packets (SYN, SYN-ACK, pure ACKs, etc.) and in retransmitted packets. The same is true for derivatives of TCP, e.g. SCTP. Similarly [RFC6679] precludes the use of ECT on RTCP datagrams, in case the path changes after it has been checked for ECN traversal.

Motivation (TCP): RFC 3168 prohibits the use of ECN on these types of TCP packet, based on a number of arguments. This means these packets are not protected from congestion loss by ECN, which considerably harms performance, particularly for short flows. [I-D.ietf-tcpm-generalized-ecn] proposes experimental use of ECN on all types of TCP packet as long as AccECN feedback [I-D.ietf-tcpm-accurate-ecn] is available (which itself satisfies the accurate feedback requirement in Section 4.2 for using a scalable congestion control).

Motivation (RTCP): L4S experiments in general will need to observe the rule in [RFC6679] that precludes ECT on RTCP datagrams. Nonetheless, as ECN usage becomes more widespread, it would be useful to conduct specific experiments with ECN-capable RTCP to gather data on whether such caution is necessary.

A.2.2. Faster than Additive Increase

Description: It would improve performance if scalable congestion controls did not limit their congestion window increase to the standard additive increase of 1 SMSS per round trip [RFC5681] during congestion avoidance. The same is true for derivatives of TCP congestion control, including similar approaches used for real-time media.

Motivation: As currently defined [RFC8257], DCTCP uses the traditional Reno additive increase in congestion avoidance phase. When the available capacity suddenly increases (e.g. when another

De Schepper & Briscoe Expires January 27, 2022 [Page 48] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

flow finishes, or if radio capacity increases) it can take very many round trips to take advantage of the new capacity. TCP Cubic was designed to solve this problem, but as flow rates have continued to increase, the delay accelerating into available capacity has become prohibitive. See, for instance, the examples in Section 1.2. Even when out of its Reno-compatibility mode, every 8x scaling of Cubic’s flow rate leads to 2x more acceleration delay.

In the steady state, DCTCP induces about 2 ECN marks per round trip, so it is possible to quickly detect when these signals have disappeared and seek available capacity more rapidly, while minimizing the impact on other flows (Classic and scalable) [LinuxPacedChirping]. Alternatively, approaches such as Adaptive Acceleration (A2DTCP [A2DTCP]) have been proposed to address this problem in data centres, which might be deployable over the public Internet.

A.2.3. Faster Convergence at Flow Start

Description: It would improve performance if scalable congestion controls converged (reached their steady-state share of the capacity) faster than Classic congestion controls or at least no slower. This affects the flow start behaviour of any L4S congestion control derived from a Classic transport that uses TCP slow start, including those for real-time media.

Motivation: As an example, a new DCTCP flow takes longer than a Classic congestion control to obtain its share of the capacity of the bottleneck when there are already ongoing flows using the bottleneck capacity. In a data centre environment DCTCP takes about a factor of 1.5 to 2 longer to converge due to the much higher typical level of ECN marking that DCTCP background traffic induces, which causes new flows to exit slow start early [Alizadeh-stability]. In testing for use over the public Internet the convergence time of DCTCP relative to a regular loss-based TCP slow start is even less favourable [Paced-Chirping] due to the shallow ECN marking threshold needed for L4S. It is exacerbated by the typically greater mismatch between the link rate of the sending host and typical Internet access bottlenecks. This problem is detrimental in general, but would particularly harm the performance of short flows relative to Classic congestion controls.

Appendix B. Compromises in the Choice of L4S Identifier

This appendix is informative, not normative. As explained in Section 2, there is insufficient space in the IP header (v4 or v6) to fully accommodate every requirement. So the choice of L4S identifier

De Schepper & Briscoe Expires January 27, 2022 [Page 49] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

involves tradeoffs. This appendix records the pros and cons of the choice that was made.

Non-normative recap of the chosen codepoint scheme:

Packets with ECT(1) and conditionally packets with CE signify L4S semantics as an alternative to the semantics of Classic ECN [RFC3168], specifically:

* The ECT(1) codepoint signifies that the packet was sent by an L4S-capable sender.

* Given shortage of codepoints, both L4S and Classic ECN sides of an AQM have to use the same CE codepoint to indicate that a packet has experienced congestion. If a packet that had already been marked CE in an upstream buffer arrived at a subsequent AQM, this AQM would then have to guess whether to classify CE packets as L4S or Classic ECN. Choosing the L4S treatment is a safer choice, because then a few Classic packets might arrive early, rather than a few L4S packets arriving late.

* Additional information might be available if the classifier were transport-aware. Then it could classify a CE packet for Classic ECN treatment if the most recent ECT packet in the same flow had been marked ECT(0). However, the L4S service ought not to need tranport-layer awareness.

Cons:

Consumes the last ECN codepoint: The L4S service could potentially supersede the service provided by Classic ECN, therefore using ECT(1) to identify L4S packets could ultimately mean that the ECT(0) codepoint was ’wasted’ purely to distinguish one form of ECN from its successor.

ECN hard in some lower layers: It is not always possible to support the equivalent of an IP-ECN field in an AQM acting in a buffer below the IP layer [I-D.ietf-tsvwg-ecn-encap-guidelines]. Then, depending on the lower layer scheme, the L4S service might have to drop rather than mark frames even though they might encapsulate an ECN-capable packet.

Risk of reordering Classic CE packets within a flow: Classifying all CE packets into the L4S queue risks any CE packets that were originally ECT(0) being incorrectly classified as L4S. If there were delay in the Classic queue, these incorrectly classified CE packets would arrive early, which is a form of reordering.

De Schepper & Briscoe Expires January 27, 2022 [Page 50] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

Reordering within a microflow can cause TCP senders (and senders of similar transports) to retransmit spuriously. However, the risk of spurious retransmissions would be extremely low for the following reasons:

1. It is quite unusual to experience queuing at more than one bottleneck on the same path (the available capacities have to be identical).

2. In only a subset of these unusual cases would the first bottleneck support Classic ECN marking while the second supported L4S ECN marking, which would be the only scenario where some ECT(0) packets could be CE marked by an AQM supporting Classic ECN then the remainder experienced further delay through the Classic side of a subsequent L4S DualQ AQM.

3. Even then, when a few packets are delivered early, it takes very unusual conditions to cause a spurious retransmission, in contrast to when some packets are delivered late. The first bottleneck has to apply CE-marks to at least N contiguous packets and the second bottleneck has to inject an uninterrupted sequence of at least N of these packets between two packets earlier in the stream (where N is the reordering window that the transport protocol allows before it considers a packet is lost).

For example consider N=3, and consider the sequence of packets 100, 101, 102, 103,... and imagine that packets 150,151,152 from later in the flow are injected as follows: 100, 150, 151, 101, 152, 102, 103... If this were late reordering, even one packet arriving out of sequence would trigger a spurious retransmission, but there is no spurious retransmission here with early reordering, because packet 101 moves the cumulative ACK counter forward before 3 packets have arrived out of order. Later, when packets 148, 149, 153... arrive, even though there is a 3-packet hole, there will be no problem, because the packets to fill the hole are already in the receive buffer.

4. Even with the current TCP recommendation of N=3 [RFC5681] spurious retransmissions will be unlikely for all the above reasons. As RACK [RFC8985] is becoming widely deployed, it tends to adapt its reordering window to a larger value of N, which will make the chance of a contiguous sequence of N early arrivals vanishingly small.

5. Even a run of 2 CE marks within a Classic ECN flow is unlikely, given FQ-CoDel is the only known widely deployed AQM

De Schepper & Briscoe Expires January 27, 2022 [Page 51] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

that supports Classic ECN marking and it takes great care to separate out flows and to space any markings evenly along each flow.

It is extremely unlikely that the above set of 5 eventualities that are each unusual in themselves would all happen simultaneously. But, even if they did, the consequences would hardly be dire: the odd spurious fast retransmission. Whenever the traffic source (a Classic congestion control) mistakes the reordering of a string of CE marks for a loss, one might think that it will reduce its congestion window as well as emitting a spurious retransmission. However, it would have already reduced its congestion window when the CE markings arrived early. If it is using ABE [RFC8511], it might reduce cwnd a little more for a loss than for a CE mark. But it will revert that reduction once it detects that the retransmission was spurious.

In conclusion, the impact of early reordering on spurious retransmissions due to CE being ambiguous will generally be vanishingly small.

Insufficient anti-replay window in some pre-existing VPNs: If delay is reduced for a subset of the flows within a VPN, the anti-replay feature of some VPNs is known to potentially mistake the difference in delay for a replay attack. Section 6.2 recommends that the anti-replay window at the VPN egress is sufficiently sized, as required by the relevant specifications. However, in some VPN implementations the maximum anti-replay window is insufficient to cater for a large delay difference at prevailing packet rates. Section 6.2 suggests alternative work-rounds for such cases, but end-users using L4S over a VPN will need to be able to recognize the symptoms of this problem, in order to seek out these work-rounds.

Hard to distinguish Classic ECN AQM: With this scheme, when a source receives ECN feedback, it is not explicitly clear which type of AQM generated the CE markings. This is not a problem for Classic ECN sources that send ECT(0) packets, because an L4S AQM will recognize the ECT(0) packets as Classic and apply the appropriate Classic ECN marking behaviour.

However, in the absence of explicit disambiguation of the CE markings, an L4S source needs to use heuristic techniques to work out which type of congestion response to apply (see Appendix A.1.5). Otherwise, if long-running Classic flow(s) are sharing a Classic ECN AQM bottleneck with long-running L4S flow(s), which then apply an L4S response to Classic CE signals, the L4S flows would outcompete the Classic flow(s). Experiments

De Schepper & Briscoe Expires January 27, 2022 [Page 52] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

have shown that L4S flows can take about 20 times more capacity share than equivalent Classic flows. Nonetheless, as link capacity reduces (e.g. to 4 Mb/s), the inequality reduces. So Classic flows always make progress and are not starved.

When L4S was first proposed (in 2015, 14 years after [RFC3168] was published), it was believed that Classic ECN AQMs had failed to be deployed, because research measurements had found little or no evidence of CE marking. In subsequent years Classic ECN was included in per-flow-queuing (FQ) deployments, however an FQ scheduler stops an L4S flow outcompeting Classic, because it enforces equality between flow rates. It is not known whether there have been any non-FQ deployments of Classic ECN AQMs in the subsequent years, or whether there will be in future.

An algorithm for detecting a Classic ECN AQM as soon as a flow stabilizes after start-up has been proposed [ecn-fallback] (see Appendix A.1.5 for a brief summary). Testbed evaluations of v2 of the algorithm have shown detection is reasonably good for Classic ECN AQMs, in a wide range of circumstances. However, although it can correctly detect an L4S ECN AQM in many circumstances, its is often incorrect at low link capacities and/or high RTTs. Although this is the safe way round, there is a danger that it will discourage use of the algorithm.

Non-L4S service for control packets: Solely for the case of TCP, the Classic ECN RFCs [RFC3168] and [RFC5562] require a sender to clear the ECN field to Not-ECT on retransmissions and on certain control packets specifically pure ACKs, window probes and SYNs. When L4S packets are classified by the ECN field, these TCP control packets would not be classified into an L4S queue, and could therefore be delayed relative to the other packets in the flow. This would not cause reordering (because retransmissions are already out of order, and these control packets typically carry no data). However, it would make critical TCP control packets more vulnerable to loss and delay. To address this problem, [I-D.ietf-tcpm-generalized-ecn] proposes an experiment in which all TCP control packets and retransmissions are ECN-capable as long as appropriate ECN feedback is available in each case.

Pros:

Should work e2e: The ECN field generally propagates end-to-end across the Internet without being wiped or mangled, at least over fixed networks. Unlike the DSCP, the setting of the ECN field is at least meant to be forwarded unchanged by networks that do not support ECN.

De Schepper & Briscoe Expires January 27, 2022 [Page 53] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

Should work in tunnels: The L4S identifiers work across and within any tunnel that propagates the ECN field in any of the variant ways it has been defined since ECN-tunneling was first specified in the year 2001 [RFC3168]. However, it is likely that some tunnels still do not implement ECN propagation at all.

Should work for many link technologies: At most, but not all, path bottlenecks there is IP-awareness, so that L4S AQMs can be located where the IP-ECN field can be manipulated. Bottlenecks at lower layer nodes without IP-awareness either have to use drop to signal congestion or a specific congestion notification facility has to be defined for that link technology, including propagation to and from IP-ECN. The programme to define these is progressing and in each case so far the scheme already defined for ECN inherently supports L4S as well (see Section 6.1).

Could migrate to one codepoint: If all Classic ECN senders eventually evolve to use the L4S service, the ECT(0) codepoint could be reused for some future purpose, but only once use of ECT(0) packets had reduced to zero, or near-zero, which might never happen.

L4 not required: Being based on the ECN field, this scheme does not need the network to access transport layer flow identifiers. Nonetheless, it does not preclude solutions that do.

Appendix C. Potential Competing Uses for the ECT(1) Codepoint

The ECT(1) codepoint of the ECN field has already been assigned once for the ECN nonce [RFC3540], which has now been categorized as historic [RFC8311]. ECN is probably the only remaining field in the Internet Protocol that is common to IPv4 and IPv6 and still has potential to work end-to-end, with tunnels and with lower layers. Therefore, ECT(1) should not be reassigned to a different experimental use (L4S) without carefully assessing competing potential uses. These fall into the following categories:

C.1. Integrity of Congestion Feedback

Receiving hosts can fool a sender into downloading faster by suppressing feedback of ECN marks (or of losses if retransmissions are not necessary or available otherwise).

The historic ECN nonce protocol [RFC3540] proposed that a TCP sender could set either of ECT(0) or ECT(1) in each packet of a flow and remember the sequence it had set. If any packet was lost or congestion marked, the receiver would miss that bit of the sequence. An ECN Nonce receiver had to feed back the least significant bit of

De Schepper & Briscoe Expires January 27, 2022 [Page 54] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

the sum, so it could not suppress feedback of a loss or mark without a 50-50 chance of guessing the sum incorrectly.

It is highly unlikely that ECT(1) will be needed for integrity protection in future. The ECN Nonce RFC [RFC3540] as been reclassified as historic, partly because other ways have been developed to protect feedback integrity of TCP and other transports [RFC8311] that do not consume a codepoint in the IP header. For instance:

o the sender can test the integrity of the receiver’s feedback by occasionally setting the IP-ECN field to a value normally only set by the network. Then it can test whether the receiver’s feedback faithfully reports what it expects (see para 2 of Section 20.2 of [RFC3168]. This works for loss and it will work for the accurate ECN feedback [RFC7560] intended for L4S.

o A network can enforce a congestion response to its ECN markings (or packet losses) by auditing congestion exposure (ConEx) [RFC7713]. Whether the receiver or a downstream network is suppressing congestion feedback or the sender is unresponsive to the feedback, or both, ConEx audit can neutralise any advantage that any of these three parties would otherwise gain.

o The TCP authentication option (TCP-AO [RFC5925]) can be used to detect any tampering with TCP congestion feedback (whether malicious or accidental). TCP’s congestion feedback fields are immutable end-to-end, so they are amenable to TCP-AO protection, which covers the main TCP header and TCP options by default. However, TCP-AO is often too brittle to use on many end-to-end paths, where middleboxes can make verification fail in their attempts to improve performance or security, e.g. by resegmentation or shifting the sequence space.

C.2. Notification of Less Severe Congestion than CE

Various researchers have proposed to use ECT(1) as a less severe congestion notification than CE, particularly to enable flows to fill available capacity more quickly after an idle period, when another flow departs or when a flow starts, e.g. VCP [VCP], Queue View (QV) [QV].

Before assigning ECT(1) as an identifier for L4S, we must carefully consider whether it might be better to hold ECT(1) in reserve for future standardisation of rapid flow acceleration, which is an important and enduring problem [RFC6077].

De Schepper & Briscoe Expires January 27, 2022 [Page 55] Internet-Draft L4S ECN Protocol for Very Low Queuing Delay July 2021

Pre-Congestion Notification (PCN) is another scheme that assigns alternative semantics to the ECN field. It uses ECT(1) to signify a less severe level of pre-congestion notification than CE [RFC6660]. However, the ECN field only takes on the PCN semantics if packets carry a Diffserv codepoint defined to indicate PCN marking within a controlled environment. PCN is required to be applied solely to the outer header of a tunnel across the controlled region in order not to interfere with any end-to-end use of the ECN field. Therefore a PCN region on the path would not interfere with the L4S service identifier defined in Section 3.

Authors’ Addresses

Koen De Schepper Nokia Bell Labs Antwerp Belgium

Email: [email protected] URI: https://www.bell-labs.com/usr/koen.de_schepper

Bob Briscoe (editor) Independent UK

Email: [email protected] URI: http://bobbriscoe.net/

De Schepper & Briscoe Expires January 27, 2022 [Page 56] TSVWG V. Roca Internet-Draft INRIA Updates: 6363 (if approved) A. Begen Intended status: Standards Track Networked Media Expires: July 15, 2019 January 11, 2019

Forward Error Correction (FEC) Framework Extension to Sliding Window Codes draft-ietf-tsvwg-fecframe-ext-08

Abstract

RFC 6363 describes a framework for using Forward Error Correction (FEC) codes to provide protection against packet loss. The framework supports applying FEC to arbitrary packet flows over unreliable transport and is primarily intended for real-time, or streaming, media. However, FECFRAME as per RFC 6363 is restricted to block FEC codes. This document updates RFC 6363 to support FEC Codes based on a sliding encoding window, in addition to Block FEC Codes, in a backward-compatible way. During multicast/broadcast real-time content delivery, the use of sliding window codes significantly improves robustness in harsh environments, with less repair traffic and lower FEC-related added latency.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

This Internet-Draft will expire on July 15, 2019.

Copyright Notice

Roca & Begen Expires July 15, 2019 [Page 1] Internet-Draft FEC Framework Extension January 2019

Table of Contents

1. Introduction ...... 2 2. Definitions and Abbreviations ...... 4 3. Summary of Architecture Overview ...... 7 4. Procedural Overview ...... 10 4.1. General ...... 10 4.2. Sender Operation with Sliding Window FEC Codes . . . . . 10 4.3. Receiver Operation with Sliding Window FEC Codes . . . . 13 5. Protocol Specification ...... 15 5.1. General ...... 15 5.2. FEC Framework Configuration Information ...... 16 5.3. FEC Scheme Requirements ...... 16 6. Feedback ...... 16 7. Transport Protocols ...... 17 8. Congestion Control ...... 17 9. Implementation Status ...... 17 10. Security Considerations ...... 17 11. Operations and Management Considerations ...... 18 12. IANA Considerations ...... 18 13. Acknowledgments ...... 18 14. References ...... 18 14.1. Normative References ...... 18 14.2. Informative References ...... 19 Appendix A. About Sliding Encoding Window Management (informational) ...... 20 Authors’ Addresses ...... 21

1. Introduction

Many applications need to transport a continuous stream of packetized data from a source (sender) to one or more destinations (receivers) over networks that do not provide guaranteed packet delivery. In particular packets may be lost, which is strictly the focus of this document: we assume that transmitted packets are either lost (e.g., because of a congested router, of a poor signal-to-noise ratio in a wireless network, or because the number of bit errors exceeds the correction capabilities of the physical-layer error correcting code)

Roca & Begen Expires July 15, 2019 [Page 2] Internet-Draft FEC Framework Extension January 2019

or received by the transport protocol without any corruption (i.e., the bit-errors, if any, have been fixed by the physical-layer error correcting code and therefore are hidden to the upper layers).

For these use-cases, Forward Error Correction (FEC) applied within the transport or application layer is an efficient technique to improve packet transmission robustness in presence of packet losses (or "erasures"), without going through packet retransmissions that create a delay often incompatible with real-time constraints. The FEC Building Block defined in [RFC5052] provides a framework for the definition of Content Delivery Protocols (CDPs) that make use of separately-defined FEC schemes. Any CDP defined according to the requirements of the FEC Building Block can then easily be used with any FEC Scheme that is also defined according to the requirements of the FEC Building Block.

Then FECFRAME [RFC6363] provides a framework to define Content Delivery Protocols (CDPs) that provide FEC protection for arbitrary packet flows over an unreliable datagram service transport such as UDP. It is primarily intended for real-time or streaming media applications, using broadcast, multicast, or on-demand delivery.

However, [RFC6363] only considers block FEC schemes defined in accordance with the FEC Building Block [RFC5052] (e.g., [RFC6681], [RFC6816] or [RFC6865]). These codes require the input flow(s) to be segmented into a sequence of blocks. Then FEC encoding (at a sender or an encoding middlebox) and decoding (at a receiver or a decoding middlebox) are both performed on a per-block basis. For instance, if the current block encompasses the 100’s to 119’s source symbols (i.e., a block of size 20 symbols) of an input flow, encoding (and decoding) will be performed on this block independently of other blocks. This approach has major impacts on FEC encoding and decoding delays. The data packets of continuous media flow(s) may be passed to the transport layer immediately, without delay. But the block creation time, that depends on the number of source symbols in this block, impacts both the FEC encoding delay (since encoding requires that all source symbols be known), and mechanically the packet loss recovery delay at a receiver (since no repair symbol for the current block can be generated and therefore received before that time). Therefore a good value for the block size is necessarily a balance between the maximum FEC decoding latency at the receivers (which must be in line with the most stringent real-time requirement of the protected flow(s), hence an incentive to reduce the block size), and the desired robustness against long loss bursts (which increases with the block size, hence an incentive to increase this size).

This document updates [RFC6363] in order to also support FEC codes based on a sliding encoding window (A.K.A. convolutional codes)

Roca & Begen Expires July 15, 2019 [Page 3] Internet-Draft FEC Framework Extension January 2019

[RFC8406]. This encoding window, either of fixed or variable size, slides over the set of source symbols. FEC encoding is launched whenever needed, from the set of source symbols present in the sliding encoding window at that time. This approach significantly reduces FEC-related latency, since repair symbols can be generated and passed to the transport layer on-the-fly, at any time, and can be regularly received by receivers to quickly recover packet losses. Using sliding window FEC codes is therefore highly beneficial to real-time flows, one of the primary targets of FECFRAME. [RLC-ID] provides an example of such FEC Scheme for FECFRAME, built upon the simple sliding window Random Linear Codes (RLC).

This document is fully backward compatible with [RFC6363]. Indeed:

o this FECFRAME update does not prevent nor compromise in any way the support of block FEC codes. Both types of codes can nicely co-exist, just like different block FEC schemes can co-exist;

o each sliding window FEC Scheme is associated to a specific FEC Encoding ID subject to IANA registration, just like block FEC Schemes;

o any receiver, for instance a legacy receiver that only supports block FEC schemes, can easily identify the FEC Scheme used in a FECFRAME session. Indeed, the FEC Encoding ID that identifies the FEC Scheme is carried in the FEC Framework Configuration Information (see section 5.5 of [RFC6363]). For instance, when the Session Description Protocol (SDP) is used to carry the FEC Framework Configuration Information, the FEC Encoding ID can be communicated in the "encoding-id=" parameter of a "fec-repair- flow" attribute [RFC6364]. This mechanism is the basic approach for a FECFRAME receiver to determine whether or not it supports the FEC Scheme used in a given FECFRAME session;

This document leverages on [RFC6363] and re-uses its structure. It proposes new sections specific to sliding window FEC codes whenever required. The only exception is Section 3 that provides a quick summary of FECFRAME in order to facilitate the understanding of this document to readers not familiar with the concepts and terminology.

2. Definitions and Abbreviations

The following list of definitions and abbreviations is copied from [RFC6363], adding only the Block/sliding window FEC Code and Encoding/Decoding Window definitions (tagged with "ADDED"):

Application Data Unit (ADU): The unit of source data provided as payload to the transport layer. For instance, it can be a

Roca & Begen Expires July 15, 2019 [Page 4] Internet-Draft FEC Framework Extension January 2019

payload containing the result of the RTP packetization of a compressed video frame.

ADU Flow: A sequence of ADUs associated with a transport-layer flow identifier (such as the standard 5-tuple {source IP address, source port, destination IP address, destination port, transport protocol}).

AL-FEC: Application-layer Forward Error Correction.

Application Protocol: Control protocol used to establish and control the source flow being protected, e.g., the Real-Time Streaming Protocol (RTSP).

Content Delivery Protocol (CDP): A complete application protocol specification that, through the use of the framework defined in this document, is able to make use of FEC schemes to provide FEC capabilities.

FEC Code: An algorithm for encoding data such that the encoded data flow is resilient to data loss. Note that, in general, FEC codes may also be used to make a data flow resilient to corruption, but that is not considered in this document.

Block FEC Code: (ADDED) An FEC Code that operates on blocks, i.e., for which the input flow MUST be segmented into a sequence of blocks, FEC encoding and decoding being performed independently on a per-block basis.

Sliding Window FEC Code: (ADDED) An FEC Code that can generate repair symbols on-the-fly, at any time, from the set of source symbols present in the sliding encoding window at that time. These codes are also known as convolutional codes.

FEC Framework: A protocol framework for the definition of Content Delivery Protocols using FEC, such as the framework defined in this document.

FEC Framework Configuration Information: Information that controls the operation of the FEC Framework.

FEC Payload ID: Information that identifies the contents and provides positional information of a packet with respect to the FEC Scheme.

FEC Repair Packet: At a sender (respectively, at a receiver), a payload submitted to (respectively, received from) the transport

Roca & Begen Expires July 15, 2019 [Page 5] Internet-Draft FEC Framework Extension January 2019

protocol containing one or more repair symbols along with a Repair FEC Payload ID and possibly an RTP header.

FEC Scheme: A specification that defines the additional protocol aspects required to use a particular FEC code with the FEC Framework.

FEC Source Packet: At a sender (respectively, at a receiver), a payload submitted to (respectively, received from) the transport protocol containing an ADU along with an optional Explicit Source FEC Payload ID.

Repair Flow: The packet flow carrying FEC data.

Repair FEC Payload ID: A FEC Payload ID specifically for use with repair packets.

Source Flow: The packet flow to which FEC protection is to be applied. A source flow consists of ADUs.

Source FEC Payload ID: A FEC Payload ID specifically for use with source packets.

Source Protocol: A protocol used for the source flow being protected, e.g., RTP.

Transport Protocol: The protocol used for the transport of the source and repair flows, using an unreliable datagram service such as UDP.

Encoding Window: (ADDED) Set of Source Symbols available at the sender/coding node that are used to generate a repair symbol, with a Sliding Window FEC Code.

Decoding Window: (ADDED) Set of received or decoded source and repair symbols available at a receiver that are used to decode erased source symbols, with a Sliding Window FEC Code.

Code Rate: The ratio between the number of source symbols and the number of encoding symbols. By definition, the code rate is such that 0 < code rate <= 1. A code rate close to 1 indicates that a small number of repair symbols have been produced during the encoding process.

Encoding Symbol: Unit of data generated by the encoding process. With systematic codes, source symbols are part of the encoding symbols.

Roca & Begen Expires July 15, 2019 [Page 6] Internet-Draft FEC Framework Extension January 2019

Packet Erasure Channel: A communication path where packets are either lost (e.g., in our case, by a congested router, or because the number of transmission errors exceeds the correction capabilities of the physical-layer code) or received. When a packet is received, it is assumed that this packet is not corrupted (i.e., in our case, the bit-errors, if any, are fixed by the physical-layer code and therefore hidden to the upper layers).

Repair Symbol: Encoding symbol that is not a source symbol.

Source Block: Group of ADUs that are to be FEC protected as a single block. This notion is restricted to Block FEC Codes.

Source Symbol: Unit of data used during the encoding process.

Systematic Code: FEC code in which the source symbols are part of the encoding symbols.

3. Summary of Architecture Overview

The architecture of [RFC6363], Section 3, equally applies to this FECFRAME extension and is not repeated here. However, we provide hereafter a quick summary to facilitate the understanding of this document to readers not familiar with the concepts and terminology.

Roca & Begen Expires July 15, 2019 [Page 7] Internet-Draft FEC Framework Extension January 2019

+------+ | Application | +------+ | | (1) Application Data Units (ADUs) | v +------+ +------+ | FEC Framework | | | | |------>| FEC Scheme | |(2) Construct source |(3) Source Block | | | blocks | |(4) FEC Encoding| |(6) Construct FEC |<------| | | Source and Repair | | | | Packets |(5) Explicit Source FEC | | +------+ Payload IDs +------+ | Repair FEC Payload IDs | Repair symbols | |(7) FEC Source and Repair Packets v +------+ | Transport Protocol | +------+

Figure 1: FECFRAME architecture at a sender.

The FECFRAME architecture is illustrated in Figure 1 from the sender’s point of view, in case of a block FEC Scheme. It shows an application generating an ADU flow (other flows, from other applications, may co-exist). These ADUs, of variable size, must be somehow mapped to source symbols of fixed size (this fixed size is a requirement of all FEC Schemes that comes from the way mathematical operations are applied to symbols content). This is the goal of an ADU-to-symbols mapping process that is FEC-Scheme specific (see below). Once the source block is built, taking into account both the FEC Scheme constraints (e.g., in terms of maximum source block size) and the application’s flow constraints (e.g., in terms of real-time constraints), the associated source symbols are handed to the FEC Scheme in order to produce an appropriate number of repair symbols. FEC Source Packets (containing ADUs) and FEC Repair Packets (containing one or more repair symbols each) are then generated and sent using an appropriate transport protocol (more precisely [RFC6363], Section 7, requires a transport protocol providing an unreliable datagram service, such as UDP). In practice FEC Source Packets may be passed to the transport layer as soon as available, without having to wait for FEC encoding to take place. In that case

Roca & Begen Expires July 15, 2019 [Page 8] Internet-Draft FEC Framework Extension January 2019

a copy of the associated source symbols needs to be kept within FECFRAME for future FEC encoding purposes.

At a receiver (not shown), FECFRAME processing operates in a similar way, taking as input the incoming FEC Source and Repair Packets received. In case of FEC Source Packet losses, the FEC decoding of the associated block may recover all (in case of successful decoding) or a subset potentially empty (otherwise) of the missing source symbols. After source-symbol-to-ADU mapping, when lost ADUs are recovered, they are then assigned to their respective flow (see below). ADUs are returned to the application(s), either in their initial transmission order (in that case ADUs received after an erased one will be delayed until FEC decoding has taken place) or not (in that case each ADU is returned as soon as it is received or recovered), depending on the application requirements.

FECFRAME features two subtle mechanisms:

o ADUs-to-source-symbols mapping: in order to manage variable size ADUs, FECFRAME and FEC Schemes can use small, fixed size symbols and create a mapping between ADUs and symbols. To each ADU this mechanism prepends a length field (plus a flow identifier, see below) and pads the result to a multiple of the symbol size. A small ADU may be mapped to a single source symbol while a large one may be mapped to multiple symbols. The mapping details are FEC-Scheme-dependent and must be defined in the associated document;

o Assignment of decoded ADUs to flows in multi-flow configurations: when multiple flows are multiplexed over the same FECFRAME instance, a problem is to assign a decoded ADU to the right flow (UDP port numbers and IP addresses traditionally used to map incoming ADUs to flows are not recovered during FEC decoding). To make it possible, at the FECFRAME sending instance, each ADU is prepended with a flow identifier (1 byte) during the ADU-to- source-symbols mapping (see above). The flow identifiers are also shared between all FECFRAME instances as part of the FEC Framework Configuration Information. This (flow identifier + length + application payload + padding), called ADUI, is then FEC protected. Therefore a decoded ADUI contains enough information to assign the ADU to the right flow.

A few aspects are not covered by FECFRAME, namely:

o [RFC6363] section 8 does not detail any congestion control mechanism, but only provides high level normative requirements;

Roca & Begen Expires July 15, 2019 [Page 9] Internet-Draft FEC Framework Extension January 2019

o the possibility of having feedbacks from receiver(s) is considered out of scope, although such a mechanism may exist within the application (e.g., through RTCP control messages);

o flow adaptation at a FECFRAME sender (e.g., how to set the FEC code rate based on transmission conditions) is not detailed, but it needs to comply with the congestion control normative requirements (see above).

4. Procedural Overview

4.1. General

The general considerations of [RFC6363], Section 4.1, that are specific to block FEC codes are not repeated here.

With a Sliding Window FEC Code, the FEC Source Packet MUST contain information to identify the position occupied by the ADU within the source flow, in terms specific to the FEC Scheme. This information is known as the Source FEC Payload ID, and the FEC Scheme is responsible for defining and interpreting it.

With a Sliding Window FEC Code, the FEC Repair Packets MUST contain information that identifies the relationship between the contained repair payloads and the original source symbols used during encoding. This information is known as the Repair FEC Payload ID, and the FEC Scheme is responsible for defining and interpreting it.

The Sender Operation ([RFC6363], Section 4.2.) and Receiver Operation ([RFC6363], Section 4.3) are both specific to block FEC codes and therefore omitted below. The following two sections detail similar operations for Sliding Window FEC codes.

4.2. Sender Operation with Sliding Window FEC Codes

With a Sliding Window FEC Scheme, the following operations, illustrated in Figure 2 for the generic case (non-RTP repair flows), and in Figure 3 for the case of RTP repair flows, describe a possible way to generate compliant source and repair flows:

1. A new ADU is provided by the application.

2. The FEC Framework communicates this ADU to the FEC Scheme.

3. The sliding encoding window is updated by the FEC Scheme. The ADU-to-source-symbols mapping as well as the encoding window management details are both the responsibility of the FEC Scheme

Roca & Begen Expires July 15, 2019 [Page 10] Internet-Draft FEC Framework Extension January 2019

and MUST be detailed there. Appendix A provides non-normative hints about what FEC Scheme designers need to consider;

4. The Source FEC Payload ID information of the source packet is determined by the FEC Scheme. If required by the FEC Scheme, the Source FEC Payload ID is encoded into the Explicit Source FEC Payload ID field and returned to the FEC Framework.

5. The FEC Framework constructs the FEC Source Packet according to [RFC6363] Figure 6, using the Explicit Source FEC Payload ID provided by the FEC Scheme if applicable.

6. The FEC Source Packet is sent using normal transport-layer procedures. This packet is sent using the same ADU flow identification information as would have been used for the original source packet if the FEC Framework were not present (e.g., the source and destination addresses and UDP port numbers on the IP datagram carrying the source packet will be the same whether or not the FEC Framework is applied).

7. When the FEC Framework needs to send one or several FEC Repair Packets (e.g., according to the target Code Rate), it asks the FEC Scheme to create one or several repair packet payloads from the current sliding encoding window along with their Repair FEC Payload ID.

8. The Repair FEC Payload IDs and repair packet payloads are provided back by the FEC Scheme to the FEC Framework.

9. The FEC Framework constructs FEC Repair Packets according to [RFC6363] Figure 7, using the FEC Payload IDs and repair packet payloads provided by the FEC Scheme.

10. The FEC Repair Packets are sent using normal transport-layer procedures. The port(s) and multicast group(s) to be used for FEC Repair Packets are defined in the FEC Framework Configuration Information.

Roca & Begen Expires July 15, 2019 [Page 11] Internet-Draft FEC Framework Extension January 2019

+------+ | Application | +------+ | | (1) New Application Data Unit (ADU) v +------+ +------+ | FEC Framework | | FEC Scheme | | |------>| | | | (2) New ADU |(3) Update of | | | | encoding | | |<------| window | |(5) Construct FEC | (4) Explicit Source | | | Source Packet | FEC Payload ID(s) |(7) FEC | | |<------| encoding | |(9) Construct FEC | (8) Repair FEC Payload ID | | | Repair Packet(s) | + Repair symbol(s) +------+ +------+ | | (6) FEC Source Packet | (10) FEC Repair Packets v +------+ | Transport Protocol | +------+

Figure 2: Sender Operation with Sliding Window FEC Codes

Roca & Begen Expires July 15, 2019 [Page 12] Internet-Draft FEC Framework Extension January 2019

+------+ | Application | +------+ | | (1) New Application Data Unit (ADU) v +------+ +------+ | FEC Framework | | FEC Scheme | | |------>| | | | (2) New ADU |(3) Update of | | | | encoding | | |<------| window | |(5) Construct FEC | (4) Explicit Source | | | Source Packet | FEC Payload ID(s) |(7) FEC | | |<------| encoding | |(9) Construct FEC | (8) Repair FEC Payload ID | | | Repair Packet(s) | + Repair symbol(s) +------+ +------+ | | |(6) Source |(10) Repair payloads | packets | | + ------+ | | RTP | | +------+ v v +------+ | Transport Protocol | +------+

Figure 3: Sender Operation with Sliding Window FEC Codes and RTP Repair Flows

4.3. Receiver Operation with Sliding Window FEC Codes

With a Sliding Window FEC Scheme, the following operations, illustrated in Figure 4 for the generic case (non-RTP repair flows), and in Figure 5 for the case of RTP repair flows. The only differences with respect to block FEC codes lie in steps (4) and (5). Therefore this section does not repeat the other steps of [RFC6363], Section 4.3, "Receiver Operation". The new steps (4) and (5) are:

4. The FEC Scheme uses the received FEC Payload IDs (and derived FEC Source Payload IDs when the Explicit Source FEC Payload ID field is not used) to insert source and repair packets into the decoding window in the right way. If at least one source packet is missing and at least one repair packet has been received, then FEC decoding is attempted to recover missing source payloads. The FEC Scheme determines whether source packets have been lost

Roca & Begen Expires July 15, 2019 [Page 13] Internet-Draft FEC Framework Extension January 2019

and whether enough repair packets have been received to decode any or all of the missing source payloads.

5. The FEC Scheme returns the received and decoded ADUs to the FEC Framework, along with indications of any ADUs that were missing and could not be decoded.

+------+ | Application | +------+ ^ |(6) ADUs | +------+ +------+ | FEC Framework | | FEC Scheme | | |<------| | |(2)Extract FEC Payload|(5) ADUs |(4) FEC Decoding | IDs and pass IDs & |------>| | | payloads to FEC |(3) Explicit Source FEC +------+ | scheme | Payload IDs +------+ Repair FEC Payload IDs ^ Source payloads | Repair payloads |(1) FEC Source | and Repair Packets +------+ | Transport Protocol | +------+

Figure 4: Receiver Operation with Sliding Window FEC Codes

Roca & Begen Expires July 15, 2019 [Page 14] Internet-Draft FEC Framework Extension January 2019

+------+ | Application | +------+ ^ |(6) ADUs | +------+ +------+ | FEC Framework | | FEC Scheme | | |<------| | |(2)Extract FEC Payload|(5) ADUs |(4) FEC Decoding| | IDs and pass IDs & |------>| | | payloads to FEC |(3) Explicit Source FEC +------+ | scheme | Payload IDs +------+ Repair FEC Payload IDs ^ ^ Source payloads | | Repair payloads |Source pkts |Repair payloads | | +-- |------+ |RTP| | RTP Processing | | | +------|-- -+ | +------|--+ | | | RTP Demux | | +------+ ^ |(1) FEC Source and Repair Packets | +------+ | Transport Protocol | +------+

Figure 5: Receiver Operation with Sliding Window FEC Codes and RTP Repair Flows

5. Protocol Specification

5.1. General

This section discusses the protocol elements for the FEC Framework specific to Sliding Window FEC schemes. The global formats of source data packets (i.e., [RFC6363], Figure 6) and repair data packets (i.e., [RFC6363], Figures 7 and 8) remain the same with Sliding Window FEC codes. They are not repeated here.

Roca & Begen Expires July 15, 2019 [Page 15] Internet-Draft FEC Framework Extension January 2019

5.2. FEC Framework Configuration Information

The FEC Framework Configuration Information considerations of [RFC6363], Section 5.5, equally applies to this FECFRAME extension and is not repeated here.

5.3. FEC Scheme Requirements

The FEC Scheme requirements of [RFC6363], Section 5.6, mostly apply to this FECFRAME extension and are not repeated here. An exception though is the "full specification of the FEC code", item (4), that is specific to block FEC codes. The following item (4-bis) applies in case of Sliding Window FEC schemes:

4-bis. A full specification of the Sliding Window FEC code

This specification MUST precisely define the valid FEC-Scheme- Specific Information values, the valid FEC Payload ID values, and the valid packet payload sizes (where packet payload refers to the space within a packet dedicated to carrying encoding symbols).

Furthermore, given valid values of the FEC-Scheme-Specific Information, a valid Repair FEC Payload ID value, a valid packet payload size, and a valid encoding window (i.e., a set of source symbols), the specification MUST uniquely define the values of the encoding symbol (or symbols) to be included in the repair packet payload with the given Repair FEC Payload ID value.

Additionally, the FEC Scheme associated to a Sliding Window FEC Code:

o MUST define the relationships between ADUs and the associated source symbols (mapping);

o MUST define the management of the encoding window that slides over the set of ADUs. Appendix A provides non normative hints about what FEC Scheme designers need to consider;

o MUST define the management of the decoding window. This usually consists in managing a system of linear equations (in case of a linear FEC code);

6. Feedback

The discussion of [RFC6363], Section 6, equally applies to this FECFRAME extension and is not repeated here.

Roca & Begen Expires July 15, 2019 [Page 16] Internet-Draft FEC Framework Extension January 2019

7. Transport Protocols

The discussion of [RFC6363], Section 7, equally applies to this FECFRAME extension and is not repeated here.

8. Congestion Control

The discussion of [RFC6363], Section 8, equally applies to this FECFRAME extension and is not repeated here.

9. Implementation Status

Editor’s notes: RFC Editor, please remove this section motivated by RFC 7942 before publishing the RFC. Thanks!

An implementation of FECFRAME extended to Sliding Window codes exists:

o Organisation: Inria

o Description: This is an implementation of FECFRAME extended to Sliding Window codes and supporting the RLC FEC Scheme [RLC-ID]. It is based on: (1) a proprietary implementation of FECFRAME, made by Inria and Expway for which interoperability tests have been conducted; and (2) a proprietary implementation of RLC Sliding Window FEC Codes.

o Maturity: the basic FECFRAME maturity is "production", the FECFRAME extension maturity is "under progress".

o Coverage: the software implements a subset of [RFC6363], as specialized by the 3GPP eMBMS standard [MBMSTS]. This software also covers the additional features of FECFRAME extended to Sliding Window codes, in particular the RLC FEC Scheme.

o Licensing: proprietary.

o Implementation experience: maximum.

o Information update date: March 2018.

o Contact: [email protected]

10. Security Considerations

This FECFRAME extension does not add any new security consideration. All the considerations of [RFC6363], Section 9, apply to this document as well. However, for the sake of completeness, the

Roca & Begen Expires July 15, 2019 [Page 17] Internet-Draft FEC Framework Extension January 2019

following goal can be added to the list provided in Section 9.1 "Problem Statement" of [RFC6363]:

o Attacks can try to corrupt source flows in order to modify the receiver application’s behavior (as opposed to just denying service).

11. Operations and Management Considerations

This FECFRAME extension does not add any new Operations and Management Consideration. All the considerations of [RFC6363], Section 10, apply to this document as well.

12. IANA Considerations

No IANA actions are required for this document.

A FEC Scheme for use with this FEC Framework is identified via its FEC Encoding ID. It is subject to IANA registration in the "FEC Framework (FECFRAME) FEC Encoding IDs" registry. All the rules of [RFC6363], Section 11, apply and are not repeated here.

13. Acknowledgments

The authors would like to thank Christer Holmberg, David Black, Gorry Fairhurst, and Emmanuel Lochin, Spencer Dawkins, Ben Campbell, Benjamin Kaduk, Eric Rescorla, Adam Roach, and Greg Skinner for their valuable feedback on this document. This document being an extension to [RFC6363], the authors would also like to thank Mark Watson as the main author of that RFC.

14. References

14.1. Normative References

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, .

[RFC6363] Watson, M., Begen, A., and V. Roca, "Forward Error Correction (FEC) Framework", RFC 6363, DOI 10.17487/RFC6363, October 2011, .

[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, .

Roca & Begen Expires July 15, 2019 [Page 18] Internet-Draft FEC Framework Extension January 2019

14.2. Informative References

[MBMSTS] 3GPP, "Multimedia Broadcast/Multicast Service (MBMS); Protocols and codecs", 3GPP TS 26.346, March 2009, .

[RFC5052] Watson, M., Luby, M., and L. Vicisano, "Forward Error Correction (FEC) Building Block", RFC 5052, DOI 10.17487/RFC5052, August 2007, .

[RFC6364] Begen, A., "Session Description Protocol Elements for the Forward Error Correction (FEC) Framework", RFC 6364, DOI 10.17487/RFC6364, October 2011, .

[RFC6681] Watson, M., Stockhammer, T., and M. Luby, "Raptor Forward Error Correction (FEC) Schemes for FECFRAME", RFC 6681, DOI 10.17487/RFC6681, August 2012, .

[RFC6816] Roca, V., Cunche, M., and J. Lacan, "Simple Low-Density Parity Check (LDPC) Staircase Forward Error Correction (FEC) Scheme for FECFRAME", RFC 6816, DOI 10.17487/RFC6816, December 2012, .

[RFC6865] Roca, V., Cunche, M., Lacan, J., Bouabdallah, A., and K. Matsuzono, "Simple Reed-Solomon Forward Error Correction (FEC) Scheme for FECFRAME", RFC 6865, DOI 10.17487/RFC6865, February 2013, .

[RFC8406] Adamson, B., Adjih, C., Bilbao, J., Firoiu, V., Fitzek, F., Ghanem, S., Lochin, E., Masucci, A., Montpetit, M-J., Pedersen, M., Peralta, G., Roca, V., Ed., Saxena, P., and S. Sivakumar, "Taxonomy of Coding Techniques for Efficient Network Communications", RFC 8406, DOI 10.17487/RFC8406, June 2018, .

[RLC-ID] Roca, V. and B. Teibi, "Sliding Window Random Linear Code (RLC) Forward Erasure Correction (FEC) Scheme for FECFRAME", Work in Progress, Transport Area Working Group (TSVWG) draft-ietf-tsvwg-rlc-fec-scheme (Work in Progress), September 2018, .

Roca & Begen Expires July 15, 2019 [Page 19] Internet-Draft FEC Framework Extension January 2019

Appendix A. About Sliding Encoding Window Management (informational)

The FEC Framework does not specify the management of the sliding encoding window which is the responsibility of the FEC Scheme. This annex only provides a few informational hints.

Source symbols are added to the sliding encoding window each time a new ADU is available at the sender, after the ADU-to-source-symbol mapping specific to the FEC Scheme.

Source symbols are removed from the sliding encoding window, for instance:

o after a certain delay, when an "old" ADU of a real-time flow times out. The source symbol retention delay in the sliding encoding window should therefore be initialized according to the real-time features of incoming flow(s) when applicable;

o once the sliding encoding window has reached its maximum size (there is usually an upper limit to the sliding encoding window size). In that case the oldest symbol is removed each time a new source symbol is added.

Several considerations can impact the management of this sliding encoding window:

o at the source flows level: real-time constraints can limit the total time source symbols can remain in the encoding window;

o at the FEC code level: theoretical or practical limitations (e.g., because of computational complexity) can limit the number of source symbols in the encoding window;

o at the FEC Scheme level: signaling and window management are intrinsically related. For instance, an encoding window composed of a non-sequential set of source symbols requires an appropriate signaling to inform a receiver of the composition of the encoding window, and the associated transmission overhead can limit the maximum encoding window size. On the opposite, an encoding window always composed of a sequential set of source symbols simplifies signaling: providing the identity of the first source symbol plus their number is sufficient, which creates a fixed and relatively small transmission overhead.

Roca & Begen Expires July 15, 2019 [Page 20] Internet-Draft FEC Framework Extension January 2019

Authors’ Addresses

Vincent Roca INRIA Univ. Grenoble Alpes France

EMail: [email protected]

Ali Begen Networked Media Konya Turkey

EMail: [email protected]

Roca & Begen Expires July 15, 2019 [Page 21] Transport Area Working Group B. Briscoe, Ed. Internet-Draft Independent Intended status: Informational K. De Schepper Expires: January 2, 2022 Nokia Bell Labs M. Bagnulo Braun Universidad Carlos III de Madrid G. White CableLabs July 1, 2021

Low Latency, Low Loss, Scalable Throughput (L4S) Internet Service: Architecture draft-ietf-tsvwg-l4s-arch-10

Abstract

This document describes the L4S architecture, which enables Internet applications to achieve Low queuing Latency, Low Loss, and Scalable throughput (L4S). The insight on which L4S is based is that the root cause of queuing delay is in the congestion controllers of senders, not in the queue itself. The L4S architecture is intended to enable _all_ Internet applications to transition away from congestion control algorithms that cause queuing delay, to a new class of congestion controls that induce very little queuing, aided by explicit congestion signaling from the network. This new class of congestion control can provide low latency for capacity-seeking flows, so applications can achieve both high bandwidth and low latency.

The architecture primarily concerns incremental deployment. It defines mechanisms that allow the new class of L4S congestion controls to coexist with ’Classic’ congestion controls in a shared network. These mechanisms aim to ensure that the latency and throughput performance using an L4S-compliant congestion controller is usually much better (and never worse) than the performance would have been using a ’Classic’ congestion controller, and that competing flows continuing to use ’Classic’ controllers are typically not impacted by the presence of L4S. These characteristics are important to encourage adoption of L4S congestion control algorithms and L4S compliant network elements.

The L4S architecture consists of three components: network support to isolate L4S traffic from classic traffic; protocol features that allow network elements to identify L4S traffic; and host support for L4S congestion controls.

Briscoe, et al. Expires January 2, 2022 [Page 1] Internet-Draft L4S Architecture July 2021

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

This Internet-Draft will expire on January 2, 2022.

Copyright Notice

Table of Contents

1. Introduction ...... 3 2. L4S Architecture Overview ...... 5 3. Terminology ...... 6 4. L4S Architecture Components ...... 7 4.1. Protocol Mechanisms ...... 7 4.2. Network Components ...... 9 4.3. Host Mechanisms ...... 11 5. Rationale ...... 12 5.1. Why These Primary Components? ...... 12 5.2. What L4S adds to Existing Approaches ...... 15 6. Applicability ...... 18 6.1. Applications ...... 18 6.2. Use Cases ...... 19 6.3. Applicability with Specific Link Technologies ...... 20

Briscoe, et al. Expires January 2, 2022 [Page 2] Internet-Draft L4S Architecture July 2021

6.4. Deployment Considerations ...... 21 6.4.1. Deployment Topology ...... 21 6.4.2. Deployment Sequences ...... 22 6.4.3. L4S Flow but Non-ECN Bottleneck ...... 25 6.4.4. L4S Flow but Classic ECN Bottleneck ...... 26 6.4.5. L4S AQM Deployment within Tunnels ...... 26 7. IANA Considerations (to be removed by RFC Editor) ...... 26 8. Security Considerations ...... 26 8.1. Traffic Rate (Non-)Policing ...... 26 8.2. ’Latency Friendliness’ ...... 27 8.3. Interaction between Rate Policing and L4S ...... 29 8.4. ECN Integrity ...... 30 8.5. Privacy Considerations ...... 30 9. Acknowledgements ...... 31 10. Informative References ...... 31 Appendix A. Standardization items ...... 39 Authors’ Addresses ...... 41

1. Introduction

It is increasingly common for _all_ of a user’s applications at any one time to require low delay: interactive Web, Web services, voice, conversational video, interactive video, interactive remote presence, instant messaging, online gaming, remote desktop, cloud-based applications and video-assisted remote control of machinery and industrial processes. In the last decade or so, much has been done to reduce propagation delay by placing caches or servers closer to users. However, queuing remains a major, albeit intermittent, component of latency. For instance spikes of hundreds of milliseconds are common, even with state-of-the-art active queue management (AQM). During a long-running flow, queuing is typically configured to cause overall network delay to roughly double relative to expected base (unloaded) path delay. Low loss is also important because, for interactive applications, losses translate into even longer retransmission delays.

It has been demonstrated that, once access network bit rates reach levels now common in the developed world, increasing capacity offers diminishing returns if latency (delay) is not addressed. Differentiated services (Diffserv) offers Expedited Forwarding (EF [RFC3246]) for some packets at the expense of others, but this is not sufficient when all (or most) of a user’s applications require low latency.

Therefore, the goal is an Internet service with very Low queueing Latency, very Low Loss and Scalable throughput (L4S). Very low queuing latency means less than 1 millisecond (ms) on average and less than about 2 ms at the 99th percentile. L4S is potentially for

Briscoe, et al. Expires January 2, 2022 [Page 3] Internet-Draft L4S Architecture July 2021

_all_ traffic - a service for all traffic needs none of the configuration or management baggage (traffic policing, traffic contracts) associated with favouring some traffic over others. This document describes the L4S architecture for achieving these goals.

It must be said that queuing delay only degrades performance infrequently [Hohlfeld14]. It only occurs when a large enough capacity-seeking (e.g. TCP) flow is running alongside the user’s traffic in the bottleneck link, which is typically in the access network. Or when the low latency application is itself a large capacity-seeking or adaptive rate (e.g. interactive video) flow. At these times, the performance improvement from L4S must be sufficient that network operators will be motivated to deploy it.

Active Queue Management (AQM) is part of the solution to queuing under load. AQM improves performance for all traffic, but there is a limit to how much queuing delay can be reduced by solely changing the network; without addressing the root of the problem.

The root of the problem is the presence of standard TCP congestion control (Reno [RFC5681]) or compatible variants (e.g. TCP Cubic [RFC8312]). We shall use the term ’Classic’ for these Reno- friendly congestion controls. Classic congestion controls induce relatively large saw-tooth-shaped excursions up the queue and down again, which have been growing as flow rate scales [RFC3649]. So if a network operator naively attempts to reduce queuing delay by configuring an AQM to operate at a shallower queue, a Classic congestion control will significantly underutilize the link at the bottom of every saw-tooth.

It has been demonstrated that if the sending host replaces a Classic congestion control with a ’Scalable’ alternative, when a suitable AQM is deployed in the network the performance under load of all the above interactive applications can be significantly improved. For instance, queuing delay under heavy load with the example DCTCP/DualQ solution cited below on a DSL or Ethernet link is roughly 1 to 2 milliseconds at the 99th percentile without losing link utilization [DualPI2Linux], [DCttH15] (for other link types, see Section 6.3). This compares with 5-20 ms on _average_ with a Classic congestion control and current state-of-the-art AQMs such as FQ- CoDel [RFC8290], PIE [RFC8033] or DOCSIS PIE [RFC8034] and about 20-30 ms at the 99th percentile [DualPI2Linux].

It has also been demonstrated [DCttH15], [DualPI2Linux] that it is possible to deploy such an L4S service alongside the existing best efforts service so that all of a user’s applications can shift to it when their stack is updated. Access networks are typically designed with one link as the bottleneck for each site (which might be a home,

Briscoe, et al. Expires January 2, 2022 [Page 4] Internet-Draft L4S Architecture July 2021

small enterprise or mobile device), so deployment at each end of this link should give nearly all the benefit in each direction. The L4S approach also requires component mechanisms at the endpoints to fulfill its goal. This document presents the L4S architecture, by describing the different components and how they interact to provide the scalable, low latency, low loss Internet service.

2. L4S Architecture Overview

There are three main components to the L4S architecture; the AQM in the network, the congestion control on the host, and the protocol between them:

1) Network: L4S traffic needs to be isolated from the queuing latency of Classic traffic. One queue per application flow (FQ) is one way to achieve this, e.g. FQ-CoDel [RFC8290]. However, just two queues is sufficient and does not require inspection of transport layer headers in the network, which is not always possible (see Section 5.2). With just two queues, it might seem impossible to know how much capacity to schedule for each queue without inspecting how many flows at any one time are using each. And it would be undesirable to arbitrarily divide access network capacity into two partitions. The Dual Queue Coupled AQM was developed as a minimal complexity solution to this problem. It acts like a ’semi-permeable’ membrane that partitions latency but not bandwidth. As such, the two queues are for transition from Classic to L4S behaviour, not bandwidth prioritization. Section 4 gives a high level explanation of how FQ and DualQ solutions work, and [I-D.ietf-tsvwg-aqm-dualq-coupled] gives a full explanation of the DualQ Coupled AQM framework.

2) Protocol: A host needs to distinguish L4S and Classic packets with an identifier so that the network can classify them into their separate treatments. [I-D.ietf-tsvwg-ecn-l4s-id] concludes that all alternatives involve compromises, but the ECT(1) and CE codepoints of the ECN field represent a workable solution.

3) Host: Scalable congestion controls already exist. They solve the scaling problem with Reno congestion control that was explained in [RFC3649]. The one used most widely (in controlled environments) is Data Center TCP (DCTCP [RFC8257]), which has been implemented and deployed in Windows Server Editions (since 2012), in Linux and in FreeBSD. Although DCTCP as-is ’works’ well over the public Internet, most implementations lack certain safety features that will be necessary once it is used outside controlled environments like data centres (see Section 6.4.3 and Appendix A). Scalable congestion control will also need to be implemented in protocols other than TCP (QUIC, SCTP, RTP/RTCP, RMCAT, etc.). Indeed,

Briscoe, et al. Expires January 2, 2022 [Page 5] Internet-Draft L4S Architecture July 2021

between the present document being drafted and published, the following scalable congestion controls were implemented: TCP Prague [PragueLinux], QUIC Prague, an L4S variant of the RMCAT SCReAM controller [SCReAM] and the L4S ECN part of BBRv2 [BBRv2] intended for TCP and QUIC transports.

3. Terminology

Classic Congestion Control: A congestion control behaviour that can co-exist with standard Reno [RFC5681] without causing significantly negative impact on its flow rate [RFC5033]. With Classic congestion controls, such as Reno or Cubic, because flow rate has scaled since TCP congestion control was first designed in 1988, it now takes hundreds of round trips (and growing) to recover after a congestion signal (whether a loss or an ECN mark) as shown in the examples in Section 5.1 and [RFC3649]. Therefore control of queuing and utilization becomes very slack, and the slightest disturbances (e.g. from new flows starting) prevent a high rate from being attained.

Scalable Congestion Control: A congestion control where the average time from one congestion signal to the next (the recovery time) remains invariant as the flow rate scales, all other factors being equal. This maintains the same degree of control over queueing and utilization whatever the flow rate, as well as ensuring that high throughput is more robust to disturbances. For instance, DCTCP averages 2 congestion signals per round-trip whatever the flow rate, as do other recently developed scalable congestion controls, e.g. Relentless TCP [Mathis09], TCP Prague [I-D.briscoe-iccrg-prague-congestion-control], [PragueLinux], BBRv2 [BBRv2] and the L4S variant of SCReAM for real-time media [SCReAM], [RFC8298]). See Section 4.3 of [I-D.ietf-tsvwg-ecn-l4s-id] for more explanation.

Classic service: The Classic service is intended for all the congestion control behaviours that co-exist with Reno [RFC5681] (e.g. Reno itself, Cubic [RFC8312], Compound [I-D.sridharan-tcpm-ctcp], TFRC [RFC5348]). The term ’Classic queue’ means a queue providing the Classic service.

Low-Latency, Low-Loss Scalable throughput (L4S) service: The ’L4S’ service is intended for traffic from scalable congestion control algorithms, such as the Prague congestion control [I-D.briscoe-iccrg-prague-congestion-control], which was derived from DCTCP [RFC8257]. The L4S service is for more general traffic than just TCP Prague--it allows the set of congestion controls with similar scaling properties to Prague to evolve, such

Briscoe, et al. Expires January 2, 2022 [Page 6] Internet-Draft L4S Architecture July 2021

as the examples listed above (Relentless, SCReAM). The term ’L4S queue’ means a queue providing the L4S service.

Both Classic and L4S services can cope with a proportion of unresponsive or less-responsive traffic as well, but in the L4S case its rate has to be smooth enough or low enough not build a queue (e.g. DNS, VoIP, game sync datagrams, etc).

Classic ECN: The original Explicit Congestion Notification (ECN) protocol [RFC3168], which requires ECN signals to be treated as equivalent to drops, both when generated in the network and when responded to by the sender.

Site: A home, mobile device, small enterprise or campus, where the network bottleneck is typically the access link to the site. Not all network arrangements fit this model but it is a useful, widely applicable generalization.

4. L4S Architecture Components

The L4S architecture is composed of the elements in the following three subsections.

4.1. Protocol Mechanisms

The L4S architecture involves: a) unassignment of an identifier; b) reassignment of the same identifier; and c) optional further identifiers:

Briscoe, et al. Expires January 2, 2022 [Page 7] Internet-Draft L4S Architecture July 2021

a. An essential aspect of a scalable congestion control is the use of explicit congestion signals. ’Classic’ ECN [RFC3168] requires an ECN signal to be treated as equivalent to drop, both when it is generated in the network and when it is responded to by hosts. L4S needs networks and hosts to support a more fine-grained meaning for each ECN signal that is less severe than a drop, so that the L4S signals:

* can be much more frequent;

* can be signalled immediately, without the signficant delay required to smooth out fluctuations in the queue.

To enable L4S, the standards track [RFC3168] has had to be updated to allow L4S packets to depart from the ’equivalent to drop’ constraint. [RFC8311] is a standards track update to relax specific requirements in RFC 3168 (and certain other standards track RFCs), which clears the way for the experimental changes proposed for L4S. [RFC8311] also reclassifies the original experimental assignment of the ECT(1) codepoint as an ECN nonce [RFC3540] as historic.

b. [I-D.ietf-tsvwg-ecn-l4s-id] recommends ECT(1) is used as the identifier to classify L4S packets into a separate treatment from Classic packets. This satisfies the requirements for identifying an alternative ECN treatment in [RFC4774].

The CE codepoint is used to indicate Congestion Experienced by both L4S and Classic treatments. This raises the concern that a Classic AQM earlier on the path might have marked some ECT(0) packets as CE. Then these packets will be erroneously classified into the L4S queue. Appendix B of [I-D.ietf-tsvwg-ecn-l4s-id] explains why five unlikely eventualities all have to coincide for this to have any detrimental effect, which even then would only involve a vanishingly small likelihood of a spurious retransmission.

c. A network operator might wish to include certain unresponsive, non-L4S traffic in the L4S queue if it is deemed to be smoothly enough paced and low enough rate not to build a queue. For instance, VoIP, low rate datagrams to sync online games, relatively low rate application-limited traffic, DNS, LDAP, etc. This traffic would need to be tagged with specific identifiers, e.g. a low latency Diffserv Codepoint such as Expedited Forwarding (EF [RFC3246]), Non-Queue-Building (NQB [I-D.ietf-tsvwg-nqb]), or operator-specific identifiers.

Briscoe, et al. Expires January 2, 2022 [Page 8] Internet-Draft L4S Architecture July 2021

4.2. Network Components

The L4S architecture aims to provide low latency without the _need_ for per-flow operations in network components. Nonetheless, the architecture does not preclude per-flow solutions--it encompasses the following combinations:

a. The Dual Queue Coupled AQM (illustrated in Figure 1) achieves the ’semi-permeable’ membrane property mentioned earlier as follows. The obvious part is that using two separate queues isolates the queuing delay of one from the other. The less obvious part is how the two queues act as if they are a single pool of bandwidth without the scheduler needing to decide between them. This is achieved by having the Classic AQM provide a congestion signal to both queues in a manner that ensures a consistent response from the two types of congestion control. In other words, the Classic AQM generates a drop/mark probability based on congestion in the Classic queue, uses this probability to drop/mark packets in that queue, and also uses this probability to affect the marking probability in the L4S queue. This coupling of the congestion signaling between the two queues makes the L4S flows slow down to leave the right amount of capacity for the Classic traffic (as they would if they were the same type of traffic sharing the same queue). Then the scheduler can serve the L4S queue with priority, because the L4S traffic isn’t offering up enough traffic to use all the priority that it is given. Therefore, on short time-scales (sub-round-trip) the prioritization of the L4S queue protects its low latency by allowing bursts to dissipate quickly; but on longer time-scales (round-trip and longer) the Classic queue creates an equal and opposite pressure against the L4S traffic to ensure that neither has priority when it comes to bandwidth. The tension between prioritizing L4S and coupling the marking from the Classic AQM results in approximate per-flow fairness. To protect against unresponsive traffic in the L4S queue taking advantage of the prioritization and starving the Classic queue, it is advisable not to use strict priority, but instead to use a weighted scheduler (see Appendix A of [I-D.ietf-tsvwg-aqm-dualq-coupled]).

When there is no Classic traffic, the L4S queue’s AQM comes into play. It starts congestion marking with a very shallow queue, so L4S traffic maintains very low queuing delay.

The Dual Queue Coupled AQM has been specified as generically as possible [I-D.ietf-tsvwg-aqm-dualq-coupled] without specifying the particular AQMs to use in the two queues so that designers are free to implement diverse ideas. Informational appendices in that draft give pseudocode examples of two different specific AQM

Briscoe, et al. Expires January 2, 2022 [Page 9] Internet-Draft L4S Architecture July 2021

approaches: one called DualPI2 (pronounced Dual PI Squared) [DualPI2Linux] that uses the PI2 variant of PIE, and a zero-config variant of RED called Curvy RED. A DualQ Coupled AQM based on PIE has also been specified and implemented for Low Latency DOCSIS [DOCSIS3.1].

(2) (1) .------^------. .------^------. ,-(3)-----. ______; ______: L4S ------. | | :|Scalable| : _\ ||___\_| mark | :| sender | : ______/ / || / |______|\ ______:|______|\; | |/ ------’ ^ \1|condit’nl| ‘------’\_| IP-ECN | Coupling : \|priority |_\ ______/ |Classifier| : /|scheduler| / |Classic |/ |______|\ ------. ___:__ / |______| | sender | \_\ || | |||___\_| mark/|/ |______| / || | ||| / | drop | Classic ------’ |______|

Figure 1: Components of an L4S Solution: 1) Isolation in separate network queues; 2) Packet Identification Protocol; and 3) Scalable Sending Host

b. A scheduler with per-flow queues such as FQ-CoDel or FQ-PIE can be used for L4S. For instance within each queue of an FQ-CoDel system, as well as a CoDel AQM, there is typically also ECN marking at an immediate (unsmoothed) shallow threshold to support use in data centres (see Sec.5.2.7 of [RFC8290]). This can be modified so that the shallow threshold is solely applied to ECT(1) packets. Then if there is a flow of non-ECN or ECT(0) packets in the per-flow-queue, the Classic AQM (e.g. CoDel) is applied; while if there is a flow of ECT(1) packets in the queue, the shallower (typically sub-millisecond) threshold is applied. In addition, ECT(0) and not-ECT packets could potentially be classified into a separate flow-queue from ECT(1) and CE packets to avoid them mixing if they share a common flow-identifier (e.g. in a VPN).

c. It should also be possible to use dual queues for isolation, but with per-flow marking to control flow-rates (instead of the coupled per-queue marking of the Dual Queue Coupled AQM). One of the two queues would be for isolating L4S packets, which would be classified by the ECN codepoint. Flow rates could be controlled by flow-specific marking. The policy goal of the marking could be to differentiate flow rates (e.g. [Nadas20], which requires additional signalling of a per-flow ’value’), or to equalize

Briscoe, et al. Expires January 2, 2022 [Page 10] Internet-Draft L4S Architecture July 2021

flow-rates (perhaps in a similar way to Approx Fair CoDel [AFCD], [I-D.morton-tsvwg-codel-approx-fair], but with two queues not one).

Note that whenever the term ’DualQ’ is used loosely without saying whether marking is per-queue or per-flow, it means a dual queue AQM with per-queue marking.

4.3. Host Mechanisms

The L4S architecture includes two main mechanisms in the end host that we enumerate next:

a. Scalable Congestion Control at the sender: Data Center TCP is the most widely used example. It has been documented as an informational record of the protocol currently in use in controlled environments [RFC8257]. A draft list of safety and performance improvements for a scalable congestion control to be usable on the public Internet has been drawn up (the so-called ’Prague L4S requirements’ in Appendix A of [I-D.ietf-tsvwg-ecn-l4s-id]). The subset that involve risk of harm to others have been captured as normative requirements in Section 4 of [I-D.ietf-tsvwg-ecn-l4s-id]. TCP Prague [I-D.briscoe-iccrg-prague-congestion-control] has been implemented in Linux as a reference implementation to address these requirements [PragueLinux].

Transport protocols other than TCP use various congestion controls that are designed to be friendly with Reno. Before they can use the L4S service, they will need to be updated to implement a scalable congestion response, which they will have to indicate by using the ECT(1) codepoint. Scalable variants are under consideration for more recent transport protocols, e.g. QUIC, and the L4S ECN part of BBRv2 [BBRv2] is a scalable congestion control intended for the TCP and QUIC transports, amongst others. Also an L4S variant of the RMCAT SCReAM controller [RFC8298] has been implemented [SCReAM] for media transported over RTP.

b. The ECN feedback in some transport protocols is already sufficiently fine-grained for L4S (specifically DCCP [RFC4340] and QUIC [RFC9000]). But others either require update or are in the process of being updated:

* For the case of TCP, the feedback protocol for ECN embeds the assumption from Classic ECN [RFC3168] that an ECN mark is equivalent to a drop, making it unusable for a scalable TCP. Therefore, the implementation of TCP receivers will have to be

Briscoe, et al. Expires January 2, 2022 [Page 11] Internet-Draft L4S Architecture July 2021

upgraded [RFC7560]. Work to standardize and implement more accurate ECN feedback for TCP (AccECN) is in progress [I-D.ietf-tcpm-accurate-ecn], [PragueLinux].

* ECN feedback is only roughly sketched in an appendix of the SCTP specification [RFC4960]. A fuller specification has been proposed in a long-expired draft [I-D.stewart-tsvwg-sctpecn], which would need to be implemented and deployed before SCTCP could support L4S.

* For RTP, sufficient ECN feedback was defined in [RFC6679], but [RFC8888] defines the latest standards track improvements.

5. Rationale

5.1. Why These Primary Components?

Explicit congestion signalling (protocol): Explicit congestion signalling is a key part of the L4S approach. In contrast, use of drop as a congestion signal creates a tension because drop is both an impairment (less would be better) and a useful signal (more would be better):

* Explicit congestion signals can be used many times per round trip, to keep tight control, without any impairment. Under heavy load, even more explicit signals can be applied so the queue can be kept short whatever the load. In contrast, Classic AQMs have to introduce very high packet drop at high load to keep the queue short. By using ECN, an L4S congestion control’s sawtooth reduction can be smaller and therefore return to the operating point more often, without worrying that more sawteeth will cause more signals. The consequent smaller amplitude sawteeth fit between an empty queue and a very shallow marking threshold (˜1 ms in the public Internet), so queue delay variation can be very low, without risk of under- utilization.

* Explicit congestion signals can be emitted immediately to track fluctuations of the queue. L4S shifts smoothing from the network to the host. The network doesn’t know the round trip times of all the flows. So if the network is responsible for smoothing (as in the Classic approach), it has to assume a worst case RTT, otherwise long RTT flows would become unstable. This delays Classic congestion signals by 100-200 ms. In contrast, each host knows its own round trip time. So, in the L4S approach, the host can smooth each flow over its own RTT, introducing no more soothing delay than strictly necessary (usually only a few milliseconds). A host can also choose not

Briscoe, et al. Expires January 2, 2022 [Page 12] Internet-Draft L4S Architecture July 2021

to introduce any smoothing delay if appropriate, e.g. during flow start-up.

Neither of the above are feasible if explicit congestion signalling has to be considered ’equivalent to drop’ (as was required with Classic ECN [RFC3168]), because drop is an impairment as well as a signal. So drop cannot be excessively frequent, and drop cannot be immediate, otherwise too many drops would turn out to have been due to only a transient fluctuation in the queue that would not have warranted dropping a packet in hindsight. Therefore, in an L4S AQM, the L4S queue uses a new L4S variant of ECN that is not equivalent to drop (see section 5.2 of [I-D.ietf-tsvwg-ecn-l4s-id]), while the Classic queue uses either Classic ECN [RFC3168] or drop, which are equivalent to each other.

Before Classic ECN was standardized, there were various proposals to give an ECN mark a different meaning from drop. However, there was no particular reason to agree on any one of the alternative meanings, so ’equivalent to drop’ was the only compromise that could be reached. RFC 3168 contains a statement that:

"An environment where all end nodes were ECN-Capable could allow new criteria to be developed for setting the CE codepoint, and new congestion control mechanisms for end-node reaction to CE packets. However, this is a research issue, and as such is not addressed in this document."

Latency isolation (network): L4S congestion controls keep queue delay low whereas Classic congestion controls need a queue of the order of the RTT to avoid under-utilization. One queue cannot have two lengths, therefore L4S traffic needs to be isolated in a separate queue (e.g. DualQ) or queues (e.g. FQ).

Coupled congestion notification: Coupling the congestion notification between two queues as in the DualQ Coupled AQM is not necessarily essential, but it is a simple way to allow senders to determine their rate, packet by packet, rather than be overridden by a network scheduler. An alternative is for a network scheduler to control the rate of each application flow (see discussion in Section 5.2).

L4S packet identifier (protocol): Once there are at least two treatments in the network, hosts need an identifier at the IP layer to distinguish which treatment they intend to use.

Scalable congestion notification: A scalable congestion control in the host keeps the signalling frequency from the network high whatever the flow rate, so that queue delay variations can be

Briscoe, et al. Expires January 2, 2022 [Page 13] Internet-Draft L4S Architecture July 2021

small when conditions are stable, and rate can track variations in available capacity as rapidly as possible otherwise.

Low loss: Latency is not the only concern of L4S. The ’Low Loss" part of the name denotes that L4S generally achieves zero congestion loss due to its use of ECN. Otherwise, loss would itself cause delay, particularly for short flows, due to retransmission delay [RFC2884].

Scalable throughput: The "Scalable throughput" part of the name denotes that the per-flow throughput of scalable congestion controls should scale indefinitely, avoiding the imminent scaling problems with Reno-friendly congestion control algorithms [RFC3649]. It was known when TCP congestion avoidance was first developed in 1988 that it would not scale to high bandwidth-delay products (see footnote 6 in [TCP-CA]). Today, regular broadband flow rates over WAN distances are already beyond the scaling range of Classic Reno congestion control. So ‘less unscalable’ Cubic [RFC8312] and Compound [I-D.sridharan-tcpm-ctcp] variants of TCP have been successfully deployed. However, these are now approaching their scaling limits.

For instance, we will consider a scenario with a maximum RTT of 30 ms at the peak of each sawtooth. As Reno packet rate scales 8x from 1,250 to 10,000 packet/s (from 15 to 120 Mb/s with 1500 B packets), the time to recover from a congestion event rises proportionately by 8x as well, from 422 ms to 3.38 s. It is clearly problematic for a congestion control to take multiple seconds to recover from each congestion event. Cubic [RFC8312] was developed to be less unscalable, but it is approaching its scaling limit; with the same max RTT of 30 ms, at 120 Mb/s the Linux implementation of Cubic is still in its Reno-friendly mode, so it takes about 2.3 s to recover. However, once the flow rate scales by 8x again to 960 Mb/s it enters true Cubic mode, with a recovery time of 10.6 s. From then on, each further scaling by 8x doubles Cubic’s recovery time (because the cube root of 8 is 2), e.g. at 7.68 Gb/s the recovery time is 21.3 s. In contrast a scalable congestion control like DCTCP or TCP Prague induces 2 congestion signals per round trip on average, which remains invariant for any flow rate, keeping dynamic control very tight.

Although work on scaling congestion controls tends to start with TCP as the transport, the above is not intended to exclude other transports (e.g. SCTP, QUIC) or less elastic algorithms (e.g. RMCAT), which all tend to adopt the same or similar developments.

Briscoe, et al. Expires January 2, 2022 [Page 14] Internet-Draft L4S Architecture July 2021

5.2. What L4S adds to Existing Approaches

All the following approaches address some part of the same problem space as L4S. In each case, it is shown that L4S complements them or improves on them, rather than being a mutually exclusive alternative:

Diffserv: Diffserv addresses the problem of bandwidth apportionment for important traffic as well as queuing latency for delay- sensitive traffic. Of these, L4S solely addresses the problem of queuing latency. Diffserv will still be necessary where important traffic requires priority (e.g. for commercial reasons, or for protection of critical infrastructure traffic) - see [I-D.briscoe-tsvwg-l4s-diffserv]. Nonetheless, the L4S approach can provide low latency for _all_ traffic within each Diffserv class (including the case where there is only the one default Diffserv class).

Also, Diffserv only works for a small subset of the traffic on a link. As already explained, it is not applicable when all the applications in use at one time at a single site (home, small business or mobile device) require low latency. In contrast, because L4S is for all traffic, it needs none of the management baggage (traffic policing, traffic contracts) associated with favouring some packets over others. This baggage has probably held Diffserv back from widespread end-to-end deployment.

In particular, because networks tend not to trust end systems to identify which packets should be favoured over others, where networks assign packets to Diffserv classes they often use packet inspection of application flow identifiers or deeper inspection of application signatures. Thus, nowadays, Diffserv doesn’t always sit well with encryption of the layers above IP. So users have to choose between privacy and QoS.

As with Diffserv, the L4S identifier is in the IP header. But, in contrast to Diffserv, the L4S identifier does not convey a want or a need for a certain level of quality. Rather, it promises a certain behaviour (scalable congestion response), which networks can objectively verify if they need to. This is because low delay depends on collective host behaviour, whereas bandwidth priority depends on network behaviour.

State-of-the-art AQMs: AQMs such as PIE and FQ-CoDel give a significant reduction in queuing delay relative to no AQM at all. L4S is intended to complement these AQMs, and should not distract from the need to deploy them as widely as possible. Nonetheless, AQMs alone cannot reduce queuing delay too far without significantly reducing link utilization, because the root cause of

Briscoe, et al. Expires January 2, 2022 [Page 15] Internet-Draft L4S Architecture July 2021

the problem is on the host - where Classic congestion controls use large saw-toothing rate variations. The L4S approach resolves this tension by ensuring hosts can minimize the size of their sawteeth without appearing so aggressive to Classic flows that they starve them.

Per-flow queuing or marking: Similarly, per-flow approaches such as FQ-CoDel or Approx Fair CoDel [AFCD] are not incompatible with the L4S approach. However, per-flow queuing alone is not enough - it only isolates the queuing of one flow from others; not from itself. Per-flow implementations still need to have support for scalable congestion control added, which has already been done in FQ-CoDel (see Sec.5.2.7 of [RFC8290]). Without this simple modification, per-flow AQMs like FQ-CoDel would still not be able to support applications that need both very low delay and high bandwidth, e.g. video-based control of remote procedures, or interactive cloud-based video (see Note 1 below).

Although per-flow techniques are not incompatible with L4S, it is important to have the DualQ alternative. This is because handling end-to-end (layer 4) flows in the network (layer 3 or 2) precludes some important end-to-end functions. For instance:

A. Per-flow forms of L4S like FQ-CoDel are incompatible with full end-to-end encryption of transport layer identifiers for privacy and confidentiality (e.g. IPSec or encrypted VPN tunnels), because they require packet inspection to access the end-to-end transport flow identifiers.

In contrast, the DualQ form of L4S requires no deeper inspection than the IP layer. So, as long as operators take the DualQ approach, their users can have both very low queuing delay and full end-to-end encryption [RFC8404].

B. With per-flow forms of L4S, the network takes over control of the relative rates of each application flow. Some see it as an advantage that the network will prevent some flows running faster than others. Others consider it an inherent part of the Internet’s appeal that applications can control their rate while taking account of the needs of others via congestion signals. They maintain that this has allowed applications with interesting rate behaviours to evolve, for instance, variable bit-rate video that varies around an equal share rather than being forced to remain equal at every instant, or scavenger services that use less than an equal share of capacity [LEDBAT_AQM].

Briscoe, et al. Expires January 2, 2022 [Page 16] Internet-Draft L4S Architecture July 2021

The L4S architecture does not require the IETF to commit to one approach over the other, because it supports both, so that the ’market’ can decide. Nonetheless, in the spirit of ’Do one thing and do it well’ [McIlroy78], the DualQ option provides low delay without prejudging the issue of flow-rate control. Then, flow rate policing can be added separately if desired. This allows application control up to a point, but the network can still choose to set the point at which it intervenes to prevent one flow completely starving another.

Note:

1. It might seem that self-inflicted queuing delay within a per- flow queue should not be counted, because if the delay wasn’t in the network it would just shift to the sender. However, modern adaptive applications, e.g. HTTP/2 [RFC7540] or some interactive media applications (see Section 6.1), can keep low latency objects at the front of their local send queue by shuffling priorities of other objects dependent on the progress of other transfers. They cannot shuffle objects once they have released them into the network.

Alternative Back-off ECN (ABE): Here again, L4S is not an alternative to ABE but a complement that introduces much lower queuing delay. ABE [RFC8511] alters the host behaviour in response to ECN marking to utilize a link better and give ECN flows faster throughput. It uses ECT(0) and assumes the network still treats ECN and drop the same. Therefore ABE exploits any lower queuing delay that AQMs can provide. But as explained above, AQMs still cannot reduce queuing delay too far without losing link utilization (to allow for other, non-ABE, flows).

BBR: Bottleneck Bandwidth and Round-trip propagation time (BBR [I-D.cardwell-iccrg-bbr-congestion-control]) controls queuing delay end-to-end without needing any special logic in the network, such as an AQM. So it works pretty-much on any path (although it has not been without problems, particularly capacity sharing in BBRv1). BBR keeps queuing delay reasonably low, but perhaps not quite as low as with state-of-the-art AQMs such as PIE or FQ- CoDel, and certainly nowhere near as low as with L4S. Queuing delay is also not consistently low, due to BBR’s regular bandwidth probing spikes and its aggressive flow start-up phase.

L4S complements BBR. Indeed BBRv2 [BBRv2] uses L4S ECN and a scalable L4S congestion control behaviour in response to any ECN signalling from the path. The L4S ECN signal complements the delay based congestion control aspects of BBR with an explicit indication that hosts can use, both to converge on a fair rate and

Briscoe, et al. Expires January 2, 2022 [Page 17] Internet-Draft L4S Architecture July 2021

to keep below a shallow queue target set by the network. Without L4S ECN, both these aspects need to be assumed or estimated.

6. Applicability

6.1. Applications

A transport layer that solves the current latency issues will provide new service, product and application opportunities.

With the L4S approach, the following existing applications also experience significantly better quality of experience under load:

o Gaming, including cloud based gaming;

o VoIP;

o Video conferencing;

o Web browsing;

o (Adaptive) video streaming;

o Instant messaging.

The significantly lower queuing latency also enables some interactive application functions to be offloaded to the cloud that would hardly even be usable today:

o Cloud based interactive video;

o Cloud based virtual and augmented reality.

The above two applications have been successfully demonstrated with L4S, both running together over a 40 Mb/s broadband access link loaded up with the numerous other latency sensitive applications in the previous list as well as numerous downloads - all sharing the same bottleneck queue simultaneously [L4Sdemo16]. For the former, a panoramic video of a football stadium could be swiped and pinched so that, on the fly, a proxy in the cloud could generate a sub-window of the match video under the finger-gesture control of each user. For the latter, a virtual reality headset displayed a viewport taken from a 360 degree camera in a racing car. The user’s head movements controlled the viewport extracted by a cloud-based proxy. In both cases, with 7 ms end-to-end base delay, the additional queuing delay of roughly 1 ms was so low that it seemed the video was generated locally.

Briscoe, et al. Expires January 2, 2022 [Page 18] Internet-Draft L4S Architecture July 2021

Using a swiping finger gesture or head movement to pan a video are extremely latency-demanding actions--far more demanding than VoIP. Because human vision can detect extremely low delays of the order of single milliseconds when delay is translated into a visual lag between a video and a reference point (the finger or the orientation of the head sensed by the balance system in the inner ear --- the vestibular system).

Without the low queuing delay of L4S, cloud-based applications like these would not be credible without significantly more access bandwidth (to deliver all possible video that might be viewed) and more local processing, which would increase the weight and power consumption of head-mounted displays. When all interactive processing can be done in the cloud, only the data to be rendered for the end user needs to be sent.

Other low latency high bandwidth applications such as:

o Interactive remote presence;

o Video-assisted remote control of machinery or industrial processes.

are not credible at all without very low queuing delay. No amount of extra access bandwidth or local processing can make up for lost time.

6.2. Use Cases

The following use-cases for L4S are being considered by various interested parties:

o Where the bottleneck is one of various types of access network: e.g. DSL, Passive Optical Networks (PON), DOCSIS cable, mobile, satellite (see Section 6.3 for some technology-specific details)

o Private networks of heterogeneous data centres, where there is no single administrator that can arrange for all the simultaneous changes to senders, receivers and network needed to deploy DCTCP:

* a set of private data centres interconnected over a wide area with separate administrations, but within the same company

* a set of data centres operated by separate companies interconnected by a community of interest network (e.g. for the finance sector)

* multi-tenant (cloud) data centres where tenants choose their operating system stack (Infrastructure as a Service - IaaS)

Briscoe, et al. Expires January 2, 2022 [Page 19] Internet-Draft L4S Architecture July 2021

o Different types of transport (or application) congestion control:

* elastic (TCP/SCTP);

* real-time (RTP, RMCAT);

* query (DNS/LDAP).

o Where low delay quality of service is required, but without inspecting or intervening above the IP layer [RFC8404]:

* mobile and other networks have tended to inspect higher layers in order to guess application QoS requirements. However, with growing demand for support of privacy and encryption, L4S offers an alternative. There is no need to select which traffic to favour for queuing, when L4S gives favourable queuing to all traffic.

o If queuing delay is minimized, applications with a fixed delay budget can communicate over longer distances, or via a longer chain of service functions [RFC7665] or onion routers.

o If delay jitter is minimized, it is possible to reduce the dejitter buffers on the receive end of video streaming, which should improve the interactive experience

6.3. Applicability with Specific Link Technologies

Certain link technologies aggregate data from multiple packets into bursts, and buffer incoming packets while building each burst. WiFi, PON and cable all involve such packet aggregation, whereas fixed Ethernet and DSL do not. No sender, whether L4S or not, can do anything to reduce the buffering needed for packet aggregation. So an AQM should not count this buffering as part of the queue that it controls, given no amount of congestion signals will reduce it.

Certain link technologies also add buffering for other reasons, specifically:

o Radio links (cellular, WiFi, satellite) that are distant from the source are particularly challenging. The radio link capacity can vary rapidly by orders of magnitude, so it is considered desirable to hold a standing queue that can utilize sudden increases of capacity;

o Cellular networks are further complicated by a perceived need to buffer in order to make hand-overs imperceptible;

Briscoe, et al. Expires January 2, 2022 [Page 20] Internet-Draft L4S Architecture July 2021

L4S cannot remove the need for all these different forms of buffering. However, by removing ’the longest pole in the tent’ (buffering for the large sawteeth of Classic congestion controls), L4S exposes all these ’shorter poles’ to greater scrutiny.

Until now, the buffering needed for these additional reasons tended to be over-specified - with the excuse that none were ’the longest pole in the tent’. But having removed the ’longest pole’, it becomes worthwhile to minimize them, for instance reducing packet aggregation burst sizes and MAC scheduling intervals.

6.4. Deployment Considerations

L4S AQMs, whether DualQ [I-D.ietf-tsvwg-aqm-dualq-coupled] or FQ, e.g. [RFC8290] are, in themselves, an incremental deployment mechanism for L4S - so that L4S traffic can coexist with existing Classic (Reno-friendly) traffic. Section 6.4.1 explains why only deploying an L4S AQM in one node at each end of the access link will realize nearly all the benefit of L4S.

L4S involves both end systems and the network, so Section 6.4.2 suggests some typical sequences to deploy each part, and why there will be an immediate and significant benefit after deploying just one part.

Section 6.4.3 and Section 6.4.4 describe the converse incremental deployment case where there is no L4S AQM at the network bottleneck, so any L4S flow traversing this bottleneck has to take care in case it is competing with Classic traffic.

6.4.1. Deployment Topology

L4S AQMs will not have to be deployed throughout the Internet before L4S will work for anyone. Operators of public Internet access networks typically design their networks so that the bottleneck will nearly always occur at one known (logical) link. This confines the cost of queue management technology to one place.

The case of mesh networks is different and will be discussed later in this section. But the known bottleneck case is generally true for Internet access to all sorts of different ’sites’, where the word ’site’ includes home networks, small- to medium-sized campus or enterprise networks and even cellular devices (Figure 2). Also, this known-bottleneck case tends to be applicable whatever the access link technology; whether xDSL, cable, PON, cellular, line of sight wireless or satellite.

Briscoe, et al. Expires January 2, 2022 [Page 21] Internet-Draft L4S Architecture July 2021

Therefore, the full benefit of the L4S service should be available in the downstream direction when an L4S AQM is deployed at the ingress to this bottleneck link. And similarly, the full upstream service will be available once an L4S AQM is deployed at the ingress into the upstream link. (Of course, multi-homed sites would only see the full benefit once all their access links were covered.)

______( ) __ __ ( ) |DQ\______/DQ|( enterprise ) ___ |__/ \__| ( /campus ) ( ) (______) ( ) ___||_ +----+ ( ) __ __ / \ | DC |-----( Core )|DQ\______/DQ|| home | +----+ ( ) |__/ \__||______| (_____) __ |DQ\__/\ __ ,===. |__/ \ ____/DQ||| ||mobile \/ \__|||_||device | o | ‘---’

Figure 2: Likely location of DualQ (DQ) Deployments in common access topologies

Deployment in mesh topologies depends on how over-booked the core is. If the core is non-blocking, or at least generously provisioned so that the edges are nearly always the bottlenecks, it would only be necessary to deploy an L4S AQM at the edge bottlenecks. For example, some data-centre networks are designed with the bottleneck in the hypervisor or host NICs, while others bottleneck at the top-of-rack switch (both the output ports facing hosts and those facing the core).

An L4S AQM would eventually also need to be deployed at any other persistent bottlenecks such as network interconnections, e.g. some public Internet exchange points and the ingress and egress to WAN links interconnecting data-centres.

6.4.2. Deployment Sequences

For any one L4S flow to work, it requires 3 parts to have been deployed. This was the same deployment problem that ECN faced [RFC8170] so we have learned from that experience.

Briscoe, et al. Expires January 2, 2022 [Page 22] Internet-Draft L4S Architecture July 2021

Firstly, L4S deployment exploits the fact that DCTCP already exists on many Internet hosts (Windows, FreeBSD and Linux); both servers and clients. Therefore, just deploying an L4S AQM at a network bottleneck immediately gives a working deployment of all the L4S parts. DCTCP needs some safety concerns to be fixed for general use over the public Internet (see Section 4.3 of [I-D.ietf-tsvwg-ecn-l4s-id]), but DCTCP is not on by default, so these issues can be managed within controlled deployments or controlled trials.

Secondly, the performance improvement with L4S is so significant that it enables new interactive services and products that were not previously possible. It is much easier for companies to initiate new work on deployment if there is budget for a new product trial. If, in contrast, there were only an incremental performance improvement (as with Classic ECN), spending on deployment tends to be much harder to justify.

Thirdly, the L4S identifier is defined so that initially network operators can enable L4S exclusively for certain customers or certain applications. But this is carefully defined so that it does not compromise future evolution towards L4S as an Internet-wide service. This is because the L4S identifier is defined not only as the end-to- end ECN field, but it can also optionally be combined with any other packet header or some status of a customer or their access link (see section 5.4 of [I-D.ietf-tsvwg-ecn-l4s-id]). Operators could do this anyway, even if it were not blessed by the IETF. However, it is best for the IETF to specify that, if they use their own local identifier, it must be in combination with the IETF’s identifier. Then, if an operator has opted for an exclusive local-use approach, later they only have to remove this extra rule to make the service work Internet-wide - it will already traverse middleboxes, peerings, etc.

Briscoe, et al. Expires January 2, 2022 [Page 23] Internet-Draft L4S Architecture July 2021

Figure 3: Example L4S Deployment Sequence

Figure 3 illustrates some example sequences in which the parts of L4S might be deployed. It consists of the following stages:

1. Here, the immediate benefit of a single AQM deployment can be seen, but limited to a controlled trial or controlled deployment. In this example downstream deployment is first, but in other scenarios the upstream might be deployed first. If no AQM at all was previously deployed for the downstream access, an L4S AQM greatly improves the Classic service (as well as adding the L4S service). If an AQM was already deployed, the Classic service will be unchanged (and L4S will add an improvement on top).

2. In this stage, the name ’TCP Prague’ [I-D.briscoe-iccrg-prague-congestion-control] is used to represent a variant of DCTCP that is safe to use in a production Internet environment. If the application is primarily unidirectional, ’TCP Prague’ at one end will provide all the benefit needed. For TCP transports, Accurate ECN feedback (AccECN) [I-D.ietf-tcpm-accurate-ecn] is needed at the other end, but it is a generic ECN feedback facility that is already planned to be deployed for other purposes, e.g. DCTCP, BBR. The two ends can be deployed in either order, because, in TCP, an L4S congestion control only enables itself if it has negotiated the use of AccECN feedback with the other end during the connection handshake. Thus, deployment of TCP Prague on a server enables L4S trials to move to a production service in one direction, wherever AccECN is deployed at the other end. This stage might

Briscoe, et al. Expires January 2, 2022 [Page 24] Internet-Draft L4S Architecture July 2021

be further motivated by the performance improvements of TCP Prague relative to DCTCP (see Appendix A.2 of [I-D.ietf-tsvwg-ecn-l4s-id]).

Unlike TCP, from the outset, QUIC ECN feedback [RFC9000] has supported L4S. Therefore, if the transport is QUIC, one-ended deployment of a Prague congestion control at this stage is simple and sufficient.

3. This is a two-move stage to enable L4S upstream. An L4S AQM or TCP Prague can be deployed in either order as already explained. To motivate the first of two independent moves, the deferred benefit of enabling new services after the second move has to be worth it to cover the first mover’s investment risk. As explained already, the potential for new interactive services provides this motivation. An L4S AQM also improves the upstream Classic service - significantly if no other AQM has already been deployed.

Note that other deployment sequences might occur. For instance: the upstream might be deployed first; a non-TCP protocol might be used end-to-end, e.g. QUIC, RTP; a body such as the 3GPP might require L4S to be implemented in 5G user equipment, or other random acts of kindness.

6.4.3. L4S Flow but Non-ECN Bottleneck

If L4S is enabled between two hosts, the L4S sender is required to coexist safely with Reno in response to any drop (see Section 4.3 of [I-D.ietf-tsvwg-ecn-l4s-id]).

Unfortunately, as well as protecting Classic traffic, this rule degrades the L4S service whenever there is any loss, even if the cause is not persistent congestion at a bottleneck, e.g.:

o congestion loss at other transient bottlenecks, e.g. due to bursts in shallower queues;

o transmission errors, e.g. due to electrical interference;

o rate policing.

Three complementary approaches are in progress to address this issue, but they are all currently research:

o In Prague congestion control, ignore certain losses deemed unlikely to be due to congestion (using some ideas from BBR [I-D.cardwell-iccrg-bbr-congestion-control] regarding isolated

Briscoe, et al. Expires January 2, 2022 [Page 25] Internet-Draft L4S Architecture July 2021

losses). This could mask any of the above types of loss while still coexisting with drop-based congestion controls.

o A combination of RACK, L4S and link retransmission without resequencing could repair transmission errors without the head of line blocking delay usually associated with link-layer retransmission [UnorderedLTE], [I-D.ietf-tsvwg-ecn-l4s-id];

o Hybrid ECN/drop rate policers (see Section 8.3).

L4S deployment scenarios that minimize these issues (e.g. over wireline networks) can proceed in parallel to this research, in the expectation that research success could continually widen L4S applicability.

6.4.4. L4S Flow but Classic ECN Bottleneck

Classic ECN support is starting to materialize on the Internet as an increased level of CE marking. It is hard to detect whether this is all due to the addition of support for ECN in the Linux implementation of FQ-CoDel, which is not problematic, because FQ inherently forces the throughput of each flow to be equal irrespective of its aggressiveness. However, some of this Classic ECN marking might be due to single-queue ECN deployment. This case is discussed in Section 4.3 of [I-D.ietf-tsvwg-ecn-l4s-id]).

6.4.5. L4S AQM Deployment within Tunnels

An L4S AQM uses the ECN field to signal congestion. So, in common with Classic ECN, if the AQM is within a tunnel or at a lower layer, correct functioning of ECN signalling requires correct propagation of the ECN field up the layers [RFC6040], [I-D.ietf-tsvwg-rfc6040update-shim], [I-D.ietf-tsvwg-ecn-encap-guidelines].

7. IANA Considerations (to be removed by RFC Editor)

This specification contains no IANA considerations.

8. Security Considerations

8.1. Traffic Rate (Non-)Policing

Because the L4S service can serve all traffic that is using the capacity of a link, it should not be necessary to rate-police access to the L4S service. In contrast, Diffserv only works if some packets get less favourable treatment than others. So Diffserv has to use traffic rate policers to limit how much traffic can be favoured. In

Briscoe, et al. Expires January 2, 2022 [Page 26] Internet-Draft L4S Architecture July 2021

turn, traffic policers require traffic contracts between users and networks as well as pairwise between networks. Because L4S will lack all this management complexity, it is more likely to work end-to-end.

During early deployment (and perhaps always), some networks will not offer the L4S service. In general, these networks should not need to police L4S traffic - they are required not to change the L4S identifier, merely treating the traffic as best efforts traffic, as they already treat traffic with ECT(1) today. At a bottleneck, such networks will introduce some queuing and dropping. When a scalable congestion control detects a drop it will have to respond safely with respect to Classic congestion controls (as required in Section 4.3 of [I-D.ietf-tsvwg-ecn-l4s-id]). This will degrade the L4S service to be no better (but never worse) than Classic best efforts, whenever a non-ECN bottleneck is encountered on a path (see Section 6.4.3).

In some cases, networks that solely support Classic ECN [RFC3168] in a single queue bottleneck might opt to police L4S traffic in order to protect competing Classic ECN traffic.

Certain network operators might choose to restrict access to the L4S class, perhaps only to selected premium customers as a value-added service. Their packet classifier (item 2 in Figure 1) could identify such customers against some other field (e.g. source address range) as well as ECN. If only the ECN L4S identifier matched, but not the source address (say), the classifier could direct these packets (from non-premium customers) into the Classic queue. Explaining clearly how operators can use an additional local classifiers (see section 5.4 of [I-D.ietf-tsvwg-ecn-l4s-id]) is intended to remove any motivation to bleach the L4S identifier. Then at least the L4S ECN identifier will be more likely to survive end-to-end even though the service may not be supported at every hop. Such local arrangements would only require simple registered/not-registered packet classification, rather than the managed, application-specific traffic policing against customer-specific traffic contracts that Diffserv uses.

8.2. ’Latency Friendliness’

Like the Classic service, the L4S service relies on self-constraint - limiting rate in response to congestion. In addition, the L4S service requires self-constraint in terms of limiting latency (burstiness). It is hoped that self-interest and guidance on dynamic behaviour (especially flow start-up, which might need to be standardized) will be sufficient to prevent transports from sending excessive bursts of L4S traffic, given the application’s own latency will suffer most from such behaviour.

Briscoe, et al. Expires January 2, 2022 [Page 27] Internet-Draft L4S Architecture July 2021

Whether burst policing becomes necessary remains to be seen. Without it, there will be potential for attacks on the low latency of the L4S service.

If needed, various arrangements could be used to address this concern:

Local bottleneck queue protection: A per-flow (5-tuple) queue protection function [I-D.briscoe-docsis-q-protection] has been developed for the low latency queue in DOCSIS, which has adopted the DualQ L4S architecture. It protects the low latency service from any queue-building flows that accidentally or maliciously classify themselves into the low latency queue. It is designed to score flows based solely on their contribution to queuing (not flow rate in itself). Then, if the shared low latency queue is at risk of exceeding a threshold, the function redirects enough packets of the highest scoring flow(s) into the Classic queue to preserve low latency.

Distributed traffic scrubbing: Rather than policing locally at each bottleneck, it may only be necessary to address problems reactively, e.g. punitively target any deployments of new bursty malware, in a similar way to how traffic from flooding attack sources is rerouted via scrubbing facilities.

Local bottleneck per-flow scheduling: Per-flow scheduling should inherently isolate non-bursty flows from bursty (see Section 5.2 for discussion of the merits of per-flow scheduling relative to per-flow policing).

Distributed access subnet queue protection: Per-flow queue protection could be arranged for a queue structure distributed across a subnet inter-communicating using lower layer control messages (see Section 2.1.4 of [QDyn]). For instance, in a radio access network user equipment already sends regular buffer status reports to a radio network controller, which could use this information to remotely police individual flows.

Distributed Congestion Exposure to Ingress Policers: The Congestion Exposure (ConEx) architecture [RFC7713] which uses egress audit to motivate senders to truthfully signal path congestion in-band where it can be used by ingress policers. An edge-to-edge variant of this architecture is also possible.

Distributed Domain-edge traffic conditioning: An architecture similar to Diffserv [RFC2475] may be preferred, where traffic is proactively conditioned on entry to a domain, rather than

Briscoe, et al. Expires January 2, 2022 [Page 28] Internet-Draft L4S Architecture July 2021

reactively policed only if it is leads to queuing once combined with other traffic at a bottleneck.

Distributed core network queue protection: The policing function could be divided between per-flow mechanisms at the network ingress that characterize the burstiness of each flow into a signal carried with the traffic, and per-class mechanisms at bottlenecks that act on these signals if queuing actually occurs once the traffic converges. This would be somewhat similar to the idea behind core stateless fair queuing, which is in turn similar to [Nadas20].

None of these possible queue protection capabilities are considered a necessary part of the L4S architecture, which works without them (in a similar way to how the Internet works without per-flow rate policing). Indeed, under normal circumstances, latency policers would not intervene, and if operators found they were not necessary they could disable them. Part of the L4S experiment will be to see whether such a function is necessary, and which arrangements are most appropriate to the size of the problem.

8.3. Interaction between Rate Policing and L4S

As mentioned in Section 5.2, L4S should remove the need for low latency Diffserv classes. However, those Diffserv classes that give certain applications or users priority over capacity, would still be applicable in certain scenarios (e.g. corporate networks). Then, within such Diffserv classes, L4S would often be applicable to give traffic low latency and low loss as well. Within such a Diffserv class, the bandwidth available to a user or application is often limited by a rate policer. Similarly, in the default Diffserv class, rate policers are used to partition shared capacity.

A classic rate policer drops any packets exceeding a set rate, usually also giving a burst allowance (variants exist where the policer re-marks non-compliant traffic to a discard-eligible Diffserv codepoint, so they may be dropped elsewhere during contention). Whenever L4S traffic encounters one of these rate policers, it will experience drops and the source will have to fall back to a Classic congestion control, thus losing the benefits of L4S (Section 6.4.3). So, in networks that already use rate policers and plan to deploy L4S, it will be preferable to redesign these rate policers to be more friendly to the L4S service.

L4S-friendly rate policing is currently a research area (note that this is not the same as latency policing). It might be achieved by setting a threshold where ECN marking is introduced, such that it is just under the policed rate or just under the burst allowance where

Briscoe, et al. Expires January 2, 2022 [Page 29] Internet-Draft L4S Architecture July 2021

drop is introduced. This could be applied to various types of rate policer, e.g. [RFC2697], [RFC2698] or the ’local’ (non-ConEx) variant of the ConEx congestion policer [I-D.briscoe-conex-policing]. It might also be possible to design scalable congestion controls to respond less catastrophically to loss that has not been preceded by a period of increasing delay.

The design of L4S-friendly rate policers will require a separate dedicated document. For further discussion of the interaction between L4S and Diffserv, see [I-D.briscoe-tsvwg-l4s-diffserv].

8.4. ECN Integrity

Receiving hosts can fool a sender into downloading faster by suppressing feedback of ECN marks (or of losses if retransmissions are not necessary or available otherwise). Various ways to protect transport feedback integrity have been developed. For instance:

o The sender can test the integrity of the receiver’s feedback by occasionally setting the IP-ECN field to the congestion experienced (CE) codepoint, which is normally only set by a congested link. Then the sender can test whether the receiver’s feedback faithfully reports what it expects (see 2nd para of Section 20.2 of [RFC3168]).

o A network can enforce a congestion response to its ECN markings (or packet losses) by auditing congestion exposure (ConEx) [RFC7713].

o The TCP authentication option (TCP-AO [RFC5925]) can be used to detect tampering with TCP congestion feedback.

o The ECN Nonce [RFC3540] was proposed to detect tampering with congestion feedback, but it has been reclassified as historic [RFC8311].

Appendix C.1 of [I-D.ietf-tsvwg-ecn-l4s-id] gives more details of these techniques including their applicability and pros and cons.

8.5. Privacy Considerations

As discussed in Section 5.2, the L4S architecture does not preclude approaches that inspect end-to-end transport layer identifiers. For instance it is simple to add L4S support to FQ-CoDel, which classifies by application flow ID in the network. However, the main innovation of L4S is the DualQ AQM framework that does not need to inspect any deeper than the outermost IP header, because the L4S identifier is in the IP-ECN field.

Briscoe, et al. Expires January 2, 2022 [Page 30] Internet-Draft L4S Architecture July 2021

Thus, the L4S architecture enables very low queuing delay without _requiring_ inspection of information above the IP layer. This means that users who want to encrypt application flow identifiers, e.g. in IPSec or other encrypted VPN tunnels, don’t have to sacrifice low delay [RFC8404].

Because L4S can provide low delay for a broad set of applications that choose to use it, there is no need for individual applications or classes within that broad set to be distinguishable in any way while traversing networks. This removes much of the ability to correlate between the delay requirements of traffic and other identifying features [RFC6973]. There may be some types of traffic that prefer not to use L4S, but the coarse binary categorization of traffic reveals very little that could be exploited to compromise privacy.

9. Acknowledgements

Thanks to Richard Scheffenegger, Wes Eddy, Karen Nielsen, David Black, Jake Holland and Vidhi Goel for their useful review comments.

Bob Briscoe and Koen De Schepper were part-funded by the European Community under its Seventh Framework Programme through the Reducing Internet Transport Latency (RITE) project (ICT-317700). Bob Briscoe was also part-funded by the Research Council of Norway through the TimeIn project, partly by CableLabs and partly by the Comcast Innovation Fund. The views expressed here are solely those of the authors.

10. Informative References

[AFCD] Xue, L., Kumar, S., Cui, C., Kondikoppa, P., Chiu, C-H., and S-J. Park, "Towards fair and low latency next generation high speed networks: AFCD queuing", Journal of Network and Computer Applications 70:183--193, July 2016.

[BBRv2] Cardwell, N., "TCP BBR v2 Alpha/Preview Release", github repository; Linux congestion control module, .

[DCttH15] De Schepper, K., Bondarenko, O., Briscoe, B., and I. Tsang, "‘Data Centre to the Home’: Ultra-Low Latency for All", RITE project Technical Report , 2015, .

Briscoe, et al. Expires January 2, 2022 [Page 31] Internet-Draft L4S Architecture July 2021

[DualPI2Linux] Albisser, O., De Schepper, K., Briscoe, B., Tilmans, O., and H. Steen, "DUALPI2 - Low Latency, Low Loss and Scalable (L4S) AQM", Proc. Linux Netdev 0x13 , March 2019, .

[Hohlfeld14] Hohlfeld , O., Pujol, E., Ciucu, F., Feldmann, A., and P. Barford, "A QoE Perspective on Sizing Network Buffers", Proc. ACM Internet Measurement Conf (IMC’14) hmm, November 2014.

[I-D.briscoe-conex-policing] Briscoe, B., "Network Performance Isolation using Congestion Policing", draft-briscoe-conex-policing-01 (work in progress), February 2014.

[I-D.briscoe-docsis-q-protection] Briscoe, B. and G. White, "Queue Protection to Preserve Low Latency", draft-briscoe-docsis-q-protection-00 (work in progress), July 2019.

Briscoe, et al. Expires January 2, 2022 [Page 32] Internet-Draft L4S Architecture July 2021

[I-D.ietf-tcpm-accurate-ecn] Briscoe, B., Kuehlewind, M., and R. Scheffenegger, "More Accurate ECN Feedback in TCP", draft-ietf-tcpm-accurate- ecn-14 (work in progress), February 2021.

[I-D.ietf-tsvwg-nqb] White, G. and T. Fossati, "A Non-Queue-Building Per-Hop Behavior (NQB PHB) for Differentiated Services", draft- ietf-tsvwg-nqb-05 (work in progress), March 2021.

[I-D.morton-tsvwg-codel-approx-fair] Morton, J. and P. G. Heist, "Controlled Delay Approximate Fairness AQM", draft-morton-tsvwg-codel-approx-fair-01 (work in progress), March 2020.

Briscoe, et al. Expires January 2, 2022 [Page 33] Internet-Draft L4S Architecture July 2021

[I-D.stewart-tsvwg-sctpecn] Stewart, R. R., Tuexen, M., and X. Dong, "ECN for Stream Control Transmission Protocol (SCTP)", draft-stewart- tsvwg-sctpecn-05 (work in progress), January 2014.

[L4Sdemo16] Bondarenko, O., De Schepper, K., Tsang, I., and B. Briscoe, "Ultra-Low Delay for All: Live Experience, Live Analysis", Proc. MMSYS’16 pp33:1--33:4, May 2016, .

[LEDBAT_AQM] Al-Saadi, R., Armitage, G., and J. But, "Characterising LEDBAT Performance Through Bottlenecks Using PIE, FQ-CoDel and FQ-PIE Active Queue Management", Proc. IEEE 42nd Conference on Local Computer Networks (LCN) 278--285, 2017, .

[Mathis09] Mathis, M., "Relentless Congestion Control", PFLDNeT’09 , May 2009, .

[McIlroy78] McIlroy, M., Pinson, E., and B. Tague, "UNIX Time-Sharing System: Foreword", The Bell System Technical Journal 57:6(1902--1903), July 1978, .

[Nadas20] Nadas, S., Gombos, G., Fejes, F., and S. Laki, "A Congestion Control Independent L4S Scheduler", Proc. Applied Networking Research Workshop (ANRW ’20) 45--51, July 2020.

[NewCC_Proc] Eggert, L., "Experimental Specification of New Congestion Control Algorithms", IETF Operational Note ion-tsv-alt-cc, July 2007.

Briscoe, et al. Expires January 2, 2022 [Page 34] Internet-Draft L4S Architecture July 2021

[QDyn] Briscoe, B., "Rapid Signalling of Queue Dynamics", bobbriscoe.net Technical Report TR-BB-2017-001; arXiv:1904.07044 [cs.NI], September 2017, .

[RFC2475] Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., and W. Weiss, "An Architecture for Differentiated Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, .

[RFC2697] Heinanen, J. and R. Guerin, "A Single Rate Three Color Marker", RFC 2697, DOI 10.17487/RFC2697, September 1999, .

[RFC2698] Heinanen, J. and R. Guerin, "A Two Rate Three Color Marker", RFC 2698, DOI 10.17487/RFC2698, September 1999, .

[RFC2884] Hadi Salim, J. and U. Ahmed, "Performance Evaluation of Explicit Congestion Notification (ECN) in IP Networks", RFC 2884, DOI 10.17487/RFC2884, July 2000, .

[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition of Explicit Congestion Notification (ECN) to IP", RFC 3168, DOI 10.17487/RFC3168, September 2001, .

[RFC3540] Spring, N., Wetherall, D., and D. Ely, "Robust Explicit Congestion Notification (ECN) Signaling with Nonces", RFC 3540, DOI 10.17487/RFC3540, June 2003, .

Briscoe, et al. Expires January 2, 2022 [Page 35] Internet-Draft L4S Architecture July 2021