AVTCORE S. Garcia Murillo Internet-Draft CoSMo Software Intended status: Standards Track Y. Fablet Expires: September 9, 2021 Apple Inc. A. Gouaillard CoSMo Software March 08, 2021 Codec agnostic RTP payload format for video draft-gouaillard-avtcore-codec-agn-rtp-payload-01 Abstract RTP Media Chains usually rely on piping encoder output directly to packetizers. Media packetization formats often support a specific codec format and optimize RTP packets generation accordingly. With the development of Selective Forward Unit (SFU) solutions, that do not process media content server side, the need for media content processing at the origin and at the destination has arised. RTP Media Chains used e.g. in WebRTC solutions are increasingly relying on application-specific transforms that sit in-between encoder and packetizer on one end and in-between depacketizer and decoder on the other end. This use case has become so important, that the W3C is standardizing the capacity to access encoded content with the [WebRTCInsertableStreams] API proposal. An extremely popular use case is application level end-to-end encryption of media content, using for instance [SFrame]. Whatever the modification applied to the media content, RTP packetizers can no longer expect to use packetization formats that mandate media content to be in a specific codec format. In the extreme cases like encryption, where the RTP Payload is made completely opaque to the SFUs, some extra mechanism must also be added for them to be able to route the packets without depending on RTP payload or payload headers. The traditionnal process of creating a new RTP Payload specification per content would not be practical as we would need to make a new one for each codec-transform pair. This document describes a solution, which provides the following features in the case the encoded content has been modified before reaching the packetizer: - a paylaod agnostic RTP packetization format that can be used on any media content, - a negotiation Garcia Murillo, et al. Expires September 9, 2021 [Page 1] Internet-Draft Codec agnostic RTP payload format for video March 2021 mechanism for the above format and the inner payload, Both of the above mechanism are backward compatible with most of (S)RTP/RTCP mechanisms used for bandwidth estimation and congestion control in RTP/SRTP/webrtc, including but not limited to SSRC, RED, FEC, RTX, NACK, SR/RR, REMB, transport-wide-CC, TMBR, .... It as illustrated by existing implementations in chrome, safari, and Medooze. This document also describes a solution to allow SFUs to continue performing packet routing on top of this generic RTP packetization format. This document complements the SFrame (media encryption), and Dependency Descriptor (AV1 payload annex) documents to provide an End-to-End-Encryption solution that would sit on top of SRTP/Webrtc, use SFUs on the media back-end, and leverage W3C APIs in the browser. A high level description of such system will be provided as an informational I-D in the SFrame WG and then cited here. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on September 9, 2021. Copyright Notice Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust’s Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of Garcia Murillo, et al. Expires September 9, 2021 [Page 2] Internet-Draft Codec agnostic RTP payload format for video March 2021 the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . 3 2. Goals . 6 3. RTP Packetization . 7 4. Payload Multiplexing . 7 5. SDP Negotiation . 8 6. SFU Packet Selection . 9 7. Redundancy Techniques Considerations . 10 7.1. Retransmission Techniques . 10 7.2. Forward Error Correction (FEC) Techniques . 10 7.3. Redundant Audio Data Techniques . 10 8. Alternatives . 11 8.1. Generic Packetization With In-Payload APT . 11 8.2. A Payload Type for Generic Packetization AND Media Format 11 8.3. A RTP Header To Choose Packetization . 13 9. Security Considerations . 13 10. IANA Considerations . 14 10.1. Registration of audio/generic . 14 11. Registration of video/generic . 14 12. References . 15 12.1. Normative References . 15 12.2. Informative References . 16 Authors’ Addresses . 17 1. Introduction As per Figure 1 of [RFC7656], a Media Packetizer transforms a single Encoded Stream into one or several RTP packets. The Encoded Stream is coming straight from the Media Encoder and is expected to follow the format produced by the Media Encoder. A number of Media Packetizer formats have been designed to process a specific format produced by Media Encoder. For instance [RFC6184] is dedicated to the processing of content produced by H.264 Media Encoders, and generates packets following NALUs organization. WebRTC applications are increasingly deploying end-to-end encryption solutions on top of RTP Media Chains. End-to-end encryption is implemented by inserting application-specific Media Transformers between Media Encoder and Media Packetizer on the sending side, and between Media Depacketizer and Media Decoder on the receiving side, as described in Figure 1 and Figure 2. To support end-to-end encryption, Media Transformers can use the [SFrame] format. In browsers, Media Transformers are implemented using Garcia Murillo, et al. Expires September 9, 2021 [Page 3] Internet-Draft Codec agnostic RTP payload format for video March 2021 [WebRTCInsertableStreams], for instance by injecting JavaScript code provided by web pages. Physical Stimulus | V +----------------------+ | Media Capture | +----------------------+ | Raw Stream V +----------------------+ | Media Source |<-- Synchronization Timing +----------------------+ | Source Stream V +----------------------+ | Media Encoder | +----------------------+ | Encoded Stream V +----------------------+ | Media Transformer |<-- NEW: application-specific transform +----------------------+ (e.g. SFrame Encryption) | Transformed Stream +------------+ V | V +----------------------+ | +----------------------+ | Media Packetizer | | | RTP-Based Redundancy | +----------------------+ | +----------------------+ | | | +-------------+ Redundancy RTP Stream Source RTP Stream | V V +----------------------+ +----------------------+ | RTP-Based Security | | RTP-Based Security | +----------------------+ +----------------------+ | | Secured RTP Stream Secured Redundancy RTP Stream V V +----------------------+ +----------------------+ | Media Transport | | Media Transport | +----------------------+ +----------------------+ Garcia Murillo, et al. Expires September 9, 2021 [Page 4] Internet-Draft Codec agnostic RTP payload format for video March 2021 Figure 1: Sender Side Concepts in the Media Chain With Application-level Media Transform These RTP packets are sent over the wire to a receiver media chain matching the sender side, reaching the Media Depacketizer that will reconstruct the Encoded Stream before passing it to the Media Decoder. +----------------------+ +----------------------+ | Media Transport | | Media Transport | +----------------------+ +----------------------+ Received | Received | Secured Secured RTP Stream Redundancy RTP Stream V V +----------------------+ +----------------------+ | RTP-Based Validation | | RTP-Based Validation | +----------------------+ +----------------------+ | | Received RTP Stream Received Redundancy RTP Stream | | | +--------------------+ V V +----------------------+ | RTP-Based Repair | +----------------------+ | Repaired RTP Stream V +----------------------+ | Media Depacketizer | +----------------------+ | Received Transformed Stream V +----------------------+ | Media Transformer |<-- NEW: application-specific transform +----------------------+ (e.g. SFrame Decryption) | Received Encoded Stream V +----------------------+ | Media Decoder | +----------------------+ | Received Source Stream V +----------------------+ | Media Sink |--> Synchronization Information Garcia Murillo, et al. Expires September 9, 2021 [Page 5] Internet-Draft Codec agnostic RTP payload format for video March 2021 +----------------------+ | Received Raw Stream V +----------------------+ | Media Render | +----------------------+ | V Physical Stimulus Figure 2: Receiver Side Concepts in the Media Chain With Application-level Media Transform This generic packetization does not change how the mapping between one or several encoded or dependant streams are mapped to the RTP streams or how the synchronization sources(s) (SSRC) are assigned. Given the use of post-encoder application-specific