APPENDIX C
W3C MSE, MEDIA SOURCE EXTENSIONS
7/15/2016 Media Source Extensions
Media Source Extensions W3C Candidate Recommendation 05 July 2016
This Version: http://www.w3.org/TR/2016/CR-media-source-20160705/ Latest Published Version: http://www.w3.org/TR/media-source/ Latest Editor's Draft: http://w3c.github.io/media-source/ Previous Version: http://www.w3.org/TR/2016/CR-media-source-20160503/ Editors: Matthew Wolenetz, Google Inc. Jerry Smith, Microsoft Corporation Mark Watson, Netflix Inc. Aaron Colwell (until April 2015), Google Inc. Adrian Bateman (until April 2015), Microsoft Corporation Repository: We are on GitHub File a bug Commit history Mailing list: public-html[email protected] Implementation: Can I use Media Source Extensions? Test Suite Test Suite repository Implementation report
Copyright © 2016 W3C® (MIT, ERCIM, Keio, Beihang). W3C liability, trademark and document use rules apply.
Abstract
This specification extends HTMLMediaElement [HTML51] to allow JavaScript to generate media streams for playback. Allowing JavaScript to generate streams facilitates a variety of use cases like
https://www.w3.org/TR/media-source/ 1/99 7/15/2016 Media Source Extensions
adaptive streaming and time shifting live streams.
If you wish to make comments or file bugs regarding this document in a manner that is tracked by the editors, please submit them via our public bug database.
Status of This Document
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.
The working group maintains a list of all bug reports that the editors have not yet tried to address; there may also be open bugs in the previous bug tracker. This draft highlights some of the pending issues that are still to be discussed in the working group. No decision has been taken on the outcome of these issues including whether they are valid.
Implementors should be aware that this specification is not stable. Implementors who are not taking part in the discussions are likely to find the specification changing out from under them in incompatible ways. Vendors interested in implementing this specification before it eventually reaches the Candidate Recommendation stage should join the mailing list mentioned below and take part in the discussions.
The following features are at risk and may be removed:
The appendStream method on the sourceBuffer object and algorithm steps that are used only by it.
The VideoPlaybackQuality object and the HTMLVideoElement Extensions that use it.
The totalFrameDelay attribute on the VideoPlaybackQuality object.
The TrackDefault object and its related objects, attributes, methods and algorithms, including: TrackDefaultList, SourceBuffer.trackDefaults, Default track language, Default track label, and Default track kinds. Implementations still must at least include trivial versions of these default track language/label/kinds algorithms that are used by the initialization segment received algorithm. These trivial versions would behave as follows: Trivial default track language and Trivial default track label algorithms must return an empty string, and Trivial default kinds algorithm must return a sequence with a single empty string element in it.
A list of changes is available.
https://www.w3.org/TR/media-source/ 2/99 7/15/2016 Media Source Extensions
For this specification to exit the Candidate Recommendation stage, two independent implementations as detailed in the CR Exit Criteria (Public Permissive version 3) document will be required to demonstrate each feature using the MSE test suite developed by the HTML Media Extensions WG.
A preliminary implementation report is available. This Candidate Recommendation is expected to advance to Proposed Recommendation no earlier than 10 August 2016.
This document was published by the HTML Media Extensions Working Group as a Candidate Recommendation. This document is intended to become a W3C Recommendation. If you wish to make comments regarding this document, please send them to [email protected] (subscribe, archives). W3C publishes a Candidate Recommendation to indicate that the document is believed to be stable and to encourage implementation by the developer community. All comments are welcome.
Publication as a Candidate Recommendation does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.
This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.
This document is governed by the 1 September 2015 W3C Process Document.
Table of Contents
1. Introduction 1.1 Goals 1.2 Definitions
2. MediaSource Object 2.1 Attributes 2.2 Methods 2.3 Event Summary 2.4 Algorithms 2.4.1 Attaching to a media element 2.4.2 Detaching from a media element 2.4.3 Seeking 2.4.4 SourceBuffer Monitoring https://www.w3.org/TR/media-source/ 3/99 7/15/2016 Media Source Extensions
2.4.5 Changes to selected/enabled track state 2.4.6 Duration change 2.4.7 End of stream algorithm
3. SourceBuffer Object 3.1 Attributes 3.2 Methods 3.3 Track Buffers 3.4 Event Summary 3.5 Algorithms 3.5.1 Segment Parser Loop 3.5.2 Reset Parser State 3.5.3 Append Error Algorithm 3.5.4 Prepare Append Algorithm 3.5.5 Buffer Append Algorithm 3.5.6 Stream Append Loop 3.5.7 Range Removal 3.5.8 Initialization Segment Received 3.5.9 Default track language 3.5.10 Default track label 3.5.11 Default track kinds 3.5.12 Coded Frame Processing 3.5.13 Coded Frame Removal Algorithm 3.5.14 Coded Frame Eviction Algorithm 3.5.15 Audio Splice Frame Algorithm 3.5.16 Audio Splice Rendering Algorithm 3.5.17 Text Splice Frame Algorithm
4. SourceBufferList Object 4.1 Attributes 4.2 Methods 4.3 Event Summary
5. VideoPlaybackQuality Object 5.1 Attributes
6. TrackDefault Object 6.1 Constructors 6.2 Attributes
https://www.w3.org/TR/media-source/ 4/99 7/15/2016 Media Source Extensions
6.3 Methods
7. TrackDefaultList Object 7.1 Constructors 7.2 Attributes 7.3 Methods
8. URL Object Extensions 8.1 Methods
9. HTMLMediaElement Extensions
10. HTMLVideoElement Extensions 10.1 Methods
11. AudioTrack Extensions 11.1 Attributes
12. VideoTrack Extensions 12.1 Attributes
13. TextTrack Extensions 13.1 Attributes
14. Byte Stream Formats
15. Examples
16. Acknowledgments
17. Revision History
A. References A.1 Normative references A.2 Informative references
1. Introduction
This specification allows JavaScript to dynamically construct media streams for
https://www.w3.org/TR/media-source/ 5/99 7/15/2016 Media Source Extensions
A buffering model is also included to describe how the user agent acts when different media segments are appended at different times. Byte stream specifications used with these extensions are available in the byte stream format registry [MSE-REGISTRY].
MediaSource
SourceBuffer SourceBuffer SourceBuffer
Media Source API HTML Media Element
Track Track Track Track Track Buffer Buffer Buffer Buffer Buffer
Video Audio Audio Video Audio Decoder Decoder Decoder Decoder Decoder
Audio Device
Video Tag Display Region
1.1 Goals
This specification was designed with the following goals in mind:
Allow JavaScript to construct media streams independent of how the media is fetched.
https://www.w3.org/TR/media-source/ 6/99 7/15/2016 Media Source Extensions
Define a splicing and buffering model that facilitates use cases like adaptive streaming, ad- insertion, time-shifting, and video editing.
Minimize the need for media parsing in JavaScript.
Leverage the browser cache as much as possible.
Provide requirements for byte stream format specifications.
Not require support for any particular media format or codec.
This specification defines:
Normative behavior for user agents to enable interoperability between user agents and web applications when processing media data.
Normative requirements to enable other specifications to define media formats to be used within this specification.
1.2 Definitions
Active Track Buffers The track buffers that provide coded frames for the enabled audioTracks, the selected videoTracks, and the "showing" or "hidden" textTracks. All these tracks are associated with SourceBuffer objects in the activeSourceBuffers list.
Append Window A presentation timestamp range used to filter out coded frames while appending. The append window represents a single continuous time range with a single start time and end time. Coded frames with presentation timestamp within this range are allowed to be appended to the SourceBuffer while coded frames outside this range are filtered out. The append window start and end times are controlled by the appendWindowStart and appendWindowEnd attributes respectively.
Coded Frame A unit of media data that has a presentation timestamp, a decode timestamp, and a coded frame duration.
Coded Frame Duration The duration of a coded frame. For video and text, the duration indicates how long the video frame or text should be displayed. For audio, the duration represents the sum of all the samples
https://www.w3.org/TR/media-source/ 7/99 7/15/2016 Media Source Extensions
contained within the coded frame. For example, if an audio frame contained 441 samples @44100Hz the frame duration would be 10 milliseconds.
Coded Frame End Timestamp The sum of a coded frame presentation timestamp and its coded frame duration. It represents the presentation timestamp that immediately follows the coded frame.
Coded Frame Group A group of coded frames that are adjacent and have monotonically increasing decode timestamps without any gaps. Discontinuities detected by the coded frame processing algorithm and abort() calls trigger the start of a new coded frame group.
Decode Timestamp The decode timestamp indicates the latest time at which the frame needs to be decoded assuming instantaneous decoding and rendering of this and any dependant frames (this is equal to the presentation timestamp of the earliest frame, in presentation order, that is dependant on this frame). If frames can be decoded out of presentation order, then the decode timestamp must be present in or derivable from the byte stream. The user agent must run the append error algorithm with the decode error parameter set to true if this is not the case. If frames cannot be decoded out of presentation order and a decode timestamp is not present in the byte stream, then the decode timestamp is equal to the presentation timestamp.
Displayed Frame Delay The delay between a frame's presentation time and the actual time it was displayed, in a double- precision value in seconds & rounded to the nearest display refresh interval. This delay is always greater than or equal to zero since frames must never be displayed before their presentation time. Non-zero delays are a sign of playback jitter and possible loss of A/V sync.
Initialization Segment A sequence of bytes that contain all of the initialization information required to decode a sequence of media segments. This includes codec initialization data, Track ID mappings for multiplexed segments, and timestamp offsets (e.g. edit lists).
NOTE
The byte stream format specifications in the byte stream format registry [MSE- REGISTRY] contain format specific examples.
Media Segment A sequence of bytes that contain packetized & timestamped media data for a portion of the media timeline. Media segments are always associated with the most recently appended initialization
https://www.w3.org/TR/media-source/ 8/99 7/15/2016 Media Source Extensions
segment.
NOTE
The byte stream format specifications in the byte stream format registry [MSE- REGISTRY] contain format specific examples.
MediaSource object URL A MediaSource object URL is a unique Blob URI [FILE-API] created by createObjectURL(). It is used to attach a MediaSource object to an HTMLMediaElement.
These URLs are the same as a Blob URI, except that anything in the definition of that feature that refers to File and Blob objects is hereby extended to also apply to MediaSource objects.
The origin of the MediaSource object URL is the effective script origin of the document that called createObjectURL().
NOTE
For example, the origin of the MediaSource object URL affects the way that the media element is consumed by canvas.
Parent Media Source The parent media source of a SourceBuffer object is the MediaSource object that created it.
Presentation Start Time The presentation start time is the earliest time point in the presentation and specifies the initial playback position and earliest possible position. All presentations created using this specification have a presentation start time of 0.
https://www.w3.org/TR/media-source/ 9/99 7/15/2016 Media Source Extensions
NOTE
For the purposes of determining if HTMLMediaElement.buffered contains a TimeRange that includes the current playback position, implementations may choose to allow a current playback position at or after presentation start time and before the first TimeRange to play the first TimeRange if that TimeRange starts within a reasonably short time, like 1 second, after presentation start time. This allowance accommodates the reality that muxed streams commonly do not begin all tracks precisely at presentation start time. Implementations should report the actual buffered range, regardless of this allowance.
Presentation Interval The presentation interval of a coded frame is the time interval from its presentation timestamp to the presentation timestamp plus the coded frame's duration. For example, if a coded frame has a presentation timestamp of 10 seconds and a coded frame duration of 100 milliseconds, then the presentation interval would be [10-10.1). Note that the start of the range is inclusive, but the end of the range is exclusive.
Presentation Order The order that coded frames are rendered in the presentation. The presentation order is achieved by ordering coded frames in monotonically increasing order by their presentation timestamps.
Presentation Timestamp A reference to a specific time in the presentation. The presentation timestamp in a coded frame indicates when the frame must be rendered.
Random Access Point A position in a media segment where decoding and continuous playback can begin without relying on any previous data in the segment. For video this tends to be the location of I-frames. In the case of audio, most audio frames can be treated as a random access point. Since video tracks tend to have a more sparse distribution of random access points, the location of these points are usually considered the random access points for multiplexed streams.
SourceBuffer byte stream format specification The specific byte stream format specification that describes the format of the byte stream accepted by a SourceBuffer instance. The byte stream format specification, for a SourceBuffer object, is selected based on the type passed to the addSourceBuffer() call that created the object.
SourceBuffer configuration A specific set of tracks distributed across one or more SourceBuffer objects owned by a single MediaSource instance.
https://www.w3.org/TR/media-source/ 10/99 7/15/2016 Media Source Extensions
Implementations must support at least 1 MediaSource object with the following configurations:
A single SourceBuffer with 1 audio track and/or 1 video track.
Two SourceBuffers with one handling a single audio track and the other handling a single video track.
MediaSource objects must support each of the configurations above, but they are only required to support one configuration at a time. Supporting multiple configurations at once or additional configurations is a quality of implementation issue.
Track Description A byte stream format specific structure that provides the Track ID, codec configuration, and other metadata for a single track. Each track description inside a single initialization segment has a unique Track ID. The user agent must run the append error algorithm with the decode error parameter set to true if the Track ID is not unique within the initialization segment.
Track ID A Track ID is a byte stream format specific identifier that marks sections of the byte stream as being part of a specific track. The Track ID in a track description identifies which sections of a media segment belong to that track.
2. MediaSource Object
The MediaSource object represents a source of media data for an HTMLMediaElement. It keeps track of the readyState for this source as well as a list of SourceBuffer objects that can be used to add media data to the presentation. MediaSource objects are created by the web application and then attached to an HTMLMediaElement. The application uses the SourceBuffer objects in sourceBuffers to add media data to this source. The HTMLMediaElement fetches this media data from the MediaSource object when it is needed during playback.
Each MediaSource object has a live seekable range variable that stores a normalized TimeRanges object. This variable is initialized to an empty TimeRanges object when the MediaSource object is created, is maintained by setLiveSeekableRange() and clearLiveSeekableRange(), and is used in HTMLMediaElement Extensions to modify HTMLMediaElement.seekable behavior.
https://www.w3.org/TR/media-source/ 11/99 7/15/2016 Media Source Extensions
WebIDL
enum ReadyState { "closed", "open", "ended" };
Enumeration description
closed Indicates the source is not currently attached to a media element. The source has been opened by a media element and is ready for data to be appended to open the SourceBuffer objects in sourceBuffers.
ended The source is still attached to a media element, but endOfStream() has been called.
WebIDL
enum EndOfStreamError { "network", "decode" };
Enumeration description
Terminates playback and signals that a network error has occured.
NOTE network JavaScript applications should use this status code to terminate playback with a network error. For example, if a network error occurs while fetching media data.
Terminates playback and signals that a decoding error has occured.
NOTE
JavaScript applications should use this status code to terminate playback with a decode error. For example, if a parsing error occurs while processing out-of- decode band media data.
https://www.w3.org/TR/media-source/ 12/99 7/15/2016 Media Source Extensions
WebIDL
[Constructor] interface MediaSource : EventTarget { readonly attribute SourceBufferList sourceBuffers; readonly attribute SourceBufferList activeSourceBuffers; readonly attribute ReadyState readyState; attribute unrestricted double duration; attribute EventHandler onsourceopen; attribute EventHandler onsourceended; attribute EventHandler onsourceclose; SourceBuffer addSourceBuffer (DOMString type); void removeSourceBuffer (SourceBuffer sourceBuffer); void endOfStream (optional EndOfStreamError error); void setLiveSeekableRange (double start, double end); void clearLiveSeekableRange (); static boolean isTypeSupported (DOMString type); };
2.1 Attributes
activeSourceBuffers of type SourceBufferList, readonly Contains the subset of sourceBuffers that are providing the selected video track, the enabled audio track(s), and the "showing" or "hidden" text track(s).
SourceBuffer objects in this list must appear in the same order as they appear in the sourceBuffers attribute. (e.g., If only sourceBuffers[0] and sourceBuffers[3] are in activeSourceBuffers, then activeSourceBuffers[0] must equal sourceBuffers[0] and activeSourceBuffers[1] must equal sourceBuffers[3].)
NOTE
The Changes to selected/enabled track state section describes how this attribute gets updated.
duration of type unrestricted double
https://www.w3.org/TR/media-source/ 13/99 7/15/2016 Media Source Extensions
Allows the web application to set the presentation duration. The duration is initially set to NaN when the MediaSource object is created.
On getting, run the following steps:
1. If the readyState attribute is "closed" then return NaN and abort these steps.
2. Return the current value of the attribute.
On setting, run the following steps:
1. If the value being set is negative or NaN then throw a TypeError exception and abort these steps.
2. If the readyState attribute is not "open" then throw an InvalidStateError exception and abort these steps.
3. If the updating attribute equals true on any SourceBuffer in sourceBuffers, then throw an InvalidStateError exception and abort these steps.
4. Run the duration change algorithm with new duration set to the value being assigned to this attribute.
NOTE
appendBuffer(), appendStream() and endOfStream() can update the duration under certain circumstances.
onsourceclose of type EventHandler The event handler for the sourceclose event.
onsourceended of type EventHandler The event handler for the sourceended event.
onsourceopen of type EventHandler The event handler for the sourceopen event.
readyState of type ReadyState, readonly Indicates the current state of the MediaSource object. When the MediaSource is created readyState must be set to "closed".
sourceBuffers of type SourceBufferList, readonly
https://www.w3.org/TR/media-source/ 14/99 7/15/2016 Media Source Extensions
Contains the list of SourceBuffer objects associated with this MediaSource. When readyState equals "closed" this list will be empty. Once readyState transitions to "open" SourceBuffer objects can be added to this list by using addSourceBuffer().
2.2 Methods
addSourceBuffer Adds a new SourceBuffer to sourceBuffers.
Parameter Type Nullable Optional Description
type DOMString ✘ ✘
Return type: SourceBuffer
When this method is invoked, the user agent must run the following steps:
1. If type is an empty string then throw a TypeError exception and abort these steps.
2. If type contains a MIME type that is not supported or contains a MIME type that is not supported with the types specified for the other SourceBuffer objects in sourceBuffers, then throw a NotSupportedError exception and abort these steps.
3. If the user agent can't handle any more SourceBuffer objects or if creating a SourceBuffer based on type would result in an unsupported SourceBuffer configuration, then throw a QuotaExceededError exception and abort these steps.
NOTE
For example, a user agent may throw a QuotaExceededError exception if the media element has reached the HAVE_METADATA readyState. This can occur if the user agent's media engine does not support adding more tracks during playback.
4. If the readyState attribute is not in the "open" state then throw an InvalidStateError exception and abort these steps.
5. Create a new SourceBuffer object and associated resources.
6. Set the generate timestamps flag on the new object to the value in the "Generate Timestamps Flag" column of the byte stream format registry [MSE-REGISTRY] entry that is associated with type.
https://www.w3.org/TR/media-source/ 15/99 7/15/2016 Media Source Extensions
7. If the generate timestamps flag equals true: Set the mode attribute on the new object to "sequence". ↪ Otherwise: Set the mode attribute on the new object to "segments". ↪ 8. Add the new object to sourceBuffers and queue a task to fire a simple event named addsourcebuffer at sourceBuffers.
9. Return the new object.
clearLiveSeekableRange Updates the live seekable range variable used in HTMLMediaElement Extensions to modify HTMLMediaElement.seekable behavior.
No parameters. Return type: void
When this method is invoked, the user agent must run the following steps:
1. If the readyState attribute is not "open" then throw an InvalidStateError exception and abort these steps.
2. If the updating attribute equals true on any SourceBuffer in sourceBuffers, then throw an InvalidStateError exception and abort these steps.
3. If live seekable range contains a range, then set live seekable range to be a new empty TimeRanges object.
endOfStream Signals the end of the stream.
Parameter Type Nullable Optional Description
error EndOfStreamError ✘ ✔
Return type: void
When this method is invoked, the user agent must run the following steps:
1. If the readyState attribute is not in the "open" state then throw an InvalidStateError exception and abort these steps.
2. If the updating attribute equals true on any SourceBuffer in sourceBuffers, then throw an InvalidStateError exception and abort these steps.
https://www.w3.org/TR/media-source/ 16/99 7/15/2016 Media Source Extensions
3. Run the end of stream algorithm with the error parameter set to error.
isTypeSupported, static Check to see whether the MediaSource is capable of creating SourceBuffer objects for the specified MIME type.
NOTE
If true is returned from this method, it only indicates that the MediaSource implementation is capable of creating SourceBuffer objects for the specified MIME type. An addSourceBuffer() call may still fail if sufficient resources are not available to support the addition of a new SourceBuffer.
NOTE
This method returning true implies that HTMLMediaElement.canPlayType() will return "maybe" or "probably" since it does not make sense for a MediaSource to support a type the HTMLMediaElement knows it cannot play.
Parameter Type Nullable Optional Description
type DOMString ✘ ✘
Return type: boolean
When this method is invoked, the user agent must run the following steps:
1. If type is an empty string, then return false.
2. If type does not contain a valid MIME type string, then return false.
3. If type contains a media type or media subtype that the MediaSource does not support, then return false.
4. If type contains a codec that the MediaSource does not support, then return false.
5. If the MediaSource does not support the specified combination of media type, media subtype, and codecs then return false.
6. Return true.
removeSourceBuffer https://www.w3.org/TR/media-source/ 17/99 7/15/2016 Media Source Extensions
Removes a SourceBuffer from sourceBuffers.
Parameter Type Nullable Optional Description
sourceBuffer SourceBuffer ✘ ✘
Return type: void
When this method is invoked, the user agent must run the following steps:
1. If sourceBuffer specifies an object that is not in sourceBuffers then throw a NotFoundError exception and abort these steps.
2. If the sourceBuffer.updating attribute equals true, then run the following steps:
1. Abort the buffer append and stream append loop algorithms if they are running.
2. Set the sourceBuffer.updating attribute to false.
3. Queue a task to fire a simple event named abort at sourceBuffer.
4. Queue a task to fire a simple event named updateend at sourceBuffer.
3. Let SourceBuffer audioTracks list equal the AudioTrackList object returned by sourceBuffer.audioTracks.
4. If the SourceBuffer audioTracks list is not empty, then run the following steps:
1. Let HTMLMediaElement audioTracks list equal the AudioTrackList object returned by the audioTracks attribute on the HTMLMediaElement.
2. Let the removed enabled audio track flag equal false.
3. For each AudioTrack object in the SourceBuffer audioTracks list, run the following steps:
1. Set the sourceBuffer attribute on the AudioTrack object to null.
2. If the enabled attribute on the AudioTrack object is true, then set the removed enabled audio track flag to true.
3. Remove the AudioTrack object from the HTMLMediaElement audioTracks list.
https://www.w3.org/TR/media-source/ 18/99 7/15/2016 Media Source Extensions
4. Queue a task to fire a trusted event named removetrack, that does not bubble and is not cancelable, and that uses the TrackEvent interface, at the HTMLMediaElement audioTracks list.
5. Remove the AudioTrack object from the SourceBuffer audioTracks list.
6. Queue a task to fire a trusted event named removetrack, that does not bubble and is not cancelable, and that uses the TrackEvent interface, at the SourceBuffer audioTracks list.
4. If the removed enabled audio track flag equals true, then queue a task to fire a simple event named change at the HTMLMediaElement audioTracks list.
5. Let SourceBuffer videoTracks list equal the VideoTrackList object returned by sourceBuffer.videoTracks.
6. If the SourceBuffer videoTracks list is not empty, then run the following steps:
1. Let HTMLMediaElement videoTracks list equal the VideoTrackList object returned by the videoTracks attribute on the HTMLMediaElement.
2. Let the removed selected video track flag equal false.
3. For each VideoTrack object in the SourceBuffer videoTracks list, run the following steps:
1. Set the sourceBuffer attribute on the VideoTrack object to null.
2. If the selected attribute on the VideoTrack object is true, then set the removed selected video track flag to true.
3. Remove the VideoTrack object from the HTMLMediaElement videoTracks list.
4. Queue a task to fire a trusted event named removetrack, that does not bubble and is not cancelable, and that uses the TrackEvent interface, at the HTMLMediaElement videoTracks list.
5. Remove the VideoTrack object from the SourceBuffer videoTracks list.
6. Queue a task to fire a trusted event named removetrack, that does not bubble and is not cancelable, and that uses the TrackEvent interface, at the SourceBuffer videoTracks list.
https://www.w3.org/TR/media-source/ 19/99 7/15/2016 Media Source Extensions
4. If the removed selected video track flag equals true, then queue a task to fire a simple event named change at the HTMLMediaElement videoTracks list.
7. Let SourceBuffer textTracks list equal the TextTrackList object returned by sourceBuffer.textTracks.
8. If the SourceBuffer textTracks list is not empty, then run the following steps:
1. Let HTMLMediaElement textTracks list equal the TextTrackList object returned by the textTracks attribute on the HTMLMediaElement.
2. Let the removed enabled text track flag equal false.
3. For each TextTrack object in the SourceBuffer textTracks list, run the following steps:
1. Set the sourceBuffer attribute on the TextTrack object to null.
2. If the mode attribute on the TextTrack object is set to "showing" or "hidden", then set the removed enabled text track flag to true.
3. Remove the TextTrack object from the HTMLMediaElement textTracks list.
4. Queue a task to fire a trusted event named removetrack, that does not bubble and is not cancelable, and that uses the TrackEvent interface, at the HTMLMediaElement textTracks list.
5. Remove the TextTrack object from the SourceBuffer textTracks list.
6. Queue a task to fire a trusted event named removetrack, that does not bubble and is not cancelable, and that uses the TrackEvent interface, at the SourceBuffer textTracks list.
4. If the removed enabled text track flag equals true, then queue a task to fire a simple event named change at the HTMLMediaElement textTracks list.
9. If sourceBuffer is in activeSourceBuffers, then remove sourceBuffer from activeSourceBuffers and queue a task to fire a simple event named removesourcebuffer at the SourceBufferList returned by activeSourceBuffers.
10. Remove sourceBuffer from sourceBuffers and queue a task to fire a simple event named removesourcebuffer at the SourceBufferList returned by sourceBuffers.
11. Destroy all resources for sourceBuffer.
https://www.w3.org/TR/media-source/ 20/99 7/15/2016 Media Source Extensions
setLiveSeekableRange Updates the live seekable range variable used in HTMLMediaElement Extensions to modify HTMLMediaElement.seekable behavior.
Parameter Type Nullable Optional Description
start double ✘ ✘
end double ✘ ✘
Return type: void
When this method is invoked, the user agent must run the following steps:
1. If the readyState attribute is not "open" then throw an InvalidStateError exception and abort these steps.
2. If the updating attribute equals true on any SourceBuffer in sourceBuffers, then throw an InvalidStateError exception and abort these steps.
3. If start is negative or greater than end, then throw a TypeError exception and abort these steps.
4. Set live seekable range to be a new normalized TimeRanges object containing a single range whose start position is start and end position is end.
2.3 Event Summary
Event name Interface Dispatched when...
sourceopen Event readyState transitions from "closed" to "open" or from "ended" to "open".
sourceended Event readyState transitions from "open" to "ended".
sourceclose Event readyState transitions from "open" to "closed" or "ended" to "closed".
2.4 Algorithms
2.4.1 Attaching to a media element
https://www.w3.org/TR/media-source/ 21/99 7/15/2016 Media Source Extensions
A MediaSource object can be attached to a media element by assigning a MediaSource object URL to the media element src attribute or the src attribute of a
If the resource fetch algorithm absolute URL matches the MediaSource object URL, ignore any preload attribute of the media element, skip any optional steps for when preload equals none, and run the following steps right before the "Perform a potentially CORS-enabled fetch" step in the resource fetch algorithm.
If readyState is NOT set to "closed" Run the "If the media data cannot be fetched at all, due to network errors, causing the user ↪ agent to give up trying to fetch the resource" steps of the resource fetch algorithm.
Otherwise
↪ 1. Set the media element's delaying-the-load-event-flag to false.
2. Set the readyState attribute to "open".
3. Queue a task to fire a simple event named sourceopen at the MediaSource.
4. Continue the resource fetch algorithm by running the "Perform a potentially CORS- enabled fetch" step, with these clarifications:
1. Text in the resource fetch algorithm that refers to "the download" or "bytes received" refers to data passed in via appendBuffer() and appendStream().
2. References to HTTP in the resource fetch algorithm do not apply because the HTMLMediaElement does not fetch media data via HTTP when a MediaSource is attached.
3. Text in the resource fetch algorithm that refers to transitions to "NETWORK_IDLE" and firing "suspend" events, and the paragraph that begins with "User agents may decide to not download more content at any time", do not apply to a media element that has a MediaSource object attached by this algorithm.
2.4.2 Detaching from a media element
The following steps are run in any case where the media element is going to transition to NETWORK_EMPTY and queue a task to fire a simple event named emptied at the media element. These steps must be run right before the transition.
https://www.w3.org/TR/media-source/ 22/99 7/15/2016 Media Source Extensions
1. Set the readyState attribute to "closed".
2. Update duration to NaN.
3. Remove all the SourceBuffer objects from activeSourceBuffers.
4. Queue a task to fire a simple event named removesourcebuffer at activeSourceBuffers.
5. Remove all the SourceBuffer objects from sourceBuffers.
6. Queue a task to fire a simple event named removesourcebuffer at sourceBuffers.
7. Queue a task to fire a simple event named sourceclose at the MediaSource.
NOTE
Going forward, this algorithm is intended to be externally called and run in any case where the attached MediaSource, if any, must be detached from the media element. It may be called on HTMLMediaElement [HTML51] operations like load() and resource fetch algorithm failures in addition to, or in place of, when the media element transitions to NETWORK_EMPTY. Resource fetch algorithm failures are those which abort either the resource fetch algorithm or the resource selection algorithm, with the exception that the "Final step" [HTML51] is not considered a failure that triggers detachment.
2.4.3 Seeking
Run the following steps as part of the "Wait until the user agent has established whether or not the media data for the new playback position is available, and, if it is, until it has decoded enough data to play back that position" step of the seek algorithm:
1. The media element looks for media segments containing the new playback position in each SourceBuffer object in activeSourceBuffers.
If one or more of the objects in activeSourceBuffers is missing media segments for the new playback position ↪ 1. If the HTMLMediaElement.readyState attribute is greater than HAVE_METADATA, then set the HTMLMediaElement.readyState attribute to HAVE_METADATA.
2. The media element waits until an appendBuffer() or an appendStream() call causes the coded frame processing algorithm to set the
https://www.w3.org/TR/media-source/ 23/99 7/15/2016 Media Source Extensions
HTMLMediaElement.readyState attribute to a value greater than HAVE_METADATA.
NOTE
The web application can use buffered to determine what the media element needs to resume playback.
Otherwise Continue ↪ 2. The media element resets all decoders and initializes each one with data from the appropriate initialization segment.
3. The media element feeds coded frames from the active track buffers into the decoders starting with the closest random access point before the new playback position.
4. Resume the seek algorithm at the "Await a stable state" step.
2.4.4 SourceBuffer Monitoring
The following steps are periodically run during playback to make sure that all of the SourceBuffer objects in activeSourceBuffers have enough data to ensure uninterrupted playback. Changes to activeSourceBuffers also cause these steps to run because they affect the conditions that trigger state transitions.
Having enough data to ensure uninterrupted playback is an implementation specific condition where the user agent determines that it currently has enough data to play the presentation without stalling for a meaningful period of time. This condition is constantly evaluated to determine when to transition the media element into and out of the HAVE_ENOUGH_DATA ready state. These transitions indicate when the user agent believes it has enough data buffered or it needs more data respectively.
NOTE
An implementation may choose to use bytes buffered, time buffered, the append rate, or any other metric it sees fit to determine when it has enough data. The metrics used may change during playback so web applications should only rely on the value of HTMLMediaElement.readyState to determine whether more data is needed or not.
https://www.w3.org/TR/media-source/ 24/99 7/15/2016 Media Source Extensions
NOTE
When the media element needs more data, the user agent should transition it from HAVE_ENOUGH_DATA to HAVE_FUTURE_DATA early enough for a web application to be able to respond without causing an interruption in playback. For example, transitioning when the current playback position is 500ms before the end of the buffered data gives the application roughly 500ms to append more data before playback stalls.
If the the HTMLMediaElement.readyState attribute equals HAVE_NOTHING:
↪ 1. Abort these steps.
If HTMLMediaElement.buffered does not contain a TimeRange for the current playback position: ↪ 1. Set the HTMLMediaElement.readyState attribute to HAVE_METADATA.
2. Abort these steps.
If HTMLMediaElement.buffered contains a TimeRange that includes the current playback position and enough data to ensure uninterrupted playback: ↪ 1. Set the HTMLMediaElement.readyState attribute to HAVE_ENOUGH_DATA.
2. Queue a task to fire a simple event named canplaythrough at the media element.
3. Playback may resume at this point if it was previously suspended by a transition to HAVE_CURRENT_DATA.
4. Abort these steps.
If HTMLMediaElement.buffered contains a TimeRange that includes the current playback position and some time beyond the current playback position, then run the following steps: ↪ 1. Set the HTMLMediaElement.readyState attribute to HAVE_FUTURE_DATA.
2. If the previous value of HTMLMediaElement.readyState was less than HAVE_FUTURE_DATA, then queue a task to fire a simple event named canplay at the media element.
3. Playback may resume at this point if it was previously suspended by a transition to HAVE_CURRENT_DATA.
4. Abort these steps.
https://www.w3.org/TR/media-source/ 25/99 7/15/2016 Media Source Extensions
If HTMLMediaElement.buffered contains a TimeRange that ends at the current playback position and does not have a range covering the time immediately after the current position: ↪ 1. Set the HTMLMediaElement.readyState attribute to HAVE_CURRENT_DATA.
2. If this is the first transition to HAVE_CURRENT_DATA, then queue a task to fire a simple event named loadeddata at the media element.
3. Playback is suspended at this point since the media element doesn't have enough data to advance the media timeline.
4. Abort these steps.
2.4.5 Changes to selected/enabled track state
During playback activeSourceBuffers needs to be updated if the selected video track, the enabled audio track(s), or a text track mode changes. When one or more of these changes occur the following steps need to be followed.
If the selected video track changes, then run the following steps:
↪ 1. If the SourceBuffer associated with the previously selected video track is not associated with any other enabled tracks, run the following steps:
1. Remove the SourceBuffer from activeSourceBuffers.
2. Queue a task to fire a simple event named removesourcebuffer at activeSourceBuffers
2. If the SourceBuffer associated with the newly selected video track is not already in activeSourceBuffers, run the following steps:
1. Add the SourceBuffer to activeSourceBuffers.
2. Queue a task to fire a simple event named addsourcebuffer at activeSourceBuffers
If an audio track becomes disabled and the SourceBuffer associated with this track is not associated with any other enabled or selected track, then run the following steps: ↪ 1. Remove the SourceBuffer associated with the audio track from activeSourceBuffers
https://www.w3.org/TR/media-source/ 26/99 7/15/2016 Media Source Extensions
2. Queue a task to fire a simple event named removesourcebuffer at activeSourceBuffers
If an audio track becomes enabled and the SourceBuffer associated with this track is not already in activeSourceBuffers, then run the following steps: ↪ 1. Add the SourceBuffer associated with the audio track to activeSourceBuffers
2. Queue a task to fire a simple event named addsourcebuffer at activeSourceBuffers
If a text track mode becomes "disabled" and the SourceBuffer associated with this track is not associated with any other enabled or selected track, then run the following steps: ↪ 1. Remove the SourceBuffer associated with the text track from activeSourceBuffers
2. Queue a task to fire a simple event named removesourcebuffer at activeSourceBuffers
If a text track mode becomes "showing" or "hidden" and the SourceBuffer associated with this track is not already in activeSourceBuffers, then run the following steps: ↪ 1. Add the SourceBuffer associated with the text track to activeSourceBuffers
2. Queue a task to fire a simple event named addsourcebuffer at activeSourceBuffers
2.4.6 Duration change
Follow these steps when duration needs to change to a new duration.
1. If the current value of duration is equal to new duration, then return.
2. If new duration is less than the highest starting presentation timestamp of any buffered coded frames for all SourceBuffer objects in sourceBuffers, then throw an InvalidStateError exception and abort these steps.
NOTE
Duration reductions that would truncate currently buffered media are disallowed. When truncation is necessary, use remove() to reduce the buffered range before updating duration.
3. Set old duration to the current value of duration.
https://www.w3.org/TR/media-source/ 27/99 7/15/2016 Media Source Extensions
4. Update duration to new duration.
5. If a user agent is unable to partially render audio frames or text cues that start before and end after the duration, then run the following steps:
NOTE
This condition can occur because the coded frame removal algorithm preserves audio frames and text cues that start before and end after the duration.
1. Update new duration to the highest end time reported by the buffered attribute across all SourceBuffer objects in sourceBuffers.
2. Update duration to new duration.
6. Update the media duration to new duration and run the HTMLMediaElement duration change algorithm.
2.4.7 End of stream algorithm
This algorithm gets called when the application signals the end of stream via an endOfStream() call or an algorithm needs to signal a decode error. This algorithm takes an error parameter that indicates whether an error will be signalled.
1. Change the readyState attribute value to "ended".
2. Queue a task to fire a simple event named sourceended at the MediaSource.
3. If error is not set
↪ 1. Run the duration change algorithm with new duration set to the highest end time reported by the buffered attribute across all SourceBuffer objects in sourceBuffers.
https://www.w3.org/TR/media-source/ 28/99 7/15/2016 Media Source Extensions
NOTE
This allows the duration to properly reflect the end of the appended media segments. For example, if the duration was explicitly set to 10 seconds and only media segments for 0 to 5 seconds were appended before endOfStream() was called, then the duration will get updated to 5 seconds.
2. Notify the media element that it now has all of the media data.
If error is set to "network"
↪ If the HTMLMediaElement.readyState attribute equals HAVE_NOTHING Run the "If the media data cannot be fetched at all, due to network errors, ↪ causing the user agent to give up trying to fetch the resource" steps of the resource fetch algorithm.
If the HTMLMediaElement.readyState attribute is greater than HAVE_NOTHING Run the "If the connection is interrupted after some media data has been ↪ received, causing the user agent to give up trying to fetch the resource" steps of the resource fetch algorithm.
If error is set to "decode"
↪ If the HTMLMediaElement.readyState attribute equals HAVE_NOTHING Run the "If the media data can be fetched but is found by inspection to be in ↪ an unsupported format, or can otherwise not be rendered at all" steps of the resource fetch algorithm.
If the HTMLMediaElement.readyState attribute is greater than HAVE_NOTHING Run the media data is corrupted steps of the resource fetch algorithm. ↪
3. SourceBuffer Object
WebIDL
enum AppendMode { "segments", "sequence" };
Enumeration description
https://www.w3.org/TR/media-source/ 29/99 7/15/2016 Media Source Extensions
The timestamps in the media segment determine where the coded frames are placed in segments the presentation. Media segments can be appended in any order.
Media segments will be treated as adjacent in time independent of the timestamps in the media segment. Coded frames in a new media segment will be placed immediately after the coded frames in the previous media segment. The timestampOffset attribute will be updated if a new offset is needed to make the new sequence media segments adjacent to the previous media segment. Setting the timestampOffset attribute in "sequence" mode allows a media segment to be placed at a specific position in the timeline without any knowledge of the timestamps in the media segment.
WebIDL
interface SourceBuffer : EventTarget { attribute AppendMode mode; readonly attribute boolean updating; readonly attribute TimeRanges buffered; attribute double timestampOffset; readonly attribute AudioTrackList audioTracks; readonly attribute VideoTrackList videoTracks; readonly attribute TextTrackList textTracks; attribute double appendWindowStart; attribute unrestricted double appendWindowEnd; attribute EventHandler onupdatestart; attribute EventHandler onupdate; attribute EventHandler onupdateend; attribute EventHandler onerror; attribute EventHandler onabort; void appendBuffer (ArrayBuffer data); void appendBuffer (ArrayBufferView data); void appendStream (ReadableStream stream, [EnforceRange] optional unsigned long long maxSize); void abort (); void remove (double start, unrestricted double end); attribute TrackDefaultList trackDefaults; };
3.1 Attributes
appendWindowEnd of type unrestricted double The presentation timestamp for the end of the append window. This attribute is initially set to positive Infinity.
https://www.w3.org/TR/media-source/ 30/99 7/15/2016 Media Source Extensions
On getting, Return the initial value or the last value that was successfully set.
On setting, run the following steps:
1. If this object has been removed from the sourceBuffers attribute of the parent media source, then throw an InvalidStateError exception and abort these steps.
2. If the updating attribute equals true, then throw an InvalidStateError exception and abort these steps.
3. If the new value equals NaN, then throw a TypeError and abort these steps.
4. If the new value is less than or equal to appendWindowStart then throw a TypeError exception and abort these steps.
5. Update the attribute to the new value.
appendWindowStart of type double The presentation timestamp for the start of the append window. This attribute is initially set to the presentation start time.
On getting, Return the initial value or the last value that was successfully set.
On setting, run the following steps:
1. If this object has been removed from the sourceBuffers attribute of the parent media source, then throw an InvalidStateError exception and abort these steps.
2. If the updating attribute equals true, then throw an InvalidStateError exception and abort these steps.
3. If the new value is less than 0 or greater than or equal to appendWindowEnd then throw a TypeError exception and abort these steps.
4. Update the attribute to the new value.
audioTracks of type AudioTrackList, readonly The list of AudioTrack objects created by this object.
buffered of type TimeRanges, readonly Indicates what TimeRanges are buffered in the SourceBuffer. This attribute is initially set to an empty TimeRanges object when the object is created.
When the attribute is read the following steps must occur:
https://www.w3.org/TR/media-source/ 31/99 7/15/2016 Media Source Extensions
1. If this object has been removed from the sourceBuffers attribute of the parent media source then throw an InvalidStateError exception and abort these steps.
2. Let highest end time be the largest track buffer ranges end time across all the track buffers managed by this SourceBuffer object.
3. Let intersection ranges equal a TimeRange object containing a single range from 0 to highest end time.
4. For each track buffer managed by this SourceBuffer, run the following steps:
1. Let track ranges equal the track buffer ranges for the current track buffer.
2. If readyState is "ended", then set the end time on the last range in track ranges to highest end time.
3. Let new intersection ranges equal the intersection between the intersection ranges and the track ranges.
4. Replace the ranges in intersection ranges with the new intersection ranges.
5. If intersection ranges does not contain the exact same range information as the current value of this attribute, then update the current value of this attribute to intersection ranges.
6. Return the current value of this attribute.
mode of type AppendMode Controls how a sequence of media segments are handled. This attribute is initially set by addSourceBuffer() after the object is created.
On getting, Return the initial value or the last value that was successfully set.
On setting, run the following steps:
1. If this object has been removed from the sourceBuffers attribute of the parent media source, then throw an InvalidStateError exception and abort these steps.
2. If the updating attribute equals true, then throw an InvalidStateError exception and abort these steps.
3. Let new mode equal the new value being assigned to this attribute.
https://www.w3.org/TR/media-source/ 32/99 7/15/2016 Media Source Extensions
4. If generate timestamps flag equals true and new mode equals "segments", then throw a TypeError exception and abort these steps.
5. If the readyState attribute of the parent media source is in the "ended" state then run the following steps:
1. Set the readyState attribute of the parent media source to "open"
2. Queue a task to fire a simple event named sourceopen at the parent media source.
6. If the append state equals PARSING_MEDIA_SEGMENT, then throw an InvalidStateError and abort these steps.
7. If the new mode equals "sequence", then set the group start timestamp to the group end timestamp.
8. Update the attribute to new mode.
onabort of type EventHandler The event handler for the abort event.
onerror of type EventHandler The event handler for the error event.
onupdate of type EventHandler The event handler for the update event.
onupdateend of type EventHandler The event handler for the updateend event.
onupdatestart of type EventHandler The event handler for the updatestart event.
textTracks of type TextTrackList, readonly The list of TextTrack objects created by this object.
timestampOffset of type double Controls the offset applied to timestamps inside subsequent media segments that are appended to this SourceBuffer. The timestampOffset is initially set to 0 which indicates that no offset is being applied.
On getting, Return the initial value or the last value that was successfully set.
On setting, run the following steps:
1. Let new timestamp offset equal the new value being assigned to this attribute. https://www.w3.org/TR/media-source/ 33/99 7/15/2016 Media Source Extensions
2. If this object has been removed from the sourceBuffers attribute of the parent media source, then throw an InvalidStateError exception and abort these steps.
3. If the updating attribute equals true, then throw an InvalidStateError exception and abort these steps.
4. If the readyState attribute of the parent media source is in the "ended" state then run the following steps:
1. Set the readyState attribute of the parent media source to "open"
2. Queue a task to fire a simple event named sourceopen at the parent media source.
5. If the append state equals PARSING_MEDIA_SEGMENT, then throw an InvalidStateError and abort these steps.
6. If the mode attribute equals "sequence", then set the group start timestamp to new timestamp offset.
7. Update the attribute to new timestamp offset.
trackDefaults of type TrackDefaultList Specifies the default values to use if kind, label, and/or language information is not available in the initialization segment when the initialization segment received algorithm needs to create track objects. This attribute is initially set to an empty TrackDefaultList object.
On getting, Return the initial value or the last value that was successfully set.
On setting, run the following steps:
1. If this object has been removed from the sourceBuffers attribute of the parent media source, then throw an InvalidStateError exception and abort these steps.
2. If the updating attribute equals true, then throw an InvalidStateError exception and abort these steps.
3. Update the attribute to the new value.
updating of type boolean, readonly Indicates whether the asynchronous continuation of an appendBuffer(), appendStream(), or remove() operation is still being processed. This attribute is initially set to false when the object is created.
videoTracks of type VideoTrackList, readonly
https://www.w3.org/TR/media-source/ 34/99 7/15/2016 Media Source Extensions
The list of VideoTrack objects created by this object.
3.2 Methods
abort Aborts the current segment and resets the segment parser.
No parameters. Return type: void
When this method is invoked, the user agent must run the following steps:
1. If this object has been removed from the sourceBuffers attribute of the parent media source then throw an InvalidStateError exception and abort these steps.
2. If the readyState attribute of the parent media source is not in the "open" state then throw an InvalidStateError exception and abort these steps.
3. If the range removal algorithm is running, then throw an InvalidStateError exception and abort these steps.
4. If the updating attribute equals true, then run the following steps:
1. Abort the buffer append and stream append loop algorithms if they are running.
2. Set the updating attribute to false.
3. Queue a task to fire a simple event named abort at this SourceBuffer object.
4. Queue a task to fire a simple event named updateend at this SourceBuffer object.
5. Run the reset parser state algorithm.
6. Set appendWindowStart to the presentation start time.
7. Set appendWindowEnd to positive Infinity.
appendBuffer Appends the segment data in an ArrayBuffer[TYPED-ARRAYS] to the source buffer.
The steps for this method are the same as the ArrayBufferView version of appendBuffer().
Parameter Type Nullable Optional Description
https://www.w3.org/TR/media-source/ 35/99 7/15/2016 Media Source Extensions
data ArrayBuffer ✘ ✘
Return type: void
appendBuffer Appends the segment data in an ArrayBufferView[TYPED-ARRAYS] to the source buffer.
Parameter Type Nullable Optional Description
data ArrayBufferView ✘ ✘
Return type: void
When this method is invoked, the user agent must run the following steps:
1. Run the prepare append algorithm.
2. Add data to the end of the input buffer.
3. Set the updating attribute to true.
4. Queue a task to fire a simple event named updatestart at this SourceBuffer object.
5. Asynchronously run the buffer append algorithm.
appendStream Appends segment data to the source buffer from a ReadableStream[WHATWG- STREAMS-API].
Parameter Type Nullable Optional Description
stream ReadableStream ✘ ✘
maxSize unsigned long ✘ ✔ long
Return type: void
When this method is invoked, the user agent must run the following steps:
1. Run the prepare append algorithm.
2. Set the updating attribute to true.
3. Queue a task to fire a simple event named updatestart at this SourceBuffer object.
4. Asynchronously run the stream append loop algorithm with stream and maxSize.
https://www.w3.org/TR/media-source/ 36/99 7/15/2016 Media Source Extensions
remove Removes media for a specific time range.
Parameter Type Nullable Optional Description
start double ✘ ✘
end unrestricted ✘ ✘ double
Return type: void
When this method is invoked, the user agent must run the following steps:
1. If this object has been removed from the sourceBuffers attribute of the parent media source then throw an InvalidStateError exception and abort these steps.
2. If the updating attribute equals true, then throw an InvalidStateError exception and abort these steps.
3. If duration equals NaN, then throw a TypeError exception and abort these steps.
4. If start is negative or greater than duration, then throw a TypeError exception and abort these steps.
5. If end is less than or equal to start or end equals NaN, then throw a TypeError exception and abort these steps.
6. If the readyState attribute of the parent media source is in the "ended" state then run the following steps:
1. Set the readyState attribute of the parent media source to "open"
2. Queue a task to fire a simple event named sourceopen at the parent media source.
7. Run the range removal algorithm with start and end as the start and end of the removal range.
3.3 Track Buffers
A track buffer stores the track descriptions and coded frames for an individual track. The track buffer is updated as initialization segments and media segments are appended to the SourceBuffer.
https://www.w3.org/TR/media-source/ 37/99 7/15/2016 Media Source Extensions
Each track buffer has a last decode timestamp variable that stores the decode timestamp of the last coded frame appended in the current coded frame group. The variable is initially unset to indicate that no coded frames have been appended yet.
Each track buffer has a last frame duration variable that stores the coded frame duration of the last coded frame appended in the current coded frame group. The variable is initially unset to indicate that no coded frames have been appended yet.
Each track buffer has a highest end timestamp variable that stores the highest coded frame end timestamp across all coded frames in the current coded frame group that were appended to this track buffer. The variable is initially unset to indicate that no coded frames have been appended yet.
Each track buffer has a need random access point flag variable that keeps track of whether the track buffer is waiting for a random access point coded frame. The variable is initially set to true to indicate that random access point coded frame is needed before anything can be added to the track buffer.
Each track buffer has a track buffer ranges variable that represents the presentation time ranges occupied by the coded frames currently stored in the track buffer.
NOTE
For track buffer ranges, these presentation time ranges are based on presentation timestamps, frame durations, and potentially coded frame group start times for coded frame groups across track buffers in a muxed SourceBuffer.
For specification purposes, this information is treated as if it were stored in a normalized TimeRanges object. Intersected track buffer ranges are used to report HTMLMediaElement.buffered, and must therefore support uninterrupted playback within each range of HTMLMediaElement.buffered.
https://www.w3.org/TR/media-source/ 38/99 7/15/2016 Media Source Extensions
NOTE
These coded frame group start times differ slightly from those mentioned in the coded frame processing algorithm in that they are the earliest presentation timestamp across all track buffers following a discontinuity. Discontinuities can occur within the coded frame processing algorithm or result from the coded frame removal algorithm, regardless of mode. The threshold for determining disjointness of track buffer ranges is implementation-specific. For example, to reduce unexpected playback stalls, implementations may approximate the coded frame processing algorithm's discontinuity detection logic by coalescing adjacent ranges separated by a gap smaller than 2 times the maximum frame duration buffered so far in this track buffer. Implementations may also use coded frame group start times as range start times across track buffers in a muxed SourceBuffer to further reduce unexpected playback stalls.
3.4 Event Summary
Event name Interface Dispatched when...
updatestart Event updating transitions from false to true.
update Event The append or remove has successfully completed. updating transitions from true to false.
updateend Event The append or remove has ended.
error Event An error occurred during the append. updating transitions from true to false.
abort Event The append or remove was aborted by an abort() call. updating transitions from true to false.
3.5 Algorithms
3.5.1 Segment Parser Loop
All SourceBuffer objects have an internal append state variable that keeps track of the high-level segment parsing state. It is initially set to WAITING_FOR_SEGMENT and can transition to the following states as data is appended.
https://www.w3.org/TR/media-source/ 39/99 7/15/2016 Media Source Extensions
Append state name Description
WAITING_FOR_SEGMENT Waiting for the start of an initialization segment or media segment to be appended. PARSING_INIT_SEGMENT Currently parsing an initialization segment. PARSING_MEDIA_SEGMENT Currently parsing a media segment.
The input buffer is a byte buffer that is used to hold unparsed bytes across appendBuffer() and appendStream() calls. The buffer is empty when the SourceBuffer object is created.
The buffer full flag keeps track of whether appendBuffer() or appendStream() is allowed to accept more bytes. It is set to false when the SourceBuffer object is created and gets updated as data is appended and removed.
The group start timestamp variable keeps track of the starting timestamp for a new coded frame group in the "sequence" mode. It is unset when the SourceBuffer object is created and gets updated when the mode attribute equals "sequence" and the timestampOffset attribute is set, or the coded frame processing algorithm runs.
The group end timestamp variable stores the highest coded frame end timestamp across all coded frames in the current coded frame group. It is set to 0 when the SourceBuffer object is created and gets updated by the coded frame processing algorithm.
NOTE
The group end timestamp stores the highest coded frame end timestamp across all track buffers in a SourceBuffer. Therefore, care should be taken in setting the mode attribute when appending multiplexed segments in which the timestamps are not aligned across tracks.
The generate timestamps flag is a boolean variable that keeps track of whether timestamps need to be generated for the coded frames passed to the coded frame processing algorithm. This flag is set by addSourceBuffer() when the SourceBuffer object is created.
When the segment parser loop algorithm is invoked, run the following steps:
1. Loop Top: If the input buffer is empty, then jump to the need more data step below.
2. If the input buffer contains bytes that violate the SourceBuffer byte stream format specification, then run the append error algorithm with the decode error parameter set to true and abort this algorithm.
https://www.w3.org/TR/media-source/ 40/99 7/15/2016 Media Source Extensions
3. Remove any bytes that the byte stream format specifications say must be ignored from the start of the input buffer.
4. If the append state equals WAITING_FOR_SEGMENT, then run the following steps:
1. If the beginning of the input buffer indicates the start of an initialization segment, set the append state to PARSING_INIT_SEGMENT.
2. If the beginning of the input buffer indicates the start of a media segment, set append state to PARSING_MEDIA_SEGMENT.
3. Jump to the loop top step above.
5. If the append state equals PARSING_INIT_SEGMENT, then run the following steps:
1. If the input buffer does not contain a complete initialization segment yet, then jump to the need more data step below.
2. Run the initialization segment received algorithm.
3. Remove the initialization segment bytes from the beginning of the input buffer.
4. Set append state to WAITING_FOR_SEGMENT.
5. Jump to the loop top step above.
6. If the append state equals PARSING_MEDIA_SEGMENT, then run the following steps:
1. If the first initialization segment received flag is false, then run the append error algorithm with the decode error parameter set to true and abort this algorithm.
2. If the input buffer contains one or more complete coded frames, then run the coded frame processing algorithm.
NOTE
The frequency at which the coded frame processing algorithm is run is implementation-specific. The coded frame processing algorithm may be called when the input buffer contains the complete media segment or it may be called multiple times as complete coded frames are added to the input buffer.
https://www.w3.org/TR/media-source/ 41/99 7/15/2016 Media Source Extensions
3. If this SourceBuffer is full and cannot accept more media data, then set the buffer full flag to true.
4. If the input buffer does not contain a complete media segment, then jump to the need more data step below.
5. Remove the media segment bytes from the beginning of the input buffer.
6. Set append state to WAITING_FOR_SEGMENT.
7. Jump to the loop top step above.
7. Need more data: Return control to the calling algorithm.
3.5.2 Reset Parser State
When the parser state needs to be reset, run the following steps:
1. If the append state equals PARSING_MEDIA_SEGMENT and the input buffer contains some complete coded frames, then run the coded frame processing algorithm until all of these complete coded frames have been processed.
2. Unset the last decode timestamp on all track buffers.
3. Unset the last frame duration on all track buffers.
4. Unset the highest end timestamp on all track buffers.
5. Set the need random access point flag on all track buffers to true.
6. If the mode attribute equals "sequence", then set the group start timestamp to the group end timestamp
7. Remove all bytes from the input buffer.
8. Set append state to WAITING_FOR_SEGMENT.
3.5.3 Append Error Algorithm
This algorithm is called when an error occurs during an append. This algorithm takes a decode error parameter that indicates whether endOfStream() should be called.
https://www.w3.org/TR/media-source/ 42/99 7/15/2016 Media Source Extensions
1. Run the reset parser state algorithm.
2. Set the updating attribute to false.
3. Queue a task to fire a simple event named error at this SourceBuffer object.
4. Queue a task to fire a simple event named updateend at this SourceBuffer object.
5. If decode error is true, then run the end of stream algorithm with the error parameter set to "decode".
3.5.4 Prepare Append Algorithm
When an append operation begins, the follow steps are run to validate and prepare the SourceBuffer.
1. If the SourceBuffer has been removed from the sourceBuffers attribute of the parent media source then throw an InvalidStateError exception and abort these steps.
2. If the updating attribute equals true, then throw an InvalidStateError exception and abort these steps.
3. If the HTMLMediaElement.error attribute is not null, then throw an InvalidStateError exception and abort these steps.
4. If the readyState attribute of the parent media source is in the "ended" state then run the following steps:
1. Set the readyState attribute of the parent media source to "open"
2. Queue a task to fire a simple event named sourceopen at the parent media source.
5. Run the coded frame eviction algorithm.
6. If the buffer full flag equals true, then throw a QuotaExceededError exception and abort these step.
NOTE
This is the signal that the implementation was unable to evict enough data to accomodate the append or the append is too big. The web application should use remove() to explicitly free up space and/or reduce the size of the append.
https://www.w3.org/TR/media-source/ 43/99 7/15/2016 Media Source Extensions
3.5.5 Buffer Append Algorithm
When appendBuffer() is called, the following steps are run to process the appended data.
1. Run the segment parser loop algorithm.
2. If the segment parser loop algorithm in the previous step was aborted, then abort this algorithm.
3. Set the updating attribute to false.
4. Queue a task to fire a simple event named update at this SourceBuffer object.
5. Queue a task to fire a simple event named updateend at this SourceBuffer object.
3.5.6 Stream Append Loop
This algorithm is invoked by appendStream()to transfer data from a ReadableStream to the SourceBuffer. This algorithm is initialized with the stream and maxSize parameters from the appendStream() call.
1. If maxSize is set, then let bytesLeft equal maxSize.
2. Loop Top: If maxSize is set and bytesLeft equals 0, then jump to the loop done step below.
3. If stream.state equals "waiting", then run the following steps: 1. Wait for either the stream.ready promise or the stream.closed promise to be resolved or rejected.
NOTE
This is intended to be similar to Promise.race([stream.ready, stream.closed], continueAppendLoop, continueAppendLoop), where the continueAppendLoop function returns control to this algorithm and starts executing the step that follows this one. If the stream append loop algorithm is aborted while waiting for the promises, continueAppendLoop must not continue the algorithm when it is eventually called.
https://www.w3.org/TR/media-source/ 44/99 7/15/2016 Media Source Extensions
NOTE
This step is only represented as a blocking wait to make it easier to describe the intended behavior. It should not block other JavaScript from executing.
2. Continue onto the next step.
4. If stream.state equals "closed", then jump to the loop done step below.
5. If stream.state equals "errored", then run the append error algorithm with the decode error parameter set to false and abort this algorithm.
6. If stream.state does not equal "readable", then run the append error algorithm with the decode error parameter set to false and abort this algorithm.
7. Let data be the value returned by a call to stream.read().
8. If data is not an ArrayBuffer or an ArrayBufferView then, run the append error algorithm with the decode error parameter set to false and abort this algorithm.
9. If maxSize is set, then run the following steps: 1. If data.byteLength is greater than bytesLeft, then run the following steps: 1. Let new data equal the value returned by calling data.slice(0, bytesLeft).
NOTE
This step and the next one are assuming data is an ArrayBuffer. If data is an ArrayBufferView subarray() should be called instead.
2. Let remaining data equal the value returned by calling data.slice(bytesLeft). 3. Push remaining data back onto the front of stream so it will be returned by the next stream.read() call. 4. Assign new data to data. 2. Subtract data.byteLength from bytesLeft.
10. Run the coded frame eviction algorithm.
https://www.w3.org/TR/media-source/ 45/99 7/15/2016 Media Source Extensions
11. If the buffer full flag equals true, then run the append error algorithm with the decode error parameter set to false and abort this algorithm.
NOTE
The web application should use remove() to free up space in the SourceBuffer.
12. Add data to the end of the input buffer.
13. Run the segment parser loop algorithm.
14. If the segment parser loop algorithm in the previous step was aborted, then abort this algorithm.
15. Jump to the loop top step above.
16. Loop Done: Set the updating attribute to false.
17. Queue a task to fire a simple event named update at this SourceBuffer object.
18. Queue a task to fire a simple event named updateend at this SourceBuffer object.
3.5.7 Range Removal
Follow these steps when a caller needs to initiate a JavaScript visible range removal operation that blocks other SourceBuffer updates:
1. Let start equal the starting presentation timestamp for the removal range.
2. Let end equal the end presentation timestamp for the removal range.
3. Set the updating attribute to true.
4. Queue a task to fire a simple event named updatestart at this SourceBuffer object.
5. Return control to the caller and run the rest of the steps asynchronously.
6. Run the coded frame removal algorithm with start and end as the start and end of the removal range.
7. Set the updating attribute to false.
8. Queue a task to fire a simple event named update at this SourceBuffer object.
https://www.w3.org/TR/media-source/ 46/99 7/15/2016 Media Source Extensions
9. Queue a task to fire a simple event named updateend at this SourceBuffer object.
3.5.8 Initialization Segment Received
The following steps are run when the segment parser loop successfully parses a complete initialization segment:
Each SourceBuffer object has an internal first initialization segment received flag that tracks whether the first initialization segment has been appended and received by this algorithm. This flag is set to false when the SourceBuffer is created and updated by the algorithm below.
1. Update the duration attribute if it currently equals NaN:
If the initialization segment contains a duration: Run the duration change algorithm with new duration set to the duration in the ↪ initialization segment.
Otherwise: Run the duration change algorithm with new duration set to positive Infinity. ↪ 2. If the initialization segment has no audio, video, or text tracks, then run the append error algorithm with the decode error parameter set to true and abort these steps.
3. If the first initialization segment received flag is true, then run the following steps: 1. Verify the following properties. If any of the checks fail then run the append error algorithm with the decode error parameter set to true and abort these steps. The number of audio, video, and text tracks match what was in the first initialization segment. The codecs for each track, match what was specified in the first initialization segment. If more than one track for a single type are present (ie 2 audio tracks), then the Track IDs match the ones in the first initialization segment. 2. Add the appropriate track descriptions from this initialization segment to each of the track buffers. 3. Set the need random access point flag on all track buffers to true.
4. Let active track flag equal false.
5. If the first initialization segment received flag is false, then run the following steps:
https://www.w3.org/TR/media-source/ 47/99 7/15/2016 Media Source Extensions
1. If the initialization segment contains tracks with codecs the user agent does not support, then run the append error algorithm with the decode error parameter set to true and abort these steps.
NOTE
User agents may consider codecs, that would otherwise be supported, as "not supported" here if the codecs were not specified in the type parameter passed to addSourceBuffer(). For example, MediaSource.isTypeSupported('video/webm;codecs="vp8,vorbis"') may return true, but if addSourceBuffer() was called with 'video/webm;codecs="vp8"' and a Vorbis track appears in the initialization segment, then the user agent may use this step to trigger a decode error.
2. For each audio track in the initialization segment, run following steps:
1. Let audio byte stream track ID be the Track ID for the current track being processed.
2. Let audio language be a BCP 47 language tag for the language specified in the initialization segment for this track or an empty string if no language info is present.
3. If audio language equals an empty string or the 'und' BCP 47 value, then run the default track language algorithm with byteStreamTrackID set to audio byte stream track ID and type set to "audio" and assign the value returned by the algorithm to audio language.
4. Let audio label be a label specified in the initialization segment for this track or an empty string if no label info is present.
5. If audio label equals an empty string, then run the default track label algorithm with byteStreamTrackID set to audio byte stream track ID and type set to "audio" and assign the value returned by the algorithm to audio label.
6. Let audio kinds be a sequence of kind strings specified in the initialization segment for this track or an empty sequence if no kind information is provided.
7. If audio kinds equals an empty sequence, then run the default track kinds algorithm with byteStreamTrackID set to audio byte stream track ID and type set to "audio" and assign the value returned by the algorithm to audio kinds.
8. For each value in audio kinds, run the following steps:
https://www.w3.org/TR/media-source/ 48/99 7/15/2016 Media Source Extensions
1. Let current audio kind equal the value from audio kinds for this iteration of the loop.
2. Let new audio track be a new AudioTrack object.
3. Generate a unique ID and assign it to the id property on new audio track.
4. Assign audio language to the language property on new audio track.
5. Assign audio label to the label property on new audio track.
6. Assign current audio kind to the kind property on new audio track.
7. If audioTracks.length equals 0, then run the following steps:
1. Set the enabled property on new audio track to true.
2. Set active track flag to true.
8. Add new audio track to the audioTracks attribute on this SourceBuffer object.
9. Queue a task to fire a trusted event named addtrack, that does not bubble and is not cancelable, and that uses the TrackEvent interface, at the AudioTrackList object referenced by the audioTracks attribute on this SourceBuffer object.
10. Add new audio track to the audioTracks attribute on the HTMLMediaElement.
11. Queue a task to fire a trusted event named addtrack, that does not bubble and is not cancelable, and that uses the TrackEvent interface, at the AudioTrackList object referenced by the audioTracks attribute on the HTMLMediaElement.
9. Create a new track buffer to store coded frames for this track.
10. Add the track description for this track to the track buffer.
3. For each video track in the initialization segment, run following steps:
1. Let video byte stream track ID be the Track ID for the current track being processed.
2. Let video language be a BCP 47 language tag for the language specified in the initialization segment for this track or an empty string if no language info is present.
3. If video language equals an empty string or the 'und' BCP 47 value, then run the default track language algorithm with byteStreamTrackID set to video byte stream track ID and type set to "video" and assign the value returned by the algorithm to video language.
4. Let video label be a label specified in the initialization segment for this track or an empty string if no label info is present.
https://www.w3.org/TR/media-source/ 49/99 7/15/2016 Media Source Extensions
5. If video label equals an empty string, then run the default track label algorithm with byteStreamTrackID set to video byte stream track ID and type set to "video" and assign the value returned by the algorithm to video label.
6. Let video kinds be a sequence of kind strings specified in the initialization segment for this track or an empty sequence if no kind information is provided.
7. If video kinds equals an empty sequence, then run the default track kinds algorithm with byteStreamTrackID set to video byte stream track ID and type set to "video" and assign the value returned by the algorithm to video kinds.
8. For each value in video kinds, run the following steps: 1. Let current video kind equal the value from video kinds for this iteration of the loop.
2. Let new video track be a new VideoTrack object.
3. Generate a unique ID and assign it to the id property on new video track.
4. Assign video language to the language property on new video track.
5. Assign video label to the label property on new video track.
6. Assign current video kind to the kind property on new video track.
7. If videoTracks.length equals 0, then run the following steps:
1. Set the selected property on new video track to true.
2. Set active track flag to true.
8. Add new video track to the videoTracks attribute on this SourceBuffer object.
9. Queue a task to fire a trusted event named addtrack, that does not bubble and is not cancelable, and that uses the TrackEvent interface, at the VideoTrackList object referenced by the videoTracks attribute on this SourceBuffer object.
10. Add new video track to the videoTracks attribute on the HTMLMediaElement.
11. Queue a task to fire a trusted event named addtrack, that does not bubble and is not cancelable, and that uses the TrackEvent interface, at the VideoTrackList object referenced by the videoTracks attribute on the HTMLMediaElement.
9. Create a new track buffer to store coded frames for this track.
10. Add the track description for this track to the track buffer.
4. For each text track in the initialization segment, run following steps: https://www.w3.org/TR/media-source/ 50/99 7/15/2016 Media Source Extensions
1. Let text byte stream track ID be the Track ID for the current track being processed.
2. Let text language be a BCP 47 language tag for the language specified in the initialization segment for this track or an empty string if no language info is present.
3. If text language equals an empty string or the 'und' BCP 47 value, then run the default track language algorithm with byteStreamTrackID set to text byte stream track ID and type set to "text" and assign the value returned by the algorithm to text language.
4. Let text label be a label specified in the initialization segment for this track or an empty string if no label info is present.
5. If text label equals an empty string, then run the default track label algorithm with byteStreamTrackID set to text byte stream track ID and type set to "text" and assign the value returned by the algorithm to text label.
6. Let text kinds be a sequence of kind strings specified in the initialization segment for this track or an empty sequence if no kind information is provided.
7. If text kinds equals an empty sequence, then run the default track kinds algorithm with byteStreamTrackID set to text byte stream track ID and type set to "text" and assign the value returned by the algorithm to text kinds.
8. For each value in text kinds, run the following steps: 1. Let current text kind equal the value from text kinds for this iteration of the loop.
2. Let new text track be a new TextTrack object.
3. Generate a unique ID and assign it to the id property on new text track.
4. Assign text language to the language property on new text track.
5. Assign text label to the label property on new text track.
6. Assign current text kind to the kind property on new text track. 7. Populate the remaining properties on new text track with the appropriate information from the initialization segment.
8. If the mode property on new text track equals "showing" or "hidden", then set active track flag to true.
9. Add new text track to the textTracks attribute on this SourceBuffer object.
10. Queue a task to fire a trusted event named addtrack, that does not bubble and is not cancelable, and that uses the TrackEvent interface, at textTracks attribute on this SourceBuffer object.
https://www.w3.org/TR/media-source/ 51/99 7/15/2016 Media Source Extensions
11. Add new text track to the textTracks attribute on the HTMLMediaElement.
12. Queue a task to fire a trusted event named addtrack, that does not bubble and is not cancelable, and that uses the TrackEvent interface, at the TextTrackList object referenced by the textTracks attribute on the HTMLMediaElement.
9. Create a new track buffer to store coded frames for this track.
10. Add the track description for this track to the track buffer.
5. If active track flag equals true, then run the following steps: 1. Add this SourceBuffer to activeSourceBuffers.
2. Queue a task to fire a simple event named addsourcebuffer at activeSourceBuffers
6. Set first initialization segment received flag to true.
6. If the HTMLMediaElement.readyState attribute is HAVE_NOTHING, then run the following steps:
1. If one or more objects in sourceBuffers have first initialization segment received flag set to false, then abort these steps.
2. Set the HTMLMediaElement.readyState attribute to HAVE_METADATA.
3. Queue a task to fire a simple event named loadedmetadata at the media element.
7. If the active track flag equals true and the HTMLMediaElement.readyState attribute is greater than HAVE_CURRENT_DATA, then set the HTMLMediaElement.readyState attribute to HAVE_METADATA.
3.5.9 Default track language
The following steps are run when the initialization segment received algorithm needs to determine what the default language for a new track should be. This algorithm is initialized with byteStreamTrackID and type parameters when invoked by the initialization segment received algorithm.
1. If trackDefaults contains a TrackDefault object with a type attribute equal to type and a byteStreamTrackID attribute equal to byteStreamTrackID, then return the value of the language attribute on this matching object and abort these steps.
2. If trackDefaults contains a TrackDefault object with a type attribute equal to type and a byteStreamTrackID attribute equal to an empty string, then return the value of the language
https://www.w3.org/TR/media-source/ 52/99 7/15/2016 Media Source Extensions
attribute on this matching object and abort these steps.
3. Return an empty string to the caller.
3.5.10 Default track label
The following steps are run when the initialization segment received algorithm needs to determine what the default label for a new track should be. This algorithm is initialized with byteStreamTrackID and type parameters when invoked by the initialization segment received algorithm.
1. If trackDefaults contains a TrackDefault object with a type attribute equal to type and a byteStreamTrackID attribute equal to byteStreamTrackID, then return the value of the label attribute on this matching object and abort these steps.
2. If trackDefaults contains a TrackDefault object with a type attribute equal to type and a byteStreamTrackID attribute equal to an empty string, then return the value of the label attribute on this matching object and abort these steps.
3. Return an empty string to the caller.
3.5.11 Default track kinds
The following steps are run when the initialization segment received algorithm needs to determine what the default kinds for a new track should be. This algorithm is initialized with byteStreamTrackID and type parameters when invoked by the initialization segment received algorithm.
1. If trackDefaults contains a TrackDefault object with a type attribute equal to type and a byteStreamTrackID attribute equal to byteStreamTrackID, then return the sequence returned by the getKinds method on this matching object and abort these steps.
2. If trackDefaults contains a TrackDefault object with a type attribute equal to type and a byteStreamTrackID attribute equal to an empty string, then return the sequence returned by the getKinds method on this matching object and abort these steps.
3. Return a sequence with a single empty string element in it to the caller.
3.5.12 Coded Frame Processing
https://www.w3.org/TR/media-source/ 53/99 7/15/2016 Media Source Extensions
When complete coded frames have been parsed by the segment parser loop then the following steps are run:
1. For each coded frame in the media segment run the following steps:
1. Loop Top:
If generate timestamps flag equals true:
↪ 1. Let presentation timestamp equal 0.
2. Let decode timestamp equal 0.
Otherwise:
↪ 1. Let presentation timestamp be a double precision floating point representation of the coded frame's presentation timestamp in seconds.
NOTE
Special processing may be needed to determine the presentation and decode timestamps for timed text frames since this information may not be explicitly present in the underlying format or may be dependent on the order of the frames. Some metadata text tracks, like MPEG2- TS PSI data, may only have implied timestamps. Format specific rules for these situations should be in the byte stream format specifications or in separate extension specifications.
2. Let decode timestamp be a double precision floating point representation of the coded frame's decode timestamp in seconds.
https://www.w3.org/TR/media-source/ 54/99 7/15/2016 Media Source Extensions
NOTE
Implementations don't have to internally store timestamps in a double precision floating point representation. This representation is used here because it is the represention for timestamps in the HTML spec. The intention here is to make the behavior clear without adding unnecessary complexity to the algorithm to deal with the fact that adding a timestampOffset may cause a timestamp rollover in the underlying timestamp representation used by the byte stream format. Implementations can use any internal timestamp representation they wish, but the addition of timestampOffset should behave in a similar manner to what would happen if a double precision floating point representation was used.
2. Let frame duration be a double precision floating point representation of the coded frame's duration in seconds.
3. If mode equals "sequence" and group start timestamp is set, then run the following steps: 1. Set timestampOffset equal to group start timestamp - presentation timestamp. 2. Set group end timestamp equal to group start timestamp. 3. Set the need random access point flag on all track buffers to true. 4. Unset group start timestamp.
4. If timestampOffset is not 0, then run the following steps:
1. Add timestampOffset to the presentation timestamp.
2. Add timestampOffset to the decode timestamp.
5. Let track buffer equal the track buffer that the coded frame will be added to.
6. If last decode timestamp for track buffer is set and decode timestamp is less than last decode timestamp: ↪ OR
If last decode timestamp for track buffer is set and the difference between decode timestamp and last decode timestamp is greater than 2 times last frame duration: ↪ 1. If mode equals "segments": Set group end timestamp to presentation timestamp. ↪ https://www.w3.org/TR/media-source/ 55/99 7/15/2016 Media Source Extensions
If mode equals "sequence": Set group start timestamp equal to the group end timestamp. ↪ 2. Unset the last decode timestamp on all track buffers.
3. Unset the last frame duration on all track buffers.
4. Unset the highest end timestamp on all track buffers.
5. Set the need random access point flag on all track buffers to true.
6. Jump to the Loop Top step above to restart processing of the current coded frame.
Otherwise: Continue. ↪ 7. Let frame end timestamp equal the sum of presentation timestamp and frame duration.
8. If presentation timestamp is less than appendWindowStart, then set the need random access point flag to true, drop the coded frame, and jump to the top of the loop to start processing the next coded frame.
NOTE
Some implementations may choose to collect some of these coded frames that are outside the append window and use them to generate a splice at the first coded frame that has a presentation timestamp greater than or equal to appendWindowStart even if that frame is not a random access point. Supporting this requires multiple decoders or faster than real-time decoding so for now this behavior will not be a normative requirement.
9. If frame end timestamp is greater than appendWindowEnd, then set the need random access point flag to true, drop the coded frame, and jump to the top of the loop to start processing the next coded frame.
10. If the need random access point flag on track buffer equals true, then run the following steps: 1. If the coded frame is not a random access point, then drop the coded frame and jump to the top of the loop to start processing the next coded frame. 2. Set the need random access point flag on track buffer to false.
https://www.w3.org/TR/media-source/ 56/99 7/15/2016 Media Source Extensions
11. Let spliced audio frame be an unset variable for holding audio splice information
12. Let spliced timed text frame be an unset variable for holding timed text splice information
13. If last decode timestamp for track buffer is unset and presentation timestamp falls within the presentation interval of a coded frame in track buffer, then run the following steps: 1. Let overlapped frame be the coded frame in track buffer that matches the condition above.
2. If track buffer contains audio coded frames: Run the audio splice frame algorithm and if a splice frame is returned, assign ↪ it to spliced audio frame.
If track buffer contains video coded frames:
↪ 1. Let remove window timestamp equal the overlapped frame presentation timestamp plus 1 microsecond.
2. If the presentation timestamp is less than the remove window timestamp, then remove overlapped frame from track buffer.
NOTE
This is to compensate for minor errors in frame timestamp computations that can appear when converting back and forth between double precision floating point numbers and rationals. This tolerance allows a frame to replace an existing one as long as it is within 1 microsecond of the existing frame's start time. Frames that come slightly before an existing frame are handled by the removal step below.
If track buffer contains timed text coded frames: Run the text splice frame algorithm and if a splice frame is returned, assign it ↪ to spliced timed text frame.
14. Remove existing coded frames in track buffer:
If highest end timestamp for track buffer is not set: Remove all coded frames from track buffer that have a presentation timestamp ↪ greater than or equal to presentation timestamp and less than frame end timestamp.
https://www.w3.org/TR/media-source/ 57/99 7/15/2016 Media Source Extensions
If highest end timestamp for track buffer is set and less than or equal to presentation timestamp: ↪ Remove all coded frames from track buffer that have a presentation timestamp greater than or equal to highest end timestamp and less than frame end timestamp
15. Remove all possible decoding dependencies on the coded frames removed in the previous two steps by removing all coded frames from track buffer between those frames removed in the previous two steps and the next random access point after those removed frames.
NOTE
Removing all coded frames until the next random access point is a conservative estimate of the decoding dependencies since it assumes all frames between the removed frames and the next random access point depended on the frames that were removed.
16. If spliced audio frame is set: Add spliced audio frame to the track buffer. ↪ If spliced timed text frame is set: Add spliced timed text frame to the track buffer. ↪ Otherwise: Add the coded frame with the presentation timestamp, decode timestamp, and ↪ frame duration to the track buffer.
17. Set last decode timestamp for track buffer to decode timestamp.
18. Set last frame duration for track buffer to frame duration.
19. If highest end timestamp for track buffer is unset or frame end timestamp is greater than highest end timestamp, then set highest end timestamp for track buffer to frame end timestamp.
NOTE
The greater than check is needed because bidirectional prediction between coded frames can cause presentation timestamp to not be monotonically increasing even though the decode timestamps are monotonically increasing.
https://www.w3.org/TR/media-source/ 58/99 7/15/2016 Media Source Extensions
20. If frame end timestamp is greater than group end timestamp, then set group end timestamp equal to frame end timestamp.
21. If generate timestamps flag equals true, then set timestampOffset equal to frame end timestamp.
2. If the HTMLMediaElement.readyState attribute is HAVE_METADATA and the new coded frames cause HTMLMediaElement.buffered to have a TimeRange for the current playback position, then run the following steps:
1. Set the HTMLMediaElement.readyState attribute to HAVE_CURRENT_DATA.
2. If this is the first transition to HAVE_CURRENT_DATA, then queue a task to fire a simple event named loadeddata at the media element.
3. If the HTMLMediaElement.readyState attribute is HAVE_CURRENT_DATA and the new coded frames cause HTMLMediaElement.buffered to have a TimeRange that includes the current playback position and some time beyond the current playback position, then run the following steps:
1. Set the HTMLMediaElement.readyState attribute to HAVE_FUTURE_DATA.
2. Queue a task to fire a simple event named canplay at the media element.
4. If the HTMLMediaElement.readyState attribute is HAVE_FUTURE_DATA and the new coded frames cause HTMLMediaElement.buffered to have a TimeRange that includes the current playback position and enough data to ensure uninterrupted playback, then run the following steps:
1. Set the HTMLMediaElement.readyState attribute to HAVE_ENOUGH_DATA.
2. Queue a task to fire a simple event named canplaythrough at the media element.
5. If the media segment contains data beyond the current duration, then run the duration change algorithm with new duration set to the maximum of the current duration and the group end timestamp.
3.5.13 Coded Frame Removal Algorithm
Follow these steps when coded frames for a specific time range need to be removed from the SourceBuffer:
1. Let start be the starting presentation timestamp for the removal range.
https://www.w3.org/TR/media-source/ 59/99 7/15/2016 Media Source Extensions
2. Let end be the end presentation timestamp for the removal range.
3. For each track buffer in this source buffer, run the following steps:
1. Let remove end timestamp be the current value of duration
2. If this track buffer has a random access point timestamp that is greater than or equal to end, then update remove end timestamp to that random access point timestamp.
NOTE
Random access point timestamps can be different across tracks because the dependencies between coded frames within a track are usually different than the dependencies in another track.
3. Remove all media data, from this track buffer, that contain starting timestamps greater than or equal to start and less than the remove end timestamp. 1. For each removed frame, if the frame has a decode timestamp equal to the last decode timestamp for the frame's track, run the following steps:
If mode equals "segments": Set group end timestamp to presentation timestamp. ↪ If mode equals "sequence": Set group start timestamp equal to the group end timestamp. ↪ 2. Unset the last decode timestamp on all track buffers. 3. Unset the last frame duration on all track buffers. 4. Unset the highest end timestamp on all track buffers. 5. Set the need random access point flag on all track buffers to true.
4. Remove all possible decoding dependencies on the coded frames removed in the previous step by removing all coded frames from this track buffer between those frames removed in the previous step and the next random access point after those removed frames.
https://www.w3.org/TR/media-source/ 60/99 7/15/2016 Media Source Extensions
NOTE
Removing all coded frames until the next random access point is a conservative estimate of the decoding dependencies since it assumes all frames between the removed frames and the next random access point depended on the frames that were removed.
5. If this object is in activeSourceBuffers, the current playback position is greater than or equal to start and less than the remove end timestamp, and HTMLMediaElement.readyState is greater than HAVE_METADATA, then set the HTMLMediaElement.readyState attribute to HAVE_METADATA and stall playback.
NOTE
This transition occurs because media data for the current position has been removed. Playback cannot progress until media for the current playback position is appended or the selected/enabled tracks change.
4. If buffer full flag equals true and this object is ready to accept more bytes, then set the buffer full flag to false.
3.5.14 Coded Frame Eviction Algorithm
This algorithm is run to free up space in this source buffer when new data is appended.
1. Let new data equal the data that is about to be appended to this SourceBuffer.
2. If the buffer full flag equals false, then abort these steps.
3. Let removal ranges equal a list of presentation time ranges that can be evicted from the presentation to make room for the new data.
https://www.w3.org/TR/media-source/ 61/99 7/15/2016 Media Source Extensions
NOTE
Implementations may use different methods for selecting removal ranges so web applications should not depend on a specific behavior. The web application can use the buffered attribute to observe whether portions of the buffered data have been evicted.
4. For each range in removal ranges, run the coded frame removal algorithm with start and end equal to the removal range start and end timestamp respectively.
3.5.15 Audio Splice Frame Algorithm
Follow these steps when the coded frame processing algorithm needs to generate a splice frame for two overlapping audio coded frames:
1. Let track buffer be the track buffer that will contain the splice.
2. Let new coded frame be the new coded frame, that is being added to track buffer, which triggered the need for a splice.
3. Let presentation timestamp be the presentation timestamp for new coded frame
4. Let decode timestamp be the decode timestamp for new coded frame.
5. Let frame duration be the coded frame duration of new coded frame.
6. Let overlapped frame be the coded frame in track buffer with a presentation interval that contains presentation timestamp.
7. Update presentation timestamp and decode timestamp to the nearest audio sample timestamp based on sample rate of the audio in overlapped frame. If a timestamp is equidistant from both audio sample timestamps, then use the higher timestamp. (eg. floor(x * sample_rate + 0.5) / sample_rate).
https://www.w3.org/TR/media-source/ 62/99 7/15/2016 Media Source Extensions
NOTE
For example, given the following values:
The presentation timestamp of overlapped frame equals 10.
The sample rate of overlapped frame equals 8000 Hz
presentation timestamp equals 10.01255
decode timestamp equals 10.01255
presentation timestamp and decode timestamp are updated to 10.0125 since 10.01255 is closer to 10 + 100/8000 (10.0125) than 10 + 101/8000 (10.012625)
8. If the user agent does not support crossfading then run the following steps: 1. Remove overlapped frame from track buffer. 2. Add a silence frame to track buffer with the following properties: The presentation timestamp set to the overlapped frame presentation timestamp. The decode timestamp set to the overlapped frame decode timestamp. The coded frame duration set to difference between presentation timestamp and the overlapped frame presentation timestamp.
NOTE
Some implementations may apply fades to/from silence to coded frames on either side of the inserted silence to make the transition less jarring.
3. Return to caller without providing a splice frame.
NOTE
This is intended to allow new coded frame to be added to the track buffer as if overlapped frame had not been in the track buffer to begin with.
9. Let frame end timestamp equal the sum of presentation timestamp and frame duration.
https://www.w3.org/TR/media-source/ 63/99 7/15/2016 Media Source Extensions
10. Let splice end timestamp equal the sum of presentation timestamp and the splice duration of 5 milliseconds.
11. Let fade out coded frames equal overlapped frame as well as any additional frames in track buffer that have a presentation timestamp greater than presentation timestamp and less than splice end timestamp.
12. Remove all the frames included in fade out coded frames from track buffer.
13. Return a splice frame with the following properties: The presentation timestamp set to the overlapped frame presentation timestamp. The decode timestamp set to the overlapped frame decode timestamp. The coded frame duration set to difference between frame end timestamp and the overlapped frame presentation timestamp. The fade out coded frames equals fade-out coded frames. The fade in coded frame equal new coded frame.
NOTE
If the new coded frame is less than 5 milliseconds in duration, then coded frames that are appended after the new coded frame will be needed to properly render the splice.
The splice timestamp equals presentation timestamp.
NOTE
See the audio splice rendering algorithm for details on how this splice frame is rendered.
3.5.16 Audio Splice Rendering Algorithm
The following steps are run when a spliced frame, generated by the audio splice frame algorithm, needs to be rendered by the media element:
1. Let fade out coded frames be the coded frames that are faded out during the splice.
2. Let fade in coded frames be the coded frames that are faded in during the splice.
https://www.w3.org/TR/media-source/ 64/99 7/15/2016 Media Source Extensions
3. Let presentation timestamp be the presentation timestamp of the first coded frame in fade out coded frames.
4. Let end timestamp be the sum of the presentation timestamp and the coded frame duration of the last frame in fade in coded frames.
5. Let splice timestamp be the presentation timestamp where the splice starts. This corresponds with the presentation timestamp of the first frame in fade in coded frames.
6. Let splice end timestamp equal splice timestamp plus five milliseconds.
7. Let fade out samples be the samples generated by decoding fade out coded frames.
8. Trim fade out samples so that it only contains samples between presentation timestamp and splice end timestamp.
9. Let fade in samples be the samples generated by decoding fade in coded frames.
10. If fade out samples and fade in samples do not have a common sample rate and channel layout, then convert fade out samples and fade in samples to a common sample rate and channel layout.
11. Let output samples be a buffer to hold the output samples.
12. Apply a linear gain fade out with a starting gain of 1 and an ending gain of 0 to the samples between splice timestamp and splice end timestamp in fade out samples.
13. Apply a linear gain fade in with a starting gain of 0 and an ending gain of 1 to the samples between splice timestamp and splice end timestamp in fade in samples.
14. Copy samples between presentation timestamp to splice timestamp from fade out samples into output samples.
15. For each sample between splice timestamp and splice end timestamp, compute the sum of a sample from fade out samples and the corresponding sample in fade in samples and store the result in output samples.
16. Copy samples between splice end timestamp to end timestamp from fade in samples into output samples.
17. Render output samples.
https://www.w3.org/TR/media-source/ 65/99 7/15/2016 Media Source Extensions
NOTE
Here is a graphical representation of this algorithm.
3.5.17 Text Splice Frame Algorithm
Follow these steps when the coded frame processing algorithm needs to generate a splice frame for two overlapping timed text coded frames:
1. Let track buffer be the track buffer that will contain the splice.
2. Let new coded frame be the new coded frame, that is being added to track buffer, which triggered the need for a splice.
3. Let presentation timestamp be the presentation timestamp for new coded frame
4. Let decode timestamp be the decode timestamp for new coded frame.
5. Let frame duration be the coded frame duration of new coded frame.
6. Let frame end timestamp equal the sum of presentation timestamp and frame duration.
7. Let first overlapped frame be the coded frame in track buffer with a presentation interval that contains presentation timestamp.
8. Let overlapped presentation timestamp be the presentation timestamp of the first overlapped frame.
9. Let overlapped frames equal first overlapped frame as well as any additional frames in track buffer that have a presentation timestamp greater than presentation timestamp and less than frame
https://www.w3.org/TR/media-source/ 66/99 7/15/2016 Media Source Extensions
end timestamp.
10. Remove all the frames included in overlapped frames from track buffer.
11. Update the coded frame duration of the first overlapped frame to presentation timestamp - overlapped presentation timestamp.
12. Add first overlapped frame to the track buffer.
13. Return to caller without providing a splice frame.
NOTE
This is intended to allow new coded frame to be added to the track buffer as if it hadn't overlapped any frames in track buffer to begin with.
4. SourceBufferList Object
SourceBufferList is a simple container object for SourceBuffer objects. It provides read-only array access and fires events when the list is modified.
WebIDL
interface SourceBufferList : EventTarget { readonly attribute unsigned long length; attribute EventHandler onaddsourcebuffer; attribute EventHandler onremovesourcebuffer; getter SourceBuffer (unsigned long index); };
4.1 Attributes
length of type unsigned long, readonly Indicates the number of SourceBuffer objects in the list.
onaddsourcebuffer of type EventHandler The event handler for the addsourcebuffer event.
onremovesourcebuffer of type EventHandler The event handler for the removesourcebuffer event.
https://www.w3.org/TR/media-source/ 67/99 7/15/2016 Media Source Extensions
4.2 Methods
SourceBuffer Allows the SourceBuffer objects in the list to be accessed with an array operator (i.e. []).
Parameter Type Nullable Optional Description
index unsigned long ✘ ✘
Return type: getter
When this method is invoked, the user agent must run the following steps:
1. If index is greater than or equal to the length attribute then return undefined and abort these steps.
2. Return the index'th SourceBuffer object in the list.
4.3 Event Summary
Event name Interface Dispatched when...
addsourcebuffer Event When a SourceBuffer is added to the list.
removesourcebuffer Event When a SourceBuffer is removed from the list.
5. VideoPlaybackQuality Object
The following section is OPTIONAL. There are several implementations of the current specification, but it is subject to change because some feel it needs further work. The community is actively investigating alternate designs to provide feedback on media playback quality. See Media Playback Quality.
WebIDL
interface VideoPlaybackQuality { readonly attribute DOMHighResTimeStamp creationTime; readonly attribute unsigned long totalVideoFrames; readonly attribute unsigned long droppedVideoFrames; readonly attribute unsigned long corruptedVideoFrames; readonly attribute double totalFrameDelay; };
https://www.w3.org/TR/media-source/ 68/99 7/15/2016 Media Source Extensions
5.1 Attributes
corruptedVideoFrames of type unsigned long, readonly The total number of corrupted frames that have been detected.
creationTime of type DOMHighResTimeStamp, readonly The timestamp returned by Performance.now() when this object was created.
droppedVideoFrames of type unsigned long, readonly The total number of frames dropped predecode or dropped because the frame missed its display deadline.
totalFrameDelay of type double, readonly The sum of all displayed frame delays for all displayed frames. (i.e., Frames included in the totalVideoFrames count, but not in the droppedVideoFrames count.
totalVideoFrames of type unsigned long, readonly The total number of frames that would have been displayed if no frames are dropped.
6. TrackDefault Object
The TrackDefault object is used to provide kind, label, and language information for tracks that do not contain this information in the initialization segments. This information is only consulted when the initialization segment received algorithm creates track objects.
WebIDL
enum TrackDefaultType { "audio", "video", "text" };
Enumeration description
Indicates that the information in the TrackDefault object should only be used when audio creating AudioTrack objects.
Indicates that the information in the TrackDefault object should only be used when video creating VideoTrack objects.
Indicates that the information in the TrackDefault object should only be used when text creating TextTrack objects.
https://www.w3.org/TR/media-source/ 69/99 7/15/2016 Media Source Extensions
WebIDL
[ Constructor (TrackDefaultType type, DOMString language, DOMString label, sequence
6.1 Constructors
TrackDefault
Parameter Type Nullable Optional Description
type TrackDefaultType ✘ ✘
language DOMString ✘ ✘
label DOMString ✘ ✘
kinds sequence
byteStreamTrackID DOMString = "" ✘ ✔
When this method is invoked, the user agent must run the following steps:
1. If language is not an empty string and language is not a BCP 47 language tag, then throw a TypeError and abort these steps.
2. If type equals "audio": If any string in kinds contains a value that is not listed as applying to audio in ↪ the kind categories table, then throw a TypeError and abort these steps.
If type equals "video": If any string in kinds contains a value that is not listed as applying to video in ↪ the kind categories table, then throw a TypeError and abort these steps.
If type equals "text": If any string in kinds contains a value that is not listed in the text track ↪ kind list, then throw a TypeError and abort these steps.
3. Set the type attribute on this new object to type.
https://www.w3.org/TR/media-source/ 70/99 7/15/2016 Media Source Extensions
4. Set the language attribute on this new object to language.
5. Set the label attribute on this new object to label.
6. Set the returned value of the getKinds method to return this new object kinds.
7. Set the byteStreamTrackID attribute on this new object to byteStreamTrackID.
6.2 Attributes
byteStreamTrackID of type DOMString, readonly The decimal string representation of the Track ID that identifies the specific track that this object should apply to. An empty string indicates that this object can be used for any track of the appropriate type. (i.e. any AudioTrack if type equals "audio").
label of type DOMString, readonly The default label to use when an initialization segment does not contain label information for a new track. See initialization segment received algorithm for details.
language of type DOMString, readonly The default language to use when an initialization segment does not contain language information for a new track. See initialization segment received algorithm for details.
type of type TrackDefaultType, readonly The type of track that can be constructed using the information in this object.
6.3 Methods
getKinds The default kinds used when an initialization segment does not contain kind information for a new track. See initialization segment received algorithm for details.
No parameters. Return type: sequence
7. TrackDefaultList Object
TrackDefaultList is a simple container object for TrackDefault objects. It provides read-only array access.
https://www.w3.org/TR/media-source/ 71/99 7/15/2016 Media Source Extensions
WebIDL
[ Constructor (optional sequence
7.1 Constructors
TrackDefaultList
Parameter Type Nullable Optional Description
trackDefaults sequence
When this method is invoked, the user agent must run the following steps:
1. If trackDefaults contains two or more TrackDefault objects with the same type and the same byteStreamTrackID, then throw a TypeError and abort these steps.
NOTE
This also applies when byteStreamTrackID contains an empty string and ensures that there is only one "byteStreamTrackID independent" default for each TrackDefaultType value.
2. Store a shallow copy of trackDefaults in this new object so the values can be returned by the accessor methods.
7.2 Attributes
length of type unsigned long, readonly Indicates the number of TrackDefault objects in the list.
7.3 Methods
TrackDefault
https://www.w3.org/TR/media-source/ 72/99 7/15/2016 Media Source Extensions
Allows the TrackDefault objects in the list to be accessed with an array operator (i.e. []).
Parameter Type Nullable Optional Description
index unsigned long ✘ ✘
Return type: getter
When this method is invoked, the user agent must run the following steps:
1. If index is greater than or equal to the length attribute then return undefined and abort these steps.
2. Return the index'th TrackDefault object in the list.
8. URL Object Extensions
This section specifies extensions to the URL[FILE-API] object definition.
WebIDL
[Exposed=Window,DedicatedWorker,SharedWorker] partial interface URL { static DOMString createObjectURL (MediaSource mediaSource); };
8.1 Methods
createObjectURL, static Creates URLs for MediaSource objects.
NOTE
This algorithm is intended to mirror the behavior of the createObjectURL()[FILE- API] method, which does not auto-revoke the created URL. Web authors are encouraged to use revokeObjectURL()[FILE-API] for any MediaSource object URL that is no longer needed for attachment to a media element.
Parameter Type Nullable Optional Description
mediaSource MediaSource ✘ ✘
https://www.w3.org/TR/media-source/ 73/99 7/15/2016 Media Source Extensions
Return type: DOMString
When this method is invoked, the user agent must run the following steps:
1. Return a unique MediaSource object URL that can be used to dereference the mediaSource argument.
9. HTMLMediaElement Extensions
This section specifies what existing attributes on the HTMLMediaElement must return when a MediaSource is attached to the element.
The HTMLMediaElement.seekable attribute returns a new static normalized TimeRanges object created based on the following steps:
If duration equals NaN: Return an empty TimeRanges object. ↪ If duration equals positive Infinity:
↪ 1. If live seekable range is not empty:
1. Let union ranges be the union of live seekable range and the HTMLMediaElement.buffered attribute.
2. Return a single range with a start time equal to the earliest start time in union ranges and an end time equal to the highest end time in union ranges and abort these steps.
2. If the HTMLMediaElement.buffered attribute returns an empty TimeRanges object, then return an empty TimeRanges object and abort these steps.
3. Return a single range with a start time of 0 and an end time equal to the highest end time reported by the HTMLMediaElement.buffered attribute.
Otherwise: Return a single range with a start time of 0 and an end time equal to duration. ↪ The HTMLMediaElement.buffered attribute returns a static normalized TimeRanges object based on the following steps.
1. Let intersection ranges equal an empty TimeRanges object.
https://www.w3.org/TR/media-source/ 74/99 7/15/2016 Media Source Extensions
2. If activeSourceBuffers.length does not equal 0 then run the following steps: 1. Let active ranges be the ranges returned by buffered for each SourceBuffer object in activeSourceBuffers. 2. Let highest end time be the largest range end time in the active ranges.
3. Let intersection ranges equal a TimeRange object containing a single range from 0 to highest end time.
4. For each SourceBuffer object in activeSourceBuffers run the following steps: 1. Let source ranges equal the ranges returned by the buffered attribute on the current SourceBuffer.
2. If readyState is "ended", then set the end time on the last range in source ranges to highest end time. 3. Let new intersection ranges equal the intersection between the intersection ranges and the source ranges. 4. Replace the ranges in intersection ranges with the new intersection ranges.
3. If the current value of this attribute has not been set by this algorithm or intersection ranges does not contain the exact same range information as the current value of this attribute, then update the current value of this attribute to intersection ranges.
4. Return the current value of this attribute.
10. HTMLVideoElement Extensions
The following section uses VideoPlaybackQuality and is OPTIONAL. There are several implementations of the current specification, but it is subject to change because some feel it needs further work. The community is actively investigating alternate designs to provide feedback on media playback quality. See Media Playback Quality.
This section specifies new attributes and internal state that are being added to the HTMLVideoElement.
Each HTMLVideoElement will maintain a total video frame count variable that keeps track of the total number of frames that have been displayed and dropped. This variable is initialized to 0 when the element is created and whenever the media element load algorithm is invoked. It is incremented when a video frame is displayed or when the dropped video frame count is incremented.
Each HTMLVideoElement will maintain a dropped video frame count variable that keeps track of the total number of frames that have been dropped. This variable is initialized to 0 when the element is
https://www.w3.org/TR/media-source/ 75/99 7/15/2016 Media Source Extensions
created and whenever the media element load algorithm is invoked. It is incremented when a video frame is dropped predecode or when a frame is decoded but dropped because it missed a display deadline.
Each HTMLVideoElement will maintain a corrupted video frame count variable that keeps track of the total number of corrupted frames detected. This variable is initialized to 0 when the element is created and whenever the media element load algorithm is invoked. It is incremented when a corrupted video frame is detected by the decoder. It is up to the implementation to determine whether to display or drop a corrupted frame. Whichever choice is made, the total video frame count and dropped video frame count must be updated appropriately.
Each HTMLVideoElement will maintain a displayed frame delay sum variable that keeps track of the sum of all displayed frame delays. This variable is initialized to 0 when the element is created and whenever the media element load algorithm is invoked. When a frame is displayed, its displayed frame delay is computed and added to this variable.
WebIDL
partial interface HTMLVideoElement { VideoPlaybackQuality getVideoPlaybackQuality (); };
10.1 Methods
getVideoPlaybackQuality Provides the current the playback quality metrics.
No parameters. Return type: VideoPlaybackQuality
When this method is invoked, the user agent must run the following steps:
1. Let playbackQuality be a new instance of VideoPlaybackQuality.
2. Set playbackQuality.creationTime to the value returned by a call to Performance.now().
3. Set playbackQuality.totalVideoFrames to the current value of the total video frame count.
4. Set playbackQuality.droppedVideoFrames to the current value of the dropped video frame count.
https://www.w3.org/TR/media-source/ 76/99 7/15/2016 Media Source Extensions
5. Set playbackQuality.corruptedVideoFrames to the current value of the corrupted video frame count.
6. Set playbackQuality.totalFrameDelay to the current value of the displayed frame delay sum.
7. Return playbackQuality.
11. AudioTrack Extensions
This section specifies extensions to the HTML AudioTrack definition.
WebIDL
partial interface AudioTrack { readonly attribute SourceBuffer? sourceBuffer; };
11.1 Attributes
sourceBuffer of type SourceBuffer, readonly , nullable Returns the SourceBuffer that created this track. Returns null if this track was not created by a SourceBuffer or the SourceBuffer has been removed from the sourceBuffers attribute of its parent media source.
12. VideoTrack Extensions
This section specifies extensions to the HTML VideoTrack definition.
WebIDL
partial interface VideoTrack { readonly attribute SourceBuffer? sourceBuffer; };
12.1 Attributes
sourceBuffer of type SourceBuffer, readonly , nullable
https://www.w3.org/TR/media-source/ 77/99 7/15/2016 Media Source Extensions
Returns the SourceBuffer that created this track. Returns null if this track was not created by a SourceBuffer or the SourceBuffer has been removed from the sourceBuffers attribute of its parent media source.
13. TextTrack Extensions
This section specifies extensions to the HTML TextTrack definition.
WebIDL
partial interface TextTrack { readonly attribute SourceBuffer? sourceBuffer; };
13.1 Attributes
sourceBuffer of type SourceBuffer, readonly , nullable Returns the SourceBuffer that created this track. Returns null if this track was not created by a SourceBuffer or the SourceBuffer has been removed from the sourceBuffers attribute of its parent media source.
14. Byte Stream Formats
The bytes provided through appendBuffer() and appendStream() for a SourceBuffer form a logical byte stream. The format and semantics of these byte streams are defined in byte stream format specifications. The byte stream format registry [MSE-REGISTRY] provides mappings between a MIME type that may be passed to addSourceBuffer() or isTypeSupported() and the byte stream format expected by a SourceBuffer created with that MIME type. Implementations are encouraged to register mappings for byte stream formats they support to facilitate interoperability. The byte stream format registry [MSE-REGISTRY] is the authoritative source for these mappings. If an implementation claims to support a MIME type listed in the registry, its SourceBuffer implementation must conform to the byte stream format specification listed in the registry entry.
https://www.w3.org/TR/media-source/ 78/99 7/15/2016 Media Source Extensions
NOTE
The byte stream format specifications in the registry are not intended to define new storage formats. They simply outline the subset of existing storage format structures that implementations of this specification will accept.
NOTE
Byte stream format parsing and validation is implemented in the segment parser loop algorithm.
This section provides general requirements for all byte stream format specifications:
A byte stream format specification must define initialization segments and media segments.
A byte stream format should provide references for sourcing AudioTrack, VideoTrack, and TextTrack attribute values from data in initialization segments.
NOTE
If the byte stream format covers a format similar to one covered in the in-band tracks spec [INBANDTRACKS], then it should try to use the same attribute mappings so that Media Source Extensions playback and non-Media Source Extensions playback provide the same track information.
It must be possible to identify segment boundaries and segment type (initialization or media) by examining the byte stream alone.
The user agent must run the append error algorithm with the decode error parameter set to true when any of the following conditions are met: 1. The number and type of tracks are not consistent.
https://www.w3.org/TR/media-source/ 79/99 7/15/2016 Media Source Extensions
NOTE
For example, if the first initialization segment has 2 audio tracks and 1 video track, then all initialization segments that follow it in the byte stream must describe 2 audio tracks and 1 video track.
2. Track IDs are not the same across initialization segments, for segments describing multiple tracks of a single type. (e.g. 2 audio tracks). 3. Codecs changes across initialization segments.
NOTE
For example, a byte stream that starts with an initialization segment that specifies a single AAC track and later contains an initialization segment that specifies a single AMR-WB track is not allowed. Support for multiple codecs is handled with multiple SourceBuffer objects.
The user agent must support the following: 1. Track IDs changing across initialization segments if the segments describes only one track of each type. 2. Video frame size changes. The user agent must support seamless playback.
NOTE
This will cause the
3. Audio channel count changes. The user agent may support this seamlessly and could trigger downmixing.
NOTE
This is a quality of implementation issue because changing the channel count may require reinitializing the audio device, resamplers, and channel mixers which tends to be audible.
https://www.w3.org/TR/media-source/ 80/99 7/15/2016 Media Source Extensions
The following rules apply to all media segments within a byte stream. A user agent must: 1. Map all timestamps to the same media timeline. 2. Support seamless playback of media segments having a timestamp gap smaller than the audio frame size. User agent must not reflect these gaps in the buffered attribute.
NOTE
This is intended to simplify switching between audio streams where the frame boundaries don't always line up across encodings (e.g. Vorbis).
The user agent must run the append error algorithm with the decode error parameter set to true when any combination of an initialization segment and any contiguous sequence of media segments satisfies the following conditions: 1. The number and type (audio, video, text, etc.) of all tracks in the media segments are not identified. 2. The decoding capabilities needed to decode each track (i.e. codec and codec parameters) are not provided. 3. Encryption parameters necessary to decrypt the content (except the encryption key itself) are not provided for all encrypted tracks. 4. All information necessary to decode and render the earliest random access point in the sequence of media segments and all subsequence samples in the sequence (in presentation time) are not provided. This includes in particular, Information that determines the intrinsic width and height of the video (specifically, this requires either the picture or pixel aspect ratio, together with the encoded resolution). Information necessary to convert the video decoder output to a format suitable for display
5. Information necessary to compute the global presentation timestamp of every sample in the sequence of media segments is not provided.
For example, if I1 is associated with M1, M2, M3 then the above must hold for all the combinations I1+M1, I1+M2, I1+M1+M2, I1+M2+M3, etc.
Byte stream specifications must at a minimum define constraints which ensure that the above requirements hold. Additional constraints may be defined, for example to simplify implementation.
https://www.w3.org/TR/media-source/ 81/99 7/15/2016 Media Source Extensions
15. Examples
Example use of the Media Source Extensions
https://www.w3.org/TR/media-source/ 82/99 7/15/2016 Media Source Extensions
16. Acknowledgments
The editors would like to thank Alex Giladi, Bob Lund, Chris Poole, Cyril Concolato, David Dorwin, David Singer, Duncan Rowden, Frank Galligan, Glenn Adams, Jer Noble, Joe Steele, John Simmons, Kevin Streeter, Mark Vickers, Matt Ward, Matthew Gregan, Michael Thornburgh, Philip Jägenstedt, Pierre Lemieux, Ralph Giles, Steven Robertson, and Tatsuya Igarashi for their contributions to this specification.
17. Revision History
Version Comment
09 June Issue 91 - Change TrackDefault.kinds to TrackDefault.getKinds. 2016 Add TrackDefaults to the at risk feature list.
08 June
2016 Issue 5 - Add support for customizing HTMLMediaElement.seekable on live streams. Issue 87 - Add VideoPlaybackQuality to the list of at risk features. Issue 69 - Add optional & subject to change language to VideoPlaybackQuality. Issue 78 - Update reference from HTML5.0 to HTML5.1.
06 June
2016 Issue 18 - Clarify fetch algorithm failure detachment trigger. Issue 24 - Clarify MSE attachment stops delaying the 'load' event, and MSE disables 'suspend' event logic.
26 May
2016 Issue 44 - Used SVG and longdesc for pipeline diagram. Issue 67 - Excluded createObjectURL() from Service Workers. Issue 68 - Made kinds attribute and ctor argument use sequence type.
https://www.w3.org/TR/media-source/ 85/99 7/15/2016 Media Source Extensions
24 May Issue 10 - Updated createObjectURL() to align to current (April 2015) non- 2016 autorevoking createObjectURL()[FILE-API] method. Encouraged web author use of revokeObjectURL()[FILE-API].
12 May
2016 Issue 15 - Clarified track buffer ranges. Clarified allowance near presentation start time. Minor whitespace and formatting fixes.
12 May
2016 Issue 19, 20 & 26 - Resolved problems with Duration Changes and Range Removals. Separated Range Removals from Duration Changes, required separate remove(). Disabled abort on Range Removals.
3 May
2016 Issue 66 - Add EventHandler attributes to MediaSource, SourceBuffer, and SourceBufferList. Minor whitespace and typo fixes.
April 28
2016 Issue 54 - Clarified use of Presentation Time Stamps (PTS) for buffered.
April 15
2016 Issue 17 - Generalize the detach MediaSource algorithm for media element load. Issue 18 - Generalize the detach MediaSource algorithm for media element fetch algorithm failures.
13 April Issue 25 - Fixed ambiguous behavior with CFP algorithm. 2016 Issue 11 - Ignore preload attribute on parent media element.
11 April
2016 Corrected the editor's draft publish date. Minor whitespace fixes for source readability. Added missing links in recent revision history.
https://www.w3.org/TR/media-source/ 86/99 7/15/2016 Media Source Extensions
5 Issue 34 - Replace InvalidAccessError usage with TypeError. February Fixed unexpected tag errors from previous changes. 2016
29
January Issue 23 - Require setting a random access point to prevent the coded frame removal
2016 algorithm from introducing a discontinuity at end of current coded frame group.
15
October Issue 30 - Remove unnecessary segment parser loop step referencing undefined
2015 media segment header.
13
October Issue 12 - Update bug tracker links to use github. 2015 17
September Bug 29133 - Maintain coded group continuity in sequence append mode when
2015 resetting parser state.
14 July
2015 Bugs 28881 and 28882 - Editorial punctuation and whitespace fixes. Also replaced tabs with spaces and removed spaces at end of line and before periods and commas.
07 April
2015 Bug 28234 - Rename highest presentation timestamp to highest end timestamp and clarify definition.
03 April
2015 Update editors.
02 April
2015 Bug 28209 - Remove redundant definition of overlapped frame presentation timestamp.
03
February Bug 27790 - Make .buffered attributes return the same object if nothing has 2015 changed. Bug 27857 - Make the 'removed from the sourceBuffers' checks in remove() and .mode setter consistent with other algorithms.
https://www.w3.org/TR/media-source/ 87/99 7/15/2016 Media Source Extensions
09
January Bug 27649 - Clarify that setting the duration in the detach algorithm does not run the
2015 attribute set algorithm. Bug 27599 - Make TrackDefaultList constructor parameter optional and have a default value of []. Bug 27758 - Add informative reference to INBANDTRACKS and track attribute sourcing requirements.
11
December Bug 27559 - Clarify that the TrackDefaultList constructor does a shallow copy. 2014 Bug 27548 - Reroute decode errors through the append error algorithm. Bug 27542 - Prevent SourceBuffer monitoring algorithm from triggering a HAVE_NOTHING to HAVE_METADATA transition.
02
December Bug 27239 - Updated appendStream() to use the new Streams API. 2014 Bug 27487 - Removed negative decode timestamp check from coded frame processing algorithm. Bug 27342 - Changed prepare append algorithm to throw an exception if HTMLMediaElement.error is not null.
18
November Update all links in the revision history to point to GitHub. 2014 17
November Changed Editor's draft link to point to GitHub. 2014 Bug 27353 - Convert exceptions from xxx_ERR to xxxError form. Changed broken exception links to point to HTML5 instead of WHATWG DOM. Bug 27352 - Changed TrackDefault constructor to throw a TypeError instead of an InvalidAccessError for invalid kinds. Bug 27318 - Fix length attribute link in TrackBufferList getter method.
11
November Bug 27296 - Clarify .seekable when duration is Infinity and .buffered is empty. 2014 04
https://www.w3.org/TR/media-source/ 88/99 7/15/2016 Media Source Extensions
November Bug 27241 - Fixed step 1 of the TrackDefault constructor so it doesn't always throw 2014 an exception. Bug 27240 - Make TrackDefaultList contructor throw an exception on any duplicate (type, byteStreamTrackID) pair. Bug 27174 - Added SourceBuffer configuration definition to clarify addSourceBuffer() behavior.
06
October Bug 26924 - Run default language algorithm if 'und' BCP 47 value is present in the
2014 init segment.
29
September Bug 26932 - Fix TrackDefault constructor kinds parameter type. 2014 Update to ReSpec 3.2.20.
10
September Bug 26777 - Rename 'first initialization segment flag' to 'first initialization segment
2014 received flag'.
09
September Bug 26721 - Add label attribute to TrackDefault object. 2014 12 August
2014 Bug 26436 - Use HTMLMediaElement.buffered for HTMLMediaElement.readyState transitions instead of individual SourceBuffer.buffered attributes.
24 July
2014 Bug 26345 - Make activeSourceBuffers order match sourceBuffers order. Bug 26316 - Changed duration change algorithm to avoid open transition on endOfStream(). Bug 26314 - Fixed coded frame removal algorithm to remove dependencies that may lie outside the removal range.
20 June
2014 Bug 26032 - Set need random access point flag on all track buffers when a new init segment is received. Bug 25845 - Clarify SourceBuffer.buffered getter behavior.
https://www.w3.org/TR/media-source/ 89/99 7/15/2016 Media Source Extensions
13 June Bug 25995 - Fix conditions in "remove existing coded frames" step. 2014 Bug 25999 - Change appendWindowStart assignments to reference presentation start time. Bug 26000 - Make it explicit that SourceBuffer.remove() throws an exception if duration equals NaN. Bug 25846 - Update end of stream algorithm to invoke the append error algorithm on decode errors. Bug 25850 - Specify SourceBuffer.trackDefaults initial value and setter/getter behavior. Bug 25998 - Move presentation & decode timestamp checks after append window filtering steps.
21 May
2014 Bug 24370 - Add TrackDefault object and remove kind/language overloads.
20 May
2014 Bug 25518 - Make remove() end parameter an unrestricted double. Bug 25580 - Add informative reference to byte stream format registry.
29 April
2014 Bug 25347 - Remove unnecessary null & invalid enum value checks. Bug 25505 - Introduce "generate timestamps flag" to handle MPEG audio timestamp generation.
01 April
2014 Bug 25157 - Fix typo in coded frame duration example. Bug 24854 - Moved negative timestamp checking to allow "sequence" mode to handle appending coded frames in reverse order Bug 24820 - Renamed highest presentation end timestamp to group end timestamp and fixed a few related issues.
03 March
2014 Bug 24347 - Fix HAVE_FUTURE_DATA transition condition in SourceBuffer monitoring algorithm.
10
December Bug 23169 - Restore totalFrameDelay units to seconds.
https://www.w3.org/TR/media-source/ 90/99 7/15/2016 Media Source Extensions
2013 Bug 23989 - Update blob URL origin text to remove stale File API reference.
02
December Bug 23818 - Make external spec references normative. 2013 Bug 23441 - Update byte stream format registry text. Bug 23169 - Update totalFrameDelay text.
14
November Bug 23663 - Clarify seeking behavior in Section 2.4.3. 2013 04
November Bug 23441 - Established MSE byte stream format registry and extracted byte stream
2013 format text into separate documents.
29
October Bug 23553 - Fixed segment parser loop so it doesn't appear to prematurely remove
2013 the media segment header. Bug 23557 - Update SPS/PPS note to not explicitly recommend specific avc versions.
28
October Bug 23549 - Add definitions for decode timestamp, presentation timestamp, an
2013 presentation order. Bug 23552 - Clarify 'this' in section 3.5.1 Bug 23554 - Introduced presentation interval and coded frame duration terms to clarify text.
15
October Bug 23525 - Fix mvex box error behavior. 2013 04
October Bug 23442 - Fix example to work when seeking in the 'ended' state. 2013 26 July
2013 Bug 22136 - Added text for Inband SPS/PPS support. Bug 22776 - Clarified that implementations are only required to support one SourceBuffer configuration at a time.
https://www.w3.org/TR/media-source/ 91/99 7/15/2016 Media Source Extensions
18 July Bug 22117 - Reword byte stream specs in terms of UA behavior. 2013 Bug 22148 - Replace VideoPlaybackQuality.playbackJitter with VideoPlaybackQuality.totalFrameDelay.
02 July
2013 Bug 22401 - Fix typo Bug 22134 - Clarify byte stream format enforcement. Bug 22431 - Convert videoPlaybackQuality attribute to getVideoPlaybackQuality() method. Bug 22109 - Renamed 'coded frame sequence' to 'coded frame group' to avoid confusion around multiple 'sequence' concepts.
05 June
2013 Bug 22139 - Added a note clarifying that byte stream specs aren't defining new storage formats. Bug 22148 - Added playbackJitter metric. Bug 22134 - Added minimal number of SourceBuffers requirements. Bug 22115 - Make algorithm abort text consistent. Bug 22113 - Address typos. Bug 22065 - Fix infinite loop in coded frame processing algorithm.
01 June
2013 Bug 21431 - Updated coded frame processing algorithm for text splicing. Bug 22035 - Update addtrack and removetrack event firing text to match HTML5 language. Bug 22111 - Remove useless playback sentence from end of stream algorithm. Bug 22052 - Add corrupted frame metric. Bug 22062 - Added links for filing bugs. Bug 22125 - Add "ended" to "open" transition to remove(). Bug 22143 - Move HTMLMediaElement.playbackQuality to HTMLVideoElement.videoPlaybackQuality.
https://www.w3.org/TR/media-source/ 92/99 7/15/2016 Media Source Extensions
13 May Bug 21954 - Add [EnforceRange] to appendStream's maxSize parameter. 2013 Bug 21953 - Add NaN handling to appendWindowEnd setter algorithm. Alphabetize definitions section. Changed endOfStream('decode') references to make it clear that JavaScript can't intercept these calls. Fix links for all types in the IDL that are defined in external specifications.
06 May
2013 Bug 20901 - Remove AbortMode and add AppendMode. Bug 21911 - Change MediaPlaybackQuality.creationTime to DOMHighResTimeStamp.
02 May
2013 Reworked ambiguous text in a variety of places. Added Acknowledgements section.
30 April
2013 Bug 21822 - Fix 'fire ... event ... at the X attribute' text. Bug 21819 & 21328 - Remove 'compressed' from coded frame definition.
24 April
2013 Bug 21796 - Removed issue box from 'Append Error' algorithm. Bug 21703 - Changed appendWindowEnd to 'unrestricted double'. Bug 20760 - Adding MediaPlaybackQuality object. Bug 21536 - Specify the origin of media data appended.
08 April
2013 Bug 21327 - Crossfade clarifications. Bug 21334 - Clarified seeking behavoir. Bug 21326 - Add a note stating some implementations may choose to add fades to/from silence. Bug 21375 - Clarified decode dependency removal. Bug 21376 - Replace 100ms limit with 2x last frame duration limit.
26 March Bug 21301 - Change timeline references to "media timeline" links.
https://www.w3.org/TR/media-source/ 93/99 7/15/2016 Media Source Extensions
2013 Bug 19676 - Clarified "fade out coded frames" definition. Bug 21276 - Convert a few append error scenarios to endOfStream('decode') errors. Bug 21376 - Changed 'time' to 'decode time' to append sequence definition. Bug 21374 - Clarify the abort() behavior. Bug 21373 - Clarified incremental parsing text in segment parser loop. Bug 21364 - Remove redundant condition from remove overlapped frame step. Bug 21327 - Clarify what to do with a splice that starts with an audio frame with a duration less than 5ms. Update to ReSpec 3.1.48
12 March
2013 Bug 21112 - Add appendWindowStart & appendWindowEnd attributes. Bug 19676 - Clarify overlapped frame definitions and splice logic. Bug 21172 - Added coded frame removal and eviction algorithms.
05 March
2013 Bug 21170 - Remove 'stream aborted' step from stream append loop algorithm. Bug 21171 - Added informative note about when addSourceBuffer() might throw an QUOTA_EXCEEDED_ERR exception. Bug 20901 - Add support for 'continuation' and 'timestampOffset' abort modes. Bug 21159 - Rename appendArrayBuffer to appendBuffer() and add ArrayBufferView overload. Bug 21198 - Remove redundant 'closed' readyState checks.
25
February Remove Source Buffer Model section since all the behavior is covered by the
2013 algorithms now. Bug 20899 - Remove media segments must start with a random access point requirement. Bug 21065 - Update example code to use updating attribute instead of old appending attribute.
19
February Bug 19676, 20327 - Provide more detail for audio & video splicing.
https://www.w3.org/TR/media-source/ 94/99 7/15/2016 Media Source Extensions
2013 Bug 20900 - Remove complete access unit constraint. Bug 20948 - Setting timestampOffset in 'ended' triggers a transition to 'open' Bug 20952 - Added update event. Bug 20953 - Move end of append event firing out of segment parser loop. Bug 21034 - Add steps to fire addtrack and removetrack events.
05
February Bug 19676 - Added a note clarifying that the internal timestamp representation
2013 doesn't have to be a double. Added steps to the coded frame processing algorithm to remove old frames when new ones overlap them. Fix isTypeSupported() return type. Bug 18933 - Clarify what top-level boxes to ignore for ISO-BMFF. Bug 18400 - Add a check to avoid creating huge hidden gaps when out-of-order appends occur w/o calling abort().
31
January Make remove() asynchronous. 2013 Added steps to various algorithms to throw an INVALID_STATE_ERR exception when async appends or remove() are pending.
30
January Remove early abort step on 0-byte appends so the same events fire as a normal
2013 append with bytes. Added definition for 'enough data to ensure uninterrupted playback'. Updated buffered ranges algorithm to properly compute the ranges for Philip's example.
15 Replace setTrackInfo() and getSourceBuffer() with AudioTrack, VideoTrack, and January TextTrack extensions. 2013
04
January Renamed append() to appendArrayBuffer() and made appending asynchronous. 2013 Added SourceBuffer.appendStream(). Added SourceBuffer.setTrackInfo() methods. https://www.w3.org/TR/media-source/ 95/99 7/15/2016 Media Source Extensions
Added issue boxes to relevant sections for outstanding bugs.
14 Pubrules, Link Checker, and Markup Validation fixes. December 2012 13
December Added MPEG-2 Transport Stream section. 2012 Added text to require abort() for out-of-order appends. Renamed "track buffer" to "decoder buffer". Redefined "track buffer" to mean the per-track buffers that hold the SourceBuffer media data. Editorial fixes.
08
December Added MediaSource.getSourceBuffer() methods. 2012 Section 2 cleanup.
06
December append() now throws a QUOTA_EXCEEDED_ERR when the SourceBuffer is full. 2012 Added unique ID generation text to Initialization Segment Received algorithm. Remove 2.x subsections that are already covered by algorithm text. Rework byte stream format text so it doesn't imply that the MediaSource implementation must support all formats supported by the HTMLMediaElement.
28
November Added transition to HAVE_METADATA when current playback position is 2012 removed. Added remove() calls to duration change algorithm. Added MediaSource.isTypeSupported() method. Remove initialization segments are optional text.
09 Converted document to ReSpec. November 2012 18 Refactored SourceBuffer.append() & added SourceBuffer.remove(). October
https://www.w3.org/TR/media-source/ 96/99 7/15/2016 Media Source Extensions
2012 8 October
2012 Defined what HTMLMediaElement.seekable and HTMLMediaElement.buffered should return. Updated seeking algorithm to run inside Step 10 of the HTMLMediaElement seeking algorithm. Removed transition from "ended" to "open" in the seeking algorithm. Clarified all the event targets.
1 October Fixed various addsourcebuffer & removesourcebuffer bugs and allow append() in ended 2012 state. 13 Updated endOfStream() behavior to change based on the value of September HTMLMediaElement.readyState. 2012
24 August
2012 Added early abort on to duration change algorithm. Added createObjectURL() IDL & algorithm. Added Track ID & Track description definitions. Rewrote start overlap for audio frames text. Removed rendering silence requirement from section 2.5.
22 August
2012 Clarified WebM byte stream requirements. Clarified SourceBuffer.buffered return value. Clarified addsourcebuffer & removesourcebuffer event targets. Clarified when media source attaches to the HTMLMediaElement. Introduced duration change algorithm and update relevant algorithms to use it.
17 August Minor editorial fixes. 2012 09 August Change presentation start time to always be 0 instead of using format specific rules about 2012 the first media segment appended. 30 July Added SourceBuffer.timestampOffset and MediaSource.duration. 2012
https://www.w3.org/TR/media-source/ 97/99 7/15/2016 Media Source Extensions
17 July Replaced SourceBufferList.remove() with MediaSource.removeSourceBuffer(). 2012 02 July Converted to the object-oriented API 2012 26 June Converted to Editor's draft. 2012 0.5 Minor updates before proposing to W3C HTML-WG. 0.4 Major revision. Adding source IDs, defining buffer model, and clarifying byte stream formats. 0.3 Minor text updates. 0.2 Updates to reflect initial WebKit implementation. 0.1 Initial Proposal
A. References
A.1 Normative references
[FILE-API] Arun Ranganathan; Jonas Sicking. File API. 21 April 2015. W3C Working Draft. URL: https://www.w3.org/TR/FileAPI/ [HTML51] Steve Faulkner; Arron Eicholz; Travis Leithead; Alex Danilo. HTML 5.1. 21 June 2016. W3C Candidate Recommendation. URL: https://www.w3.org/TR/html51/ [TYPED-ARRAYS] David Herman; Kenneth Russell. Typed Array Specification. 26 June 2013. Khronos Working Draft. URL: https://www.khronos.org/registry/typedarray/specs/latest/ [WHATWG-STREAMS-API] Domenic Denicola; Takeshi Yoshino. WHATWG Streams API. URL: https://streams.spec.whatwg.org/
A.2 Informative references
[INBANDTRACKS]
https://www.w3.org/TR/media-source/ 98/99 7/15/2016 Media Source Extensions
Bob Lund; Silvia Pfeiffer. Sourcing In-band Media Resource Tracks from Media Containers into HTML. URL: http://dev.w3.org/html5/html-sourcing-inband-tracks/ [MSE-REGISTRY] Aaron Colwell. Media Source Extensions Byte Stream Format Registry. URL: byte-stream- format-registry.html
↑
https://www.w3.org/TR/media-source/ 99/99