Mediacapture-Worker

mediacapture-worker

Manipulate the MediaStream video data on the worker

Mozilla Corporation Confidential - Internal Use Only 2015/10/29 @ W3C TPAC2015 Info

Repository: https://github.com/w3c/mediacapture-worker

Editors: ● Chia-hung Tai (CTai), Mozilla ● Robert O'Callahan (ROC), Mozilla ● Tzuhao Kuo (Kaku), Mozilla ● Anssi Kostiainen, Intel

Mozilla Corporation Confidential - Internal Use Only The situation of processing video data now

Mozilla Corporation Confidential - Internal Use Only The situation & the problems of processing video data now Redundant canvas element

Only Stick to the main thread pull-based mechanism

Mozilla Corporation Confidential - Internal Use Only MediaStream with worker

Mozilla Corporation Confidential - Internal Use Only ImageBitmap extensions

Mozilla Corporation Confidential - Internal Use Only Use cases - video editing

Mozilla Corporation Confidential - Internal Use Only Use cases - camera effects

Mozilla Corporation Confidential - Internal Use Only Use cases - stream manipulation

Mozilla Corporation Confidential - Internal Use Only Use cases - computer vision depth camera

Mozilla Corporation Confidential - Internal Use Only DEMO Pure JavaScript code and off the main thread

Mozilla Corporation Confidential - Internal Use Only Issues for discussion

Mozilla Corporation Confidential - Internal Use Only Decouple from worker

interface VidoeMonitor { // Q: A MediaStreamTrack can only be associated with one attribute EventHandler onvideomonitor; Processor? → How to ensure it? }; partial interface MediaStreamTrack { Promise createVideoMonitor(); // Need a way to get the result MediaStream out. Promise createVideoProcessor(); interface VidoeMonitor : VideoProcessor { }; attribute EventHandler onvideoprocess; };

VidoeMonitor implements Transferable; VidoeProcessor implements Transferable; Proposed by Mathieu Hofman

● Worklets Level 1, proposed by Paul

Mozilla Corporation Confidential - Internal Use Only Offline processing

Is it a good idea to extend the MediaStream with offline property? There are OfflineAudioContext and AudioBuffer in the WebAudio.

● Streams, proposed by Paul ● https://github.com/whatwg/streams/ ● BugZilla bug (which now is P5……)

Mozilla Corporation Confidential - Internal Use Only Up-sampling / down-sampling

Use case: Interpolate a 30 FPS video to 240 FPS?

Mozilla Corporation Confidential - Internal Use Only Backup slides

Mozilla Corporation Confidential - Internal Use Only WebGL support

1. OffscreenCanvas https://wiki.whatwg.org/wiki/OffscreenCanvas

2. ImageBitmap as WebGLRenderingContext::texImage2D() source.

Mozilla Corporation Confidential - Internal Use Only The situation & one more problem

Only Redundant real-time canvas palyback element

Only Stick to the main thread pull-based mechanism

Mozilla Corporation Confidential - Internal Use Only OfflineMediaContext

Mozilla Corporation Confidential - Internal Use Only The spec. For Ad-hoc meeting only http://w3c.github.io/mediacapture-worker

Mozilla Corporation Confidential - Internal Use Only Worker extensions

partial interface MediaStreamTrack { [Exposed=Worker] Promise addWorkerMonitor(Worker worker); interface VideoMonitorEvent : Event { Promise removeWorkerMonitor(Worker worker); readonly attribute DOMString trackId; Promise addWorkerProcessor(Worker worker); readonly attribute double playbackTime; Promise removeWorkerProcessor(); readonly attribute ImageBitmap? inputImageBitmap; }; };

partial interface DedicatedWorkerGlobalScope { [Exposed=Worker] attribute EventHandler onvideomonitor; interface VideoProcessEvent : VideoMonitorEvent { attribute EventHandler onvideoprocess; readonly attribute ImageBitmap? outputImageBitmap; }; };

Mozilla Corporation Confidential - Internal Use Only ImageBitmap extensions

[Exposed=(Window,Worker)] partial interface ImageBitmap { ImageFormat findOptimalFormat (optional sequence possibleFormats); [Throws] long mappedDataLength (ImageFormat format); [Throws] Promise mapDataInto (ImageFormat format, BufferSource buffer, long offset, long length); };

[Exposed=(Window,Worker), NoInterfaceObject] partial interface ImageBitmapFactories { Promise createImageBitmap (BufferSource buffer, long offset, long length, ImageFormat format, ImageFormatPixelLayout layout); Promise createImageBitmap (BufferSource buffer, long offset, long length, ImageFormat format, ImageFormatPixelLayout layout, long sx, long sy, long sw, long sh); };

Mozilla Corporation Confidential - Internal Use Only ImageBitmap extensions

[Exposed=(Window,Worker), enum ImageFormat { enum DataType { Constructor(unsigned long offset, unsigned long width, unsigned long height, "RGBA32", "uint8", DataType dataType, unsigned long stride, unsigned long skip)] "BGRA32", "int8", interface ChannelPixelLayout { "RGB24", "uint16", readonly attribute unsigned long offset; "BGR24", "int16", readonly attribute unsigned long width; "GRAY8", "uint32", readonly attribute unsigned long height; "YUV444P", "int32", readonly attribute DataType dataType; "YUV422P", "float32", readonly attribute unsigned long stride; "YUV420P", "float64" readonly attribute unsigned long skip; "YUV420SP_NV12", }; }; "YUV420SP_NV21", "HSV", [Exposed=(Window,Worker), "Lab", Constructor(ChannelPixelLayout[] channels)] "DEPTH", interface ImageFormatPixelLayout { "" readonly attribute ChannelPixelLayout[] channels; }; };

Mozilla Corporation Confidential - Internal Use Only