mediacapture-worker Manipulate the MediaStream video data on the worker Mozilla Corporation Confidential - Internal Use Only 2015/10/29 @ W3C TPAC2015 Info Repository: https://github.com/w3c/mediacapture-worker Editors: ● Chia-hung Tai (CTai), Mozilla ● Robert O'Callahan (ROC), Mozilla ● Tzuhao Kuo (Kaku), Mozilla ● Anssi Kostiainen, Intel Mozilla Corporation Confidential - Internal Use Only The situation of processing video data now Mozilla Corporation Confidential - Internal Use Only The situation & the problems of processing video data now Redundant canvas element Only Stick to the main thread pull-based mechanism Mozilla Corporation Confidential - Internal Use Only MediaStream with worker Mozilla Corporation Confidential - Internal Use Only MediaStream with worker Mozilla Corporation Confidential - Internal Use Only MediaStream with worker Mozilla Corporation Confidential - Internal Use Only ImageBitmap extensions Mozilla Corporation Confidential - Internal Use Only ImageBitmap extensions Mozilla Corporation Confidential - Internal Use Only ImageBitmap extensions Mozilla Corporation Confidential - Internal Use Only ImageBitmap extensions Mozilla Corporation Confidential - Internal Use Only Use cases - video editing Mozilla Corporation Confidential - Internal Use Only Use cases - camera effects Mozilla Corporation Confidential - Internal Use Only Use cases - stream manipulation Mozilla Corporation Confidential - Internal Use Only Use cases - computer vision depth camera Mozilla Corporation Confidential - Internal Use Only DEMO Pure JavaScript code and off the main thread Mozilla Corporation Confidential - Internal Use Only Issues for discussion Mozilla Corporation Confidential - Internal Use Only Decouple from worker interface VidoeMonitor { // Q: A MediaStreamTrack can only be associated with one attribute EventHandler onvideomonitor; Processor? → How to ensure it? }; partial interface MediaStreamTrack { Promise<VideoMonitor> createVideoMonitor(); // Need a way to get the result MediaStream out. Promise<VideoProcessor> createVideoProcessor(); interface VidoeMonitor : VideoProcessor { }; attribute EventHandler onvideoprocess; }; VidoeMonitor implements Transferable; VidoeProcessor implements Transferable; Proposed by Mathieu Hofman ● Worklets Level 1, proposed by Paul Mozilla Corporation Confidential - Internal Use Only Offline processing Is it a good idea to extend the MediaStream with offline property? There are OfflineAudioContext and AudioBuffer in the WebAudio. ● Streams, proposed by Paul ● https://github.com/whatwg/streams/ ● BugZilla bug (which now is P5……) Mozilla Corporation Confidential - Internal Use Only Up-sampling / down-sampling Use case: Interpolate a 30 FPS video to 240 FPS? Mozilla Corporation Confidential - Internal Use Only Backup slides Mozilla Corporation Confidential - Internal Use Only WebGL support 1. OffscreenCanvas https://wiki.whatwg.org/wiki/OffscreenCanvas 2. ImageBitmap as WebGLRenderingContext::texImage2D() source. Mozilla Corporation Confidential - Internal Use Only The situation & one more problem Only Redundant real-time canvas palyback element Only Stick to the main thread pull-based mechanism Mozilla Corporation Confidential - Internal Use Only OfflineMediaContext Mozilla Corporation Confidential - Internal Use Only OfflineMediaContext Mozilla Corporation Confidential - Internal Use Only The spec. For Ad-hoc meeting only http://w3c.github.io/mediacapture-worker Mozilla Corporation Confidential - Internal Use Only Worker extensions partial interface MediaStreamTrack { [Exposed=Worker] Promise<void> addWorkerMonitor(Worker worker); interface VideoMonitorEvent : Event { Promise<void> removeWorkerMonitor(Worker worker); readonly attribute DOMString trackId; Promise<MediaStreamTrack> addWorkerProcessor(Worker worker); readonly attribute double playbackTime; Promise<void> removeWorkerProcessor(); readonly attribute ImageBitmap? inputImageBitmap; }; }; partial interface DedicatedWorkerGlobalScope { [Exposed=Worker] attribute EventHandler onvideomonitor; interface VideoProcessEvent : VideoMonitorEvent { attribute EventHandler onvideoprocess; readonly attribute ImageBitmap? outputImageBitmap; }; }; Mozilla Corporation Confidential - Internal Use Only ImageBitmap extensions [Exposed=(Window,Worker)] partial interface ImageBitmap { ImageFormat findOptimalFormat (optional sequence<ImageFormat> possibleFormats); [Throws] long mappedDataLength (ImageFormat format); [Throws] Promise<ImageFormatPixelLayout> mapDataInto (ImageFormat format, BufferSource buffer, long offset, long length); }; [Exposed=(Window,Worker), NoInterfaceObject] partial interface ImageBitmapFactories { Promise<ImageBitmap> createImageBitmap (BufferSource buffer, long offset, long length, ImageFormat format, ImageFormatPixelLayout layout); Promise<ImageBitmap> createImageBitmap (BufferSource buffer, long offset, long length, ImageFormat format, ImageFormatPixelLayout layout, long sx, long sy, long sw, long sh); }; Mozilla Corporation Confidential - Internal Use Only ImageBitmap extensions [Exposed=(Window,Worker), enum ImageFormat { enum DataType { Constructor(unsigned long offset, unsigned long width, unsigned long height, "RGBA32", "uint8", DataType dataType, unsigned long stride, unsigned long skip)] "BGRA32", "int8", interface ChannelPixelLayout { "RGB24", "uint16", readonly attribute unsigned long offset; "BGR24", "int16", readonly attribute unsigned long width; "GRAY8", "uint32", readonly attribute unsigned long height; "YUV444P", "int32", readonly attribute DataType dataType; "YUV422P", "float32", readonly attribute unsigned long stride; "YUV420P", "float64" readonly attribute unsigned long skip; "YUV420SP_NV12", }; }; "YUV420SP_NV21", "HSV", [Exposed=(Window,Worker), "Lab", Constructor(ChannelPixelLayout[] channels)] "DEPTH", interface ImageFormatPixelLayout { "" readonly attribute ChannelPixelLayout[] channels; }; }; Mozilla Corporation Confidential - Internal Use Only.