Bringing the Power of the GPU to the Web Neil Trevett Vice President NVIDIA President Khronos © Copyright Khronos Group 2013 - Page 1 Mobile is the New Epicenter of Innovation © Copyright Khronos Group 2013 - Page 2 Khronos Standards 3D Asset Handling - Advanced Authoring pipelines - 3D Asset Transmission Format with streaming and compression Visual Computing - Object and Terrain Visualization - Advanced scene construction Over 100 companies defining royalty-free APIs to connect software to silicon Camera Control API Acceleration in the Browser - WebGL for 3D in browsers - WebCL – Heterogeneous Sensor Processing Computing for the web - Mobile Vision Acceleration - On-device Sensor Fusion © Copyright Khronos Group 2013 - Page 3 Mobile Web is a Real Time Application 2048x1536 3100K Pixels 326 DPI 1024x768 786K Pixels 132 DPI + 320x480 = 153K Pixels 163 DPI Apple Apple Apple iPhone iPad iPad Mini Buttery smooth touch In 5 years the number of Need GPU interaction needs continuous pixels to process on Acceleration for 60Hz updates mobile screens has gone everything Web! up by factor of TWENTY © Copyright Khronos Group 2013 - Page 4 How are GPUs Accessible to the Web? • Hardware composition - Within the browser stack – under the hood • Vector Acceleration for SVG - Using NVIDIA OpenGL extensions • 3D Developer Functionality - OpenGL ES functionality through JavaScript • Compute Acceleration - Offloading compute intensive code to GPU • Compression and streaming of 3D assets - For network transmission • Camera, vision and sensor processing - Future JavaScript bindings to native APIs? © Copyright Khronos Group 2013 - Page 5 Mobile OS Adoption of Khronos APIs OpenGL ES 2.0 Shipping - Android 2.2 OpenSL ES 1.0 (subset) Shipping – Android 2.3 OpenMAX AL 1.0 (subset) Shipping - Android 4.0 EGL 1.4 Shipping under SDK -> NDK Opera and Firefox WebGL now Chrome soon OpenGL 3.2 on MacOS OpenCL 1.2 on MacOS OpenGL ES 3.0 on iOS Can enable on MacOS Safari iOS5 enables WebGL for iAds © Copyright Khronos Group 2013 - Page 6 WebGL – 3D on the Web – No Plug-in! • Leveraging HTML 5 and <canvas> element - WebGL defines JavaScript binding to OpenGL ES 2.0 - Enables a 3D context for the canvas • Low-level foundational Web API for accessing the GPU - Flexibility and direct GPU access - Enables higher-level frameworks and middleware JavaScript binding to OpenGL ES 2.0 Availability of OpenGL and Increasing JavaScript OpenGL ES on almost every performance. web-capable device HTML 5 Canvas Tag © Copyright Khronos Group 2013 - Page 7 WebGL Implementation Anatomy Content Much WebGL JavaScript, HTML, CSS, ... Content downloaded from the Web. content uses Middleware can make WebGL accessible to three.js non-expert 3D programmers library: JavaScript Middleware http://threejs.org/ HTML5 Browser provides WebGL functionality alongside other HTML5 technologies - no plug-in required JavaScript CSS OpenGL ES 2.0 OS Provided Drivers. WebGL on Windows can OpenGL use Direct3D - for example Angle open source DX9/Angle project creates OpenGL ES 2.0 over DX9 © Copyright Khronos Group 2013 - Page 8 WebGL Availability in Browsers - Microsoft – “where you have IE11, you have WebGL – turned on by default and working all the time” - Microsoft - WebGL also enabled for Windows applications - web app framework and web view - Apple - WebGL must be explicitly turned on MAC Safari and only exposed on iOS for iAds - Chrome OS - WebGL is the only cross-platform API to program the GPU - Google IO announcement - Chrome on Android will soon launch with WebGL © Copyright Khronos Group 2013 - Page 9 Cross-OS Portability HTML5 provides cross HTML/CSS HTML/CSS HTML/CSS platform portability. GPU accessibility through WebGL available soon on ~90% mobile systems Dalvik Preferred development SDK Objective C C# environments not (Java) designed for portability Native code is portable- but apps must cope with C/C++ DirectX different available APIs and libraries © Copyright Khronos Group 2013 - Page 10 WebGL First Wave Application Categories • Maps and Navigation • Modeling Tools and Repositories • Games • 3D Printing • Visualization • Music Videos and Promotion • Education • Photo Editors • Music Visualizers • Vision/Video Processing © Copyright Khronos Group 2013 - Page 11 Google Maps • All rendering (2D and 3D) in Google Maps uses WebGL © Copyright Khronos Group 2013 - Page 12 Microsoft PhotoSynth2 • Demonstrated at Build 2013 http://channel9.msdn.com/Events/Build/2013/4-072 1:50 © Copyright Khronos Group 2013 - Page 13 WebGL on Logan Android Tablet © Copyright Khronos Group 2013 - Page 14 WebGL on Logan Android Tablet © Copyright Khronos Group 2013 - Page 15 OpenGL 3D API Family Tree ES3 is backward compatible Fixed function Programmable vertex so new features can be 3D Pipeline and fragment shaders added incrementally WebGL 2.0 is in development now - will bring OpenGL ES 3.0 functionality to the Web http://www.khronos.org/webgl/public-mailing-list/ http://www.khronos.org/registry/webgl/specs/latest/ http://www.khronos.org/webgl/wiki/Testing/Conformance OpenGL ES 1.1 OpenGL ES 2.0 OpenGL ES 3.0 Content Content Content Mobile 3D WebGL 1.0 WebGL 2.0 OpenGL ES 1.1 OpenGL ES 2.0 OpenGL ES 3.0 ES-Next OpenGL ES 1.0 OpenGL 1.3 OpenGL 1.5 OpenGL 2.0 OpenGL 2.1 OpenGL 3.1 OpenGL 3.3 OpenGL 4.2 OpenGL 4.3 OpenGL 4.4 GL-Next OpenGL 3.0 OpenGL 3.2 OpenGL 4.0 OpenGL 4.1 OpenGL 4.4 is a Desktop 3D superset of DX11 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 © Copyright Khronos Group 2013 - Page 16 OpenGL ES 3.0 Highlights • Better looking, faster performing games and apps – at lower power - Incorporates proven features from OpenGL 3.3 / 4.x - 32-bit integers and floats in shader programs - NPOT, 3D textures, depth textures, texture arrays - Multiple Render Targets for deferred rendering, Occlusion Queries - Instanced Rendering, Transform Feedback … • Make life better for the programmer - Tighter requirements for supported features to reduce implementation variability • Backward compatible with OpenGL ES 2.0 - OpenGL ES 2.0 apps continue to run unmodified • Standardized Texture Compression - #1 developer request! © Copyright Khronos Group 2013 - Page 17 Why Khronos for WebGL? • Hardware API standards must take into account silicon design cycles - Multi-year pipeline of APIs that affect chips that take $100Ms to execute - Deep insights into silicon and driver architectures - Rigorous conformance tests and infrastructure • Khronos is committed to being a good citizen in the larger Web community - Opened Khronos WebGL processes to enable cooperation with web community • Khronos is the industry forum to drive hardware consensus and cooperation - Help create foundational support for higher-level Web standards that access hardware capabilities © Copyright Khronos Group 2013 - Page 18 Leveraging Proven Native APIs into HTML5 • Khronos and W3C liaison - Leverage proven native API investments into the Web - Fast API development and deployment - Designed by the hardware community - Familiar foundation reduces developer learning curve HTML WebCAM(!) Canvas WebVX? WebStream? Camera control and JavaScript Vision Sensor Fusion Processing video processing Camera Control Native Path Rendering JavaScript API shipping, Possible future Native APIs shipping acceleration being developed JavaScript APIs or or Khronos working group or work underway acceleration © Copyright Khronos Group 2013 - Page 19 OpenCL as Parallel Compute Foundation C++ AMP OpenCL HLM WebCL Aparapi River Trail PyOpenCL Harlan Compiler C++ JavaScript binding to Java language Language Python wrapper High level Shevlin Park directives for syntax/compiler OpenCL for initiation extensions for extensions to around language for GPU Uses Clang Fortran C and C++ and LLVM extensions of OpenCL C kernels parallelism JavaScript OpenCL programming OpenCL provides vendor optimized, cross-platform, cross-vendor access to heterogeneous compute resources © Copyright Khronos Group 2013 - Page 20 WebCL • WebCL is a JavaScript binding to OpenCL APIs - Enables initiation of Kernels written in OpenCL C within the browser - Requires a conformant underlying OpenCL on the host system • Leverage heterogeneous computing resources - 3D asset codecs, video codecs and processing, imaging and vision processing - Physics for WebGL games, Online data visualization, Augmented Reality • WebCL 1.0 based on OpenCL 1.1 Embedded Profile: - Implementations may utilize OpenCL 1.1 or 1.2 • WebCL API is designed for complete security - Restriction of some OpenCL native functionality - WebCL kernel validation – similar to WebGL © Copyright Khronos Group 2013 - Page 21 WebCL 1.0 Kernels • HTML data interoperability - <canvas>, <image>, ImageData sources bindable to WebCLBuffer & WebCLImage - <video> tag can be bound to a WebCLImage • Interoperability between WebCL and WebGL - Through GL_SHARING extension • WebCL may support the following extensions - KHR_FP16 — 16-bit float support in kernels - KHR_FP64 — 64-bit float support in kernels • No 3D image support in WebCL 1.0 - May change in future WebCL versions © Copyright Khronos Group 2013 - Page 22 WebCL 1.0 Security • Leverages OpenCL 1.2 robustness/security extensions - Context Termination - to prevent DOS from long running kernels - Memory Initialization - so no leakage from out of bounds memory access • Kernels passed through open source WebCL Kernel Validator - https://github.com/KhronosGroup/webcl-validator - Initializes local and private memory if underlying OpenCL implementation does not implement memory initialization extension