Khronos Overview The State of the Art in Open Standards for Visual Computing Neil Trevett Khronos President Vice President Mobile Content, NVIDIA © Copyright Khronos Group 2013 - Page 1 Khronos Connects Software to Silicon ROYALTY-FREE, OPEN STANDARD APIs for advanced hardware acceleration Low level silicon to software interfaces needed on every platform Graphics, video, audio, compute, vision, sensor and camera processing Defines the forward looking roadmap for the silicon community Shipping on billions of devices across multiple operating systems Rigorous conformance tests for cross-vendor consistency Khronos is OPEN for any company to join and participate Acceleration APIs BY the Industry FOR the Industry © Copyright Khronos Group 2013 - Page 2 Making a Difference – One API at a Time Well over 1 BILLION people are using what the Khronos members have created together - Every Day… © Copyright Khronos Group 2013 - Page 3 Khronos Standards glTF cooperation with MPEG for 3D Asset Compression! 3D Asset Handling - Advanced Authoring pipelines - 3D Asset Transmission Format with OpenCL 2.0 Finalized! streaming and compression Visual Computing - Object and Terrain Visualization - Advanced scene construction Over 100 companies defining royalty-free APIs to connect software to silicon Camera Control API OpenVX 1.0 Acceleration in the Browser Provisional - WebGL for 3D in browsers Released! - WebCL – Heterogeneous Computing for the web Sensor Processing - Mobile Vision Acceleration WebGL and WebCL - On-device Sensor Fusion Momentum! © Copyright Khronos Group 2013 - Page 4 OpenCL Milestones • 24 month cadence for major OpenCL 2.0 update - Slightly longer than 18 month cadence between versions of OpenCL 1.X • Significant feedback from the developer community on Provisional Specification - Many suggestions were incorporated into the final 2.0 specification - Other feedback will be considered for future specification versions OpenCL 1.1 OpenCL 2.0 Specification and Provisional Specification conformance tests released for public released review Dec08 Nov11 Nov13 Jun10 Jul13 OpenCL 1.0 released. OpenCL 1.2 OpenCL 2.0 Conformance tests Specification and Specification finalized released Dec08 conformance tests and conformance released tests released © Copyright Khronos Group 2013 - Page 5 Key OpenCL 2.0 Features • Shared Virtual Memory - Host and device kernels can directly share complex, pointer-containing data structures such as trees and linked lists, providing significant programming flexibility and eliminating costly data transfers between host and devices • Nested Parallelism - Device kernels can enqueue kernels to the same device with no host interaction, enabling flexible work scheduling paradigms and avoiding the need to transfer execution control and data between the device and host, often significantly offloading host processor bottlenecks • Generic Address Space - Functions can be written without specifying a named address space for arguments, especially useful for those arguments that are declared to be a pointer to a type, eliminating the need for multiple functions to be written for each named address space used in an application © Copyright Khronos Group 2013 - Page 6 Broad OpenCL Implementer Adoption • Multiple conformant implementations shipping on desktop and mobile - For CPUs and GPUs on multiple OS • Android ICD extension released in latest extension specification - OpenCL implementations can be discovered and loaded as a shared object • Multiple implementations shipping in Android NDK - ARM, Imagination, Vivante, Qualcomm, Samsung … © Copyright Khronos Group 2013 - Page 7 OpenCL as Parallel Compute Foundation • 100+ tool chains and languages leveraging OpenCL - Heterogeneous solutions emerging for the most popular programming languages C++ AMP OpenCL HLM WebCL Aparapi River Trail PyOpenCL Harlan Shevlin Park C++ JavaScript binding to Java language Language Python wrapper High level Compiler Uses Clang syntax/compiler OpenCL for initiation extensions for extensions to around language for GPU directives for and LLVM extensions of OpenCL C kernels parallelism JavaScript OpenCL programming Fortran C and C++ OpenCL provides vendor optimized, cross-platform, cross-vendor access to heterogeneous compute resources © Copyright Khronos Group 2013 - Page 8 Widespread Developers Leveraging OpenCL • Broad uptake of OpenCL in commercial applications - For desktop and increasingly mobile apps • “OpenCL” on Sourceforge, Github, Google Code, BitBucket finds over 2,000 projects - x264 - Handbrake - FFMPEG - JPEG - VLC - OpenCV - GIMP - ImageMagick - IrfanView - Hadoop, Memcched - Aparapi – A parallel API (for Java) - Bolt – a Unified Heterogeneous Library - Sumatra – next generation of compute enabled Java - WinZip - Crypto++ - Bullet physics library - Etc. Etc. © Copyright Khronos Group 2013 - Page 9 OpenCL Academic Traction • OpenCL at over 100 Universities Worldwide Teaching multi-faceted programming courses - Research with top-tier Universities globally • Complete University Kits available - Presentation w/instructor & speaker notes - Example code, & sample application • Growing textbook ecosystem - US, Japan, Europe, China and India • Number of papers referencing OpenCL on Google Scholar is growing rapidly - Over 2000 papers in 2012 • Commercial OpenCL training courses - http://www.accelereyes.com/services/training http://developer.amd.com/Resources/library/Pages/default.aspx © Copyright Khronos Group 2013 - Page 10 Leveraging Proven Native APIs into HTML5 • Khronos and W3C liaison - Leverage proven native API investments into the Web - Fast API development and deployment - Designed by the hardware community - Familiar foundation reduces developer learning curve HTML WebCAM(!) Canvas WebVX? WebStream? Camera control and JavaScript Vision Sensor Fusion Processing video processing Camera Control Native Path Rendering JavaScript API shipping, Possible future Native APIs shipping acceleration being developed JavaScript APIs or or Khronos working group or work underway acceleration © Copyright Khronos Group 2013 - Page 11 Mobile Web is a Real Time Application 2048x1536 3100K Pixels 326 DPI 1024x768 786K Pixels 132 DPI + 320x480 = 153K Pixels 163 DPI Apple Apple Apple iPhone iPad iPad Mini Buttery smooth touch In 5 years the number of Need GPU interaction needs continuous pixels to process on Acceleration for 60Hz updates mobile screens has gone everything Web! up by factor of TWENTY © Copyright Khronos Group 2013 - Page 12 WebGL Availability in Browsers Much WebGL content uses three.js library: http://threejs.org/ - Microsoft – “where you have IE11, you have WebGL – turned on by default and working all the time” - Microsoft - WebGL also enabled for Windows applications - web app framework and web view - Apple - WebGL must be explicitly turned on MAC Safari and only exposed on iOS for iAds - Chrome OS - WebGL is the only cross-platform API to program the GPU - Google IO announcement - Chrome on Android will soon launch with WebGL © Copyright Khronos Group 2013 - Page 13 Microsoft PhotoSynth2 • Demonstrated at Build 2013 http://channel9.msdn.com/Events/Build/2013/4-072 1:50 © Copyright Khronos Group 2013 - Page 14 Cross-OS Portability HTML5 provides cross HTML/CSS HTML/CSS HTML/CSS platform portability. GPU accessibility through WebGL available soon on ~90% mobile systems Dalvik Preferred development SDK Objective C C# environments not (Java) designed for portability Native code is portable- but apps must cope with C/C++ DirectX different available APIs and libraries © Copyright Khronos Group 2013 - Page 15 OpenGL 3D API Family Tree ES3 is backward compatible Fixed function Programmable vertex so new features can be 3D Pipeline and fragment shaders added incrementally WebGL 2.0 is in development now - will bring OpenGL ES 3.0 functionality to the Web http://www.khronos.org/webgl/public-mailing-list/ http://www.khronos.org/registry/webgl/specs/latest/ http://www.khronos.org/webgl/wiki/Testing/Conformance OpenGL ES 1.1 OpenGL ES 2.0 OpenGL ES 3.0 Content Content Content Mobile 3D WebGL 1.0 WebGL 2.0 OpenGL ES 1.1 OpenGL ES 2.0 OpenGL ES 3.0 ES-Next OpenGL ES 1.0 OpenGL 1.3 OpenGL 1.5 OpenGL 2.0 OpenGL 2.1 OpenGL 3.1 OpenGL 3.3 OpenGL 4.2 OpenGL 4.3 OpenGL 4.4 GL-Next OpenGL 3.0 OpenGL 3.2 OpenGL 4.0 OpenGL 4.1 OpenGL 4.4 is a Desktop 3D superset of DX11 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 © Copyright Khronos Group 2013 - Page 16 OpenGL ES 3.0 Highlights • Better looking, faster performing games and apps – at lower power - Incorporates proven features from OpenGL 3.3 / 4.x - 32-bit integers and floats in shader programs - NPOT, 3D textures, depth textures, texture arrays - Multiple Render Targets for deferred rendering, Occlusion Queries - Instanced Rendering, Transform Feedback … • Make life better for the programmer - Tighter requirements for supported features to reduce implementation variability • Backward compatible with OpenGL ES 2.0 - OpenGL ES 2.0 apps continue to run unmodified • Standardized Texture Compression - #1 developer request! © Copyright Khronos Group 2013 - Page 17 3D Needs a Transmission Format! • Compression and streaming of 3D assets becoming essential - Mobile and connected devices need access to increasingly large asset databases • 3D is the last media type to define a compressed format - 3D is more complex – diverse asset types and use cases • Needs to be royalty-free - Avoid an ‘internet video codec war’ scenario • Eventually enable hardware implementations of successful codecs - High-performance