Khronos Overview Taiwan Feb 2012
Total Page:16
File Type:pdf, Size:1020Kb
Khronos Overview Taiwan, February 2012 © Copyright Khronos Group, 2012 - Page 1 Khronos - Connecting Software to Silicon • Creating open, royalty-free API standards - Focus on graphics, dynamic media, compute and sensor hardware • Low-level - just above raw silicon - “Foundation” functionality needed on every platform • Safe forum for industry cooperation - „By the industry for the industry‟ - Open to any company to join - IP framework to protect members and industry APIs enable software developers to turn silicon functionality into rich end user experiences © Copyright Khronos Group, 2012 - Page 2 Khronos API Ecosystem - Trends High-end API Breakthrough apps embrace technology is created on mobility‟s strengths – not just treat high-end platforms phones as small PCs - will need complex, interoperating APIs e.g. Augmented Reality As platforms diversify – mobile, Mobile is the new platform for apps TV, embedded – HTML5 will innovation. Mobile APIs are needed become increasingly important to unlock hardware potential while as a universal app platform conserving battery life © Copyright Khronos Group, 2012 - Page 3 Apple Over 100 members – any company worldwide is welcome to join Board of Promoters © Copyright Khronos Group, 2012 - Page 4 Khronos Ecosystem of Standards Advanced Audio Camera, Images and Streaming Media Computer Vision Surface Hardware accelerated Management vision functions 3D Digital Asset Exchange format Parallel Computing Embedded and Unified Sensor and Mobile 3D Input Processing Vector 2D Cross platform Web desktop 3D Compute Plugin-free 3D Web Content Khronos creates royalty-free specifications to meet real market needs and helps drive industry adoption across multiple platforms © Copyright Khronos Group, 2012 - Page 5 Khronos Working Group Process Academic Members - Participation in WGs Working Groups (WGs) Contributor Members One working group per API - Participation and vote in WGs Promoter Members - Participation and vote in WGs - Board seat for strategy, budget and spec ratification $$$ $ Conformance Tests Ratified SDKs, Sample, Ref Cards Educator Guidelines and Adopters Program Specifications and Man Pages and Courseware Feedback Members Wider Industry $ Adopters Developers Educators Build conformant Develop applications Collaboration for Khronos implementation and products using the APIs API Education © Copyright Khronos Group, 2012 - Page 6 Milestones This Year at Khronos WebCL Computer Vision OpenMAX AL 1.1 Working Group Working Group Spec Released Announced Announced OpenSL ES 1.1 OpenGL 4.2 EGLStreams Spec Released Spec Released Spec Release Mar11 Nov11 Jan11 Aug11 Dec11 WebGL 1.0 OpenCL 1.2 Spec Released Spec Release StreamInput Working Group Announced COLLADA 1.4 Conformance Tests Released © Copyright Khronos Group, 2012 - Page 7 OpenGL for Each GPU Generation Tessellation and Compute „Shape 4.X realism‟ Geometry Shaders 3.X Vertex and Fragment Shaders 2.X Fixed Function 1.X „Surface realism‟ © Copyright Khronos Group, 2012 - Page 8 Accelerating OpenGL Innovation Bringing state-of-the- art functionality to cross-platform graphics OpenGL 4.1 OpenGL 3.3/4.0 OpenGL 3.2 OpenGL 3.1 OpenGL 2.0 OpenGL 2.1 OpenGL 3.0 2004 2005 2006 2007 2008 2009 2010 2011 DirectX 9.0c DirectX 10.0 DirectX 10.1 DirectX 11 © Copyright Khronos Group, 2012 - Page 9 OpenGL ES Adoption • OpenGL for embedded and mobile devices - Eliminates redundant and legacy features - Adds extensions to make it mobile-friendly • The dominant 3D API for mobile devices - Widely adopted for STB, DTV, automotive,… - Hundreds and hundreds of millions shipped • OpenGL ES 2.0 - Fully programmable pipeline • Runs high-end content and engines - UE3, Unity, Unigine, Rage © Copyright Khronos Group, 2012 - Page 10 Processor Parallelism CPUs GPUs Multiple cores driving Emerging Increasingly general performance increases Intersection purpose data-parallel computing Multi- Heterogeneous Graphics processor Computing APIs and programming Shading – e.g. OpenMP Languages OpenCL is a programming framework for heterogeneous compute resources © Copyright Khronos Group, 2012 - Page 11 OpenCL – Heterogeneous Computing • Framework for programming diverse parallel computing resources in a system • Platform Layer API - Query, select and initialize compute devices • Kernel Language Specification - Subset of ISO C99 with language extensions • Runtime API - Execute compute kernels – gather results • OpenCL has Embedded profile - No need for a separate “ES” spec © Copyright Khronos Khronos 2009 Group, 2012 - Page 12 OpenCL 1.2 Announced in December • Significant updates - Khronos being responsive to developer requests - Updated OpenCL 1.2 conformance tests available • Backward compatible upgrade to OpenCL 1.1 - OpenCL 1.2 will run any OpenCL 1.0 and OpenCL 1.1 programs - OpenCL 1.2 platform can contain 1.0, 1.1 and 1.2 devices - Maintains embedded profile for mobile and embedded devices • Significant functionality added in response to developer demands - Partition Devices into sub devices - Separate compilation and linking of programs - Enhanced Image Support - Custom devices and built-in kernels - DX9 and DX11 Surface Sharing - Installable Client Drivers (optional) © Copyright Khronos Group, 2012 - Page 13 Video and Audio Acceleration • OpenSL ES 1.1 and OpenMAX AL 1.1 launched in January 2011 - Additional functionality including improved buffer queuing - OpenMAX AL Digital TV Extension - technology-neutral access to Digital TV • OpenMAX AL is designed as application API - OpenMAX IL is for system integration – more flexibility, less portability • OpenMAX AL enables key image, camera and video use cases - Allows optimal hardware acceleration with application portability Advanced image capture and photography HD content playback with robust DRM HD video teleconferencing Augmented Reality © Copyright Khronos Group, 2012 - Page 14 OpenSL ES – Advanced Audio • OpenSL ES does for audio what OpenGL ES does for graphics - Advanced audio functionality from simple playback to 3D audio • Object-based native audio API for simplicity and high performance - Reduces development time • Same API regardless of underlying implementation - Software or hardware accelerated • Cross OS portability - Preserves application investment © Copyright Khronos Group, 2012 - Page 15 OpenSL ES and OpenMAX AL Advanced Audio Multimedia Video 3D Audio playback Audio Video Playback recording Audio Effects Radio and RDS Audio Advanced Recording Camera MIDI Basic Image capture Buffer MIDI & display queues Common Object-oriented API Framework © Copyright Khronos Group, 2012 - Page 16 © Copyright Khronos Group, 2012 - Page 17 Visual-based Augmented Reality Camera video stream sent to the compositor Camera 3D images used augmentations to track the composited with camera‟s video stream location and orientation Camera 3D Augmentation Tracking Camera-to-scene transform locks Rendering the 3D rendering to the real world © Copyright Khronos Group, 2012 - Page 18 EGLStream – Video/Graphics Interop OpenMAX AL Media Player EGLStreams enables and hides details of OpenGL ES GL_TEXTURE_EXTERNAL is the EGLStream video frame transport. Enables multiple is the EGLStream “Consumer” and “Producer” and controls buffering modes for different uses cases converts video format into RGB production of frames. eg: FIFO and explicit latch/release OpenGL ES texture Camera OpenMAX AL OpenGL ES File MEDIA PLAYER EGLStream GL_TEXTURE_EXTERNAL URL Etc. Object ? © Copyright Khronos Group, 2012 - Page 19 StreamInput Connects Sensors to Apps Apps request semantic sensor information StreamInput defines list of possible semantic requests “Am I in an elevator?” “Give me gestures and face position” Standardized Node Universal Intercommunication Timestamps Input Input Input Device Device Device Filter Filter Node Node Advanced Sensors Everywhere Standard cameras, depth cameras Apps Need Sophisticated motion and position, touch, microphones Filter App Access to Sensor Data wireless controllers Node Without coding to specific systems or sensor hardware SHOULD NOT FORCE Sensor graph created to provide sensor information APPLICATIONS TO StreamInput defines graph creation API and node interconnects ACCESS INDIVIDUAL Low-level sensor processing encapsulated in nodes – unleashes fusion innovation SENSORS Apps gain „magical‟ situational awareness © Copyright Khronos Group, 2012 - Page 20 Current StreamInput Participants • Aiming for specification release in 2012 © Copyright Khronos Group, 2012 - Page 21 Khronos Computer Vision Working Group • Vision Hardware Acceleration Layer - Enable hardware vendors to implement accelerated imaging and vision algorithms • CV HAL* can be used by high-level Application libraries or applications directly - Primary focus on enabling mobile and embedded systems OpenCV open Other higher-level • OpenCV is widely used open source source library CV libraries library for vision projects - Future version will leverage CV HAL CV HAL - * CV Hal is placeholder name Open source sample Hardware vendor implementation? implementations © Copyright Khronos Group, 2012 - Page 22 Possible Implementation of Vision Stack Semantics and fusion of Implement camera and positional sensors StreamInput vision sensor modules with OpenCV High-level computer vision library Accelerate OpenCV library with CV HAL Camera input from Functions Data and event OpenMAX AL or Accelerated computer interop with CL / other camera GL / ES for display subsystems CV HAL vision algorithms and compute Use OpenCL processing to implement CV HAL with parallel execution