Khronos Overview

Mobile APIs BOF

SIGGRAPH 2010

- …nothing works without them - Jon Peddie, JPR

The Khronos Handheld API Ecosystem - Neil Trevett, NVIDIA

EGL - Working Group report – Jon Leech, Khronos

OpenGL ES - Working Group report – Tom Olson, ARM - KTX: A light-weight file format for OpenGL and OpenGL ES textures – Mark Callow, HI Corp - Real-time, on-device OpenGL ES profiling using gDebugger – Yaki Tabaka, Graphic Remedy - OpenGL ES 2.0: 3D Graphics for In-Vehicle Infotainment – Petri Talala, Symbio

OpenVG - OpenVG Implementations and Applications – Hwanyong Lee, HUONE

© Copyright Khronos Group, 2009 - Page 2 Jon Peddie Research - nothing works without them 3 Jon Peddie Research Jon Peddie Research Jon Peddie Research Agenda Future developments and trends • Status and Momentum of current APIs • Trends in the mobile graphics markets • (Some interesting, others FYI) (You may be surprised, or maybe not) (It’s been a long, hard trail…) - a bunch of overviews Jon Peddie Research Fusix Fusix PX/GL PS3 IRIS Open Open GL PHIGS Plot Plot 10 MESA CORE PIKS GLX GKS PEX ADI API - -- RSX GL Sony/Nvidia A little 30 year History of APIs Tektronix Devloper Autodesk ISO/IEC Du Du Pont ACM ACM ACM SGI SGI SGI Tek 4010 Tek&4010 HG Various Various Various Various Various Various Various Various Various Various Various PS3 OS ReleaseDate 1977 1972 1971 2006 1999 1995 1992 1991 1991 1989 1988 1981 1979 Fahrenheit Fahrenheit Project QuickDraw3D WGF(LDDM) DirectX DirectX 10.1 DirectX 9.0b DirectX DirectX 9.0c DirectX 9.0a DirectX DirectX 9.0 DirectX 8.1 DirectX 8.0 DirectX 7.0 DirectX 6.0 DirectX 5.0 DirectX 4.0 DirectX 3.0 DirectX 2.0 DirectX 1.0 DirectX DirectX 11 DirectX 10 Direct3D Glide 8514 GDI S3G API GL Compaq (ACE) Developer Microsoft Microsoft Microsoft Microsoft Microsoft Microsoft Microsoft Microsoft Microsoft Microsoft Microsoft Microsoft Microsoft Microsoft Microsoft Microsoft Microsoft Microsoft Microsoft Apple 3Dfx IBM S3 MS MS &DOS others 98 98 SE and ME 95 95 4.0 and NT Windows 3.1 NT 4.0 NT4.0 SP3 Windows 7 Vista Vista SP1 MS MS DOS MS DOS MS DOS Apple Vista 2000 2000 OS XP XP XP XP XP XP n/a 98 98 Neverreleased ReleaseDate Sept. Sept. 1999 Sept. 1996 Sept. 1995 Nov. 2006 Aug.2004 Aug.2003 Nov. 2001 Nov. 2000 Aug.1998 Mar. Mar. 2003 Dec. 2002 Feb. Feb. 2008 Oct. 2009 Jun. Jun. 1996 Jul. Jul. 1998 Jul. 1997 2004 1995 1995 1995 1991 1988 Jon Peddie Research API and Graphics Application Libraries Future Trends Hardware Operating System Driver Jon Peddie Research Jon Peddie Research 2,500 2,000 1,500 1,000 500 0 BILLIONS of thingies get suppliers excited… suppliers get BILLIONSof thingies 2009 1,026 598 The Mobile Market Overall Mobile Mobile Devices shipments WW annual (M units) Mobile Mobile (non PC)devices All other phones 2010 1,091 691 2011 1,114 784 2012 1,141 895 2013 1,031 1,129 2014 1,197 1,107 2015 1,335 1,062 Jon Peddie Research ...... Main Main Suppliers Zoran TI ** Telechips ST Samsung ** Renesas * Qualcomm ** Nvidia ** NEC * Marvell * Intel ** Freescale * Broadcom * - Ericsson Ericsson * * KHRONOS Member The Suppliers ( There’s a lot… ) ...... IP Providers ...... Others Vivante * Takumi * MIPS Imagination Technologies ** DMP * ARM ** ARC/ Wondermedia NetLogic Sigma Design Mtekvision MediaTek A nyka Virage /Synopsys * * /VIA* Jon Peddie Research 1,400 1,200 1,000 800 600 400 200 0 Digital Still Cameras (DSC) MIDs &Gadgets Navigation (PND) DAB Tablets &Smartbooks The Sub or Adjacent Markets 2009 2010 TOTAL DEVICESHIPMENTS (M units) 2011 Digital Picture Frames (DPF) e Mobile Game Console Personal Media Devices (PMP) Smartphones - 2012 Books 2013 - Gen Gen 2 2014 2015 Jon Peddie Research MP3 music player MP3 music The Sub or Adjacent Markets Smartphone Tablet PC Netbook Desktop Laptop Consumer Consumer plansfuture purchase(PriceGrabber.com) 0% 5% 10% 15% 20% 25% 30% 35% Jon Peddie Research Thank you Chasing pixels – finding gems [email protected] OpenGL ES Update

Tom Olson, ARM Chair, OpenGL ES Working Group

• OpenGL for mobile devices - Gets rid of redundant / legacy features - Extensions to make it mobile-friendly - All the functionality of the desktop API

• Versions - ES 1.0 / 1.1: fixed function - ES 2.0: shader-based via GLSL ES

• A huge success! - The dominant native 3D API for mobile devices - Widely used in other markets: STB, printers, NAV… - Now shipping > 100M units a year

• Available in all the latest high-end handsets • SDKs, books are now available • Starting point for other APIs, e.g. WebGL

…and the fun is just starting

• Next Generation OpenGL ES - Working Group’s main focus since mid-2009 - Will be released when market needs it - Will not disrupt the growing OpenGL ES 2.0 ecosystem

• ARB / ES Convergence Group - Joint subcommittee of OpenGL and OpenGL ES working groups - Goal is to align roadmaps and avoid unnecessary divergence

• Ecosystem Support - OpenGL ES 2.0 Reference Card - KTX file format

Mark Callow, HI Corporation July 2010

• KTX – Khronos TeXture

http://www.khronos.org/opengles/sdk/tools/KTX/file_format_spec/

• A new file format for storing texture images • Oh no! Not Another image file format? Groan...! • NO • It is a lightweight format specifically for storing OpenGL® textures

• Designed around how textures are loaded in OpenGL. • KTX files contain all the parameters needed for texture loading. • The loader is very simple.

• Image data can be stored in any of the compressed formats supported by OpenGL family APIs and extensions - e.g. ETC1, ATITC, S3TC, etc. • Or can be stored uncompressed. • A single .ktx file can contain any type of GL texture from a 1D texture to an array texture with all mipmap levels.

} my_mipmap.ktx

} my_cubemap.ktx

• libktx - Library of functions for writing ktx files and loading textures from them - Loader decompresses ETC1 compressed textures when hardware does not support ETC1. - Easy to use. - E.g. app. can change from using a base level 2D texture to providing a full mipmap pyramid simply by changing the file it loads. • toktx - Create KTX files from a list of .pam, .pgm or .ppm files - Does not compress images • Both are open source under BSD-like license

© Copyright Khronos Group, 2010 - Page 25 • Specification, downloads, source and documentation can all be found on the KTX home page in new OpenGL ES SDK at: http://www.khronos.org/opengles/sdk/tools/KTX/

• etcpack - Creates KTX files for RGB textures compressed with ETC1. • Available from Ericsson’s developer site: http://devtools.ericsson.com/etc

• etcpack and toktx by default map the first pixel in the input file to s0, t0. • First pixel in a netpbm file is logical top-left of the image - so image will be inverted if displayed with GL defaults • This behavior was chosen to match existing compression tools

• etcpack and toktx provide options to change the mapping • toktx inserts a KTXorientation key-value pair into the file that indicates the logical orientation of the image allowing smart applications to display the image correctly.

• We expect other compression tools such as The Compressionator to add an option to output the KTX format. • Applications will then be able to use a single file format across all vendors proprietary texture compression schemes.

Yaki Tebeka, CTO Graphic Remedy

• Today’s smart phones and devices are equipped with GPUs, offering high performance graphics and low power consumption • Implementations are available from PowerVR, ARM, Qualcomm, NVIDIA, … • The OpenGL ES API is widely used over these platforms

• Debugging and profiling 3D applications are hard and time consuming tasks • Companies find it extremely hard to deliver robust and bug-free 3D applications • It is almost impossible to optimize a 3D application to fully utilize the graphic system performance

© 2004 – 2009 Graphic Remedy. All Rights Reserved • OpenGL, OpenGL ES and Open CL Debugger, Profiler and Memory Analyzer • Exposes the internal graphic system information needed to find bugs, optimize render performance and memory usage

© 2004 – 2009 Graphic Remedy. All Rights Reserved Automatic errors detection mechanism gDEBugger can automatically break on: • OpenGL ES errors • gDEBugger detected errors: a unique and more comprehensive errors detection mechanism • Graphic memory leaks • Redundant functions • And more

• View the call stack and source code that led to the error / OpenGL ES function call

• Displays the number of times each OpenGL ES function was called • Display redundant state changes • Break on redundant state change

Displays performance counters graphs from different sources • Operating Systems: Windows, Mac OS X and Linux • GPU and driver: SGX Graphic driver, etc • gDEBugger OpenGL ES Server

• Turn off the graphic pipeline stages one after the other • If the performance improves when turning off a certain stage, you have found a graphic pipeline bottleneck

• Eliminate draw commands • Eliminate raster operations • Eliminate fixed pipeline lights • Eliminate texture data fetch operations • Eliminate geometry shader operations • Eliminate fragment shader operations

Displays all texture types, FBOs, pbuffers, Depth, Stencil, VBOs, etc.

© 2004 – 2009 Graphic Remedy. All Rights Reserved Shader's Source Code Editor • Displays shader's source code • Displays programs active uniform values • Edit shader's source code, re-compile, link and validate "on the fly"

• Displays information about graphic memory allocated objects • Displays graphic memory leaks • Use the Object Creation Calls Stack to see the scenario that led to each object's creation

• Displays a statistical view of application activities • Warns about none best practice usage • Suggest best practice alternatives

Vehicle SERIOUSInfotainment ABOUT SOFTWARE (IVI)

Confidential July 29th, 2010 Petri Talala,9/3/2010 Symbio 46 Audi V8.... 20 Years Ago

9/3/2010 Audi R8... Today

9/3/2010 Audi e-Tron Concept 2010

9/3/2010 In Vehicle Infotainment (IVI) Direction

• IVI systems are becoming a key differentiator between automotive OEMs • Huge need for graphics design and programming expertise • IVI systems are moving towards open platforms – Genivi platform – Ubuntu – Android • IVIs heavily based on same technology as mobile graphics – In majority cases… same HW platforms, same OS base, same tools – OpenGL ES x.x, OpenVG for graphics 9/3/2010 Use Cases

• Different forms of graphics and multimedia acceleration are becoming more common in automotive IVI systems – UI’s for IVI and entertainment systems – Instrumentation • Gauges and information screens rendered with real-time graphics • Heads Up Displays – Video compress/decompress • Parking cameras – Video postprocessing • Graphics overlay • Night vision • Navigation – Content playback for entertainment • Video decode • Games

9/3/2010 Use cases

Instrumentation cluster: • Gauges (2D/OpenVG/OpenGL ES 2.0) • Information system UI (2D/OpenVG/OpenGL ES 2.0) • HUD (graphics overlay) • Night vision (graphics overlay, composition, video postprocessing)

Entertainment cluster: • UI (content browsing, climate control, etc) • Video decode for 1 or more screens from 1 or more sources • Multiple application content

9/3/2010 3D Navigation Demo Source: Kishonti Informatics (www.navigenie.com)

9/3/2010 Ecosystem Overview

HW Platforms Content Adaptation Work Creation OpenGL ES 2.0 Toolchain: • Customization -> Model, animate • Optimized -> Export / convert booting -> Integrate • Optimized -> Run it on target usability OS / • Scheduling Frameworks • Layering • Smooth animations • Smooth transitions • Automotive Grade

9/3/2010 Few Words about

• Non-profit industry alliance committed to IVI reference platform based on open source IVI community – Members such as ARM, BMW, Continental, Freescale, GM, Intel, Marvell, NVIDIA, Renesas, Symbio, and Texas Instruments • Genivi platform is a common SW architecture – Linux based core services, middleware and open application layer API – OpenGL ES 2.0 is preferred for graphics – Allows new compelling automotive applications and new business models

9/3/2010 Content Creation Example

HW Platform: NVIDIA Tegra2 OS: Linux OpenGL ES 2.0 based content Full HD resolution FPS > 30 9/3/2010 Final Notes to Takeaway

1. IVI market is a good business opportunity with potentially long term commitments 2. Automotive OEMs and Tier-1 suppliers urgently need graphics expertise 3. IVIs will be mainly based on familiar HW platforms, OS, OGLES 2.0 graphics from mobile industry 4. Usage of familiar Digital Content Creation Tools 5. Importance to select a trusted and experienced SW Integration Partner

9/3/2010 www.symbio.com

Symbio’s value is based on an extensive track record plus years of experience in demanding software integration and user interface implementation projects in mobile and embedded industry. We know how to optimize graphics rendering solutions for the best possible visual performance and user experience, while minimizing power consumption through an optimized software stack.

Petri Talala Director, Business Development

[email protected] Mobile: +1 408 888 8927

Confidential 9/3/2010 Applications and Implementations

Hwanyong LEE CTO and Technical Marketing Director HUONE 60 The Khronos Ecosystem

3D asset interchange, WebGL 3D web content and 3D Asset Interchange 3D Web Content Mobile OS Abstraction Format Mobile OS portability

High-level streaming media and audio High-level Steaming Media Recording and Playback High-level Enhanced Audio frameworks

Surface and Low-level silicon Embedded 3D synch abstraction acceleration of graphics, Cross platform desktop 3D media and parallel computation Heterogeneous Streaming Media and Parallel Programming Image Processing Vector 2D

Hundreds of man years invested by industry experts to create a coordinated visual computing ecosystem for accelerated parallel computation, 3D, video, audio and image processing on desktop, embedded and mobile systems 61 OpenVG

 Royalty-free open standard API

 Low-level 2D vector graphics rendering API

 OpenGL-style programming model

 Advanced feature set enables  SVG,  Flash,  PDF, Postscript,  Java (JSR 287, 271, 226)  etc. Applications and UI

 Portable content SVG/Flash/ Font Packages etc..  Map Applications

 Hardware Acceleration Hardware Acceleration 62 OpenVG Rendering Pipeline 63 OpenVG with Native Graphics Processor

 CPU sending data and commands to OVG HW.  OpenVG rendering pipeline is in the hardware

CPU Native Vector Graphics Hardware 64 Software Implementation • Many implementation alternatives • Everything is processed on the CPU (or +FPU) - Power consumption is often the CPU maximum power

CPU 65 OpenVG with a 3D GPU (tessellation approach)

• 3D hardware is not tailored for tessellating arbitrary paths to triangles which causes significant pre-processing to the CPU

• Frame-rates can be discontinuous when tessellation is computed infrequently

CPU Tessellation- based implementations require complex polygon GPU tessellation in the software driver 66 OpenVG with a 3D GPU (the stencil approach)

• Path subdivision is computationally simpler than path tessellation

• Stencil buffer rasterization consumes a lot of memory bandwidth

CPU

Triangles are rasterized into GPU stencil buffer to determine path areas Stencil-based implementations require path subdivision in the software driver 67 OpenVG with Multimedia Hardware

• Using double buffer, Fast image processing, Bilblt

Most of rasterization CPU (Vector Graphics)

Selective hardware acceleration 68 Implementations

 Conformant Products

 Takumi GV series

 DMP dandelion VG,

 Vivente GC series

 ARM Mali series

 Inst. for Information Industry DeltaVG

 Imagination PowerVR SGX, MBX with VGP

 Broadcom – VideoCore III

 HUONE – AlexVG engine

 NVIDIA – NVIDIA AP 69 Implementations

 Not conformant but based on OpenVG

 ShivaVG – Open Source OpenVG on OpenGL

 Mazatech – AmanithVG (Software and on OpenGL ES)

 Hooked Wireless – Hooked OpenVG (on OpenGL ES)

 HUONE – AlexVG forge (on OpenGL ES) 70 Applications

 GUI

 Mobile Devices GUI – Samsung

 Bitmap Graphics, Flash lite, OpenVG, Flash lite on OpenVG

 Qt Rendering on OpenVG

 Mapping

 GPS map drawing

 Flash lite player

 SVG Mobile Player

 MMS, DCD, MPEG4-LASeR, WAP

 Java Bindings - JSR226, JSR271, JSR287 71 OpenVG Roadmap

 OpenVG Inari – Future Version

 Investigations

 Various kind of features including “shading language”

 OpenVG Express – A Streamlined, refactored

 Investigations

 Add no new functionality

 Reduce HW and SW complexity

 Fix Color Space

 Objectives - Supports SVGT, Flash Lite

 CTS - Making it more effective and more accessible to new vendor 72 BOOK & Study Materials

 AlexVG™ StudyKit – Sep. 2010 (www.alexvg.com)

 Book “OpenVG Programming”

 Korean Version - Jan. 2010

 English Version – Jan. 2011?

 Quick Ref. Card

 KHRONOS Booth