Mobile APIs BOF SIGGRAPH 2010 © Copyright Khronos Group, 2009 - Page 1 Agenda - …nothing works without them - Jon Peddie, JPR The Khronos Handheld API Ecosystem - Neil Trevett, NVIDIA EGL - Working Group report – Jon Leech, Khronos OpenGL ES - Working Group report – Tom Olson, ARM - KTX: A light-weight file format for OpenGL and OpenGL ES textures – Mark Callow, HI Corp - Real-time, on-device OpenGL ES profiling using gDebugger – Yaki Tabaka, Graphic Remedy - OpenGL ES 2.0: 3D Graphics for In-Vehicle Infotainment – Petri Talala, Symbio OpenVG - OpenVG Implementations and Applications – Hwanyong Lee, HUONE © Copyright Khronos Group, 2009 - Page 2 - nothing works without them 3 JonPeddieResearch Jon Peddie Research Jon Peddie Research Peddie Jon Agenda - a bunch of overviews Status and Momentum of current APIs • (It’s been a long, hard trail…) Future developments and trends • (You may be surprised, or maybe not) Trends in the mobile graphics markets • (Some interesting, others FYI) JonPeddieResearch API Devloper OS Release Date API Developer OS Release Date Plot 10 Tektronix Tek 4010 & HG 1971 CORE AACM littleVarious 30 year1972 History of APIs GKS ACM Various 1977 PHIGS ACM Various 1979 IRIS--GL SGI Various 1981 ADI Autodesk Various 1988 8514 IBM MS DOS 1988 PEX Various 1989 S3G S3 MS DOS 1991 Fusix PX/GL Du Pont Various 1991 GL Compaq (ACE) MESA Various 1991 Glide 3Dfx MS DOS & others 1995 Open GL SGI Various 1992 QuickDraw 3D Apple Apple 1995 Direct3D Microsoft MS DOS 1995 PIKS ISO/IEC Various 1995 DirectX 1.0 Microsoft Windows 3.1 Sept. 1995 DirectX 2.0 Microsoft 95 and NT 4.0 Jun. 1996 DirectX 3.0 Microsoft NT 4.0 SP3 Sept. 1996 Fahrenheit Project DirectX 4.0 Microsoft n/a Never released DirectX 5.0 Microsoft 98 Jul. 1997 GDI Microsoft 98 Jul. 1998 DirectX 6.0 Microsoft 98 SE and ME Aug. 1998 GLX SGI Various 1999 DirectX 7.0 Microsoft 2000 Sept. 1999 DirectX 8.0 Microsoft 2000 Nov. 2000 DirectX 8.1 Microsoft XP Nov. 2001 DirectX 9.0 Microsoft XP Dec. 2002 DirectX 9.0a Microsoft XP Mar. 2003 DirectX 9.0b Microsoft XP Aug. 2003 WGF (LDDM) Microsoft XP 2004 DirectX 9.0c Microsoft XP Aug. 2004 DirectX 10 Microsoft Vista Nov. 2006 PS3-RSX Sony/Nvidia PS3 2006 DirectX 10.1 Microsoft Vista SP1 Feb. 2008 JonPeddieResearch DirectX 11 Microsoft Windows 7 Oct. 2009 Future Trends Application Operating System API and Graphics Driver Libraries Hardware JonPeddieResearch Jon Peddie Research The Mobile Market Overall Mobile Devices WW annual shipments (M units) 2,500 All other phones 2,000 Mobile (non PC) devices 1,062 1,500 1,107 1,129 1,141 1,114 1,091 1,026 1,000 500 598 691 784 895 1,031 1,197 1,335 0 2009 2010 2011 2012 2013 2014 2015 BILLIONS of thingies get suppliers excited… JonPeddieResearch The Suppliers ( There’s a lot… ) Main Suppliers Others . Broadcom * . Anyka . Freescale * . MediaTek . Intel ** . Mtekvision * . Marvell * . Sigma Design . NEC * . NetLogic * . Nvidia ** . Wondermedia/VIA * . Qualcomm ** . Renesas * IP Providers . Samsung ** . ARC/Virage/Synopsys . ST-Ericsson * . ARM ** . Telechips . DMP * . TI ** . Imagination Technologies ** . Zoran . MIPS * KHRONOS Member . Takumi * . Vivante * JonPeddieResearch The Sub or Adjacent Markets TOTAL DEVICE SHIPMENTS (M units) Tablets & Smartbooks Smartphones 1,400 DAB Personal Media Devices (PMP) Navigation (PND) Mobile Game Console - Gen 2 1,200 MIDs & Gadgets e-Books 1,000 Digital Still Cameras (DSC) Digital Picture Frames (DPF) 800 600 400 200 0 2009 2010 2011 2012 2013 2014 2015 JonPeddieResearch The Sub or Adjacent Markets Consumer future purchase plans (PriceGrabber.com) Tablet PC Netbook Desktop MP3 music player Smartphone Laptop 0% 5% 10% 15% 20% 25% 30% 35% JonPeddieResearch Thank you Chasing pixels – finding gems [email protected] JonPeddieResearch OpenGL ES Update Tom Olson, ARM Chair, OpenGL ES Working Group © Copyright Khronos Group, 2009 - Page 14 What is OpenGL ES? • OpenGL for mobile devices - Gets rid of redundant / legacy features - Extensions to make it mobile-friendly - All the functionality of the desktop API • Versions - ES 1.0 / 1.1: fixed function - ES 2.0: shader-based via GLSL ES • A huge success! - The dominant native 3D API for mobile devices - Widely used in other markets: STB, printers, NAV… - Now shipping > 100M units a year © Copyright Khronos Group, 2009 - Page 15 OpenGL ES 2.0 is here! • Available in all the latest high-end handsets • SDKs, books are now available • Starting point for other APIs, e.g. WebGL …and the fun is just starting © Copyright Khronos Group, 2009 - Page 16 Working Group Activities • Next Generation OpenGL ES - Working Group’s main focus since mid-2009 - Will be released when market needs it - Will not disrupt the growing OpenGL ES 2.0 ecosystem • ARB / ES Convergence Group - Joint subcommittee of OpenGL and OpenGL ES working groups - Goal is to align roadmaps and avoid unnecessary divergence • Ecosystem Support - OpenGL ES 2.0 Reference Card - KTX file format © Copyright Khronos Group, 2009 - Page 17 KTX Mark Callow, HI Corporation July 2010 © Copyright Khronos Group, 2010 - Page 18 The Khronos Group Announces • KTX – Khronos TeXture http://www.khronos.org/opengles/sdk/tools/KTX/file_format_spec/ © Copyright Khronos Group, 2010 - Page 19 KTX • A new file format for storing texture images • Oh no! Not Another image file format? Groan...! • NO • It is a lightweight format specifically for storing OpenGL® textures © Copyright Khronos Group, 2010 - Page 20 KTX • Designed around how textures are loaded in OpenGL. • KTX files contain all the parameters needed for texture loading. • The loader is very simple. © Copyright Khronos Group, 2010 - Page 21 Compression • Image data can be stored in any of the compressed formats supported by OpenGL family APIs and extensions - e.g. ETC1, ATITC, S3TC, etc. • Or can be stored uncompressed. • A single .ktx file can contain any type of GL texture from a 1D texture to an array texture with all mipmap levels. © Copyright Khronos Group, 2010 - Page 22 E.G. Mipmap Pyramid in Single KTX file } my_mipmap.ktx © Copyright Khronos Group, 2010 - Page 23 E.G. Cube Map + Mip Levels in Single KTX File } my_cubemap.ktx © Copyright Khronos Group, 2010 - Page 24 Tools • libktx - Library of functions for writing ktx files and loading textures from them - Loader decompresses ETC1 compressed textures when hardware does not support ETC1. - Easy to use. - E.g. app. can change from using a base level 2D texture to providing a full mipmap pyramid simply by changing the file it loads. • toktx - Create KTX files from a list of .pam, .pgm or .ppm files - Does not compress images • Both are open source under BSD-like license © Copyright Khronos Group, 2010 - Page 25 • Specification, downloads, source and documentation can all be found on the KTX home page in new OpenGL ES SDK at: http://www.khronos.org/opengles/sdk/tools/KTX/ © Copyright Khronos Group, 2010 - Page 26 More Tools • etcpack - Creates KTX files for RGB textures compressed with ETC1. • Available from Ericsson’s developer site: http://devtools.ericsson.com/etc © Copyright Khronos Group, 2010 - Page 27 A Note About Orientation • etcpack and toktx by default map the first pixel in the input file to s0, t0. • First pixel in a netpbm file is logical top-left of the image - so image will be inverted if displayed with GL defaults • This behavior was chosen to match existing compression tools • etcpack and toktx provide options to change the mapping • toktx inserts a KTXorientation key-value pair into the file that indicates the logical orientation of the image allowing smart applications to display the image correctly. © Copyright Khronos Group, 2010 - Page 28 Future • We expect other compression tools such as The Compressionator to add an option to output the KTX format. • Applications will then be able to use a single file format across all vendors proprietary texture compression schemes. © Copyright Khronos Group, 2010 - Page 29 Debugging and Profiling OpenGL ES based applications using gDEBugger ES Yaki Tebeka, CTO Graphic Remedy © 2004 – 2009 Graphic Remedy. All Rights Reserved Smart phones and devices era • Today’s smart phones and devices are equipped with GPUs, offering high performance graphics and low power consumption • Implementations are available from PowerVR, ARM, Qualcomm, NVIDIA, … • The OpenGL ES API is widely used over these platforms © 2004 – 2009 Graphic Remedy. All Rights Reserved Debugging and profiling 3D apps • Debugging and profiling 3D applications are hard and time consuming tasks • Companies find it extremely hard to deliver robust and bug-free 3D applications • It is almost impossible to optimize a 3D application to fully utilize the graphic system performance © 2004 – 2009 Graphic Remedy. All Rights Reserved • OpenGL, OpenGL ES and Open CL Debugger, Profiler and Memory Analyzer • Exposes the internal graphic system information needed to find bugs, optimize render performance and memory usage © 2004 – 2009 Graphic Remedy. All Rights Reserved Debugging and Profiling Demo © 2004 – 2009 Graphic Remedy. All Rights Reserved Automatic errors detection mechanism gDEBugger can automatically break on: • OpenGL ES errors • gDEBugger detected errors: a unique and more comprehensive errors detection mechanism • Graphic memory leaks • Redundant functions • And more © 2004 – 2009 Graphic Remedy. All Rights Reserved Call stack and Source code views • View the call stack and source code that led to the error / OpenGL ES function call © 2004 – 2009 Graphic Remedy. All Rights Reserved Statistical and Redundant calls viewer • Displays the number of times each OpenGL ES function was called • Display redundant state changes • Break on redundant state change © 2004 – 2009 Graphic Remedy. All Rights Reserved Performance graph Displays performance counters graphs from different sources • Operating Systems: Windows, Mac OS X and Linux • GPU and driver: SGX Graphic driver, etc • gDEBugger OpenGL ES Server © 2004 – 2009 Graphic Remedy. All Rights Reserved Performance Analysis Toolbar • Turn off the graphic pipeline stages one after the other • If the performance improves when turning off a certain stage, you have found a graphic pipeline bottleneck © 2004 – 2009 Graphic Remedy.