Embedded Graphics with Freescale Devices AMF-AUT-T0697 Francisco Sandoval | Automotive FAE S E P T . 2 0 1 4 TM External Use Agenda • Graphics in Freescale Devices • Graphics Cores and APIs • Run-time vs Pre-rendered content • 3D Programming − Concepts − Pipeline − Shader Programs • Q&A TM External Use 1 Freescale Devices • i.MX6x – Most of them have 2D and 3D capabilities − DQ – GC2000, GC355, GC320 − DL, S – GC880, GC320 − SL – GC355 − SLX - GC400T • Spectrum – PowerPC core, has DCU • Rainbow – PowerPC core, has DCU and z160 • Vybrid, Halo – ARM core, has GC355, 2D-ACE TM External Use 3 Graphics cores GC2000 GC320 VPU GC880 Z430 2D-ACE VIU GC355 Z160 IPU* etc GC255 GCxxx cores are Vivante’s IP, newer Video input encoding Zxxx cores are AMD’s, older decoding 2D-ACE (DCUx), IPU are Freescale’s *ipu CSI Content creation Composition 2D/3D Real time rendering Memory manipulation TM External Use 4 Graphics APIs GC2000 OpenGL ES 1.1 GC880 OpenGL ES 2.0 Z430 OpenGL ES 3.0 3D GC355 OpenVG 1.1 Z160 2D/Vector GC320 DirectFB IPU 2D-ACE gfxlibs 2D-ACE/DCUx Composition TM External Use 5 GPU vs Sprites • GPU generates content at run-time • Sprites are pre-generated, stored in memory • Hard to compare performance-wise, they are different • There is no “best”, they are useful for different scenarios TM External Use 6 2.5D? 3D-like? • 2D-ACE is very powerful, i.MX’s gpus are not by default better: − Locked framerate − Hardware layers − On the fly HUD warping (Halo) − Cheaper solution (memory requirements) − Can look as good, sometimes even better frame rate. Example: TM External Use 7 i.MX6 Linux graphics layers Layer/ Example IPU Frame buffer EGL Qt, DirectFB OGLES/OVG glClear(); Engine Unity3D, EBGuide Application Fully 3D cluster TM External Use 8 Vybrid graphics layers Layer/ Example DCU Configure DCU Scheduler/RTOS MQX OVG/GFXLibs vgClear(); Altia Application 2D cluster, pre-rendered 3D effects TM External Use 9 Key concepts • OpenGL – Open Graphics Library • OpenGL ES – OpenGL for Embedded Systems • EGL – Embedded-systems Graphics Library • OpenCL – Open Computing Library • OpenVG – Open Vector Graphics-library • OpenCV – Open-source Computer Vision library. NOT Khronos • 2D-ACE – 2D Animation and Composition Engine (sexy for DCU) • DCU – Display Controller Unit • IPU – Image processing Unit • Wayland – computer display server protocol, like X • Weston – reference implementation of a Wayland compositor • Frame buffer – contiguous chunk of memory that stores a frame TM External Use 10 Different OpenGL flavours • OpenGL – Desktop • OpenGL ES 1.0, 1.1 (i.mx35, mpc5121, etc) − Fixed pipeline − Not compatible with 2.0 • OpenGL ES 2.0, 3.0 (i.mx5x, 6x) − Programmable pipeline (shader programs) − Compatible with 1.1 by a shader program TM External Use 11 3D Graphics Programming TM External Use 12 Objects in 3D programming • 3D objects are built from one or more series or “strips” of triangles 1 1 2 1 1 2 3 2 3 2 3 1 2 3 4 4 4 4 4 5 6 5 5 3 5 6 TM External Use 13 Objects in 3D programming • 3D models are made from a series of triangles and UV textures TM External Use 14 The Graphics Pipeline Sequence of steps that turns points in the computer-defined space to pixels in a display TM External Use 15 Fixed (1.0) Pipeline TM External Use 16 The OpenGL ES 2.0 Pipeline TM External Use 17 Graphics process TM External Use 18 How to place this in a 3D world? • Everything is referenced using three coordinate systems: . Global space . Model space . Camera space • The Matrix 1 0 0 0 0 1 0 0 0 0 1 0 [ x y z 1 ] TM External Use 19 The 3D world (Global space) • The 3D Cartesian coordinates that define the triangle vertices are multiplied by a matrix to move it from the object’s space to the “screen” space Camera System Projection Primitive TM External Use 20 Frustum TM External Use 21 Transformations (Modelview Matrix) 1 0 0 0 • 22 0 1 0 0 Translation 0 0 1 0 [ x y z 1 ] x 0 0 0 0 y 0 0 Scale 0 0 z 0 [ 0 0 0 1 ] cos a 0 sin a 0 Rotation (y) 0 1 0 0 [ -sin a 0 cos a 0 ] 0 0 0 1 TM External Use 22 Texture Mapping • If a texture (2D still image) is being applied to the surface, the interpolated texture coordinates are used to derive what color should be sampled from the texture. TM External Use 23 Texture Mapping • To every vertex in our mesh, a pair of 2D coordinates (UV) is assigned and pixels are interpolated UV Textures go from (0.0,0.0) to (1.0, 1.0) TM External Use 24 Text-tures • Render string to texture • PROS −Easiest method • CONS −Least flexible method −Texture space in memory −Not scalable (resolution) TM External Use 25 Text-tures • Create a sprite-based alphabet (1,1) • PROS −More efficient −Good for plain text • CONS −Complexity ++ −Check alignment, spacing, etc (0,0) TM External Use 26 Text-tures • Use library • PROS −Most flexible, robust −Easy to use • CONS −Most complex −Port engine to system −Third party SW TM External Use 27 OpenGL ES 1.x Lighting • Lighting is calculated at each vertex • Phong Lighting 3 components: − Ambient. Do not depend of the light source. − Diffuse. The light is evenly reflected in all direction (think flashlight). − Specular. The reflection is predominant in a given direction. The specular coefficient can be varied. • The light intensity can decrease according the distance (constant, linear or quadratic). TM External Use 28 OpenGL ES 1.x Lighting By combining all the lighting components, we can create very realistic real-time renders Ambient Only Ambient + Diffuse Ambient + Diffuse + Specular TM External Use 29 Shading Modes (ES 1.0) • A normal vector is perpendicular to an object (face, vertex) • You can use normal vectors to calculate the orientation of a surface relative to a light source TM External Use 30 Shading Modes (ES 2.0) Flat color, vertex Cartoony look position only Per pixel lighting Color information Moving the Textures vertices around TM External Use 31 Rasterization • After the 3 vertex coordinates are transformed into screen space, they are then bilinearly interpolated and aliased to ‘fragments’ • Other data associated with the vertices is also interpolated (colors, normals, texture coordinates, etc.) Rasterization Linear Color Interpolation TM External Use 32 Z-test • After interpolation, fragments have a ‘depth’ into the screen that is tested against the fragments that have already been rasterized. - If a fragment has already been produced that is “closer” to the screen, then the new value is not stored. - Otherwise, the new color value overwrites the old and the new depth is written to a “Depth Buffer” or “Z Buffer”. Final image Final z-buffer TM External Use 33 3D programming concepts • Resolution (hor x vert pixels being output by display) − Higher res = more detail. Prevents aliasing • Frame buffer (contiguous space in graphics memory − Raster – 2D rectangular grid of pixels − Refresh Rate, raster scan (row by row, top to bottom, left to right) − Double /triple buffer • Anti aliasing (jagged edges in straight lines) − Change color in pixels around edges so lines look smooth . MSAA – Multi Sample Anti Aliasing (selective on parts on screen) . FXAA – Fast Approximated AA (fastest, uses output resolution, so so qual) . TXAA – (MLAA) Temporal AA (like MSAA but takes in account previous frame) . Higher resolution (FSAA) TM External Use 34 3D programming concepts • Vsync − Screen tearing. Gfx card and display aren’t synced. FPS vs Refresh rate − Locks frame rate output of graphics card to display refresh rate TM External Use 35 3D programming concepts • Tessellation (Break polygon in smaller pieces to apply displacement maps, more depth info} − Bump/normal/parallax mapping − Displacement mapping TM External Use 36 3D programming concepts • Ambient Occlusion − Used to add soft shadows depending on surface exposure to ambient light − Cast rays from surface, if it touches the sky box brightness ++, otherwise no brightness. − Does not take direction into account! Hence can be pre-baked into static objects but looks wrong in dynamic objects − SSAO, HBAO, HDAO (crytek’s Screen Space samples depth of pixels, Horizon Based more samples per pixel half res, High Res AO AMD like HBAO) TM External Use 37 3D programming concepts • Field of View (FOV) − Observable angle, used when calculating the frustum TM External Use 38 3D programming concepts • Depth of Field (DOF) − Focus, near and far distance. Can be used to optimize by blurring low quality textures TM External Use 39 3D programming concepts • Bi-linear / Tri-linear / Anisotropic Filtering (AF) − Selective blurring, as things are farther apart. “non same direction filtering” TM External Use 40 3D programming concepts • Post-Processing effects (work on whole rendered frame instead of single objects) − Motion Blur − Bloom/HDR for lighting (more light when moving from dark to light) TM External Use 41 Shader Programs TM External Use 42 GLSL • GL Shading Language • Supported in OpenGL ES 2.0 • Lets you define specific parts of the graphics pipeline . Vertex shader . Fragment shader • C like language to write shaders • Shaders have to be written, compiled and linked TM External Use 43 GLSL overview • Data types • float float a,b; // two vectors • bool int c = 2; // c is initialized with 2 • Int bool d = true; // d is true • vec{2,3,4} vec3 direction; • bvec{2,3,4} bvec2 lightFlags; • ivec{2,3,4} ivec3 color; • mat{2,3,4} square matrices • sampler{1,2,3}D for textures • samplerCube for cube map textures • sampler{1D,2D}Shadow for shadow maps • arrays struct dirlight { // type definition • structs vec3 direction; vec3 color; }; TM External Use 44 GLSL overview • Variable qualifiers − const .