S07 Rendering, Metal, Sprite Kit, Scene
Total Page:16
File Type:pdf, Size:1020Kb
iPhone Application Programming Lecture 7: Rendering, Metal, Sprite kit, Scene kit Simon Voelker Media Computing Group RWTH Aachen University Winter Semester 2015/2016 http://hci.rwth-aachen.de/iphone Media Computing Group Your App 2D Graphics Standards-Based High Efficiency and Imaging 3D Graphics GPU Access Scene Graphs SpriteKit Core Animation SceneKit Core Image Core Graphics Open GL ES Metal GPU Apple, WWDC 2014 Media Computing 2 Simon Voelker: iPhone Programming Group Your App 2D Graphics Standards-Based High Efficiency and Imaging 3D Graphics GPU Access Scene Graphs SpriteKit Core Animation SceneKit Core Image Core Graphics Open GL ES Metal GPU Apple, WWDC 2014 Media Computing 3 Simon Voelker: iPhone Programming Group OpenGL ES • 3D graphics rendering engine • OpenGL with reduced instruction set • Cross-platform • C based • OpenGL is pixel based • Version 1.1: Fixed rendering pipeline • Version 2.0: Shader-based rendering pipeline Media Computing 5 Simon Voelker: iPhone Programming Group GLKit • Introduced iOS 5 • More comfort for basic rendering tasks • Helper classes • GLKView & GLKViewController • GLKTextureLoader • GLKEffect • Math Library • Lighting • For More details: iPhone Lecture 6 (2012/2013) Media Computing 6 Simon Voelker: iPhone Programming Group Your App 2D Graphics Standards-Based High Efficiency and Imaging 3D Graphics GPU Access Scene Graphs SpriteKit Core Animation SceneKit Core Image Core Graphics Open GL ES Metal GPU Apple, WWDC 2014 Media Computing 7 Simon Voelker: iPhone Programming Group Metal • Highly optimised rendering pipeline (OpenGL overhead reduced) • C++ • Precompiled shaders • Multithreading • Speed increase by reducing CPU load • Works only on A7 and A8 processors (iPhone 5s) Media Computing 8 Simon Voelker: iPhone Programming Group Improve Rendering Performance 60 FPS (16.6 ms/frame) CPU Work (frame N) CPU Work (frame N+1) CPU Work (frame N+2) CPU Work (frame N+3) GPU Work (frame N - 1) GPU Work (frame N) GPU Work (frame N+1) GPU Work (frame N+2) 0 ms 16.6 ms 34.2 ms 50.8 ms 67.4 ms Media Computing 9 Simon Voelker: iPhone Programming Group Improve Rendering Performance 60 FPS (16.6 ms/frame) CPU Work (frame N) GPU Work (frame N -1) 0 ms 16.6 ms Media Computing 10 Simon Voelker: iPhone Programming Group Improve Rendering Performance 60 FPS (16.6 ms/frame) CPU Work (frame N) GPU Work (frame N -1) 0 ms 16.6 ms Media Computing 11 Simon Voelker: iPhone Programming Group Improve Rendering Performance 60 FPS (16.6 ms/frame) CPU Work (frame N) GPU Work (frame N -1) 0 ms 16.6 ms 20 ms Media Computing 12 Simon Voelker: iPhone Programming Group Improve Rendering Performance 60 FPS (16.6 ms/frame) Application GPU API GPU Work (frame N -1) 0 ms 16.6 ms Media Computing 13 Simon Voelker: iPhone Programming Group Improve Rendering Performance 60 FPS (16.6 ms/frame) Application GPU Work (frame N -1) 0 ms 16.6 ms Media Computing 14 Simon Voelker: iPhone Programming Group Improve Rendering Performance 60 FPS (16.6 ms/frame) Application GPU Work (frame N -1) 0 ms 16.6 ms Media Computing 15 Simon Voelker: iPhone Programming Group Expansive GPU API • State validation • Check if API use is valid • Encode API state to hardware state • Shader compilation • Run-time generation of shader code • Transferring work to GPU • Resources • Commands Media Computing 16 Simon Voelker: iPhone Programming Group Expansive GPU API When Frequency Before Metal Metal Application build never Content loading Rare Draw time every frame Apple, WWDC 2013 Media Computing 17 Simon Voelker: iPhone Programming Group Expansive GPU API When Frequency Before Metal Metal Application build never Content loading Rare State validation Draw time every frame Shader compilation Start work on the GPU Apple, WWDC 2013 Media Computing 18 Simon Voelker: iPhone Programming Group Expansive GPU API When Frequency Before Metal Metal Application build never Shader compilation Content loading Rare State validation State validation Draw time every frame Shader compilation Start work on the GPU Start work on the GPU Apple, WWDC 2013 Media Computing 19 Simon Voelker: iPhone Programming Group Data Buffer DataData Buffer Buffer Resource ResourceResource Vertex Texture Texture Vertex Format TextureTexture TextureTexture Descriptor Resource DescriptorDescriptor ResourceResource Descriptor Vertex Function Function Render Render Pipeline Render Pipeline RenderRender CommandCommand State Descriptor EncoderCommand Fragment EncoderEncoder Function Function Depth Stencil Depth Stencil Blend Blend State Descriptor Descriptor Key CommandCommand Resource CommandBuffer BufferBuffer Code State Command Queue Device Descriptor Apple WWDC 2013 System Metal Demo Media Computing 21 Simon Voelker: iPhone Programming Group Your App 2D Graphics Standards-Based High Efficiency and Imaging 3D Graphics GPU Access Scene Graphs SpriteKit Core Animation SceneKit Core Image Core Graphics Open GL ES Metal GPU Apple, WWDC 2014 Media Computing 22 Simon Voelker: iPhone Programming Group Sprite Kit Objects Actions Physics Media Computing 23 Simon Voelker: iPhone Programming Group Demo Media Computing 24 Simon Voelker: iPhone Programming Group Root Object: SKScene Application UIView SKView ViewController SKScene //Setup SKView /* Your Code here */ //Create Scene Media Computing 25 Simon Voelker: iPhone Programming Group Scene Graph SKScene background spaceship crater projectile asteroid crater propellant asteroid crater asteroid Nodes crater asteroid Actions Physics Media Computing 26 Simon Voelker: iPhone Programming Group Sprite Kit Nodes SKNode SKSpriteNode SKShapeNode SKLabelNode SKEmitterNode SKEffectNode SKScene SKCropNode SKVideoNode Media Computing 27 Simon Voelker: iPhone Programming Group SKNode (0,0) • Basic node (used for grouping) • Position, rotation, scale (0,0) • zPosition (0,0) //Hit Test node.containsPoint(aCGPoint) //Converts a point from the coordinate system node.convertPoint(aCGPoint, fromNode: aSKNode) //Converts a point in this node’s coordinate system node.convertPoint(aCGPoint, toNode: aSKNode) Media Computing 28 Simon Voelker: iPhone Programming Group SKSpriteNode solid color texture color blending let sprite = SKSpriteNode(color: SKColor.greenColor(), size: CGSizeMake(100, 100)) let asteroid = SKSpriteNode(imageNamed:"asteroid.png") asteroid.color = SKColor.greenColor() asteroid.colorBlendFactor = 0.5 Media Computing 29 Simon Voelker: iPhone Programming Group SKShapeNode • Draws CGPath • Stroke, Fill Color Media Computing 30 Simon Voelker: iPhone Programming Group Particle Editor Media Computing 31 Simon Voelker: iPhone Programming Group SKEmitterNode let thrust = SKEmitterNode(fileNamed: "thrust.sks") thrust.position = CGPointMake(0, self.spaceship.size.height -10); self.addChild(thrust!) thrust.particleScale = 2; thrust.particleScaleSpeed = -10; Media Computing 32 Simon Voelker: iPhone Programming Group SKCropNode • Creates a mask the children • Mask is defined as a SKNode Mask child result Media Computing 33 Simon Voelker: iPhone Programming Group SKVideoNode • Video as Node • AVPPlayer (AVFoundation.framework) Video • All the functionality from AVFoundation let videoNode = SKVideoNode(fileNamed: “video”) import AVFoundation let videoNode = SKVideoNode(AVPlayer: player) videoNode.play() Media Computing 34 Simon Voelker: iPhone Programming Group SKLightNode • Creates Light • lightColor • shadowColor • ambientColor • Can be set for each Node using a bitMask Media Computing 35 Simon Voelker: iPhone Programming Group SKEffectNode • Applies CIFilter to its children • CIFilter is a powerful Core Image filter • Can be used on the entire Scene let filter = CIFilter(name: “CIColorInvert") filter?.setDefaults() Media Computing 36 Simon Voelker: iPhone Programming Group SKEffectNode: CIFilter CIBloom • More then 100 different Filter • Glow effects: • CIBloom let filter = CIFilter(name: “CIBloom") self.filter = filter self.shouldEnableEffects = true Media Computing 37 Simon Voelker: iPhone Programming Group SKShaders void main() • Basic OpenGL ES Fragment Shader { gl_FragColor = SKDefaultShading(); • Can be applied to } • SKSpriteNodes, SKShapeNodes (stroke, fill), SKScenes.. • Two types of input variables: void main() • Uniforms (value is the same across all shader instances) { gl_FragColor = vec4(255,0,0,255); • Varying (value is different for each shader instances. e. g. } pixel position) Large library of Shaders: www.shadertoy.com Media Computing 38 Simon Voelker: iPhone Programming Group Sprite Kit Shader Demo Media Computing 39 Simon Voelker: iPhone Programming Group Sprite Kit Render Loop Apple iOS 9 API Media Computing 40 Simon Voelker: iPhone Programming Group Sprite Kit: Actions Media Computing 41 Simon Voelker: iPhone Programming Group Simple Actions Create Action SKAction.moveTo(CGPoint(100,100), duration: 1.0) SKAction.rotateByAngle(M_PI, duration: 1.0) Move the spaceship let action = SKAction.moveTo(CGPointMake(100,100), duration: 1.0) spaceship.runAction(action) spaceship.runAction(SKAction.moveTo(CGPointMake(100,100), duration: 1.0)) Media Computing 42 Simon Voelker: iPhone Programming Group Repeating Actions let action = SKAction.moveTo(CGPointMake(100,100), duration: 1.0) shape.runAction(SKAction.repeatAction(action, count: 3)) shape.runAction(SKAction.repeatActionForever(action) shape.runAction(SKAction.sequence([action1,action2,action3])); Media Computing 43 Simon Voelker: iPhone Programming Group Combining Actions 1 sec 2 sec 1.5 sec Sequence: action1 action2 action3 node.runAction(SKAction.sequence([action1,action2,action3])) Groups: action1 1 sec action2