S07 Rendering, Metal, Sprite Kit, Scene

iPhone Application Programming Lecture 7: Rendering, Metal, Sprite kit, Scene kit

Simon Voelker Media Computing Group RWTH Aachen University

Winter Semester 2015/2016

http://hci.rwth-aachen.de/iphone

Media Computing Group Your App

2D Graphics Standards-Based High Efﬁciency and Imaging 3D Graphics GPU Access Scene Graphs

SpriteKit Core Animation SceneKit Core Image Core Graphics Open GL ES Metal

GPU Apple, WWDC 2014 Media Computing 2 Simon Voelker: iPhone Programming Group Your App

2D Graphics Standards-Based High Efﬁciency and Imaging 3D Graphics GPU Access Scene Graphs

SpriteKit Core Animation SceneKit Core Image Core Graphics Open GL ES Metal

GPU Apple, WWDC 2014 Media Computing 3 Simon Voelker: iPhone Programming Group

OpenGL ES

• 3D graphics rendering engine

• OpenGL with reduced instruction set

• Cross-platform

• C based

• OpenGL is pixel based

• Version 1.1: Fixed rendering pipeline

• Version 2.0: Shader-based rendering pipeline

Media Computing 5 Simon Voelker: iPhone Programming Group GLKit

• Introduced iOS 5

• More comfort for basic rendering tasks

• Helper classes

• GLKView & GLKViewController

• GLKTextureLoader

• GLKEffect

• Math Library

• Lighting

• For More details: iPhone Lecture 6 (2012/2013)

Media Computing 6 Simon Voelker: iPhone Programming Group Your App

2D Graphics Standards-Based High Efﬁciency and Imaging 3D Graphics GPU Access Scene Graphs

SpriteKit Core Animation SceneKit Core Image Core Graphics Open GL ES Metal

GPU Apple, WWDC 2014 Media Computing 7 Simon Voelker: iPhone Programming Group Metal

• Highly optimised rendering pipeline (OpenGL overhead reduced)

• C++

• Precompiled shaders

• Multithreading

• Speed increase by reducing CPU load

• Works only on A7 and A8 processors (iPhone 5s)

Media Computing 8 Simon Voelker: iPhone Programming Group Improve Rendering Performance 60 FPS (16.6 ms/frame)

CPU Work (frame N) CPU Work (frame N+1) CPU Work (frame N+2) CPU Work (frame N+3)

GPU Work (frame N - 1) GPU Work (frame N) GPU Work (frame N+1) GPU Work (frame N+2)

0 ms 16.6 ms 34.2 ms 50.8 ms 67.4 ms

Media Computing 9 Simon Voelker: iPhone Programming Group Improve Rendering Performance 60 FPS (16.6 ms/frame)

CPU Work (frame N)

GPU Work (frame N -1)

0 ms 16.6 ms

Media Computing 10 Simon Voelker: iPhone Programming Group Improve Rendering Performance 60 FPS (16.6 ms/frame)

CPU Work (frame N)

GPU Work (frame N -1)

0 ms 16.6 ms

Media Computing 11 Simon Voelker: iPhone Programming Group Improve Rendering Performance 60 FPS (16.6 ms/frame)

CPU Work (frame N)

GPU Work (frame N -1)

0 ms 16.6 ms 20 ms

Media Computing 12 Simon Voelker: iPhone Programming Group Improve Rendering Performance 60 FPS (16.6 ms/frame)

Application GPU API

GPU Work (frame N -1)

0 ms 16.6 ms

Media Computing 13 Simon Voelker: iPhone Programming Group Improve Rendering Performance 60 FPS (16.6 ms/frame)

Application

GPU Work (frame N -1)

0 ms 16.6 ms

Media Computing 14 Simon Voelker: iPhone Programming Group Improve Rendering Performance 60 FPS (16.6 ms/frame)

Application

GPU Work (frame N -1)

0 ms 16.6 ms

Media Computing 15 Simon Voelker: iPhone Programming Group Expansive GPU API

• State validation

• Check if API use is valid

• Encode API state to hardware state

• Shader compilation

• Run-time generation of shader code

• Transferring work to GPU

• Resources

• Commands

Media Computing 16 Simon Voelker: iPhone Programming Group Expansive GPU API

When Frequency Before Metal Metal

Application build never

Content loading Rare

Draw time every frame

Apple, WWDC 2013 Media Computing 17 Simon Voelker: iPhone Programming Group Expansive GPU API

When Frequency Before Metal Metal

Application build never

Content loading Rare

State validation Draw time every frame Shader compilation Start work on the GPU Apple, WWDC 2013 Media Computing 18 Simon Voelker: iPhone Programming Group Expansive GPU API

When Frequency Before Metal Metal

Application build never Shader compilation

Content loading Rare State validation

State validation Draw time every frame Shader compilation Start work on the GPU Start work on the GPU Apple, WWDC 2013 Media Computing 19 Simon Voelker: iPhone Programming Group Data Buffer DataData Buffer Buffer Resource ResourceResource

Vertex Texture Texture Vertex Format TextureTexture TextureTexture Descriptor Resource DescriptorDescriptor ResourceResource Descriptor Vertex Function Function Render Render Pipeline Render Pipeline RenderRender CommandCommand State Descriptor EncoderCommand Fragment EncoderEncoder Function Function

Depth Stencil Depth Stencil Blend Blend State Descriptor Descriptor

Key

CommandCommand Resource CommandBuffer BufferBuffer Code

State

Command Queue Device Descriptor Apple WWDC 2013 System Metal Demo

Media Computing 21 Simon Voelker: iPhone Programming Group Your App

2D Graphics Standards-Based High Efﬁciency and Imaging 3D Graphics GPU Access Scene Graphs

SpriteKit Core Animation SceneKit Core Image Core Graphics Open GL ES Metal

GPU

Apple, WWDC 2014 Media Computing 22 Simon Voelker: iPhone Programming Group Sprite Kit

Objects Actions Physics

Media Computing 23 Simon Voelker: iPhone Programming Group Demo

Media Computing 24 Simon Voelker: iPhone Programming Group Root Object: SKScene

Application UIView SKView

ViewController SKScene //Setup SKView /* Your Code here */ //Create Scene

Media Computing 25 Simon Voelker: iPhone Programming Group Scene Graph SKScene

background spaceship

crater projectile

asteroid crater propellant asteroid

crater asteroid Nodes

crater asteroid Actions Physics

Media Computing 26 Simon Voelker: iPhone Programming Group Sprite Kit Nodes

SKNode SKSpriteNode

SKShapeNode

SKLabelNode

SKEmitterNode

SKEffectNode SKScene

SKCropNode

SKVideoNode

Media Computing 27 Simon Voelker: iPhone Programming Group SKNode (0,0) • Basic node (used for grouping)

• Position, rotation, scale (0,0)

• zPosition (0,0)

//Hit Test node.containsPoint(aCGPoint)

//Converts a point from the coordinate system node.convertPoint(aCGPoint, fromNode: aSKNode)

//Converts a point in this node’s coordinate system node.convertPoint(aCGPoint, toNode: aSKNode)

Media Computing 28 Simon Voelker: iPhone Programming Group SKSpriteNode

solid color texture color blending

let sprite = SKSpriteNode(color: SKColor.greenColor(), size: CGSizeMake(100, 100))

let asteroid = SKSpriteNode(imageNamed:"asteroid.png")

asteroid.color = SKColor.greenColor() asteroid.colorBlendFactor = 0.5

Media Computing 29 Simon Voelker: iPhone Programming Group SKShapeNode

• Draws CGPath

• Stroke, Fill Color

Media Computing 30 Simon Voelker: iPhone Programming Group Particle Editor

Media Computing 31 Simon Voelker: iPhone Programming Group SKEmitterNode

let thrust = SKEmitterNode(fileNamed: "thrust.sks")

thrust.position = CGPointMake(0, self.spaceship.size.height -10);

self.addChild(thrust!)

thrust.particleScale = 2; thrust.particleScaleSpeed = -10;

Media Computing 32 Simon Voelker: iPhone Programming Group SKCropNode

• Creates a mask the children

• Mask is deﬁned as a SKNode

Mask child result

Media Computing 33 Simon Voelker: iPhone Programming Group SKVideoNode

• Video as Node

• AVPPlayer (AVFoundation.framework) Video

• All the functionality from AVFoundation

let videoNode = SKVideoNode(fileNamed: “video”)

import AVFoundation let videoNode = SKVideoNode(AVPlayer: player)

videoNode.play()

Media Computing 34 Simon Voelker: iPhone Programming Group SKLightNode

• Creates Light

• lightColor

• shadowColor

• ambientColor

• Can be set for each Node using a bitMask

Media Computing 35 Simon Voelker: iPhone Programming Group SKEffectNode

• Applies CIFilter to its children

• CIFilter is a powerful Core Image ﬁlter

• Can be used on the entire Scene

let filter = CIFilter(name: “CIColorInvert")

filter?.setDefaults()

Media Computing 36 Simon Voelker: iPhone Programming Group SKEffectNode: CIFilter CIBloom • More then 100 different Filter • Glow effects: • CIBloom

let filter = CIFilter(name: “CIBloom")

self.filter = filter self.shouldEnableEffects = true

Media Computing 37 Simon Voelker: iPhone Programming Group SKShaders

void main() • Basic OpenGL ES Fragment Shader { gl_FragColor = SKDefaultShading(); • Can be applied to }

• SKSpriteNodes, SKShapeNodes (stroke, ﬁll), SKScenes.. • Two types of input variables: void main() • Uniforms (value is the same across all shader instances) { gl_FragColor = vec4(255,0,0,255); • Varying (value is different for each shader instances. e. g. } pixel position)

Large library of Shaders: www.shadertoy.com Media Computing 38 Simon Voelker: iPhone Programming Group Sprite Kit Shader Demo

Media Computing 39 Simon Voelker: iPhone Programming Group Sprite Kit Render Loop

Apple iOS 9 API Media Computing 40 Simon Voelker: iPhone Programming Group Sprite Kit: Actions

Media Computing 41 Simon Voelker: iPhone Programming Group Simple Actions Create Action

SKAction.moveTo(CGPoint(100,100), duration: 1.0)

SKAction.rotateByAngle(M_PI, duration: 1.0)

Move the spaceship

let action = SKAction.moveTo(CGPointMake(100,100), duration: 1.0)

spaceship.runAction(action)

spaceship.runAction(SKAction.moveTo(CGPointMake(100,100), duration: 1.0))

Media Computing 42 Simon Voelker: iPhone Programming Group Repeating Actions

let action = SKAction.moveTo(CGPointMake(100,100), duration: 1.0)

shape.runAction(SKAction.repeatAction(action, count: 3))

shape.runAction(SKAction.repeatActionForever(action)

shape.runAction(SKAction.sequence([action1,action2,action3]));

Media Computing 43 Simon Voelker: iPhone Programming Group Combining Actions 1 sec 2 sec 1.5 sec

Sequence: action1 action2 action3

node.runAction(SKAction.sequence([action1,action2,action3]))

Groups: action1 1 sec

action2 2 sec

action3 1.5 sec

node.runAction(SKAction.group([action1,action2,action3]))

Media Computing 44 Simon Voelker: iPhone Programming Group Texture animate Other Actions

SKAction.animateWithTextures([action1,action2,action3] timePerFrame:0.1)

Path animate

SKAction.followPath(aPath timePerFrame:2.5)

and many more: colors, sounds, custom blocks ...

Media Computing 45 Simon Voelker: iPhone Programming Group Sprite Kit: Physics

Media Computing 46 Simon Voelker: iPhone Programming Group Scene Graph SKScene

background spaceship

crater propellant

asteroid crater projectile asteroid

crater asteroid

physicsBody crater asteroid

Media Computing 47 Simon Voelker: iPhone Programming Group SKPhysicsBody

asteroid.physicsBody = SKPhysicsBody(circleOfRadius: 2.0)

asteroid.physicsBody = SKPhysicsBody(texture: sprite.texture, size: sprite.size)

Edge

EdgeLoopFromRect Polygon

EdgeChain EdgeLoopFromPath Rectangle

Media Computing 48 Simon Voelker: iPhone Programming Group SKPhysicsWorld

• Each scene as its own PhysicsWorld

• Performs contact and collision tests

/* normal gravity */ self.physicsWorld.gravity = CGPointMake(0.0, -9.8);

/* inverted gravity */ self.physicsWorld.gravity = CGPointMake(0.0, +9.8);

Media Computing 49 Simon Voelker: iPhone Programming Group SKPhysicsContact Contact Delegate

self.physicsWorld.contactDelegate = self

projectile asteroid

func didBeginContact(_ contact: SKPhysicsContact)

var bodyA: SKPhysicsBody { get }

var bodyB: SKPhysicsBody { get }

var contactPoint: CGPoint { get }

var collisionImpulse: CGFloat { get }

var contactNormal: CGVector { get }

Media Computing 50 Simon Voelker: iPhone Programming Group Physics Groups

/** Defines what logical 'categories' this body belongs to. Defaults to all bits set (all categories). */ public var categoryBitMask: UInt32

/** Defines what logical 'categories' of bodies this body responds to collisions with. Defaults to all bits set (all categories). */ public var collisionBitMask: UInt32

/** Defines what logical 'categories' of bodies this body generates intersection notifications with. Defaults to all bits cleared (no categories). */ public var contactTestBitMask: UInt32

Media Computing 51 Simon Voelker: iPhone Programming Group More Physics

• Inverse Kinematics

• Physics Fields

• Fields simulate physical forces

• Linear gravity ﬁeld

• Radial gravity ﬁeld

• Electric ﬁeld

Media Computing 52 Simon Voelker: iPhone Programming Group Physics Field Demo

Media Computing 53 Simon Voelker: iPhone Programming Group SpriteKit Tools SpriteKitSprite Tools Kit Editor 2D editor Referenced actions

Full visual editor for 2D games Create once, reuse on multiple nodes, projects • 2D editor in Xcode Reference via editor or code No more compile and check • Visual layout Leveraging the new SKAction actionNamed: No code required Build a custom library of actions Visual layout • Animation editor Build very complex actions Live physics simulation • Physics simulation Live shader editing Apple (2015)

Media Computing 54 Simon Voelker: iPhone Programming Group Your App

2D Graphics Standards-Based High Efﬁciency and Imaging 3D Graphics GPU Access Scene Graphs

SpriteKit Core Animation SceneKit Core Image Core Graphics Open GL ES Metal

GPU Apple, WWDC 2014 Media Computing 55 Simon Voelker: iPhone Programming Group Scene Kit

Apple (2013)

Media Computing 56 Simon Voelker: iPhone Programming Group Scene Kit

COLLADA (*.dae)

Scene Kit

Media Computing 57 Simon Voelker: iPhone Programming Group Scene Kit

• Similar to Sprite Kit but in 3D SCNScene

• Scene class: SCNScene background spaceship

• Root class SCNNode crater projectile

• Scene Graph asteroid propellant crater • Geometry asteroid

crater asteroid • Light

crater asteroid • Camera

Media Computing 58 Simon Voelker: iPhone Programming Group SCNNode

• SCNGeometry

• Triangles

• Vertices

• Normals

• UVs

• Materials

• SCNCamera

Apple WWDC 2014

Media Computing 59 Simon Voelker: iPhone Programming Group SCNMaterials

• Determines the appearance of the geometry

• SCNMaterialProperty

• Contains colors or images

• Lighting and shading attributes

• Defuse

• Ambient

• Specular

• Normal

• …

Media Computing 60 Simon Voelker: iPhone Programming Group Animations and Actions Transaction like Core Animation

// highlight it SCNTransaction.begin() SCNTransaction.setAnimationDuration(0.5)

// on completion - unhighlight SCNTransaction.setCompletionBlock { SCNTransaction.begin() SCNTransaction.setAnimationDuration(0.5)

material.emission.contents = UIColor.blackColor()

SCNTransaction.commit() }

Action like in Sprite Kit (SCNAction) SCNAction.rotateByAngle(M_PI, aroundAxis: SCNVector3Make(0, 1, 0), duration: 5)

Media Computing 61 Simon Voelker: iPhone Programming Group Scene Kit Render Loop

Apple WWDC 2014

Media Computing 62 Simon Voelker: iPhone Programming Group Physics

• Similar to Sprite Kit

aNode.physicsBody?.applyForce(aVector3, atPosition: aVector3, impulse: true)

aNode.physicsBody?.physicsShape = SCNPhysicsShape(aGeometry,options)

• Inverse Kinematics

• Constraints (SCNConstraints)

• Force ﬁelds

Media Computing 63 Simon Voelker: iPhone Programming Group Scene Kit and Sprite Kit

• Include Sprite Kit in Scene Kit

let spriteScene = SKScene(size: aSize) let material = SCNMaterial()

material.diffuse.contents = spriteScene

• Include Scene Kit in Sprite Kit

let node = SK3DNode(WithViewportSize:aPort) node.scnScene = node

Media Computing 64 Simon Voelker: iPhone Programming Group Scene Editor SceneAvailable Editor in Xcode 7

• 3D scene editor in Xcode

• Visual layout

• Animation editor

• Physics simulation

Apple (2015)

Media Computing 65 Simon Voelker: iPhone Programming Group Scene Kit Demo

Media Computing 66 Simon Voelker: iPhone Programming Group Summary

• OpenGL (WS 2012/2013)

• Metal (WWDC 2014/2015)

• Powerful low level API

• Sprite Kit

• Easy to use 2D game engine

• Scene Kit

• Easy to use 3D game engine

Media Computing 67 Simon Voelker: iPhone Programming Group