Achieving High Quality Mobile VR Games

Title 44pt Title Case Achieving High Quality Mobile VR Games

Roberto Lopez Mendez, Senior Software Engineer Affiliations 24pt sentence case Carl Callewaert - Americas Director & Global Leader of Evangelism, Unity

Patrick O'Luanaigh – CEO, nDreams

20pt sentence case GDC 2016 Title 40pt Title Case Agenda

Bullets 24pt sentence case . Ice Cave Demo (Roberto) Sub-bullets 20pt sentence case . Porting Ice Cave Demo to Samsung Gear VR . Improving VR quality & performance . Dynamic soft shadows and reflections based on local cubemaps . Stereo reflections

. VR Integration into Unity (Carl)

. Movement in Mobile VR (Patrick)

2 © ARM2016 Title 40pt Title Case Demo Ice Cave

Bullets 24pt sentence case Sub-bullets 20pt sentence case

3 © ARM2016 Text 54pt Sentence Case Porting Ice Cave Demo to Samsung Gear VR

4 © ARM 2016 Title 40pt Title Case Porting Ice Cave for Samsung Gear VR

Enabling Gear VR developer mode Bullets 24pt sentence case Native Plugin Sub-bullets 20pt sentence case or Native? • Go to your device Settings Application Acquire Manager Gear VR Service device ID • Tap on "Manage storage" • Tap on the "VR Service Version" six times • Wait for scan process to complete and you should now see the Developer Mode toggle

Place in Assets/ Plugins/Android/assets

5 © ARM2016 Title 40pt Title Case Considering VR Specifics

Bullets 24pt sentence case Sub-bullets 20pt sentence case

. Removed existing UI based on . Ice Cave was designed big so virtual joysticks users don’t feel claustrophobic . Added very simple UI through . Removed dirty lens effect as it Gear VR touchpad doesn’t translate well to VR . Removed camera animation to . Added camera collision and avoid motion sickness sliding as going through geometry . Carefully set the camera speed leads to bad VR experience

6 © ARM2016 Title 40pt Title Case Ice Cave VR Extras

Bullets 24pt sentence case . Added streaming to another device Sub-bullets 20pt sentence case to show what the user is actually experiencing (camera position and orientation) . Added an alternative UI by means of a mini Bluetooth controller

7 © ARM2016 Quality and Performance for VR Text 54pt Sentence Case Optimized Rendering Techniques Based On Local Cubemaps

8 © ARM 2016 Title 40pt Title Case The Concept of Local Cubemaps

Bounding Box Bullets 24pt sentence case Sub-bullets 20pt sentence case Local environment Local environment Local environment cubemap cubemap P cubemap V P V’ V’ C C C

V V V

If we use the view vector V We need to use a new vector V’ = CP We introduce a proxy geometry to defined in WCS to fetch the texel to fetch the correct texel . We need to simplify the problem of finding the from the cubemap we will get find the intersection point P of the view intersection point P. The simplest the smiley face instead of the vector V with the boundaries of the proxy geometry is the bounding box. star. local environment.

Local Cubemap = Cubemap + Cubemap Position + Scene Bounding Box + Local Correction

9 © ARM2016 Text 54pt Sentence Case Dynamic Soft Shadows Based on Local Cubemaps

10 © ARM 2016 Title 40pt Title Case Dynamic Soft Shadows Based on Local Cubemaps Generation stage Bullets 24pt sentence case Top +Y Sub-bullets 20pt sentence case Render the Left –X transparency of the Front Right Back

-Z +X +Z scene in the alpha channel Bottom Y -Y

X Z Camera background alpha colour = 0 We have a map of the zones where light rays Opaque geometry is rendered with alpha = 1 can potentially come Semi-transparent geometry is rendered with from and reach the alpha different from 1 geometry. Fully transparent geometry is rendered with No light information is alpha 0 processed at this stage.

11 © ARM2016 Title 40pt Title Case Dynamic Soft Shadows Based on Local Cubemaps Runtime stage Bullets 24pt sentence case . Create a vertex to light source L vector in the vertex shader.

Sub-bullets 20pt sentence case . Pass this vector to the fragment shader to obtain the vector

from the pixel to the light position piL. L . Find the intersection of the vector piL with the bounding box. cubemap . Q Build the vector CP from the cubemap position C to the C intersection point P. P shadowed . Use the new vector CP to fetch the texture from the cubemap. pixel pk float texShadow = texCUBE(_CubeShadows, CP).a;

lit pixel pi

Source code in the ARM Guide for Unity Developers at MaliDeveloper.arm.com

Bounding Box

12 © ARM2016 Title 40pt Title Case Dynamic Soft Shadows Based on Local Cubemaps Why soft? Bullets 24pt sentence case float texShadow = texCUBE(_CubeShadows, CP).a;

Sub-bullets 20pt sentence case

float4 newVec = float4`(CP, factor * length(piP)) L float texShadow = texCUBElod(_CubeShadows, newVec ).a; cubemap Q C P shadowed pixel The further from the object pk the softer the shadows.

lit pixel pi

Source code in the ARM Guide for Unity Developers at MaliDeveloper.arm.com

Bounding Box

13 © ARM2016 Title 40pt Title Case Dynamic Soft Shadows Based on Local Cubemaps

Bullets 24pt sentence case Sub-bullets 20pt sentence case

14 © ARM2016 Title 40pt Title Case Handling Shadows from Different Types of Geometries

Bullets 24pt sentence case Sub-bullets 20pt sentence case High Quality Shadows from Shadows from Dynamic Objects Static Environment

Use Local Cubemap technique Use Shadow Mapping technique

All Shadows

Combine both types of shadows

15 © ARM2016 Title 40pt Title Case Combined Shadows in Ice Cave VR

Bullets 24pt sentence case Shadows from dynamic Sub-bullets 20pt sentence case Shadows from static Geometry using shadow Geometry using local cubemps mapping

16 © ARM2016 Text 54pt Sentence Case Reflections Based on Local Cubemaps

17 © ARM 2016 Title 40pt Title Case Local Correction to Reflection Vector

float3 R = reflect(D, N); Bullets 24pt sentence case

Sub-bullets 20pt sentence case float4 col = texCUBE(Cubemap, R);

cubemap Find intersection point P C R’ Find vector R’ = CP float4 col = texCUBE(Cubemap, R’); P N GPU Gems. Chapter 19. Image-Based Lighting. Kevin Bjork, 2004. R http://http.developer.nvidia.com/GPUGems/gpugems_ch19.html

Cubemap Environment Mapping. 2010. D http://www.gamedev.net/topic/568829-box-projected-cubemap-environment-mapping/?&p=4637262

Image-based Lighting approaches and parallax-corrected cubemap. Sebastien Lagarde. SIGGRAPH 2012. http://seblagarde.wordpress.com/2012/09/29/image-based-lighting-approaches-and-parallax-corrected-cubemap/

Reflective surface Source code in the ARM Guide for Unity Developers at MaliDeveloper.arm.com

Bounding Box

18 © ARM2016 Title 40pt Title Case Combined Reflections in Ice Cave VR

Bullets 24pt sentence case Sub-bullets 20pt sentence case Reflections from Static Planar Reflections from Dynamic Reflections from Distant Geometry Geometry Environments

Use Local Cubemap technique Use Mirrored Camera technique Use standard infinite cubemap

All Reflections

Combine all types of reflections

19 © ARM2016 Title 40pt Title Case Combined Reflections in Ice Cave VR

Bullets 24pt sentence case Reflections from static Sub-bullets 20pt sentence case geometry using local cubemap

Reflections from the sky using infinite cubemap

Reflections from dynamic geometry using a mirrored camera

20 © ARM2016 Text 54pt Sentence Case Stereo Reflections in Unity

21 © ARM 2016 Title 40pt Title Case Why Stereo Reflections?

Bullets 24pt sentence case . Using the same texture for right/left eyes reflections in VR looks plain and Sub-bullets 20pt sentence case breaks the sensation of full immersion.

22 © ARM2016 Title 40pt Title Case Implementing Stereo Reflections in Unity

Bullets 24pt sentence case . Add two new cameras targeting left/right eye void OnPreRender() Sub-bullets 20pt sentence case respectively and disable them as you will render { SetUpCamera (); them manually. // Invert winding GL.invertCulling = true; . Create a target texture the cameras will render to } void OnPostRender() { . Attach the script to each camera, the SetUpCamera // Restore winding function places the camera in the right position for GL.invertCulling = false; } rendering reflections for each eye.

23 © ARM2016 Title 40pt Title Case Implementing Stereo Reflections in Unity

Bullets 24pt sentence case Attach the script to the main camera Sub-bullets 20pt sentence case public class RenderStereoReflections : MonoBehaviour { public GameObject reflectiveObj; public GameObject leftReflCamera; public GameObject rightReflCamera; int eyeIndex = 0;

void OnPreRender(){  Optimize further by adding a condition about reflective object’s visibility if (eyeIndex == 0){ // Render Left camera leftReflCamera.GetComponent().Render(); reflectiveObj.GetComponent().material.SetTexture("_DynReflTex", leftReflCamera.GetComponent().targetTexture ); } else{ // Render right camera rightReflCamera.GetComponent().Render(); reflectiveObj.GetComponent().material.SetTexture("_DynReflTex", rightReflCamera.GetComponent().targetTexture ); } eyeIndex = 1 - eyeIndex; } }

24 © ARM2016 Title 40pt Title Case Check Left/Right Reflection Synchronization

Bullets 24pt sentence case Sub-bullets 20pt sentence case

25 © ARM2016 Title 40pt Title Case Stereo Reflections in Ice Cave VR

Bullets 24pt sentence case Sub-bullets 20pt sentence case

26 © ARM2016 Title 40pt Title Case Wrap Up

Bullets 24pt sentence case . Unity has made a great contribution . Stereo reflections in VR improve Sub-bullets 20pt sentence case to VR democratization user experience https:// community.arm.com/groups/arm-mali- graphics/blog/2016/03/10/combined-reflections- . In mobile VR the user is no longer stereo-reflections-in-vr limited to the size of the screen and is now fully embedded in a virtual world . Check out The ARM Guide for Unity Developers for optimization tips, . It is possible to run high quality VR recommendations and very efficient and non-VR content on current rendering techniques to make the mobile devices using optimized most out of Unity when developing rendering techniques VR mobile games

27 © ARM2016

VR Integration Into Unity

Carl Callewaert Americas Director & Global Leader of Evangelism Unity Technologies

Movement in Mobile VR

Patrick O'Luanaigh CEO - nDreams VR Games & Experiences

Latest Focus Test

• 22 experienced gamers • 20-30 minute sessions

• Inexperienced in VR

0hrs 7

0-2hrs 8

3-5hrs 2 Hours Spent in VRinSpentHours

5hrs+ 5

No. Participants Previous Tests

Internal Events Oculus suffer | discomfort 40-50% | when walking around in VR*.

*Varying depending on headset, framerate, room temperature etc. suffer | discomfort 50-60% | when walking around in VR. Title 40pt Title Case

Bullets 24pt sentence case Sub-bullets 20pt sentence case

37 © ARM2016 An Easy Solution? Traditional Made as accessible as possible

Alternative Designed to be comfortable for all Title 40pt Title Case

Bullets 24pt sentence case Sub-bullets 20pt sentence case

Traditional

Realistic movement Don’t touch the camera No perceived acceleration

7 5 4 6 SPRINT 3 JOG 7

2 8 1.5 WALK ON/OFF 1 9

0 10 M/S Findings

Natural High Speed

20 20 18 18 16 16

14 14

12 12 10 10 8 8

6 6 No. Participants No. Participants 4 4 2 2 0 0 Moving Strafing Rotating Moving Strafing Rotating 1.5 m/s 1.0 m/s 37.5 θ/s 2.4 m/s 2.4 m/s 250 θ/s

Fine Uncomfortable Fine Uncomfortable Rotation

participants 18 preferred a high Of 22 rotation speed. Strafing

strafe <1m/s speed Summary

• Tailor specifically for VR: • Natural movement speed. • High speed rotation. • Consider adding an alternate rotation option. • No perceived acceleration. Not a solution for everyone! Title 40pt Title Case

Bullets 24pt sentence case Sub-bullets 20pt sentence case

Alternative

*As far as we can tell Alternative Rotation

Alternative controls encourage players to move their body to look around Trigger Rotation Snap Rotation Snap Rotation Snap Rotation Alternative Movement Teleport Title 40pt Title Case

Bullets 24pt sentence case Sub-bullets 20pt sentence case

Teleport “More precise”

Blink “More immersive”

0 2 4 6 8 10 12 No. Participants

Rotation Further usability “More immediate/accessible” testing required Trigger

“Offers more flexibility” Snap

0 2 4 6 8 10 12 14 16 No. Participants Summary

• Remove movement and rotation altogether. • Add alternatives! • We’re going with teleport or blink for moving, • Trigger or snap look for rotation. Google Cardboard

• Much of this is appropriate for even simple headset controls such as the Cardboard. • Use natural rotation. • Use gaze plus button to “blink” to another location. ? Your Projects

• Experiment with first person movement if appropriate for your title. • Implement comfortable alternative controls as default that avoid natural movement and rotation. • You may want to include optional traditional controls tailored to be as comfortable as possible in VR. • Test everything! What works for your game may be different. Thank you!

The trademarks featured in this presentation are registered and/or unregistered trademarks of ARM Limited (or its subsidiaries) in the EU and/or elsewhere. All rights reserved. All other marks featured may be trademarks of their respective owners. Copyright © 2016 ARM Limited To Find Out More….

Text 54pt Sentence Case

ARM Booth #1624 on Expo Floor: . Live demos of the techniques shown in this session . In-depth Q&A with ARM engineers . More tech talks at the ARM Lecture Theatre http://malideveloper.arm.com/gdc2016: . Revisit this talk in PDF and video format post GDC . Download the tools and resources

64 © ARM 2016 Title 40pt Title Case More Talks From ARM at GDC 2016 Available post-show at the Mali Developer Center: malideveloper.arm.com/

Bullets 24pt sentence case Vulkan on Mobile with Unreal Engine 4 Case Study Sub-bullets 20pt sentence case Weds. 9:30am, West Hall 3022

Making Light Work of Dynamic Large Worlds Weds. 2pm, West Hall 2000

Achieving High Quality Mobile VR Games Thurs. 10am, West Hall 3022

Optimize Your Mobile Games With Practical Case Studies Thurs. 11:30am, West Hall 2404