OpenXR: On an HMD near you now!

Brent Insko

® This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 1 Unifying Reality

OpenXR: On an HMD near you now! Brent E. Insko, PhD Lead XR Architect at & OpenXR Working Group Chair © Khronos® Group Inc. 2020 - Page 2 What is OpenXR?

OpenXR is a royalty-free, open standard that provides high-performance access to (AR) and (VR)—collectively known as XR—platforms and devices.

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 3 OpenXR handles communication to and from an application and an XR platform

Application XR device

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 4 OpenXR handles communication to and from an application and an XR platform

• Image(s) to display • Audio • Haptic responses

Application XR device

• Head & hand pose information • Controller/input state • Display configuration / form factor

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 5 Consider the variety of XR Platforms available…

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 6 Cross Platform XR Application Development

Before OpenXR, program to each platform’s proprietary API

With OpenXR, program to a single, common, high-performance API

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 7 OpenXR is used with a 3D API Application or Engine

High-performance, low-latency Display, Cross-platform access to XR composition and 3D rendering and composition* optical correction HMDs and sensors Multiview parameters XR application lifecycle Context priority Input device discovery and events Front buffer rendering Sensor tracking and pose calculation Tiled rendering (beam racing) Frame timing and display composition Variable rate rendering Haptics Control

OpenXR is strongly influenced by Vulkan with a * OpenXR can be used with other 3D shared spec toolchain and support for API layers. OpenXR is a “lower-frequency” API than Vulkan such as Direct3D, OpenGL and OpenGL ES and is a much smaller spec This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 8 OpenXR Flexibility

VR

Oculus Quest / Quest 2 AR VR

Valve Index and other SteamVR compatibile HMDs Rift S

Microsoft HoloLens 2 HTC Vive Cosmos Windows All In One Tethered

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 9 Applications, Game Engines, Rendering Engines, Development Browsers using OpenXR! - Fall 2020 - 2021

Conformant AR and VR platforms released - Summer & Fall 2020

Conformance suite developed and Adopters program released - July 2020

OpenXR 1.0 Specification Released - SIGGRAPH, July 2019

0.90 Provisional Specification Released - GDC, March 2019

This workOpenXR is licensed underWorking a Creative Group Commons Formed Attribution - 20174.0 International License © The Khronos® Group Inc. 2021 - Page 10 Latest Developments with OpenXR

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 11 OpenXR Widespread Industry Adoption Conformant Implementations

Microsoft HoloLens 2 S HTC Vive Cosmos Headsets Quest and Quest 2

Valve SteamVR Varjo Transitioning from OpenVR to OpenXR Preliminary support shipping now Open- Implementation Engine Support

Unreal Engine Engine 81 for Chrome and Edge Browsers Support in 4.24. Optimizations in 4.25 OpenXR Plugin Preview in 2020.2+ OpenXR default backend for WebXR

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 12 All the Major Platforms Transitioning to OpenXR for Future Development

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 13 OpenXR Applications

Blender 2.83 integrates OpenXR to deliver native VR scene inspection capabilities

Minecraft’s RenderDragon rendering engine uses OpenXR for desktop VR support

Google Chrome

Google Chromium uses OpenXR as its default backend for WebXR, Microsoft Flight Simulator Sim Update 2 enabling and Microsoft Edge uses OpenXR for VR functionality browsers to use any OpenXR-compatible hardware

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 14 Latest OpenXR Game

• Single Executable

• Multiple Platforms

• Built in Unity

Nominated for 'VR Game of the Year' at the 2021 VR Awards, ‘Zombieland VR: Headshot Fever’ XR Games, Sony Pictures Virtual Reality This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 15 What’s Next for OpenXR?

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 16 OpenXR Overlays Extension • Simultaneous in-world applications in OpenXR • Can overlay one app’s 2D content “on top” of an existing application’s content • Overlay applications add a rich content into other XR Applications: e.g. Desktop OS windows in-world, In-game HUD, Virtual keyboard, Chat (e.g. Pluto VR's Pluto)

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 17 Advanced UI Cross-Vendor OpenXR Extensions

Developers can build cross-platform applications that use advanced UI solutions from different technology vendors OpenXR API layers can be used implement extensions

Hand Tracking Eye Tracking

26 unique joints per hand for fully Eye gaze interaction for intuitive interfaces articulated hands visible to the user 2-Step Interaction Shipping on HoloLens 2 Hand-eye coordination and Microsoft Hand Mesh Extension Natural aiming for HoloLens 2 layers over it Shipping on HoloLens 2 Ultraleap developer preview available

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 18 OpenXR and Edge Server Applications with 5G

OpenXR will enable AR Significant industry applications to run portably effort in developing

on edge server architectures Low latency this use case Sensor Data

Runtime using 5G implemented across device and server and accessed through OpenXR API

Sensor handling

MEC (Multi-access Edge Display composition Computing) Server 1. Processes sensor data, can include Wireless mobile device machine learning for environmental with display and sensors lighting, occlusion, scene semantics, object reconstruction and UI Generated 2. Generates imagery from 3D Augmentations & models, can include stereo, foveal rendering, ray-tracing, optics pre- Scenes distortion, varifocal processing

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 19 What is in the future for of OpenXR?

•Clarifying and fixing a few things from OpenXR 1.0

•Expanding support for AR and MR - Scene understanding

•Additional User input/output capabilities - Expanded haptics support - Expanded body tracking support - Expanded controller and device support

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 20 With OpenXR on an HMD near you, now is the time to get involved!

Officially conformant runtimes from: Microsoft, Oculus, HTC, Valve, Varjo and Collabora

OpenXR being used by real : Unreal! ! Chrome! Edge! Unity! Flight Simulator! Minecraft! Games!

Virtual Reality Now is the time for application V developers to leverage OpenXR for widespread deployment! A Augmented Reality

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 21 How to get started… https://openxr.org

• OpenXR Forum and Slack Channel - Forum: https://khr.io/openxrfeedback - Discussion: https://khr.io/slack

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 22 Join Us… BOF

August 10th, 1PM Pacific

This work is licensed under a Creative Commons Attribution 4.0 International License © The Khronos® Group Inc. 2021 - Page 23