FOSS Virtual & Augmented Reality Monado & OpenXR
Jakob Bornecrantz jakob@collabora.com 01/02/2020 1 Jakob Bornecrantz
FDO GH @Wallbraker [email protected]
Open First 2 ● What is AR & VR ● OpenXR ● Monado ● Status ● Demos ● Wrapping up 3 ● What is AR & VR OpenXR Monado Status Demos Wrapping up 4 What is Augmented Reality? ● Augment ━ Verb ━ /ɔːɡˈmɛnt/ ━ make greater by adding to it; increase
5 Open First 6 Open First 7 What is Virtual Reality? ● I reject your reality and substitute my own ━ Dungeonmaster, 1984 ■ (popularized by Adam Savage)
8 Open First 9 Open First 10 AR + VR = ? ● Also known as XR ● X = A | V ● Not eXtended Reality
11 What is AR & VR ● OpenXR Monado Status Demos Wrapping up 12 How do we XR?
Program
Platform
Hardware
13 How do we XR?
Program API Platform
Hardware
14 15 What’s cool about OpenXR? ● It’s just like OpenGL & Vulkan ━ Made by Khronos ● Out now ● Action Based ● FOSS Implementation! :D
16 Action Based ● Type of controller not in focus ━ Only after the fact ● Context ━ From program ━ To runtime ━ Runtime decides what button does what
17 What is AR & VR OpenXR ● Monado Status Demos Wrapping up 18 How do we XR?
Program
Platform
Hardware
19 Where does Monado fit in?
Program
Monado
Hardware
20 Monado Program
OpenXR
OpenXR State Tracker
xrt_prober xrt_device xrt_compositor
Platform Compositor Drivers
Hardware
21 Complex Truth Program
OpenXR Loader OpenXR OpenXR State Tracker
xrt_prober xrt_device xrt_gl_compositor
GL Wrapper Platform xrt_fd_compositor Drivers
Compositor Auxiliary Hardware
22 Desktop Stack
Program
OpenXR
OpenXR State Tracker
xrt_prober xrt_device xrt_compositor
Platform Drivers Compositor
Misc APIs Linux APIs Vulkan, X11, Wayland and KMS
Linux Hardware Hardware
23 With IPC - Anticipated
OpenXR State Tracker
xrt_prober xrt_device xrt_fd_compositor
Process boundary IPC Layer
xrt_prober xrt_device xrt_fd_compositor
Platform Drivers Compositor
Misc APIs Linux APIs Vulkan and KMS
Linux Hardware Hardware
24 XRT Interfaces ● Between different components ━ Auxiliary code has their own interfaces ● Purely C ● Not stable![1]
25 xrt_device ● HMDs ● Functions ● Controllers ━ Input Sync ● Aspect pattern ━ Output Set ━ Optional structs ━ View/Position Retrieval ━ HMD info ● Info (immutable) ━ Input/Outputs ━ Tracking Origin ━ Screen position (HMD) ━ Distortion (HMD)
26 xrt_compositor ● Base class ● Functions ━ xrt_[gl|vk|fd]_compositor ━ More explicit then OpenXR ━ GL & VK are wrappers ━ Create Swapchain ■ Client side only ━ Prepare, Begin & End ━ FD is “real” compositor ━ Wait, Begin, Discard & End ■ Written in Vulkan ━ ■ Only “talks” in FDs Poll Events (Refactor?) ● xrt_swapchain ● Info (immutable) ━ xrt_[gl|vk|fd]_swapchain ━ Supported Formats ━ acquire, wait & release
27 xrt_prober ● Policy holder ● Functions ━ Reads configuration ━ Look for devices ━ Decides what devices open ━ Select (opens devices) ● Device prober ━ List & Open Video devices ━ Used by OpenXR ST ━ Misc USB ● Platform code ● Info (immutable) ━ Abstract native interfaces ━ None ━ Used by device drivers ● Should be split in two ━ xrt_prober ━ xrt_platform
28 What is AR & VR OpenXR Monado ● Status Demos Wrapping up 29 Status ● Mostly complete OpenXR support ● In-process Compositor ━ Mesh shader ● Video processing framework ● Tracking PS Move ● Nifty debug UI
30 Hardware support ● North Star* ● PlayStation VR ● Daydream Ctrl* ● PlayStation Move ● OSVR HDK ● Razer Hydra ● Vive (Pro) & Index ● OpenHMD ━ Positional tracking ━ Port to native drivers through libsurvive where it makes sense.
31 Code ● 35.5K lines of codes ━ Drivers: 6.9K ■ 9 drivers ■ Vive: 1.5K ■ HDK: 0.5K ━ Compositor: 6.6K ━ Auxiliary: 7.6K ━ State Trackers: 10.1K ━ Targets: 1.3K
32 Next Up ● Complete OpenXR support ● Improve PS Move Tracking ● PSVR Tracking ● Out of process compositor
33 Even further out ● Safe space ● System UI ● Setup UI ● Lighthouse Tracking ● AR and SLAM
34 What is AR & VR OpenXR Monado Status ● Demos Wrapping up 35 Tracking
36 Debug UI
37 Tracking (back-up)
38 What is AR & VR OpenXR Monado Status Demos ● Wrapping up 39 Rant ● Gnome-Shell ━ Output hot-plug ━ non-desktop ● Intel Missing Extension ━ GL_EXT_memory_object_fd
40 Some thoughts ● Lots of work needed ● GNOME & KDE UIs? ━ Monado interface library? ● Switching between apps?
41 FOSS XR Community @FossXR
42 Links ● Monado ━ https://monado.dev/ ━ https://monado.pages.freedesktop.org/monado ━ https://gitlab.freedesktop.org/monado/monado ● OpenXR ━ https://www.khronos.org/openxr ● Work on GFX & XR ━ Internships @ Collabora ━ Work @ Collabora
43 Talk to me about
● XR, VR & AR ● FOSS XR ● Volt Programming Language ● FPGAs & FOSS ● Amiga (FPGA), mc68k (LLVM) ● Voxel/SVO rendering ● FOSS & Society ● Joining Collabora!
Open First 44 FOSS Virtual & Augmented Reality Questions?
45