FOSS Virtual & Augmented Reality Monado & OpenXR Jakob Bornecrantz [email protected] 01/02/2020 1 Jakob Bornecrantz FDO GH @Wallbraker [email protected] Open First 2 ● What is AR & VR ● OpenXR ● Monado ● Status ● Demos ● Wrapping up 3 ● What is AR & VR OpenXR Monado Status Demos Wrapping up 4 What is Augmented Reality? ● Augment ━ Verb ━ /ɔːɡˈmɛnt/ ━ make greater by adding to it; increase 5 Open First 6 Open First 7 What is Virtual Reality? ● I reject your reality and substitute my own ━ Dungeonmaster, 1984 ■ (popularized by Adam Savage) 8 Open First 9 Open First 10 AR + VR = ? ● Also known as XR ● X = A | V ● Not eXtended Reality 11 What is AR & VR ● OpenXR Monado Status Demos Wrapping up 12 How do we XR? Program Platform Hardware 13 How do we XR? Program API Platform Hardware 14 15 What’s cool about OpenXR? ● It’s just like OpenGL & Vulkan ━ Made by Khronos ● Out now ● Action Based ● FOSS Implementation! :D 16 Action Based ● Type of controller not in focus ━ Only after the fact ● Context ━ From program ━ To runtime ━ Runtime decides what button does what 17 What is AR & VR OpenXR ● Monado Status Demos Wrapping up 18 How do we XR? Program Platform Hardware 19 Where does Monado fit in? Program Monado Hardware 20 Monado Program OpenXR OpenXR State Tracker xrt_prober xrt_device xrt_compositor Platform Compositor Drivers Hardware 21 Complex Truth Program OpenXR Loader OpenXR OpenXR State Tracker xrt_prober xrt_device xrt_gl_compositor GL Wrapper Platform xrt_fd_compositor Drivers Compositor Auxiliary Hardware 22 Desktop Stack Program OpenXR OpenXR State Tracker xrt_prober xrt_device xrt_compositor Platform Drivers Compositor Misc APIs Linux APIs Vulkan, X11, Wayland and KMS Linux Hardware Hardware 23 With IPC - Anticipated OpenXR State Tracker xrt_prober xrt_device xrt_fd_compositor Process boundary IPC Layer xrt_prober xrt_device xrt_fd_compositor Platform Drivers Compositor Misc APIs Linux APIs Vulkan and KMS Linux Hardware Hardware 24 XRT Interfaces ● Between different components ━ Auxiliary code has their own interfaces ● Purely C ● Not stable![1] 25 xrt_device ● HMDs ● Functions ● Controllers ━ Input Sync ● Aspect pattern ━ Output Set ━ Optional structs ━ View/Position Retrieval ━ HMD info ● Info (immutable) ━ Input/Outputs ━ Tracking Origin ━ Screen position (HMD) ━ Distortion (HMD) 26 xrt_compositor ● Base class ● Functions ━ xrt_[gl|vk|fd]_compositor ━ More explicit then OpenXR ━ GL & VK are wrappers ━ Create Swapchain ■ Client side only ━ Prepare, Begin & End ━ FD is “real” compositor ━ Wait, Begin, Discard & End ■ Written in Vulkan ━ ■ Only “talks” in FDs Poll Events (Refactor?) ● xrt_swapchain ● Info (immutable) ━ xrt_[gl|vk|fd]_swapchain ━ Supported Formats ━ acquire, wait & release 27 xrt_prober ● Policy holder ● Functions ━ Reads configuration ━ Look for devices ━ Decides what devices open ━ Select (opens devices) ● Device prober ━ List & Open Video devices ━ Used by OpenXR ST ━ Misc USB ● Platform code ● Info (immutable) ━ Abstract native interfaces ━ None ━ Used by device drivers ● Should be split in two ━ xrt_prober ━ xrt_platform 28 What is AR & VR OpenXR Monado ● Status Demos Wrapping up 29 Status ● Mostly complete OpenXR support ● In-process Compositor ━ Mesh shader ● Video processing framework ● Tracking PS Move ● Nifty debug UI 30 Hardware support ● North Star* ● PlayStation VR ● Daydream Ctrl* ● PlayStation Move ● OSVR HDK ● Razer Hydra ● Vive (Pro) & Index ● OpenHMD ━ Positional tracking ━ Port to native drivers through libsurvive where it makes sense. 31 Code ● 35.5K lines of codes ━ Drivers: 6.9K ■ 9 drivers ■ Vive: 1.5K ■ HDK: 0.5K ━ Compositor: 6.6K ━ Auxiliary: 7.6K ━ State Trackers: 10.1K ━ Targets: 1.3K 32 Next Up ● Complete OpenXR support ● Improve PS Move Tracking ● PSVR Tracking ● Out of process compositor 33 Even further out ● Safe space ● System UI ● Setup UI ● Lighthouse Tracking ● AR and SLAM 34 What is AR & VR OpenXR Monado Status ● Demos Wrapping up 35 Tracking 36 Debug UI 37 Tracking (back-up) 38 What is AR & VR OpenXR Monado Status Demos ● Wrapping up 39 Rant ● Gnome-Shell ━ Output hot-plug ━ non-desktop ● Intel Missing Extension ━ GL_EXT_memory_object_fd 40 Some thoughts ● Lots of work needed ● GNOME & KDE UIs? ━ Monado interface library? ● Switching between apps? 41 FOSS XR Community @FossXR 42 Links ● Monado ━ https://monado.dev/ ━ https://monado.pages.freedesktop.org/monado ━ https://gitlab.freedesktop.org/monado/monado ● OpenXR ━ https://www.khronos.org/openxr ● Work on GFX & XR ━ Internships @ Collabora ━ Work @ Collabora 43 Talk to me about ● XR, VR & AR ● FOSS XR ● Volt Programming Language ● FPGAs & FOSS ● Amiga (FPGA), mc68k (LLVM) ● Voxel/SVO rendering ● FOSS & Society ● Joining Collabora! Open First 44 FOSS Virtual & Augmented Reality Questions? 45.