Capture, Reconstruction, and Representation of the Visual Real World for Virtual Reality Christian Richardt1[0000−0001−6716−9845], James Tompkin2[0000−0003−2218−2899], and Gordon Wetzstein3[0000−0002−9243−6885] 1 University of Bath
[email protected] https://richardt.name/ 2 Brown University james
[email protected] http://jamestompkin.com/ 3 Stanford University
[email protected] http://www.computationalimaging.org/ Abstract. We provide an overview of the concerns, current practice, and limita- tions for capturing, reconstructing, and representing the real world visually within virtual reality. Given that our goals are to capture, transmit, and depict com- plex real-world phenomena to humans, these challenges cover the opto-electro- mechancial, computational, informational, and perceptual fields. Practically pro- ducing a system for real-world VR capture requires navigating a complex design space and pushing the state of the art in each of these areas. As such, we outline several promising directions for future work to improve the quality and flexibility of real-world VR capture systems. Keywords: Cameras · Reconstruction · Representation · Image-Based Rendering · Novel-view synthesis · Virtual reality 1 Introduction One of the high-level goals of virtual reality is to reproduce how the real world looks in a way which is indistinguishable from reality. Achieving this arguably-quixotic goal requires us to solve significant problems across capture, reconstruction, and represen- tation, and raises many questions: “Which camera system should we use to sample enough of the environment for our application?”; “How should we model the world and which algorithm should we use to recover these models?”; “How should we store the data for easy compression and transmission?”, and “How can we achieve simple and high-quality rendering for human viewing?”.