Visual Computing in Virtual Environments

+0 ( 1(2 3 & 1 & 2 0 %' % " ! "+ "-.// -../'-.//0 & # %'( %%)%* + #%' %& , ! "#$$% %%$ Abstract his thesis covers research on new and alternative ways of interaction with computers. Virtual Reality and multi touch setups Tare discussed with a focus on three dimensional rendering and photographic applications in the field of Computer Graphics. Virtual Reality (VR) and Virtual Environments (VE) were once thought to be the future interface to computers. However, a lot of problems prevent an everyday use. This work shows solutions to some of the problems and discusses remaining issues. Hardware for Virtual Reality is diverse and many new devices are still being developed. An overview on historic and current devices and VE setups is given and our setups are described. The DAVE, an immersive projection room, and the HEyeWall Graz, a large high resolution display with multi touch input are presented. Available processing power and in some parts rapidly decreasing prices lead to a continuous change of the best choice of hardware. A major influence of this choice is the application. VR and multi touch setups often require sensing or tracking the user, optical tracking being a common choice. Hardware and software of an optical 3D marker tracking and an optical multi touch system are explained. The Davelib, a software framework for rendering 3D models in Virtual Environments is presented. It allows to easily port existing 3D applications to immersive setups with stereoscopic rendering and head tracking. Display calibration and rendering issues that are special to VR setups are explained. User interfaces for navigation and manipu- lation are described, focusing on interaction techniques for the DAVE and for multi touch screens. Intuitive methods are shown that are easy to learn and use, even for computer illiterates. Exemplary applications demonstrate the potential of immersive and non-immersive setups, showing which applications can most benefit from Virtual En- vironments. Also, some image processing applications in the area of computational photography are explained, that help to better depict the captured scene. ii ABSTRACT Contents Abstract ii 1 Introduction 2 1.1 Motivation ......................... 3 1.2 Covered Work . .................... 4 1.3 Virtual Environments and Virtual Reality . ..... 4 1.4 Computational Photography and Image Processing . 5 2 Hardware Devices 6 2.1 HardwareOverview................... 8 2.2 Virtual Environment Setups . ............ 19 2.3 Custom Devices . .................... 30 3 Input: Optical Tracking 44 3.1 3D Marker Tracking ................... 46 3.2 Multi Touch Tracking .................. 67 3.3 Miscellaneous . .................... 78 4 Output: Image Rendering 82 4.1 Related Work . .................... 84 4.2 Davelib: A VR Framework . ............. 91 4.3 Frequency Split Display .................108 5 User Interaction 118 5.1 Interaction in the DAVE .................120 5.2 Interaction on the HEyeWall . .............129 5.3 Touch Screen Interaction .................130 5.4 Miscellaneous . ....................141 6 Applications 146 6.1 Related Work . ....................148 6.2 Non-Immersive Applications . .............150 6.3 Immersive 3D Applications . .............174 7 Conclusion 201 7.1 Accomplishments . ....................202 7.2 Contributions and Benefit . .............202 7.3 Limitations ........................203 7.4 Future Work ........................206 A Publications and Collaborations 207 A.1 Publications ........................208 A.2 Supervising Activities and Collaborations . .....209 References 212 Index 222 CONTENTS iii Acknowledgements arts of the work for this thesis were done at ENSIMAG at the INPG, Grenoble, France in 2001-2002, the Computer Graphics lab of Braunschweig University of Technology, Germany in P2002-2004 and the HITLabNZ at the University of Canterbury, New Zealand in 2004-2005. The majority however was done at the CGV at the Graz University of Technology, Austria in 2005-2011. I thank my colleagues for collaboration and support. Fruitful discussions and talks with members of the neighboring ICG were helpful and interesting. I thank my parents for their loving support as well as my girlfriend and friends for their understanding of my priorities for work, especially in the last year of this thesis. Finally, I thank my supervisors for advice and everyone involved in taking care of funding and administration. The author. Photo courtesy of Oliver Sacherer. 1 Introduction desktop PC with keyboard and mouse is certainly quite uni- versal. However, it is easy to see that for some tasks, it is A not the best interface. Computing power has increased a lot in the past decades, but user interfaces have hardly changed. This well known finding among Human Computer Interaction (HCI) re- searchers has led to a number of user interface concepts for specific purposes. This thesis focuses on the display and interaction of 3D content. How can visual perception of 3D data be improved? How is immersion into a virtual world possible? How can a user explore, navigate and ma- nipulate the data in the best way? Which hardware is necessary? The content of this work shows steps on the way to answer these questions. 1.1 Motivation . ......................... 3 1.2 Covered Work ......................... 4 1.3 Virtual Environments and Virtual Reality ......... 4 1.4 Computational Photography and Image Processing . 5 2 1. INTRODUCTION 1.1 MOTIVATION Users can comfortably work or play on a PC for many hours using a mouse and a keyboard. In most 2D and 3D applications, tasks can be accomplished quickly and precisely. Complex graphical user interfaces (GUIs) are possible and there are standard GUI elements such as text input fields or scroll bars that are commonly known. Especially pushed by the games industry, normal PCs are nowadays equipped with powerful 3D graphics hardware, developed and optimized for real-time rendering of shaded and textured triangle meshes. Limitations of Conventional Desktop Setups. Many of such applications are or at least seem to be complex to novice users. The inhibition threshold may be too high, especially for people with limited experience with PCs. They may be overwhelmed by the interface and be afraid not to be able to use the system or even to break some- thing. Capabilities of the programs are unclear and the interaction is often indirect and limited. Players of 3D games can be observed instinctively moving their head sideways in order to look around a corner, without the computer reacting on this subconscious request by the user. Two dimensional interfaces for three dimensional worlds lack a degree of freedom, resulting in a less intuitive control. Also, the interaction is performed dislocated from the displayed content, requiring a further indirection. Immersion, the sense of feeling to really be in a virtual world or 3D scene, is also limited. A very limited field of view and the lack of parallax and stereoscopic effects only allow a low level of spatial immersion. Novel User Interfaces. For suited applications, we would like to immerse in 3D worlds, exploring and modifying them in an easy way, even for computer illiterates. We would also like to interact with large amounts of data for non-immersive applications in an intuitive way. Imaginative concepts for natural interaction in public spaces are given by Valli in [Val07], such as technology enhanced spaces that react on a user, provoking spontaneous interactions. A large variety of different devices exists, but how can they be used best? We tried out many different setups and applications in order to understand and analyze problems and find similarities to develop generalized solutions. Computational Photography. In digital photography, the images are often still used like in analog film photography. However, with the help of image processing, some hardware restrictions can be bypassed by using multiple photos, usually taken with different settings. This area of Computational Photography slowly makes its way to the common users, panoramic image stitching being a good example. We would like to explore more of these new ways to make additional use of digital photos. 1.1. MOTIVATION 3 Contributions. This work presents three major contributions on the way to fulfill these goals: • Hardware and software developments for the DAVE, an immersive environment. • The HEyeWall Graz, a large high resolution multi touch screen. • Improvements for applications in the area of Computational Photography. 1.2 COVERED WORK Figure 1.1: The basic components of a Virtual Environment. All of these aspects are covered in the following chapters. The subjects illustrated in the figure above represent the building blocks of Virtual Environments and related applications. They are covered in the following chapters, but first, introductions to Virtual Reality and Computational Photography are given. 1.3 VIRTUAL ENVIRONMENTS AND VIRTUAL REALITY The term ’Virtual Reality’ appeared in the 1930ies, meaning a fictional world, later also described with cyberspace. Only for about 20 years it is also used for its second and now more common scientific meaning. Similar to the term robot, high expectations raised by science fiction literature and movies led to blatant misconceptions in the public, concerning the actual capabilities of VR. But also in science the term is fuzzy, as illustrated in the figure below. Figure 1.2: The virtuality continuum. Many varieties of mixed setups

Load more