A Hitchhiker's Guide to Virtual Reality
Total Page:16
File Type:pdf, Size:1020Kb
i i i i A Hitchhiker’s Guide to Virtual Reality i i i i i i i i A Hitchhiker’s Guide to Virtual Reality Karen McMenemy Stuart Ferguson A K Peters, Ltd. Wellesley, Massachusetts i i i i i i i i Editorial, Sales, and Customer Service Office A K Peters, Ltd. 888 Worcester Street, Suite 230 Wellesley, MA 02482 www.akpeters.com Copyright c 2007 by A K Peters, Ltd. All rights reserved. No part of the material protected by this copyright notice may be repro- duced or utilized in any form, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without written permission from the copy- right owner. Library of Congress Cataloging-in-Publication Data McMenemy, Karen, 1976- – A hitchhiker’s guide to virtual reality / by Karen McMenemy, Stuart Ferguson. p. cm. Includes bibliographical references and index. ISBN 13: 978-1-56881-303-5 (alk. paper) ISBN 10: 1-56881-303-1 (alk. paper) 1. Human-computer interaction. 2. Virtual reality. I. Ferguson, Stuart, 1953- II. Title. QA76.9.H85M425 2007 006.8–dc22 2006102280 Cover image: The front cover illustrates a virtual figure exploring a virtual landscape. The theory and practical implementation making this possible are described in Chapters 5, 7, 13 and 14. The inset images on the back cover illustrate the projection system designed in Chapter 4 and some of the practical projects of Chapter 18. Printed in Canada 1110090807 10987654321 i i i i i i i i To Cathy i i i i i i i i Contents Preface xiii I What, Why and How 1 1 Introduction 3 1.1WhatisVirtualReality?.................. 3 1.2AQuickTouroftheBook................. 4 1.3AssumedKnowledgeandTargetAudience......... 7 1.4ValuableResourcesforVR................. 8 1.5TheCodeListings..................... 9 Bibliography........................... 10 2 The Human Senses and VR 13 2.1HumanVisualSystem................... 14 2.2HumanAuditorySystem.................. 18 2.3HumanTactileSystem................... 21 2.4Olfaction—TheSenseofSmell.............. 23 2.5Gustation—TheSenseofTaste.............. 24 2.6 Summary . ........................ 25 Bibliography........................... 26 3 Applications and Implications of VR 29 3.1Entertainment....................... 30 3.2Visualization........................ 31 3.3Training.......................... 33 vii i i i i i i i i viii Contents 3.4Education......................... 35 3.5OtherApplications..................... 36 3.6DistributedVR...................... 38 3.7TheImplicationsofVR.................. 39 3.8 Summary . ...................... 42 Bibliography........................... 43 4 Building a Practical VR System 45 4.1TechnologyofVisualization................ 47 4.2TechnologyforInteraction................. 64 4.3TechnologyforMotionTrackingandCapture....... 71 4.4 Summary . ...................... 83 Bibliography........................... 84 5 Describing and Storing the VR World 87 5.1TheGeometricDescriptionofa3DVirtualWorld.... 88 5.2SceneGraphs....................... 94 5.3PuttingVRContentontheWEB............. 102 5.4StoringImagesandTextures................ 105 5.5StoringVideo....................... 111 5.6 Summary . ...................... 116 Bibliography........................... 116 6 A Pocket 3D Theory Reference 119 6.1CoordinateSystems.................... 120 6.2Vectors........................... 123 6.3TheLine.......................... 124 6.4ThePlane......................... 125 6.5ReflectioninaPlane ................... 129 6.6Transformations...................... 131 6.7 Summary . ...................... 145 Bibliography........................... 145 7 The Rendering Pipeline 147 7.1GeometryandVertexOperations............. 149 7.2CullingandClipping................... 150 7.3ScreenMapping...................... 155 7.4ScanConversionorRasterization............. 156 i i i i i i i i Contents ix 7.5 Fragment Processing . ................... 159 7.6Texturing......................... 172 7.7 Summary . ........................ 179 Bibliography........................... 180 8 Computer Vision in VR 181 8.1 The Mathematical Language of Geometric Computer Vision 183 8.2Cameras.......................... 185 8.3 A Brief Look at Some Advanced Ideas in Computer Vision 189 8.4 Software Libraries for Computer Vision . ........ 214 8.5 Summary . ........................ 216 Bibliography........................... 217 9 Image-Based Rendering 219 9.1GeneralApproachestoIBR................ 220 9.2AcquiringImagesforIBR................. 223 9.3 Mosaicing and Making Panoramic Images . ........ 224 9.4 Summary . ........................ 230 Bibliography........................... 231 10 Stereopsis 235 10.1Parallax.......................... 237 10.2Head-MountedDisplays.................. 246 10.3 Active, Passive and Other Stereoscopic Systems ...... 249 10.4 Summary . ........................ 256 Bibliography........................... 256 11 Navigation and Movement in VR 257 11.1ComputerAnimation................... 258 11.2MovingandRotatingin3D................ 260 11.3RoboticMotion...................... 266 11.4InverseKinematics..................... 271 11.5 Summary . ........................ 281 Bibliography........................... 282 Summing Up 283 i i i i i i i i x Contents II Practical Programs for VR 287 12 Tools, Libraries and Templates for VR 289 12.1TheHardwareEnvironment................ 290 12.2SoftwareTools....................... 293 12.3AFrameworkfortheApplications............. 294 12.4TheComponentObjectModel.............. 296 12.5 Summary . ...................... 300 Bibliography........................... 301 13 Programming 3D Graphics in Real Time 303 13.1VisualizingaVirtualWorldUsingOpenGL........ 304 13.2SystemIndependencewiththeGLUTLibraries...... 333 13.3VisualizingaVirtualWorldUsingDirect3D........ 336 13.4 Summary . ...................... 349 Bibliography........................... 351 14 High-Quality 3D with OpenGL 2 353 14.1OpenGLVersion2..................... 354 14.2UsingProgrammableShaders............... 365 14.3SomeBasicShaderPrograms................ 370 14.4ProgrammableShadersinDirect3D............ 382 14.5 Summary . ...................... 382 Bibliography........................... 383 15 Using Multimedia in VR 385 15.1ABriefOverviewofDirectShow ............. 386 15.2Methodology....................... 391 15.3AMoviePlayer....................... 393 15.4VideoSources....................... 401 15.5CustomFiltersandUtilityPrograms............ 406 15.6PlayingMoviesorLiveVideointoTextures........ 413 15.7 Summary . ...................... 431 Bibliography........................... 431 16 Programming Stereopsis 433 16.1DisplayAdapterHardwareforStereopsis......... 434 16.2FileFormatsforStoringStereoscopicImages........ 438 16.3AFileFormatforStereoMovies.............. 441 i i i i i i i i Contents xi 16.4DisplayingStereoscopicImages.............. 441 16.5AStereoscopicMoviePlayer................ 446 16.6 Capturing Stereo Pictures from a Pair of Video Cameras . 459 16.7MakingaStereoMoviefromImageSequences....... 467 16.8 Summary . ........................ 475 Bibliography........................... 475 17 Programming Input and Force Feedback 477 17.1DirectInput........................ 479 17.2ForceFeedbackwithDirectInput............. 484 17.3HapticSoftwareLibrariesandTools............ 489 17.4Middleware........................ 494 17.5 Interfacing Custom VR Hardware with the Applications . 496 17.6 Summary . ........................ 505 Bibliography........................... 505 18 Building on the Basics, Some Projects in VR 507 18.1VideoEditing....................... 508 18.2 Stereo Movie and Picture Output for a Head-Mounted Display........................... 510 18.3 Video Processing . ................... 510 18.4Chroma-Keying...................... 511 18.5AMulti-ProjectorVRTheaterSystem........... 514 18.6 Using Image-Processing and Computer-Vision Libraries . 520 18.7 Augmented Reality (AR) .................. 521 18.8VirtualSculptingin3D.................. 526 18.9 Summary . ........................ 534 Bibliography........................... 534 A Rotation with Quaternions 537 A.1TheQuaternion...................... 538 A.2QuaternionsandRotation................. 539 A.3ConvertingaMatrixtoaQuaternion........... 542 A.4ConvertingaQuaterniontoEulerAngles......... 542 A.5InterpolatingQuaternions................. 544 B The Generalized Inverse 547 C Aligning Two Images in a Panoramic Mosaic 549 i i i i i i i i xii Contents D A Minimal Windows Template Program 557 E An MFC Template Program 565 F The COM in Practice 579 F.1 UsingCOM........................ 579 F.2 COMRecipes....................... 581 Index 585 i i i i i i i i Preface The seeds of the ideas that have become this book were sown on the breakfast terrace of a hotel in Beverly Hills during SIGGRAPH 2005. Whilst preparing for another interesting day at the conference, we faced the immediate task of how to journey downtown, no mean task in LA. So as we turned to a small guidebook for advice, the thought struck us that it would be really useful if we had a similar informative and practical guide to virtual reality (VR). At that time, we were preparing to build the next phase of our own VR research lab, and to be able to do that, it isn’t enough just to know about the background theory and technology that might be useful. We needed practical solutions— hardware and software solutions. Not finding anything suitable, we decided to compile our own—a guide that would bring together under one cover all the aspects of graphics, video, audio and haptics that have to work together to make virtual