Projector-Based Location Discovery and Tracking
Total Page:16
File Type:pdf, Size:1020Kb
Projector-Based Location Discovery and Tracking Johnny Chung Lee May 2008 CMU-HCII—xx-xxxx Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 Thesis Committee: Scott E. Hudson (Chair), Carnegie Mellon University Jodi Forlizzi, Carnegie Mellon University Luis Von Ahn, Carnegie Mellon University Paul Dietz, Mitsubishi Electric Research Labs Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy Copyright © 2008 Johnny Chung Lee. All rights reserved. This research was supported in part by the National Science Foundation under grants CCR-03244770, IIS-0121560, IIS-0205219, IIS-0325351, IIS-0329090, IIS-9800597, and IIS-9980013, and by the author’s internship at Mitsubishi Electric Research Labs. Keywords: Projector alignment, projector calibration, augmented reality, location discovery, motion capture, incremental tracking, high-speed projection, infrared projection, multi-projector applications. i Abstract Today, the primary use of projection technology is for creating large flat displays that provide a shared viewing experience for presentations or entertainment applications. While research projects have explored the powerful ability for projected light to create illusions that can reshape our perception and our interaction with surfaces in the environment, very few of these systems have had success in terms of commercial and consumer adoption. Part of this market place failure can be attributed to the lack of practicality in the cost-of-operation due to the complexity of installation and reliability of execution. Often these systems require expert knowledge to perform system setup and calibration between the projected and the physical surfaces to make these illusions effective. In this thesis, we present a technique for inherently adding object location discovery and tracking capabilities to commercial projectors. This is accomplished by introducing light sensors into the projection area and then spatially encoding the image area using a series of structured light patterns. This delivers a unique pattern of light to every pixel in the projector’s screen space directly encoding the location data using the projector itself. By unifying the image projection and location tracking technologies, many of the difficult calibration and alignment issues related to interactive projection and projected spatial augmented reality applications can be eliminated simplifying their implementation and execution. Furthermore, by creating a hybrid visible light and infrared light projector, a single calibration-free device can perform invisible location tracking of input devices while simultaneously presenting visible application content. We present a detailed description of the projector- based location discovery and tracking technique, a description of three prototype ii implementations, and a demonstration the effectiveness of this simplification by re-implementing, and in some cases improving upon, several location-sensitive projector applications that have been previously executed using external calibration and tracking technologies. iii Acknowledgements The primary influences in this work most certainly are my thesis advisor, Scott Hudson, and research mentor, Paul Dietz. Scott provided the research guidance and outlook that has taken me from a naïve new graduate student to the researcher I have become today. When I began the doctoral program at Carnegie Mellon University, I had little knowledge and appreciation of what research should be and a lack of confidence that I could make a meaningful contribution to the field. Scott has been integral in helping me gain perspective to select problems of substance and taught me the approach to develop solutions that are practical. These two skills have become essential in defining my path of work as a researcher. Paul provided me with an exposure to a sense of technical creativity and personal enjoyment in invention that I have not found in any other individual. As mentor, he has provided me with a perspective on life and the impact that work, research, and creativity can have on the world outside the typical academic boundaries. For this, I will be eternally grateful In addition to these two individuals, I have had the pleasure to work with and be in the company of brilliant individuals that have inspired me on both professional and personal levels This includes my co-authors Ramesh Raskar, Desney Tan, Dan Maynes-Aminzade, Jay Summit, Chris Kyriakakis, and Darren Leigh. My classmates, colleagues, and friends James Fogarty, Daniel Avrahami, Darren Gergle, Luis Von Ahn, and Laura Dabbish have been an incomparable set of individuals to have begun my research career with. I felt sad to see them graduate and move on, but I now look forward to joining them “on the other side”. My brother, David Lee, was the one who first got me interested in Human- Computer Interaction when he was taking classes at University of Virginia with iv Randy Paucsh. Once I entered college, my first exposure to research was with Denny Proffitt who gave a young undergrad an unusual opportunity to participate in a large research project with Carnegie Mellon. It was through this work, that I was able to develop a research relationship with Randy Paucsh and Jodi Forlizzi. Without these individuals, I most certainly would not be where I am today. Thank you. v Table of Contents Abstract .................................................................................................................... i Acknowledgements ................................................................................................ iii Index of Figures .................................................................................................... vii 1: Introduction ......................................................................................................... 1 Research Summary and Components ................................................................. 3 Location-Sensitive Projector Applications ......................................................... 4 2: Background and Approach ............................................................................... 10 Automatic Projector Calibration ....................................................................... 12 Structured Light ................................................................................................ 15 3: Projector-Based Location Discovery ................................................................ 18 Applications of Projector Based-Location Discovery ...................................... 22 Limitations and Discussion ............................................................................... 27 4: Projector-Based Tracking ................................................................................. 30 Anatomy of a Projector ..................................................................................... 31 Low-Perceptability Tracking Patterns .............................................................. 32 Achieving Interactive Tracking Using 60Hz Refresh Rate .............................. 34 Localized Pattern Size and Shape ..................................................................... 37 Motion Modeling .............................................................................................. 39 Tracking Loss Strategies ................................................................................... 39 Occlusion Detection and Behavior ................................................................... 41 Applications of Projector-Based Location Tracking ........................................ 45 5: Hybrid Infrared-Visible Light Projection ......................................................... 50 Creating the Hybrid Infrared and Visible Light Projector ................................ 51 Invisible Location Discovery ............................................................................ 54 Applications of Hybrid Projection .................................................................... 58 6: High-Speed Motion Tracking ........................................................................... 61 7: Foldable Interactive Surfaces............................................................................ 64 Tracking ............................................................................................................ 65 Foldable Shapes ................................................................................................ 66 Newspaper..................................................................................................... 67 vi Scroll ............................................................................................................. 67 Fan................................................................................................................. 69 Umbrella ....................................................................................................... 69 Orientation Sensitivity ...................................................................................... 70 Interactivity ....................................................................................................... 73 10: Comparison to Computer Vision Approaches ................................................ 74 11: Error Modeling of Moving Sensors ................................................................ 79 12: Summary and Conclusion ..............................................................................