Iowa State University Capstones, Theses and Graduate Theses and Dissertations Dissertations 2010 The AugmenTable: markerless hand manipulation of virtual objects in a tabletop augmented reality environment Michael Van Waardhuizen Iowa State University Follow this and additional works at: https://lib.dr.iastate.edu/etd Part of the Mechanical Engineering Commons Recommended Citation Van Waardhuizen, Michael, "The AugmenTable: markerless hand manipulation of virtual objects in a tabletop augmented reality environment" (2010). Graduate Theses and Dissertations. 11525. https://lib.dr.iastate.edu/etd/11525 This Thesis is brought to you for free and open access by the Iowa State University Capstones, Theses and Dissertations at Iowa State University Digital Repository. It has been accepted for inclusion in Graduate Theses and Dissertations by an authorized administrator of Iowa State University Digital Repository. For more information, please contact [email protected]. The AugmenTable: markerless hand manipulation of virtual objects in a tabletop augmented reality environment by Michael Van Waardhuizen A thesis submitted to the graduate faculty in partial fulfillment of the requirements for the degree of MASTER OF SCIENCE Major: Human Computer Interaction Program of Study Committee: Jim Oliver, Major Professor Eliot Winer Stephen Gilbert Iowa State University Ames, Iowa 2010 Copyright © Michael Van Waardhuizen, 2010. All rights reserved. ii Table of Contents Table of Contents .......................................................................................................................ii List of Figures ............................................................................................................................ iv Abstract ...................................................................................................................................... v Chapter One: Introduction ........................................................................................................ 1 Direct Manipulation Interfaces ............................................................................................. 1 Augmented/Mixed Reality .................................................................................................... 2 Increased use of VR in design/manufacturing ...................................................................... 3 User Needs ............................................................................................................................ 3 The AugmenTable ................................................................................................................. 4 Chapter Two: Related Work...................................................................................................... 6 Augmented Reality................................................................................................................ 6 Gesture Interaction ............................................................................................................... 6 Hand Tracking ................................................................................................................... 7 Hand Interaction ............................................................................................................. 10 Comparable Systems........................................................................................................... 13 Chapter Three: Method .......................................................................................................... 17 Apparatus ............................................................................................................................ 17 Software Libraries ............................................................................................................... 19 Algorithm/Process .............................................................................................................. 22 Initialization..................................................................................................................... 22 Background Segmentation .............................................................................................. 24 Skin Detection ................................................................................................................. 26 Fingertip Detection ......................................................................................................... 30 2D to 3D .......................................................................................................................... 33 Tracking and Filtering ...................................................................................................... 36 Occlusions and the Illusion of Depth .............................................................................. 39 Chapter Four: Application & Discussion ................................................................................. 42 Object Manipulation Prototype .......................................................................................... 42 Discussion............................................................................................................................ 45 Realized Benefits ............................................................................................................. 46 iii Research Contributions ................................................................................................... 46 Challenges ....................................................................................................................... 47 System Limitations .......................................................................................................... 49 Future Work ........................................................................................................................ 49 Application ...................................................................................................................... 50 Apparatus ........................................................................................................................ 51 Concept ........................................................................................................................... 51 Chapter Six: Conclusion .......................................................................................................... 54 Bibliography ............................................................................................................................ 55 Appendix: Project Source Code .............................................................................................. 62 Background.h ...................................................................................................................... 62 Background.cpp .................................................................................................................. 62 Cam.h .................................................................................................................................. 64 Cam.cpp .............................................................................................................................. 67 Fingerpoint.h ....................................................................................................................... 76 Fingerpoint.cpp ................................................................................................................... 77 IntersectionTester.h ............................................................................................................ 85 IntersectionTester.cpp ........................................................................................................ 87 ManipulatorWidget.h ......................................................................................................... 95 ManipulatorWidget.cpp ...................................................................................................... 96 PtFilter.h ............................................................................................................................ 101 Main.cpp ........................................................................................................................... 103 iv List of Figures FIGURE 1: AUGMEN TABLE - FRONT ................................................................................................... 4 FIGURE 2: AUGMEN TABLE - BACK ..................................................................................................... 4 FIGURE 3: SIMPLIFIED HAND MODEL OF FING ART IPS [13] .................................................................... 7 FIGURE 4: SPIDAR-8 HAPTIC AR [17] ............................................................................................. 8 FIGURE 5: IBID . .............................................................................................................................. 8 FIGURE 6: EXAMPLE RAY -CASTING SYSTEM SYTEM [33] ....................................................................... 11 FIGURE 7: HAND AR SNAPSHOTS [43] ............................................................................................. 14 FIGURE 8: SIMILAR WORKSPACE CONCEPT [47] ................................................................................ 15 FIGURE 9: AUGMEN TABLE APPARATUS ............................................................................................ 17 FIGURE 10: THREADS /A LGORITHMS