DOCSLIB.ORG

3D Gesture Recognition and Tracking for Next Generation of Smart Devices

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

3D Gesture Recognition and Tracking for Next Generation of Smart Devices Theories, Concepts, and Implementations SHAHROUZ YOUSEFI Department of Media Technology and Interaction Design School of Computer Science and Communication KTH Royal Institute of Technology Doctoral Thesis in Media Technology Stockholm, February 2014 3D Gesture Recognition and Tracking for Next Generation of Smart De- vices: Theories, Concepts, and Implementations Shahrouz Yousefi Department of Media Technology and Interaction Design (MID) School of Computer Science and Communication (CSC) KTH Royal Institute of Technology SE-100 44, Stockholm, Sweden Author's e-mail: [email protected] Akademisk avhandling som med tillst˚andav Kungliga Tekniska H¨ogskolan framl¨aggstill offentlig granskning f¨oravl¨aggandeav Teknologie Doktorsex- amen i Medieteknik, m˚andagenden 17 mars 2014 kl 13:15 i sal F3 Lindst- edsv¨agen26, Kungliga Tekniska H¨ogskolan, Stockholm. TRITA-CSC-A-2014-02 ISSN-1653-5723 ISRN-KTH/CSC/A{14/02-SE ISBN-978-91-7595-031-0 Copyright c 2014 by Shahrouz Yousefi, All rights reserved. Typeset in LATEX by Shahrouz Yousefi E-version available at http://kth.diva-portal.org Printed by E-print AB, Stockholm, Sweden, 2014 Distributor: KTH School of Computer Science and Communication Abstract The rapid development of mobile devices during the recent decade has been greatly driven by interaction and visualization technologies. Al- though touchscreens have significantly enhanced the interaction technol- ogy, it is predictable that with the future mobile devices, e.g., augmented reality glasses and smart watches, users will demand more intuitive in- puts such as free-hand interaction in 3D space. Specifically, for manipu- lation of the digital content in augmented environments, 3D hand/body gestures will be extremely required. Therefore, 3D gesture recognition and tracking are highly desired features for interaction design in future smart environments. Due to the complexity of the hand/body motions, and limitations of mobile devices in expensive computations, 3D gesture analysis is still an extremely difficult problem to solve. This thesis aims to introduce new concepts, theories and technologies for natural and intuitive interaction in future augmented environments. Contributions of this thesis support the concept of bare-hand 3D ges- tural interaction and interactive visualization on future smart devices. The introduced technical solutions enable an effective interaction in the 3D space around the smart device. High accuracy and robust 3D mo- tion analysis of the hand/body gestures is performed to facilitate the 3D interaction in various application scenarios. The proposed technologies enable users to control, manipulate, and organize the digital content in 3D space. Keywords: 3D gestural interaction, gesture recognition, gesture tracking, 3D visualization, 3D motion analysis, augmented environments. Shahrouz Yousefi February 2014 Sammanfattning Den snabba utvecklingen av mobila enheter under det senaste decenniet har i stor utstr¨ackning drivits av interaktion och visualiseringsteknologi. Aven¨ om peksk¨armaravsev¨arthar f¨orb¨attratinteraktions tekniken ¨ar det f¨oruts¨agbartatt med framtida mobila enheter, t.ex. augmented real- ity glas¨ogonoch smarta klockor, kommer anv¨andarekr¨aver mer intuitiva s¨attatt interagera, s˚asomex. fri hand interaktion i 3D-rymden. Speciellt viktigt blir det vid manipulation av digitalt inneh˚alli ut¨okade milj¨oer d¨ar3D hand/kropp gester kommer att vara ytterst n¨odv¨andig. D¨arf¨or ¨ar3D gest igenk¨anning och sp˚arningh¨ogt¨onskade egenskaper f¨orin- teraktionsdesign i framtida smarta milj¨oer. P˚agrund av komplexiteten i hand/kroppsr¨orelser,och begr¨ansningarav mobila enheter vid dyra ber¨akningar,¨ar3D- gest analys fortfarande ett mycket sv˚artproblem att l¨osa. Avhandlingen syftar till att inf¨oranya begrepp, teorier och tekniker f¨or naturlig och intuitiv interaktion i framtida ut¨okade milj¨oer.Bidrag fr˚an denna avhandling st¨oderbegreppet naken-hand 3D gest interaktion och interaktiv visualisering p˚aframtida smarta enheter. De inf¨ordatekniska l¨osningarm¨ojligg¨oreffektiv interaktion i 3D-rymden runt den smarta enheten. H¨ognoggrannhet och robust 3D r¨orelseanalysav hand/kropp gester utf¨orsf¨oratt underl¨atta3D-interaktion i olika till¨ampningsscenar- ier. De f¨oreslagnateknik g¨ordet m¨ojligtf¨oranv¨andareatt kontrollera, manipulera, och organiserar digitalt inneh˚alli 3D-rymden. Nyckelord: 3D gest interaktion, gest igenk¨annande,gest sp˚arning,3D visualisering, 3D r¨orelseanalys, ut¨okademilj¨oer. Shahrouz Yousefi februari 2014 Acknowledgements First of all, I wish to express my sincere gratitude to my main ad- visor, Prof. Haibo Li, for providing me this research opportunity. Thank you for your motivation, enthusiasm, and support during these years. Without your supervision and mentoring this thesis would not have been possible. You inspired me to be more adven- turous in research. I would like to thank my second advisor, Dr. Li Liu, for all the motivational and fruitful discussions. Special thanks to my dear friend and colleague, Farid Kondori. We had many collaborations, interesting discussions and enjoyable moments during these years. I would like to thank my former colleagues at Digital Media Lab, Ume˚aUniversity for their helpful suggestions and comments on my research projects. Special thanks to Annemaj Nilsson, Mona-Lisa Gunnarsson, and the friendly staff of the department of Applied Physics and Electronics, Ume˚aUniversity. My time at KTH was really enjoyable due to the friendly col- leagues of the department of Media Technology and Interaction Design. I am grateful for the time spent with them at work meet- ings, seminars and social events. I must especially thank Prof. Ann Lantz for providing an excellent research environment at the MID department. Thanks for your support, encouragement and kind- ness. I would also like to thank Henrik Artman, Cristian Bogdan, Ambj¨ornNaeve, Olle B¨alter,Eva-Lotta Salln¨as,and other senior researchers at MID department for their support and guidance. Many thanks should go to Dr. Roberto Bresin and Prof. Yngve Sundblad for reviewing my thesis. Your constructive ideas, in- sightful comments, and suggestions made a great improvement in the quality of my PhD thesis. Winning the first prize in KTH Innovation Idea Competition, best project work in Uminova Academic Business Challenge, and being selected as one of the top PhD works at ACM Multimedia Doctoral Symposium, motivated me to work harder on the development of my research ideas. I would especially like to thank H˚akan Borg and Cecilia Sandell from KTH Innovation for their great support on patentability analysis, business development and commercial- ization of my research results. Finally, and most importantly, I am grateful to my loving par- ents, my brother, and his family for giving me the endless intellec- tual support and encouragement to pursue my studies during these years. I would especially like to thank my best friend and com- panion Shora. Thanks for the wonderful and precious moments we shared together. Shahrouz Yousefi February 2014 Contents Contents v 1 Introduction 1 1.1 Motivation . 1 1.2 Research Problem . 3 1.2.1 Future Mobile Devices . 4 1.2.2 Experience Design . 4 1.2.3 Limitations in Interaction Facilities . 6 1.2.4 Limitations in Visualization . 8 1.2.5 Technical Challenges in 3D Gestural Interaction . 8 1.3 Future Trends in Multimedia Context . 10 1.3.1 3D Interaction Technology . 10 1.3.2 3D Visualization . 10 1.3.3 Passive Vision to Active/Interactive Vision . 10 1.3.4 Gesture Analysis: from Computer Vision Methods to Image-based Search Methods . 11 1.4 Research Strategy . 11 2 Related Work 13 2.1 Terminology . 13 2.2 Related Work . 14 2.2.1 3D Motion Capture Technologies in Available Interactive Systems . 15 v CONTENTS 2.2.1.1 Passive Motion Tracking and Its Applications 15 2.2.1.2 Active Motion Tracking and Its Applications . 16 2.2.1.3 Comparison Between Active and Passive Meth- ods . 16 2.2.2 3D Motion Estimation for Mobile Interaction . 18 2.2.3 3D Gesture Recognition and Tracking . 19 2.2.4 3D Visualization on Mobile Devices . 21 3 General Concept and Methodology 23 3.1 General Concept . 23 3.1.1 Interaction/Visualization Space . 25 3.1.2 Sharing the Interaction/Visualization space . 27 3.1.2.1 Single-user, Single-device . 27 3.1.2.2 Multi-user, Multi-device with Shared Interac- tion Space . 28 3.1.2.3 Multi-user, Single-device with Shared Visual- ization Space . 28 3.1.2.4 Interaction from Different Locations for Multi- user Multi-device . 28 3.2 Evolution of Interaction/Visualization Spaces . 28 3.3 Enabling Media Technologies . 31 3.3.1 Vision-based Motion Tracking in 3D Space . 32 3.3.2 3D Visualization . 33 3.4 Methodology Overview . 36 3.5 Gesture Analysis through the Pattern Recognition Methods . 38 3.6 Gesture Analysis through the Large-scale Image Retrieval . 40 4 Enabling Media Technologies 43 4.1 Gesture Detection and Tracking Based on Low-level Pattern Recognition . 44 vi CONTENTS 4.1.1 3D Motion Analysis . 46 4.2 Gesture Detection and Tracking Based on Gesture Search Engine . 48 4.2.1 Providing the Database of Gesture Images . 49 4.2.2 Query Processing and Matching . 50 4.2.3 Scoring System . 50 4.2.4 Quality of Hand Gesture Database . 52 4.3 Interactive 3D Visualization . 54 4.4 Methods for 3D Visualization . 56 4.4.1 Depth Recovery and 3D Visualization from a Single View 56 4.4.2 3D Visualization from Multiple 2D Views . 57 4.5 3D Channel Coding . 57 5 Experimental Results 59 5.1 Experiments on Gesture Detection, Tracking and 3D Motion Analysis . 59 5.1.1 Camera and Experiment Condition . 59 5.1.2 Algorithm . 60 5.1.3 Programming Environment and Results . 62 5.2 Experiments on Gesture Search Framework . 63 5.2.1 Constructing the Database . 63 5.2.2 Forming the Vocabulary Table .
Recommended publications
  • Gesture Recognition for Human-Robot Collaboration: a Review

    Gesture Recognition for Human-Robot Collaboration: a Review

    International Journal of Industrial Ergonomics xxx (2017) 1e13 Contents lists available at ScienceDirect International Journal of Industrial Ergonomics journal homepage: www.elsevier.com/locate/ergon Gesture recognition for human-robot collaboration: A review * Hongyi Liu, Lihui Wang KTH Royal Institute of Technology, Department of Production Engineering, Stockholm, Sweden article info abstract Article history: Recently, the concept of human-robot collaboration has raised many research interests. Instead of robots Received 20 July 2016 replacing human workers in workplaces, human-robot collaboration allows human workers and robots Received in revised form working together in a shared manufacturing environment. Human-robot collaboration can release hu- 15 November 2016 man workers from heavy tasks with assistive robots if effective communication channels between Accepted 15 February 2017 humans and robots are established. Although the communication channels between human workers and Available online xxx robots are still limited, gesture recognition has been effectively applied as the interface between humans and computers for long time. Covering some of the most important technologies and algorithms of Keywords: Human-robot collaboration gesture recognition, this paper is intended to provide an overview of the gesture recognition research Gesture and explore the possibility to apply gesture recognition in human-robot collaborative manufacturing. In Gesture recognition this paper, an overall model of gesture recognition for human-robot collaboration is also proposed. There are four essential technical components in the model of gesture recognition for human-robot collabo- ration: sensor technologies, gesture identification, gesture tracking and gesture classification. Reviewed approaches are classified according to the four essential technical components. Statistical analysis is also presented after technical analysis.
  • Inverse Kinematic Infrared Optical Finger Tracking

    Inverse Kinematic Infrared Optical Finger Tracking

    Inverse Kinematic Infrared Optical Finger Tracking Gerrit Hillebrand1, Martin Bauer2, Kurt Achatz1, Gudrun Klinker2 1Advanced Realtime Tracking GmbH ∗ Am Oferl¨ 3 Weilheim, Germany 2Technische Universit¨atM¨unchen Institut f¨urInformatik Boltzmannstr. 3, 85748 Garching b. M¨unchen, Germany e·mail: [email protected] Abstract the fingertips are tracked. All further parameters, such as the angles between the phalanx bones, are Human hand and finger tracking is an important calculated from that. This approach is inverse input modality for both virtual and augmented compared to the current glove technology and re- reality. We present a novel device that overcomes sults in much better accuracy. Additionally, the the disadvantages of current glove-based or vision- device is lightweight and comfortable to use. based solutions by using inverse-kinematic models of the human hand. The fingertips are tracked by an optical infrared tracking system and the pose of the phalanxes is calculated from the known anatomy of the hand. The new device is lightweight and accurate and allows almost arbitrary hand movement. Exami- nations of the flexibility of the hand have shown that this new approach is practical because ambi- guities in the relationship between finger tip posi- tions and joint positions, which are theoretically possible, occur only scarcely in practical use. Figure 1: The inverse kinematic infrared optical finger tracker Keywords: Optical Tracking, Finger Tracking, Motion Capture 1 Introduction 1.1 Related Work While even a standard keyboard can be seen For many people, the most important and natural as an input device based on human finger mo- way to interact with the environment is by using tion, we consider only input methods that allow their hands.
  • Study of Gesture Recognition Methods and Augmented Reality

    Study of Gesture Recognition Methods and Augmented Reality

    Study of Gesture Recognition methods and augmented reality Amol Palve ,Sandeep Vasave Department of Computer Engineering MIT COLLEGE OF ENGINEERING ( [email protected] ,[email protected]) Abstract: However recognizing the gestures in the noisy background has always been a With the growing technology, we difficult task to achieve. In the proposed humans always need something that stands system, we are going to use one such out from the other thing. Gestures are most technique called Augmentation in Image desirable source to Communicate with the processing to control Media Player. We will Machines. Human Computer Interaction recognize Gestures using which we are going finds its importance when it comes to to control the operations on Media player. working with the Human gestures to control Augmentation usually is one step ahead when the computer applications. Usually we control it comes to virtual reality. It has no the applications using mouse, keyboard, laser restrictions on the background. Moreover it pointers etc. but , with the recent advent in also doesn’t rely on certain things like gloves, the technology it has even left behind their color pointers etc. for recognizing the gesture. usage by introducing more efficient This system mainly appeals to those users techniques to control applications. There are who always looks out for a better option that many Gesture Recognition techniques that makes their interaction with computer more have been implemented using image simpler or easier. processing in the past. Keywords: Gesture Recognition, Human Computer Interaction, Augmentation I.INTRODUCTION: Augmentation in controlling the media player operations. Media player always In Human Computer interaction, the first requires mouse or keyboard to control it.
  • Airpen: a Touchless Fingertip Based Gestural Interface for Smartphones and Head-Mounted Devices

    Airpen: a Touchless Fingertip Based Gestural Interface for Smartphones and Head-Mounted Devices

    AirPen: A Touchless Fingertip Based Gestural Interface for Smartphones and Head-Mounted Devices Varun Jain Ramya Hebbalaguppe [email protected] [email protected] TCS Research TCS Research New Delhi, India New Delhi, India ABSTRACT Hand gestures are an intuitive, socially acceptable, and a non-intrusive interaction modality in Mixed Reality (MR) and smartphone based applications. Unlike speech interfaces, they tend to perform well even in shared and public spaces. Hand gestures can also be used to interact with smartphones in situations where the user’s ability to physically touch the device is impaired. However, accurate gesture recognition can be achieved through state-of-the-art deep learning models or with the use of expensive sensors. Despite the robustness of these deep learning models, they are computationally heavy and memory hungry, and obtaining real-time performance on-device without additional hardware is still a challenge. To address this, we propose AirPen: an analogue to pen on paper, but in air, for in-air writing and gestural commands that works seamlessly in First and Second Person View. The models are trained on a GPU machine and ported on an Android smartphone. AirPen comprises of three deep learning models that work in tandem: MobileNetV2 for hand localisation, our custom fingertip regression architecture followed by a Bi-LSTM model for gesture classification. The overall framework works in real-time on mobile devices and achieves a classification accuracy of 80% with an average latency of only 0:12 s. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.
  • Auraring: Precise Electromagnetic Finger Tracking

    Auraring: Precise Electromagnetic Finger Tracking

    AuraRing: Precise Electromagnetic Finger Tracking FARSHID SALEMI PARIZI∗, ERIC WHITMIRE∗, and SHWETAK PATEL, University of Washington Fig. 1. AuraRing is 5-DoF electromagnetic tracker that enables precise, accurate, and fine-grained finger tracking for AR, VR and wearable applications. Left: A user writes the word "hello" in the air. Right: Using AuraRing to play a songinamusic application on a smart glass. Wearable computing platforms, such as smartwatches and head-mounted mixed reality displays, demand new input devices for high-fidelity interaction. We present AuraRing, a wearable magnetic tracking system designed for tracking fine-grained finger movement. The hardware consists of a ring with an embedded electromagnetic transmitter coil and a wristband with multiple sensor coils. By measuring the magnetic fields at different points around the wrist, AuraRing estimates thefive degree-of-freedom pose of the ring. We develop two different approaches to pose reconstruction—a first-principles iterative approach and a closed-form neural network approach. Notably, AuraRing requires no runtime supervised training, ensuring user and session independence. AuraRing has a resolution of 0:1 mm and a dynamic accuracy of 4:4 mm, as measured through a user evaluation with optical ground truth. The ring is completely self-contained and consumes just 2:3 mW of power. CCS Concepts: • Human-centered computing → Interaction devices; Ubiquitous and mobile computing. Additional Key Words and Phrases: Electromagnetic tracking, mixed reality, wearable, input, finger tracking ACM Reference Format: Farshid Salemi Parizi, Eric Whitmire, and Shwetak Patel. 2019. AuraRing: Precise Electromagnetic Finger Tracking. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 4, Article 150 (December 2019), 28 pages.
  • Hand Interface Using Deep Learning in Immersive Virtual Reality

    Hand Interface Using Deep Learning in Immersive Virtual Reality

    electronics Article DeepHandsVR: Hand Interface Using Deep Learning in Immersive Virtual Reality Taeseok Kang 1, Minsu Chae 1, Eunbin Seo 1, Mingyu Kim 2 and Jinmo Kim 1,* 1 Division of Computer Engineering, Hansung University, Seoul 02876, Korea; [email protected] (T.K.); [email protected] (M.C.); [email protected] (E.S.) 2 Program in Visual Information Processing, Korea University, Seoul 02841, Korea; [email protected] * Correspondence: [email protected]; Tel.: +82-2-760-4046 Received: 28 September 2020; Accepted: 4 November 2020; Published: 6 November 2020 Abstract: This paper proposes a hand interface through a novel deep learning that provides easy and realistic interactions with hands in immersive virtual reality. The proposed interface is designed to provide a real-to-virtual direct hand interface using a controller to map a real hand gesture to a virtual hand in an easy and simple structure. In addition, a gesture-to-action interface that expresses the process of gesture to action in real-time without the necessity of a graphical user interface (GUI) used in existing interactive applications is proposed. This interface uses the method of applying image classification training process of capturing a 3D virtual hand gesture model as a 2D image using a deep learning model, convolutional neural network (CNN). The key objective of this process is to provide users with intuitive and realistic interactions that feature convenient operation in immersive virtual reality. To achieve this, an application that can compare and analyze the proposed interface and the existing GUI was developed. Next, a survey experiment was conducted to statistically analyze and evaluate the positive effects on the sense of presence through user satisfaction with the interface experience.
  • Lightweight Palm and Finger Tracking for Real-Time 3D Gesture Control

    Lightweight Palm and Finger Tracking for Real-Time 3D Gesture Control

    Lightweight Palm and Finger Tracking for Real-Time 3D Gesture Control Georg Hackenberg* Rod McCall‡ Wolfgang BrollD Fraunhofer FIT Fraunhofer FIT Fraunhofer FIT Freehand multi-touch has been explored within various ABSTRACT approaches, e.g. Oblong’s g-speak1, after initially having been We present a novel technique implementing barehanded introduced in the popular Hollywood movie “Minority Report”. interaction with virtual 3D content by employing a time-of-flight They usually depended heavily on hand-worn gloves, markers, camera. The system improves on existing 3D multi-touch systems wrists, or other input devices, and typically did not achieve the by working regardless of lighting conditions and supplying a intuitiveness, simplicity and efficiency of surface (2D) based working volume large enough for multiple users. Previous multi-touch techniques2. In contrast to those approaches, the goal systems were limited either by environmental requirements, of our approach was to use barehanded interaction as a working volume, or computational resources necessary for real- replacement for surface based interaction. time operation. By employing a time-of-flight camera, the system Only a vision-based approach will allow for freehand and is capable of reliably recognizing gestures at the finger level in barehanded 3D multi-touch interaction. The system must also real-time at more than 50 fps with commodity computer hardware provide sufficient solutions for the following four steps: detection using our newly developed precision hand and finger-tracking of hand position without prior knowledge of existence; for each algorithm. Building on this algorithm, the system performs appearance determine pose from image cues; track appearances gesture recognition with simple constraint modeling over over time; recognize gestures based on their trajectory and pose statistical aggregations of the hand appearances in a working information.
  • Improving Gesture Recognition Accuracy on Touch Screens for Users with Low Vision

    Improving Gesture Recognition Accuracy on Touch Screens for Users with Low Vision

    Improving Touch Interfaces CHI 2017, May 6–11, 2017, Denver, CO, USA Improving Gesture Recognition Accuracy on Touch Screens for Users with Low Vision Radu-Daniel Vatavu MintViz Lab MANSiD Research Center University Stefan cel Mare of Suceava Suceava 720229, Romania [email protected] Figure 1. Gesture articulations produced by people with low vision present more variation than gestures produced by people without visual impairments, which negatively affects recognizers’ accuracy rates. Ten (10) superimposed executions of a “star” gesture are shown for six people: a person without visual impairments (a); three people with congenital nystagmus and high myopia (b), (c), (d); and two people with chorioretinal degeneration (e), (f). ABSTRACT INTRODUCTION We contribute in this work on gesture recognition to improve Today’s touch screen devices are little accessible to people the accessibility of touch screens for people with low vision. with visual impairments, who need to employ workaround We examine the accuracy of popular recognizers for gestures strategies to be able to use them effectively and indepen- produced by people with and without visual impairments, and dently [17,18,42]. Because smart devices expose touch screens we show that the user-independent accuracy of $P, the best not adapted to non-visual input, touch and gesture inter- recognizer among those evaluated, is small for people with action pose many challenges to people with visual impair- low vision (83.8%), despite $P being very effective for ges- ments [11,12,20,30], which can only be addressed with a thor- tures produced by people without visual impairments (95.9%).
  • Controlling Robots with Wii Gestures

    Controlling Robots with Wii Gestures

    WiiRobot: Controlling Robots with Wii Gestures Aysun Bas¸c¸etinc¸elik Department of Computer Science Brown University Providence, RI, 02906 [email protected] May 15, 2009 Abstract In this project, we introduce WiiRobot, a sim- ple and reproducable gesture based robot con- troller. Our proposed system uses the Nintendo Wii Remote Controller to interact with a robot. The user first trains the system using Wii Re- mote in order to recognize future gestures. For the recognition process, we implemented two simple algorithms; K-Nearest Neighbour and sum of square differences. Our results show that using only a few number of repetitions per gesture is enough for recognition. 1 Introduction Finding a natural and effective way of communication with robots is important in order to enhance their contribution in our daily lives. For this purpose, there have been various developments in the area of Human- Figure 1: User controlling the robot with Wii Remote Robot Interaction. It is known that using a computer, a gamepad or a keyboard can be teleoperative but it is still unnatural and difficult to control. Hence, it is our fingers or with a mouse. We use gestures while we still one of the major research directions to design ease- are doing work with our laptops and phones. Moreoever, of-use and intuitive modes of communication with robots. after the introduction of Nintendo Wii Console, we were introduced to a new hardware, Wii Remote Controller, Another form of communication is gestures. Currently, which has the capability of creating gestures and using it gestures are not just limited to face, hand and body for a different intuitive control of video games.
  • Perception-Action Loop in the Experience of Virtual Environments

    Perception-Action Loop in the Experience of Virtual Environments

    Perception-Action Loop in the Experience of Virtual Environments by Seung Wook Kim A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Architecture and the Designated Emphasis in New Media in the Graduate Division of the University of California, Berkeley Committee in charge: Professor Yehuda E. Kalay, Chair Professor Galen Cranz Professor John F. Canny Fall 2009 Abstract Perception-Action Loop in the Experience of Virtual Environments by Seung Wook Kim Doctor of Philosophy in Architecture and the Designated Emphasis in New Media University of California, Berkeley Professor Yehuda Kalay, Chair The goal of this work is to develop an approach to the design of natural and immersive interaction methods for three-dimensional virtual environments. My thesis is that habitation and presence in any environment are based on a continuous process of perception and action between a person and his/her surroundings. The current practice of virtual environments, however, disconnects this intrinsic loop, separating perception and action into two different ‘worlds’—a physical one (for perception) and a virtual one (for action). This research is aimed at bridging the gap between those two worlds. Being drawn from perceptual philosophy and psychology, the theoretical study in this dissertation identifies three embodiments of natural perception-action loop: direct perceptual acts, proprioceptive locomotion, and motor intentionality. These concepts form the basis for the interaction methods proposed in this work, and I demonstrate these methods by implementing pertinent prototype systems. First, I suggest a view-dependent, non-planar display space that supports natural perceptual actions, thereby enhancing our field of view as well as depth perception.
  • A Wii-Based Gestural Interface for Computer Conducting Systems

    A Wii-Based Gestural Interface for Computer Conducting Systems

    A Wii-based gestural interface for computer conducting systems Lijuan Peng David Gerhard Computer Science Department Computer Science Department University of Regina University of Regina SK, Canada, S4S 0A2 SK, Canada, S4S 0A2 [email protected] [email protected] Abstract This system is developed using Max/MSP/Jitter 1 and Java. With the increase of sales of Wii game consoles, it is be- The user stands in front of the computer and the Wii remote coming commonplace for the Wii remote to be used as an is mounted with the camera pointing toward the user. The alternative input device for other computer systems. In this accelerometer functionality and buttons on the Wii remote paper, we present a system which makes use of the infrared are ignored. Unlike other video-camera-based infrared sys- camera within the Wii remote to capture the gestures of a tems, no positional configuration is required. The infrared conductor using a baton with an infrared LED and battery. baton is held in the user’s right hand. It should be noted that Our system then performs data analysis with gesture classi- as long as the camera in the Wii remote can ”see” the entire fication and following, and finally displays the gestures us- conducting window, the specific location of the Wii remote ing visual baton trajectories and audio feedback. Gesture is irrelevant. The setup should be configured so that the user trajectories are displayed in real time and can be compared has full range of conducting motion and is comfortable dur- to the corresponding diagram shown in a textbook.
  • Analysis and Categorization of 2D Multi-Touch Gesture Recognition Techniques THESIS Presented in Partial Fulfillment of the Requ

    Analysis and Categorization of 2D Multi-Touch Gesture Recognition Techniques THESIS Presented in Partial Fulfillment of the Requ

    Analysis and Categorization of 2D Multi-Touch Gesture Recognition Techniques THESIS Presented in Partial Fulfillment of the Requirements for the Degree Master of Science in the Graduate School of The Ohio State University By Aditi Singhal Graduate Program in Computer Science and Engineering The Ohio State University 2013 Master's Examination Committee: Dr. Rajiv Ramnath, Adviser Dr. Jay Ramanathan, Co-Adviser Copyright by Aditi Singhal 2013 Abstract Various techniques have been implemented for wide variety of gesture recognition applications. However, discerning the best technique to adopt for a specific gesture application is still a challenge. Wide variety, complexity in the gesture and the need for complex algorithms make implementation of recognition difficult even for basic gesture applications. In this thesis, different techniques for two dimensional (2D) touch gestures are reviewed, compared and categorized based on user requirements of the gesture applications. This work introduces two main paradigms for gesture applications: a) gestures for direct manipulation and navigation, b) gesture-based languages. These two paradigms have specific and separate roles in 2D touch gesture systems. To provide the clear distinction between two paradigms, three different algorithms are implemented for basic to complex 2D gestures using simple sometimes, as well as complex techniques such as linear regression and Hidden Markov Models. Thereafter, these algorithms are analyzed based on their performance and their fit with application requirements. ii Dedication I lovingly dedicate this thesis to my husband Shashnk Agrawal, who supported me each step of the way, my son Aryan Agrawal who in his 3 year old wisdom knew that mommy should not be disturbed while studying, and my late father Dr.