DOCSLIB.ORG

Supporting Everyday Activities Through Always-Available Mobile Computing

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Supporting Everyday Activities Through Always-Available Mobile Computing ©Copyright 2010 Timothy Scott Saponas Supporting Everyday Activities through Always-Available Mobile Computing Timothy Scott Saponas A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy University of Washington 2010 Program Authorized to Offer Degree: Department of Computer Science and Engineering University of Washington Graduate School This is to certify that I have examined this copy of a doctoral dissertation by Timothy Scott Saponas and have found that it is complete and satisfactory in all respects, and that any and all revisions required by the final examining committee have been made. Co-Chairs of the Supervisory Committee: James A. Landay Desney S. Tan Reading Committee: James A. Landay Desney S. Tan James A. Fogarty Date:______________________________________ In presenting this dissertation in partial fulfillment of the requirements for the doctoral degree at the University of Washington, I agree that the Library shall make its copies freely available for inspection. I further agree that extensive copying of the dissertation is allowable only for scholarly purposes, consistent with ―fair use‖ as prescribed in the U.S. Copyright Law. Requests for copying or reproduction of this dissertation may be referred to ProQuest Information and Learning, 300 North Zeeb Road, Ann Arbor, MI 48106-1346, 1-800-521-0600, to whom the author has granted ―the right to reproduce and sell (a) copies of the manuscript in microform and/or (b) printed copies of the manuscript made from microform.‖ Signature __________________________ Date ______________________________ University of Washington Abstract Supporting Everyday Activities through Always-Available Mobile Computing T. Scott Saponas Co-Chairs of the Supervisory Committee: Professor James A. Landay Computer Science and Engineering Affiliate Professor Desney S. Tan Computer Science and Engineering Ubiquitous Computing provides us a vision of computing fading into the background and gracefully supporting our everyday activities. However, our interactions with mobile devices still demand so much of our physical, visual, and cognitive attention that mobile applications are tasks unto themselves, often interrupting the activities they seek to support. For example, the physical interfaces of portable music players are well designed for situations where people can see the controls while holding the device in their hand. However, when people use these devices to listen to music or manage their workout while exercising, they often are forced to interrupt their activity just to advance to the next song. As our primary use of computing continues to move off the desktop, we need new interfaces to mobile computing that expand the types of situations in which people can make use of computing. In this dissertation, I explore always-available interfaces to improve our current interactions and to enable new mobile computing opportunities. The experiments I have conducted and the systems I have built demonstrate my thesis that: Finger level gestures detected and classified through forearm electromyography can enable an always-available interaction paradigm for mobile computing. TABLE OF CONTENTS List of Figures ................................................................................................................................. iii List of Tables ................................................................................................................................. vii Chapter 1 Introduction ................................................................................................................ 1 1.1 Hypotheses ..................................................................................................................... 8 1.2 Contributions .................................................................................................................. 8 1.3 Research Approach ....................................................................................................... 12 1.4 Dissertation Outline ...................................................................................................... 13 Chapter 2 Related Work ........................................................................................................... 15 2.1 Natural User Interfaces ................................................................................................. 15 2.2 On-Body and Wearable Interfaces ............................................................................... 19 2.3 Muscle Sensing with Electromyography (EMG) ......................................................... 24 2.4 Summary....................................................................................................................... 28 Chapter 3 Offline Classification of Finger-Level Movements from the Upper Forearm ......... 29 3.1 Introduction .................................................................................................................. 29 3.2 Gesture Sets .................................................................................................................. 30 3.3 Experiment ................................................................................................................... 32 3.4 Discussion..................................................................................................................... 44 3.5 Summary....................................................................................................................... 46 Chapter 4 Real-Time Classification of Free-Space and Hands Busy Gestures ........................ 49 4.1 Natural Human Grips ................................................................................................... 50 4.2 Gesture Sets .................................................................................................................. 51 4.3 Gesture Classification ................................................................................................... 52 4.4 Experiment ................................................................................................................... 56 4.5 Summary....................................................................................................................... 66 Chapter 5 Making Muscle-Computer Interfaces More Practical .............................................. 71 5.1 Wireless Platform ......................................................................................................... 71 5.2 Cross-Session Classification Experiment ..................................................................... 75 5.3 Summary....................................................................................................................... 79 Chapter 6 Applications of Muscle-Computer Interfaces .......................................................... 81 6.1 Air Guitar Hero ............................................................................................................. 81 6.2 Augmenting Interactive Surfaces with Muscle-Computer Input .................................. 88 i 6.3 Summary ..................................................................................................................... 104 Chapter 7 Future Work ........................................................................................................... 105 7.1 On-Body Input ............................................................................................................ 106 7.2 Output for Always-Available Computing .................................................................. 110 7.3 Applications ................................................................................................................ 115 7.4 Summary ..................................................................................................................... 118 Chapter 8 Conclusion.............................................................................................................. 119 8.1 Thesis .......................................................................................................................... 120 8.2 Contributions .............................................................................................................. 120 8.3 Limitations .................................................................................................................. 122 8.4 Reflections and Insights.............................................................................................. 124 8.5 Final Remarks ............................................................................................................. 126 References .................................................................................................................................... 127 Appendix A. Questionnaire from Muscle-Computer Interfaces Feasibility Experiment ........ 151 Appendix B. Questionnaire from Real-Time, In-Air Gesture Classification Experiment ...... 153 Appendix C. Questionnaire from Cross-Session Classification Experiment .......................... 155 Appendix D. Questionnaire from Surfaces with Muscle-Computer Input Experiment .......... 157 ii LIST OF FIGURES Figure 1-1: Marketing image of people playing the Kinect Sports game using the forthcoming Kinect Sensor for the XBOX 360. The Kinect Sensor is a no-controller approach to controlling video games employing computer vision using the combination of a depth camera and a traditional
Load more

Recommended publications
  • Surface Computing.Pdf
    SURFACE COMPUTING QIS INSTITUE OF TECHNOLOGY, VENGAMUKKAPALEM, ONGOLE PRESENTED BY, GANGAVARAPU SRAVANI, K.SIVA KRISHNA, IIND B.TECH. CONTENTS : 1.Abstract 2. Introduction 3. Microsoft Surface Computer - Multi Touch Technology 4. Experience Microsoft Surface 5. Intelligent tables 6. Surface Computing 7. Surface Computing Technology 8. Features 9. Applications 10. Future enhancements 11. Conclusion 12.References ABSTRACT: commercially available towards the end of the year. A Surface computer is able to recognize physical objects from a paintbrush It’s an interesting product in that it’s to a cell phone and allows hands-on, direct completely out of left field. Microsoft gives control of content such as photos, music and examples of ordering a beverage during a maps. Surface turns an ordinary tabletop meal with just the tap of a finger and quickly into a dynamic surface that provides browsing through music and dragging interaction with all forms of digital content favorite songs onto a personal playlist by through natural gestures, touch and physical moving a finger across the screen. Build this objects. into a bar and you’d get one-touch beer service although I’m not sure if they’ve found a way to work out when your beer glass is empty so replenishment becomes automatic, maybe in a later version. All Things Digital conference Wednesday, Microsoft CEO Steve Ballmer will unveil Microsoft Surface, the first in a new category of surface computing products from Microsoft that will “break down traditional barriers between people and technology”. INTRODUCTION: Bill Gates introduced a new 150-pound coffee table ‘surface computer’ computer on May 30th.
    [Show full text]
  • Designing Tabletop and Surface Applications Using Interactive Prototypes
    UNIVERSITY OF CALGARY Designing Tabletop and Surface Applications Using Interactive Prototypes by Túlio de Souza Alcantara A THESIS SUBMITTED TO THE FACULTY OF GRADUATE STUDIES IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE DEPARTMENT OF COMPUTER SCIENCE CALGARY, ALBERTA JANUARY, 2013 © Túlio de Souza Alcantara 2013 ABSTRACT Innovative possibilities of interacting with touch-based devices come with the cost of having interaction designers to determine if users consider the interactions natural and easy to use. It is safe to assume that user-centered design helps creating applications that better fit user requirements and one of its steps is prototyping. However, interaction designers sometimes skip the prototyping phase due to time or economic constraints or for a lack of proper tools. In order to help designers of touch-based applications this research presents an approach that allows designers to create their own application-specific hand, finger or tag-based gestures and evaluate these gestures using sketch-based prototypes for touch-based applications without requiring any textual programming. The development of a sketch based prototyping tool followed a user centered design approach with requirements provided by user experience designers from industry. The whole application suite was evaluated to determine its fitness for helping designers of touch-based applications. The evaluation process was conducted in three steps: first, a study with software developers determined the efficacy of the tool to create custom gestures for performing tasks in sketch based prototypes. A follow up evaluation investigated the performance of the tool as a prototyping tool for tangible applications. Finally this thesis reports two cases where this solution helped in the design of real touch-based applications.
    [Show full text]
  • Exploring the Use of Tangible Interfaces for Occupational Therapy in Dementia Care
    Tangible Memories: Exploring the use of tangible interfaces for occupational therapy in dementia care Patrizia Murko, Cristophe Kunze Furtwangen University, GE Abstract In the care of people with dementia (PwD), occupational therapies are taking on a more and more important role. Only a few pilot projects could be identified which report on positive effects of computer-assisted therapy or technology-based memory care for PwD, mostly based on tablet PCs or touch screen computers. So far, the use of tangible interfaces such as surface computing and interactive objects is entirely unexplored. The goal of the presented research is to explore the application of surface computing and tangible interfaces for PwD. Three application prototypes were designed, implemented and tested during a two-week trial period with the use of a surface computer in an elderly care facility in order to examine the usability and the impact on people with dementia. A total of 14 participants with dementia were included in the evaluation. Among the participants, all stages of dementia were represented. The applications led to enjoyable conversations between caregivers and residents and encouraged wallowing in memories. For the majority of participants fun and joy were clearly visible. The results show considerable effects in terms of increasing interaction of the participants compared to conventional touch-screen applications. Keywords: Dementia, computer-aided occupational therapy, memory care, Surface Computing, Tangible Interaction, Usability & User Experience 1 Proceedings of the Third European Conference on Design4Health 2015, Sheffield, 13 - 16 July 2015 ISBN 978-1-84387-385-3 Introduction Background The number of older people with dementia increases.
    [Show full text]
  • Surface Computing
    SEMINAR ON Surface Computing BY RABINDRA KUMAR SAHU REGD. NO: 0601294084 B.Tech, 7th SEM. INFORMATION TECHNOLOGY Raajdhani Engineering College. IN THIS SESSION WHAT IS COMPUTING? WHAT IS SURFACE COMPUTING? TIME LINE. WHAT SURFACE CAN DO? APPLICATIONS DEVELOPMENTS SMART TABLE TABLE SPECIFICATION SOFTWARE / HARDWARE USED. THE TECHNOLOGY. HOW FLEXIBLE IT IS? THE POND EFFECT. DIGITAL CAMERA TO MOBILE PRACTICAL USE +VE AND –VE CONCLUSION. What is computing? • Computing to mean any goal-oriented activity requiring, benefiting from or creating computers. • Computing includes designing and building hardware and software system for a wide range of purposes; • Processing, structuring, and managing various kinds of information; • Doing scientific studies using computers; making computer systems behave intelligently; • Creating and using communications and entertainment media; finding and gathering information relevant to any particular purpose, and so on. The list is virtually endless, and the possibilities are vast. WHAT IS SURFACE COMPUTING? Microsoft Surface (Codename: Milan), is a Multi-touch product from Microsoft which is developed as a software and hardware combination technology that allows a user, or multiple users, to manipulate digital content by the use of natural motions, hand gestures, or physical objects Surface features a 30-inch tabletop display whose unique abilities allow for several people to work independently or simultaneously. All without using a mouse or a keyboard. TIME LINE • 2001: Microsoft researchers
    [Show full text]
  • SURFACE COMPUTING.Pdf
    Surface Computing SURFACE COMPUTING A SEMINAR REPORT Submitted by CHITHIRA SAJEEV In partial fulfillment for the award of the degree Of B-TECH DEGREE In COMPUTER SCIENCE & ENGINEERING SCHOOL OF ENGINEERING COCHIN UNIVERSITY OF SCIENCE & TECHNOLOGY KOCHI- 682022 JULY, 2010 Division of Computer Science, SOE 1 Surface Computing Division of Computer Engineering School of Engineering Cochin University of Science & Technology Kochi-682022 ____________________________________________________ _____ CERTIFICATE Certified that this is a bonafide record of the seminar work titled Surface computing Done by Chithira Sajeev of VII semester Computer Science & Engineering in the year 2010 in partial fulfillment of the requirements for the award of Degree of Bachelor of Technology in Computer Science & Engineering of Cochin University of Science & Technology Dr.David Peter S Deepa Paul Head of the Division Seminar Guide Division of Computer Science, SOE 2 Surface Computing ACKNOWLEDGEMENT I thank GOD almighty for guiding me throughout the seminar. I would like to thank all those who have contributed to the completion of the seminar and helped me with valuable suggestions for improvement. I am extremely grateful to Dr. David Peter, HOD, Division of Computer Science, for providing me with best facilities and atmosphere for the creative work guidance and encouragement. I would like to thank my coordinator, Mr.Sudheep Elayidom, Sr. Lecturer, Division of Computer Science, and my guide Mrs. Deepa Paul , Lecturer , Division of Computer Science , SOE for all help and support extended to me. I thank all the Staff members of my college and my friends for extending their cooperation during my seminar. Above all I would like to thank my parents without whose blessings; I would not have been able to accomplish my goal.
    [Show full text]
  • A Hybrid Touch Interface for Prêt À Voter
    A Hybrid Touch Interface for Prêt à Voter Chris Culnane, University of Surrey Trustworthy Voting Systems Project Overview • Brief Introduction to Prêt à Voter • Motivation • Hybrid Touch Concept • Implementation Prototypes • Future Work Voting in Prêt à Voter • Fill in ballot form • Detach and destroy Left Hand Side • Submit Right Hand Side to the system (scan) • Receive signed receipt from Web Bulletin Board (WBB) • Right Hand Side is published on WBB Properties • Privacy – The Right Hand Side does not reveal how you voted – No machine learns how you voted • End-to-end verifiability – Voters can check the inclusion of their vote – 3rd parties can check each operation • Strong integrity guarantees • Accessibility? Accessibility of Voting in the UK • Limited – Tactile Voting Device (2001) • Polls Apart Campaign – Reports after each election – Works with Electoral Commission Motivation • Accessibility is a requirement of any new voting scheme in the UK • Round table discussion with Polls Apart 2010 • Polls Apart Report 2010 recommendation: – “Ensure disabled people can vote independently and in secret by diversifying voting methods...” Motivation • We often have to make a pragmatic choice about accessibility and privacy – If we want the system to help the voter cast their vote, it generally has to learn how they vote • Can we develop a front-end, for Prêt à Voter, which does not compromise on privacy? Previous Ideas • Previously suggestions have been to line up a left hand side paper ballot with a touch screen – Not really usable, let alone
    [Show full text]
  • Smart Workstation with a Kinect-Projector Assistance System for Manual Assembly
    Lorenzo Andreoli Smart Workstation with a Kinect-Projector Assistance System for Manual Assembly Master’s thesis in Global Manufacturing Management Supervisor: Fabio Sgarbossa June 2019 Master’s thesis Master’s NTNU Faculty of Engineering Faculty Norwegian University of Science and Technology of Science University Norwegian Department of Mechanical and Industrial Engineering and Industrial Department of Mechanical Lorenzo Andreoli Smart Workstation with a Kinect-Projector Assistance System for Manual Assembly Master’s thesis in Global Manufacturing Management Supervisor: Fabio Sgarbossa June 2019 Norwegian University of Science and Technology Faculty of Engineering Department of Mechanical and Industrial Engineering Executive Summary Although automation has increased more and more in manufacturing companies over the last decades, manual labor is still used in a variety of complex tasks and is currently irreplaceable, especially in assembly operations. The problem of assisting and supporting the human worker during potentially complex assembly tasks is, therefore, very relevant. Clear and easy-to-read assembly instructions, error-proofing methods, and an intuitive user interface for the worker have the potential to reduce the cognitive workload of the operator, increase the productivity, improve the quality, reduce defects, and consequently reduce costs. Digital technologies such as augmented reality and motion recognition sensors can help assembly workers in their tasks and have been subject of research with increased interest over the years. In this thesis, we develop a prototype of a smart workstation equipped with a Kinect-projector assistance system for manual assembly. By following a V-model for systems development approach, we identify key requirements for assistance systems for both continuously supporting workers and teaching assembly steps to workers.
    [Show full text]
  • Most Flexible Human Computer Interactions Using Surface Computing Technology
    ISSN No. 0976-5697 Volume 2, No. 4, July-August 2011 International Journal of Advanced Research in Computer Science TECHNICAL NOTE Available Online at www.ijarcs.info Most Flexible Human Computer Interactions Using Surface Computing Technology Nousherwan Malik* M. Aqeel Iqbal, Department of Software Engineering Department of Software Engineering Faculty of Engineering and IT, FUIEMS Faculty of Engineering and IT, FUIEMS Rawalpindi, Pakistan Rawalpindi, Pakistan [email protected] [email protected] Haider Ali Sardar Waqar Shahroon Department of Software Engineering Department of Software Engineering Faculty of Engineering and IT, FUIEMS Faculty of Engineering and IT, FUIEMS Rawalpindi, Pakistan Rawalpindi, Pakistan [email protected] Muhammad Naqash Department of Software Engineering Faculty of Engineering and IT, FUIEMS Rawalpindi, Pakistan [email protected] Abstract: Surface computing introduces multi touch technology which is multi figure and multi user interface which is dancing around computer science since last few decades. In this technology computer vision is created by series of infrared cameras are installed in it. It has a lot of wireless cameras which is built in it surface computing platform. Now it is being used in commercial purpose in hotels, casinos. It can be used by gestures. The main functions of this technology is direct interaction, multi touch interaction, multi user experience and object recognition. In this paper a critical review has been presented about the surface computing technology which is an unusual forum of human computer interactions. Keywords: Surface; computing; multi touch; wireless cameras touching and grabbing. Music player or music files can be I. INTRODUCTION easily dragged around between two devices. Digital pictures can be easily resized and can be easily With the passage of time technology is shifting gears moved around table by using fingers.
    [Show full text]
  • Understanding Multi-Touch Manipulation for Surface Computing
    Understanding Multi-touch Manipulation for Surface Computing Chris North1, Tim Dwyer2, Bongshin Lee2, Danyel Fisher2, Petra Isenberg3, George Robertson2 and Kori Inkpen2 1 Virginia Tech, Blacksburg, VA, USA {[email protected]} 2 Microsoft Research, Redmond, WA, USA {t-tdwyer,danyelf,bongshin,ggr,[email protected]} 3 University of Calgary, Alberta, Canada {[email protected]} Abstract. Two-handed, multi-touch surface computing provides a scope for in- teractions that are closer analogues to physical interactions than classical win- dowed interfaces. The design of natural and intuitive gestures is a difficult problem as we do not know how users will approach a new multi-touch inter- face and which gestures they will attempt to use. In this paper we study whether familiarity with other environments influences how users approach interaction with a multi-touch surface computer as well as how efficiently those users complete a simple task. Inspired by the need for object manipulation in infor- mation visualization applications, we asked users to carry out an object sorting task on a physical table, on a tabletop display, and on a desktop computer with a mouse. To compare users’ gestures we produced a vocabulary of manipula- tion techniques that users apply in the physical world and we compare this vo- cabulary to the set of gestures that users attempted on the surface without train- ing. We find that users who start with the physical model finish the task faster when they move over to using the surface than users who start with the mouse. Keywords: Surface, Multi-touch, Gestures, Tabletop 1 Introduction The rapidly-developing world of multi-touch tabletop and surface computing is open- ing up new possibilities for interaction paradigms.
    [Show full text]
  • Enhancement in Surface Computing
    Journal of Basic and Applied Engineering Research p-ISSN: 2350-0077; e-ISSN: 2350-0255; Volume 3, Issue 12; October-December, 2016, pp. 1061-1063 © Krishi Sanskriti Publications http://www.krishisanskriti.org/Publication.html Enhancement in Surface Computing Chetanya Ahuja ITM University, Gurgaon E-mail: [email protected] Abstract—Our research summarizes the work of surface computing. acts as a mediator between devices (e.g. data exchange Surface computing is the term that is used for a specialized computer between digital camera and mobile phone). GUI in which traditional GUI components or elements are replaced by real world objects. The user interacts directly with the touch- 2. HISTORY sensitive screen rather than using keyboard or mouse. The device that uses this technology is known as surface computer. In this Microsoft surface computer is much like a blend of Disk research we have covered the basic features, history, working and Operating System and GUI (Graphical User Interface). It is a structure of the surface computing, applications, advantages and collaborative work of Microsoft Hardware and Microsoft disadvantages. Research. It is an idea inspired by Cross-Division Keywords: GUI,touch-sensitive,technology,real. Collaboration. In 2001, Stevie Bathiche from the department of Microsoft hardware and Andy Wilson from the department 1. INTRODUCTION of Microsoft Research began working together on various projects. In one of their regular brainstorm session, they thought of an idea of an interactive table that could possibly understand the manipulation of physical elements. They both saw the need for device where interaction was richer and more instinctive and at the same time practical for everyone to use.
    [Show full text]
  • Understanding Multi-Touch Manipulation for Surface Computing
    Understanding Multi-touch Manipulation for Surface Computing Chris North1, Tim Dwyer2, Bongshin Lee2, Danyel Fisher2, Petra Isenberg3, George Robertson2, Kori Inkpen Quinn2 1 Virginia Tech, Blacksburg, VA, USA {[email protected]} 2 Microsoft Research, Redmond, WA, USA {t-tdwyer,danyelf,bongshin,ggr,[email protected]} 3 University of Calgary, Alberta, Canada {[email protected]} Abstract. Two-handed, multi-touch surface computing provides a scope for in- teractions that are closer analogues to physical interactions than classical win- dowed interfaces. The design of natural and intuitive gestures is a difficult problem as we do not know how users will approach a new multi-touch inter- face and which gestures they will attempt to use. In this paper we study whether familiarity with other environments influences how users approach interaction with a multi-touch surface computer as well as how efficiently those users complete a simple task. Inspired by the need for object manipulation in infor- mation visualization applications, we asked users to carry out an object sorting task on a physical table, on a tabletop display, and on a desktop computer with a mouse. To compare users’ gestures we produced a vocabulary of manipula- tion techniques that users apply in the physical world and we compare this vo- cabulary to the set of gestures that users attempted on the surface without train- ing. We find that users who start with the physical model finish the task faster when they move over to using the surface than users who start with the mouse. Keywords: Surface, Multi-touch, Gestures, Tabletop 1 Introduction The rapidly-developing world of multi-touch tabletop and surface computing is open- ing up new possibilities for interaction paradigms.
    [Show full text]
  • 5. What Is Surface Computer ?
    SURFACE COMPUTER 1. INTRODUCTION Surface computing is the term for the use of a specialized computer GUI in which traditional GUI elements are replaced by intuitive, everyday objects. Instead of a keyboard and mouse, the user interacts directly with a touch-sensitive screen. It is a natural user interface. Surface computer was created by Microsoft with surface (code named Milan) The Surface is a horizontal display on a table-like form. A surface computer is a computer that interacts with the user through the surface of an ordinary object, rather than through a monitor and keyboard. A surface computer is able to recognize physical objects from a paintbrush to a cell phone and allows hands-on, direct control of content such as photos, music and maps. Surface turns an ordinary tabletop into a dynamic surface that provides interaction with all forms of digital content through natural gestures, touch and physical objects. 1 SURFACE COMPUTER 2. HISTORY An Idea Inspired by Cross-Division Collaboration Microsoft Research began working together on various projects that took advantage of their complementary expertise in the areas of hardware and software. In one of their regular brainstorm sessions, they started talking about an idea for an interactive table that could understand the manipulation of physical pieces. Although there were related efforts happening in academia, Bathiche and Wilson saw the need for a product where theinteraction was richer and more intuitive, and at the same time practical for everyone to use. This conversation was the beginning of an idea that would later result in the development of Surface, and over the course of the following year, various people at Microsoft involved in developing new product concepts, including the gaming-specific PlayTable, continued to think through the possibilities and feasibility of the project.
    [Show full text]