University of Nevada Reno Non-Visual Natural User Interfaces A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Computer Science and Engineering by Anthony Morelli Dr. Eelke Folmer/Dissertation Advisor December, 2011 i Abstract Natural user interfaces (NUI) have recently become popular due to their character- istics that capitalize on a user's ability to utilize skills acquired through real world experiences. NUI's provide a method of device interaction that may be easier when compared to a traditional graphical user interfaces because users can interact by using gestures and motions which mimic motions that are used outside of device interaction. One example of NUI's is the recent trend in exergames - video games that require actions such as running in place, moving the arm in a bowling motion, or swing- ing the arm as if to swing a tennis racquet. Unfortunately, most of the cues given to the player to perform these natural motions are visual cues which makes it diffi- cult for people who have visual impairments to participate in these activities. This dissertation investigates non-visual natural user interfaces. Three exergames were created to show techniques for using a combination of haptic and audio cues in order to promote physical activity for people who are visually impaired. A method, Real Time Sensory Substitution, was developed which allowed people who are visually im- paired to participate in a commercially available exergame by introducing additional haptic cues. The exergames and Real Time Sensory Substitution were effective in promoting physical activity for temporal based challenges, however they lacked the information to assist users in complex spatial challenges. Techniques were developed and compared using proprioception (the body's ability to sense its own position) to assist users in finding targets in one, two, and three dimensions without using any visual cues. Proprioception was also used as a cell phone interface where the phone can convey information to the user without using the graphical display of the phone, but using the human body as the display mechanism. The techniques developed in these studies have set the stage for enhanced access to technology without the use of a video display. ii Acknowledgments First, I would like to thank John Foley and Lauren Lieberman. Their assistance and collaboration was very important to this research. I would also like to thank the members of my committee, Sergiu Dascalu, Kostas Bekris, Dwight Egbert, and Nora Constantino. I would also like to thank the kids at Camp Abilities for their inspiring way of life. The Northern Nevada Chapter of the National Federation for the Blind has been a huge help with testing our software. Finally, I would like to thank my advisor, Eelke Folmer, his guidance has made this research possible. November 30, 2011 iii Contents Abstract i List of Tables vi List of Figures vii 1 Introduction 1 1.1 Background and Related Work . .3 1.2 NV NUI Overview . .6 2 Exergames 9 2.1 Background and Related Work . .9 2.2 VI Tennis Methodology . 15 2.2.1 Wii Tennis Gameplay & Feedback . 16 2.2.2 Gameplay . 18 2.2.3 Feedback . 19 2.3 VI Tennis User Study . 22 2.3.1 Participants . 22 2.3.2 Instrumentation . 23 2.3.3 Experimental Trial . 23 2.3.4 Results . 23 2.3.5 Qualitative Analysis . 25 2.4 VI Tennis Discussion . 26 2.4.1 Sensory Substitution . 26 2.4.2 Active Energy Expenditure . 27 2.5 VI Tennis - Future Work . 28 2.5.1 Whole Body Exercise . 28 2.5.2 Sensorimotor Skills . 29 2.5.3 Motor Learning . 29 2.5.4 Barriers to Physical Activity . 30 2.6 VI Tennis Conclusion . 30 2.7 VI Bowling Design . 31 2.7.1 Controls . 33 2.7.2 Sensory Substitution . 34 iv 2.7.3 Tactile Dowsing . 37 2.8 VI Bowling User Study . 40 2.8.1 Participants . 40 2.8.2 Instrumentation & Experimental Trial . 41 2.8.3 Results . 41 2.8.4 Qualitative Analysis . 42 2.9 VI Bowling Discussion and Future Work . 44 2.9.1 Active Energy Expenditure . 44 2.9.2 Tactile Dowsing Based Motor Learning . 44 2.9.3 Temporal-Spatial Challenges . 45 2.9.4 Barriers to Physical Activity . 45 2.10 VI Bowling Conclusion . 46 2.11 Pet-N-Punch Game Design . 46 2.11.1 Game Play . 47 2.11.2 Technical Implementation . 50 2.12 Pet-N-Punch User Study . 53 2.12.1 Participants . 54 2.12.2 Physical Activity Measurement . 55 2.13 Pet-N-Punch Results . 56 2.13.1 Error Rates . 56 2.13.2 Physical Activity . 58 2.13.3 Player Survey Results . 59 2.14 Pet-N-Punch Discussion . 60 2.14.1 Visual Observations . 60 2.14.2 Physical Activity . 60 2.14.3 Accuracy . 61 2.14.4 Maximizing Results . 63 2.14.5 Socialization . 63 2.15 Pet-N-Punch Future Work . 63 2.15.1 Higher Activity Intensity . 63 2.15.2 Health Benefits . 64 2.15.3 Socialization . 64 2.16 Pet-N-Punch Conclusion . 64 3 Real Time Sensory Substitution 66 3.1 Introduction . 66 3.2 Background and Related Work . 67 3.3 Real Time Sensory Substitution . 70 3.3.1 How it Works . 71 3.3.2 Runtime Video Analysis . 72 3.4 User Study 1 - Sighted Players . 76 3.4.1 User Study 2 - Players with Visual Impairments . 78 v 3.5 Results . 78 3.5.1 Sighted Player Performance Results . 78 3.5.2 Player Performance Results - Players with VI . 80 3.6 Discussion . 81 3.6.1 Limitations . 82 3.7 Future Work . 84 3.8 Conclusion . 85 4 Proprioceptive Displays 87 4.1 Introduction . 87 4.2 Discrete Proprioceptive Display Background . 90 4.3 Discrete Proprioceptive Display . 92 4.3.1 Scanning . 92 4.3.2 Auto-Semaphoring . 93 4.4 Twist-N-Lock . 94 4.5 User Studies . 96 4.6 Discrete Proprioceptive Display Discussion and Future Work . 100 4.7 Discrete Proprioceptive Display Conclusion . 101 4.8 2D Target Selection - Background and Related Work . 102 4.9 Tactile-Proprioceptive Displays . 104 4.9.1 Information Space . 105 4.9.2 Gesture Based Interaction . 105 4.10 2D Target Acquisition Study 1: Target Acquisition . 106 4.10.1 Instrumentation . 106 4.10.2 Participants . 108 4.10.3 Procedure . 108 4.10.4 Results . 109 4.11 2D Target Acquisition Study 2: Performing Directed Gestures . 111 4.11.1 Instrumentation . 111 4.11.2 Participants . 112 4.11.3 Procedure . 112 4.11.4 Results . 112 4.12 2D Target Acquisition Discussion . 113 4.13 2D Target Acquisition Future Work . 114 4.14 2D Target Acquisition Conclusion . 115 4.15 3D Target Acquisition Related Work . 115 4.16 3D Target Selection Prior Work and Motivation . 116 4.17 3D Scanning . 117 4.18 3D Target Selection Methods . 118 4.18.1 Instrumentation . 118 4.18.2 Participants . 119 4.18.3 Procedure . 119 4.19 3D Target Selection Results . 120 vi 4.20 3D Target Selection Discussion and Future Work . 122 5 Conclusions and Future Work 128 Bibliography 130 vii List of Tables 2.1 Participants' characteristics . 22 2.2 Average Active Energy Expenditure Kcal/Kg/Min . 24 2.3 Total time spent in MVPA . ..