Keep the Phone in Your Pocket: Enabling Smartphone Operation with an IMU Ring for Visually Impaired People

Keep the Phone in Your Pocket: Enabling Smartphone Operation with an IMU Ring for Visually Impaired People GUANHONG LIU∗, YIZHENG GU∗, YIWEN YIN, CHUN YU‡, YUNTAO WANG†, HAIPENG MI, and YUANCHUN SHI‡, Tsinghua University, China Previous studies have shown that visually impaired users face a unique set of pain points in smartphone interaction including locating and removing the phone from a pocket, two-handed interaction while holding a cane, and keeping personal data private in a public setting. In this paper, we present a ring-based input interaction that enables in-pocket smartphone operation. By wearing a ring with an Inertial Measurement Unit on the index finger, users can perform gestures on any surface (e.g., tables, thighs) using subtle, one-handed gestures and receive auditory feedback via earphones. We conducted participatory studies to obtain a set of versatile commands and corresponding gestures. We subsequently trained an SVM model to recognize these gestures and achieved a mean accuracy of 95.5% on 15 classifications. Evaluation results showed that our ring interaction is more efficient than some baseline phone interactions and is easy, private, and fun touse. CCS Concepts: • Human-centered computing → Accessibility technologies; Gestural input. Additional Key Words and Phrases: accessibility, gestural input ACM Reference Format: Guanhong Liu, Yizheng Gu, Yiwen Yin, Chun Yu, Yuntao Wang, Haipeng Mi, and Yuanchun Shi. 2020. Keep the Phone in Your Pocket: Enabling Smartphone Operation with an IMU Ring for Visually Impaired People. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 2, Article 58 (June 2020), 23 pages. https://doi.org/10.1145/3397308 1 INTRODUCTION Smartphones are an indispensable tool for visually impaired (VI) people. As shown in Figure1(a), VI users operate smartphones via a screen reader (e.g., VoiceOver or Talkback), following a touch and listen model. Previous studies have demonstrated that VI users encounter the following issues when using a smartphone. Getting the phone out of a pocket is inconvenient and time-consuming [8, 43]. It’s difficult for VI users to access or locate smartphones in a timely manner [32]. When using smartphones, VI people are constrained to two-handed interaction: one holding the phone, the other one interacting with the screen by performing gestures or touch-and-explore [39]. Therefore, using a smartphone is difficult when only one hand is available, such as when walking withacaneor ∗Both authors contributed equally to this study †This is the corresponding author ‡Also with Institute for Accessibility Development, Tsinghua University Authors’ address: Guanhong Liu, [email protected]; Yizheng Gu, [email protected]; Yiwen Yin, yyw17@mails. tsinghua.edu.cn; Chun Yu, [email protected]; Yuntao Wang, [email protected]; Haipeng Mi, [email protected]; Yuanchun Shi, [email protected], Key Laboratory of Pervasive Computing, Ministry of Education, Beijing Key Lab of Networked Multimedia, Beijing National Research Center for Information Science and Technology, Department of Computer Science and Technology, Department of Information Art Design, Tsinghua University, Beijing, China. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation onthefirst page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy 58 otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2018 Association for Computing Machinery. 2474-9567/2020/6-ART58 $15.00 https://doi.org/10.1145/3397308 Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 2, Article 58. Publication date: June 2020. 58:2 • Liu et al. Fig. 1. (a) A visually impaired user operates a smartphone. (b), (c) and (d): Using an IMU ring to perform gestures on a table, on the top of the thigh, and on the side of the thigh to operate the phone without taking it out of her pocket a dog. To address these issues, we can develop a socially acceptable phone controlling system that allows users to use one hand to interact with smartphones without touching the phone. Prior works have provided a handful of wearable solutions. Previous studies using wristband-based interaction [43] allowed users to perform a set five of predefined gestures on a wristband to operate a smartphone. This interaction method addressed the pain points of acquiring the phone but failed to provide one-handed interaction. Ring-based surface-specific interaction [8], in which users wear a ring embedded with sensors worn on the index finger and use their thumb to rub on the side of the index finger, allowed users to perform gestures toissuesix commands to screen reader software. Location-specific on-body interaction [31, 35], enabled users wearing a ring equipped with a camera, IMU, and IR reflectance sensors, to quickly complete simple tasks like checking the date and time by performing different gestures on different body locations such as the thighs, ears, andsoon. In this paper, we present a ring-based gesture input system to enhance phone interaction. Our system uses a smart ring with a single Inertial Measurement Unit (IMU) to sense gesture inputs and delivers feedback to the user via earphones. Wearing the ring on the middle phalanx of the index finger, users can perform 15 subtle gestures on any surface (e.g., tables, thighs) to operate their phones. This interaction not only addresses the pain points associated with fishing a phone out of a pocket and one-handed operation, but also mitigates privacy and social concerns, such as unintentionally exposing personal data to bystanders, and unwanted attention in public settings due to special postures used to operate the phone [39, 43]. We use an IMU ring as a gesture sensing device. Compared with other sensing form factors like wrist-worn devices [15, 21, 40, 51], when using the index finger to perform gestures on a given surface, a ring is betterat sensing subtle gestures like taps and swipes. We believe it reduces interaction fatigue and is more efficient, for one-handed, subtle, and socially inconspicuous interaction. We address three research questions in this work: 1) What gestures should be included for ring-based interactions that work on any surface? 2) What is the optimal technical implementation to recognize those gestures? 3) How usable is this interaction? To address the first question, we conducted focus groups to obtain a set of 15 versatile commands forphone operation. We then designed a gesture vocabulary and conducted a gesture ease-of-use scoring study to understand the participants’ preferences. Finally, we mapped gestures to commands based on the idiomatic gestures of existing screen reader software, semantic relationships between commands and gestures, and gesture ease-of-use score. Our interaction surfaces include the user’s thighs, which are soft, not flat, and covered with the user’s clothing. For the second question, we collected data from 20 participants with different types of clothing. We then used this training data to identify the moment when a user’s finger touches down on the interaction surface [10], trained an SVM model, and achieved an average of 95.5% accuracy on 15 classifications. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 4, No. 2, Article 58. Publication date: June 2020. Keep the Phone in Your Pocket: Enabling Smartphone Operation with an IMU Ring for Visually Impaired People • 58:3 For the third question, we evaluated the gestures’ practice time, compared the accuracy and efficiency on three interaction surfaces (table surfaces, top of the thigh, side of the thigh), and compared the efficiency with the baseline – phone interaction. Results showed that users acquired ring interaction gestures in less than 10 minutes; When the phone was in the user’s pocket, the ring interaction was more efficient than the baseline interaction on all three surfaces. Specifically, our contributions are as follows: (1) This work enables VI people to operate smartphones via gestures input on any surface with an IMU ring for the first time. Based on data we collected from user studies, we designed a set of versatile commands and suitable gestures to control smartphones. (2) This work shows that a ring with a single IMU sensor has a good performance on recognizing 15 touch gestures on three different surfaces. (3) This work provides empirical results for usability evaluation, demonstrating that our ring interaction is learnable and and is more efficient than the baseline interaction when the phone is in the user’s pocket. 2 RELATED WORK In this section, we briefly review prior research, including input techniques that enable users to operatea smartphone without touching it, use smart rings to control devices, and phone-free interaction for VI users. 2.1 Input Techniques for a Smartphone Operation without Touching It In the future, smartphones will become smaller and smaller and even allow for invisible integration into ubiquitous surfaces and human skin [28]. In anticipation of this trend, some researches have explored phone-free interaction. For example, OmmiTouch [14] and Imaginary Phone [12] focused on using a projector to project the phone’s interface onto an ordinary surface (e.g., wall, hand, etc.), and then using a camera to track users’ hands to enable input. In terms of smartphones’ current form, screen-touch is the primary input method. However, this kind of interaction is difficult in some scenarios (e.g., when users’ hands are wet). Therefore, many researchers have explored screen-touch-free interactions, such as mid-air gesture input [7, 11], lip-motion input [36], acoustic input [50] and so on.

Load more