The Impact of User Characteristics and Preferences on Performance with an Unfamiliar Voice User Interface
Total Page：16
File Type：pdf, Size：1020Kb
The Impact of User Characteristics and Preferences on Performance with an Unfamiliar Voice User Interface Chelsea M. Myers Anushay Furqan Jichen Zhu Drexel University Drexel University Drexel University Philadelphia, PA, USA Philadelphia, PA, USA Philadelphia, PA, USA [email protected] [email protected] [email protected] ABSTRACT 1 INTRODUCTION Voice User Interfaces are increasing in popularity. However, Voice User Interfaces (VUIs), systems controlled primarily their invisible nature with no or limited visuals makes it through voice input, are becoming more integrated into our difficult for users to interact with unfamiliar VUIs. Wean- lives. With Amazon’s Alexa and the Google Home growing alyze the impact of user characteristics and preferences on in popularity, VUIs are now more commonly found in a home how users interact with a VUI-based calendar, DiscoverCal. setting. These VUIs provide a hands-free and eye-free interac- While recent VUI studies analyze user behavior through self- tion method for users to request information or even use for reported data, we extend this research by analyzing both VUI entertainment. However, issues such as the invisible nature usage data and self-reported data to observe correlations be- of VUIs make interaction challenging [4, 17, 19, 22] and limit tween both data types. Results from our user study (n=50) common uses of VUIs to simple interactions (e.g., checking led to four key findings: 1) programming experience did not the weather). Unlike Graphical User Interfaces (GUIs) that have a wide-spread impact on performance metrics while 2) can rely on permanently visible user interface features (e.g., assimilation bias did, 3) participants with more technical con- menu systems), VUIs currently have limited means to com- fidence exhibited a trial-and-error approach, and 4) desiring municate their affordances and limitations. Users must learn more guidance from our VUI correlated with performance and memorize the supported features and commands. The re- metrics that indicate cautious users. liance on stored mental knowledge, rather than information available in the world, indicates that a user’s background CCS CONCEPTS and preferences may play an important role in how well • Human-centered computing → Natural language in- she interacts with an unfamiliar VUI, such as the types of terfaces; Empirical studies in HCI; obstacles encountered [19] and initial expectations of the system [17]. KEYWORDS Existing research has started to examine how user fac- voice user interfaces; voice; user experience; empirical anal- tors impact performance with VUIs. Studies show that user ysis; multimodal interfaces characteristics such as technical knowledge [17] and cultural background [5] influence how people behave with VUIs. ACM Reference Format: Chelsea M. Myers, Anushay Furqan, and Jichen Zhu. 2019. The With few exceptions [19, 24], most existing studies on mod- Impact of User Characteristics and Preferences on Performance with ern VUIs analyze self-reported data on how participants an Unfamiliar Voice User Interface. In CHI Conference on Human interact with VUIs to observe behavior patterns. Factors in Computing Systems Proceedings (CHI 2019), May 4–9, 2019, To extend this work, we investigated the impact of user Glasgow, Scotland Uk. ACM, New York, NY, USA, 9 pages. https: characteristics and preferences by analyzing usage data on //doi.org/10.1145/3290605.3300277 how exactly people interact with an unfamiliar VUI. In our user study (n=50), we collected information on user char- Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies acteristics (e.g., age, gender, programming experience, VUI are not made or distributed for profit or commercial advantage and that experience) and preferences (e.g., initiative) based on existing copies bear this notice and the full citation on the first page. Copyrights literature. We analyzed these factors impact on how users for components of this work owned by others than the author(s) must interact with a VUI-based calendar, called DiscoverCal, in a be honored. Abstracting with credit is permitted. To copy otherwise, or home setting. In particular, we seek to answer these research republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. questions: CHI 2019, May 4–9, 2019, Glasgow, Scotland Uk • RQ 1: How do user characteristics influence a user’s © 2019 Copyright held by the owner/author(s). Publication rights licensed performance with an unfamiliar VUI? to ACM. ACM ISBN 978-1-4503-5970-2/19/05...$15.00 • RQ 2: How do VUI design preferences influence a user’s https://doi.org/10.1145/3290605.3300277 performance with an unfamiliar VUI? Our findings suggest: 1) programming experience did not Other demographic characteristics. A limited number have a wide-spread impact on performance metrics while of studies have looked into the impact of age and gender on 2) assimilation bias did, 3) participants with more technical VUIs [19]. Previous research has found no correlation be- confidence exhibited a trial-and-error approach, and 4) de- tween age, gender, and obstacles encountered with a VUI [19]. siring more guidance correlated with performance metrics However, this study recruited younger participants who were that indicate cautious users. mostly technically knowledgeable. A study comparing se- Our main contribution is that this is among the first study niors interacting with a keyboard and VUIs highlights the with a modern VUI to examine the impact of user character- factors affecting their perceptions of VUIs [24]. For exam- istics and preferences on how users actually interact with ple, participants who preferred the VUI over the keyboard an unfamiliar VUI in a home setting. Our study widens the were less likely to be technically knowledgeable or were knowledge on this topic and re-evaluates previously self- experiencing hand dexterity issues. It is unknown how these reported results on user interaction with VUIs. Our findings factors impact performance metrics. lead to design implications that can be useful, especially to With two exceptions [19, 24], all above-mentioned studies adaptive VUI research to design personalized systems based analyze self-reported data on how users interact with VUIs. on these user differences. In our work, we extend existing research by additionally The rest of the paper is structured as follows. First, we analyzing usage data on how people interact with an unfa- review related work in this area and our selection of factors. miliar VUI in a home setting. Our study provides another Next, we describe our methodology and our VUI system. prospective through usage data to re-evaluate and confirm After, we present our results and discuss their implications previous findings based on self-reported data. on VUI design. VUI research also designs and evaluates on-boarding tech- niques to address its invisibility [4, 7]. These tactics try to 2 RELATED WORK improve the mental model of VUI users and increase their With VUIs growing in popularity, many existing studies performance. We measure our participants’ self-reported focus on improving user satisfaction and learnability [4, 7, confidence in understanding the technical functionality of 18–21, 23]. While general design guidelines are important, DiscoverCal as a way to measure their mental model. We the invisible nature of VUIs requires users to rely more on are not collecting the description of their mental model, but knowledge in their head. Hence individual differences in user instead the confidence they have in it being correct. Finally, characteristics and preferences may have a strong impact a recent VUI study found that participants using an unfamil- on how people interact with unfamiliar VUIs. Studies have iar VUI were not relying on the menu provided and instead shown the wide range of interaction between users trying were “guessing” intents and utterances [19]. Based on this, to accomplish the same tasks in the same systems [16, 19] we also collect self-reported data on our participants menu and suggest VUIs are more effective when they can adapt to usage and their initial strategy when planning what to say user characteristics and preferences (e.g., initiative) [14]. to DiscoverCal. User Characteristic Factors VUI Design Preferences A growing number of research has looked into how different Existing work on adaptive interfaces focuses on two main user characteristics influence VUI interaction. They can be VUI elements based on user preferences; feedback and ini- categorized into three main types. tiative. We also collect our participants’ System Usability Technical/programming experience. Recent research Scale [1] to compare subjective usability to performance. has found that a user’s technical knowledge and cultural Feedback. VUI feedback includes how much information influences may impact their mental model and behavior to- is provided to users and the way the VUI confirms an action. wards VUIs [5, 17]. Luger and Sellen [17] found through Feedback design can range from explicit to implicit [22]. Ex- interviews that a user’s technical knowledge may influence plicit feedback strives to be transparent and may repeat to the their patience when encountering VUI errors. Less techni- user their previous utterances. However, explicit feedback cally knowledgeable participants reported quitting faster