CHI 2019 Late-Breaking Work CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK ML-Process Canvas: A Design Tool to Support the UX Design of Machine Learning-Empowered Products Zhibin Zhou Qing Gong Alibaba-Zhejiang University Joint Institute of Key Laboratory of Design Intelligence and Frontier Technologies, Zhejiang University Digital Creativity of Zhejiang Province Hangzhou, China Hangzhou, China [email protected] [email protected] Zheting Qi Lingyun Sun Alibaba-Zhejiang University Joint Institute of Alibaba-Zhejiang University Joint Institute of Frontier Technologies, Zhejiang University Frontier Technologies, Zhejiang University Hangzhou, China Hangzhou, China [email protected] [email protected] ABSTRACT1 Machine learning (ML) is now widely used to empower products and services, but there is a lack of research on the tools that involve designers in the entire ML process. Thus, designers who are new to ML technology may struggle to fully understand the capabilities of ML, users, and scenarios when designing ML-empowered products. This paper describes a design tool, ML-Process Canvas (see Fig. 1), which assists designers in considering the specific factors of the user, ML system, and scenario throughout the whole ML process. The Canvas was applied to a design project, and was observed to contribute in the conceptual phase of UX design practice. In the future, we hope that the Canvas will become more practical through continued use in design practice. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact owner/author(s). CHI’19 Extended Abstracts, May 4–9, 2019, Glasgow, Scotland UK. © 2019 Copyright is held by the author/owner(s). ACM ISBN 978-1-4503-5971-9/19/05. DOI: https://doi.org/10.1145/3290607.3312859 LBW1420, Page 1 CHI 2019 Late-Breaking Work CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK KEYWORDS INTRODUCTION Machine learning; user experience; design tool; design method Machine learning (ML) is now widely used to improve the quality of user experience (UX) in existing products or service, and even to develop new forms of UX[14]. However, recent research indicates that UX designers are not sufficiently prepared to leverage ML capabilities[3]. Issues such as the anthropomorphism[8] and interactivity of ML-empowered products bother designers because of the unpredictable and ever-changing nature of ML. Existing studies on the integration of UX and ML include attempts to understand users[1, 11] as well as design methods and processes [12, 13]. In particular, several researchers noted a lack of understanding of the user in the design of ML systems and advocated “power to the people”. These researchers allowed users to freely pick and modify the training data [1] or personalization labels [11] so that the ML systems could better understand the user preferences. The UX community also encouraged designers to involve users in the whole ML process, including data annotation, model construction, training, inference, and model updating, as a means of gaining valuable insights. Recent research offered some perspectives on the design value of ML [12] and demonstrated a design process without radically changing design activities. The process identifies the problem for ML to solve, validates technical feasibility of solution, and then iterates the ideation process [13]. The existing practice consists of reasonable methods that support the UX design of ML- empowered products, including valuing users, improving UX in the whole process of ML, and retaining familiar design activities. However, there was only limited research on specific UX issues related to ML and how to design interactions that may avoid such issues. Therefore, we present a brief summary of the issues we are aware of (see Fig. 2). In addition to the description of Holmquist et al. [6], who basically summarized almost all UX issues in the ML context, we also draw on the work of Yang et al. [13], Forlizzi et al. [4], and Dove et al. [3]. As ML technology is quite new to designers, they may be unaware of the information required to overcome the above UX issues, and may not have viable tools or methods for participating in the whole process of ML. However, there are different UX issues at various stages of the ML process, and overcoming these issues requires designers to participate in the entire ML process to Figure 1: ML-Process Canvas. understand how ML works and what innovations may be possible at each stage of the process. Therefore, the design tools should be tailored to the context of ML-related UX issues and should give designers the chance to participate in the entire ML process. Otherwise, designers will struggle to fully understand the limitations and capabilities of ML, and will have no suitable way of analyzing information about users and scenarios to deal with the specific UX issues. In this paper, we describe ML-Process Canvas (see Fig. 1) that assists designers in gathering essential information throughout the whole ML process without changing the familiar design activities. The Canvas identifies the specific UX issues caused by ML, and then summarizes the factors related to the user, scenario, and ML system that may influence those UX issues. The Canvas allows designers to organize and illustrate their findings related to those factors in order to seek potential design opportunities in the conceptual phase of UX design. Finally, we describe the application of the tool to a real design project and the feedback obtained for further improvement. LBW1420, Page 2 CHI 2019 Late-Breaking Work CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK Figure 3: Persona cards (a) and ML-Process Canvas I (b). ML-PROCESS CANVAS ML-Process Canvas consists of Canvas I and Canvas II. Canvas I maps the specific UX issues caused by ML to the whole ML process (see Fig. 3b), which reminds designers of the issues they need to work on at certain stages of the ML process. Designers can also write their final solutions on the corresponding issues cards (see Fig. 3a) after using Canvas II. The content of these issues cards is consistent with the content in Figure 2, and some issues will occur several times at different stages. Canvas II (see Fig. 4b) guides designers through the organization and analysis of the essential information related to the UX issues among the ML process. We divided the relevant information into three categories: user, system, and scenario. Designers need to list relevant information in different categories of columns at specific stages of the ML process, such as data annotation or inference. Furthermore, the listed information about the user, system, and scenario will be further discussed and analyzed to provide a technically viable solution that meets the needs of the user and scenario. Similar to using a journey map[7], designers can participate in various stages of the ML process during the conceptual phase of UX design to propose creative solutions that address the specific UX issues. Figure 2: UX issues caused by ML. Figure 4: Persona card (a) and Canvas II (b). LBW1420, Page 3 CHI 2019 Late-Breaking Work CHI 2019, May 4–9, 2019, Glasgow, Scotland, UK Table 1: Factors Related to the Scenario Research on human factors, such as user traits, that influence interactions with intelligent Factor Brief description systems is quite common, but the relationships between these factors and the identified UX issues Societal Culture, political system, are somewhat complex. However, we found that users can imagine the type of ML system they impact aspirations concerning want if asked the right questions. Users can clearly state their opinions on the anthropomorphism, safety and the future.[2] control, and etc. of the ML system, and the reasons for their choices. Thus, we provided Persona Task types Cognitive aids, control cards for designers to summarize the expectations of these user issues (see Fig. 4a). Sometimes, aids or perceptual aids, designers encounter different users at different stages of the ML process, and these users etc. occasionally have completely different expectations. Designers need to record the results of their Physical Light, noise, location, etc. user research on the Persona cards, especially the user traits and their preferences for UX issues, environment and then consider this content along with the system and scenario factors in determining potential solutions. Tables 1 and 2 summarize the factors related to the scenario and the ML system [10] Table 2: Factors Related to the ML System respectively, which may influence the UX issues. Designers are encouraged to list all available information about these factors at the corresponding stage of the ML process, as this allows Factor Brief description information to be collected and enhances their understanding of the scenario and ML systems. At Level of Predictability and stability the conceptual stage of UX design, ML systems with different attributes are optional, and the automation of automation[9]. scenario can be freely changed based on the design practice. This enables designers to carefully Capability Types of tasks that the ML select the appropriate ML system and application scenario by considering and discussing all trade- system can support. offs within the available information on the Canvas. After repeated discussions and analysis, the Aesthetics Aesthetic appearance, designers fill in the issues cards of Canvas I with the final solutions. Therefore, the Canvas can also which is related to be used by designers to share their ideas and solutions. automation likeability and use[8].