Grierson, Mick and Fiebrink, Rebecca. 2018. User-Centred Design Actions for Lightweight Evaluation of an Interactive Machine Learning Toolkit
Total Page:16
File Type:pdf, Size:1020Kb
Bernardo, Francisco; Grierson, Mick and Fiebrink, Rebecca. 2018. User-Centred Design Actions for Lightweight Evaluation of an Interactive Machine Learning Toolkit. Journal of Science and Technology of the Arts (CITARJ), 10(2), ISSN 1646-9798 [Article] (In Press) http://research.gold.ac.uk/23667/ The version presented here may differ from the published, performed or presented work. Please go to the persistent GRO record above for more information. If you believe that any material held in the repository infringes copyright law, please contact the Repository Team at Goldsmiths, University of London via the following email address: [email protected]. The item will be removed from the repository while any claim is being investigated. For more information, please contact the GRO team: [email protected] Journal of Science and Technology of the Arts, Volume 10, No. 2 – Special Issue eNTERFACE’17 User-Centred Design Actions for Lightweight Evaluation of an Interactive Machine Learning Toolkit Francisco Bernardo Mick Grierson Rebecca Fiebrink Department of Computing Department of Computing Department of Computing Goldsmiths, University of London Goldsmiths, University of London Goldsmiths, University of London London SE14 6NW London SE14 6NW London SE14 6NW ----- ----- ----- [email protected] [email protected] [email protected] ----- ----- ----- ABSTRACT KEYWORDS Machine learning offers great potential to developers User-centred Design; Action Research; Interactive and end users in the creative industries. For Machine Learning; Application Programming example, it can support new sensor-based Interfaces; Toolkits; Creative Technology. interactions, procedural content generation and end- user product customisation. However, designing ARTICLE INFO machine learning toolkits for adoption by creative Received: 03 April 2018 developers is still a nascent effort. This work focuses Accepted: 09 May 2018 on the application of user-centred design with Published: 11 July 2018 creative end-user developers for informing the https://dx.doi.org/10.7559/citarj.v10i2.509 design of an interactive machine learning toolkit. We introduce a framework for user-centred design 1 | INTRODUCTION actions that we developed within the context of an Recent developments in artificial intelligence are European Union innovation project, RAPID-MIX. We generating a surge of interest around making illustrate the application of the framework with two machine learning (ML) more accessible to new actions for lightweight formative evaluation of our groups of people—particularly people who are not toolkit—the JUCE Machine Learning Hackathon and ML experts—for problem-solving and practical the RAPID-MIX API workshop at eNTERFACE’17. applications in various domains. However, using ML We describe how we used these actions to uncover is still often difficult for those who are not ML experts conceptual and technical limitations. We also or data scientists. Nonetheless, as with other discuss how these actions provided us with a better computer technologies that have seen mass understanding of users, helped us to refine the adoption (e.g., email interfaces, word processing scope of the design space, and informed software, web technologies, etc.), broader adoption improvements to the toolkit. We conclude with a of ML can be facilitated by improving the usability of reflection about the knowledge we obtained from tools for employing ML. applying user-centred design to creative technology, in the context of an innovation project in the creative Many software developers within the creative industries. industries, as well as many non-professional CITARJ 2-25 developers working with creative technology, can framework we devised to employ user-centred benefit from ML techniques. For instance, ML can design techniques systematically. We describe two facilitate the analysis of audio, visual, and sensor user interventions that involved different user data. ML can support the creation of new systems groups, methods and data collection strategies. for embodied interaction and expression, as well as Finally, we discuss the emerging knowledge about provide new creative workflows for rapid prototyping the application of user-centred design in the context and product customisation (Hartmann et al., 2007; of innovation projects and creative technology. Fiebrink, Cook, & Trueman, 2011; Katan, Grierson, & Fiebrink, 2015). 2 | BACKGROUND WORK This paper describes the development of new ML 2.1 USER-CENTRED DESIGN tools within the context of a European Commission- User-centred design (UCD), a term coined by funded “Innovation Action”—a joint effort between Norman and Draper (1986), is a design approach academic institutions and companies aimed at based on understanding users, their tasks and technology transfer and the production of new or environments. UCD has been characterised as improved products or services. This project, called “philosophy and methods, which focus on designing RAPID-MIX (“Realtime Adaptive Prototyping for for and involving users in the design of computerised Industrial Design of Multimodal Interactive systems” (Abras, Maloney-Krichmar, & Preece, eXpressive technology”), has involved a consortium 2004). of three European research labs and five small and medium-sized enterprises (SMEs) collaborating in When adopting UCD, we look for ways to better the development of creative technology tools for understand users, their characteristics, skills and rapid prototyping and product development. Central behaviour in specific tasks. Users are at the centre to RAPID-MIX’s goals is the creation of new of this process; they are involved at an early stage machine learning tools targeting developers in the and throughout the design process. This approach creative industries. enables the design team to communicate and negotiate a better and shared understanding of the A user-centred approach to design has been critical right problem, and to reason about and inform to ensure that these new tools are usable and useful design decisions for the right solution (Monk, 2007). for creative developers. The needs of creative However, Norman notes that in UCD, “even though developers using ML are not well-understood. ML the ideal can seldom be met in practice, it is always presents unique challenges to developers (Patel et good to aim for the ideal, but to be realistic about the al., 2010), and the needs of people employing time and budgetary challenges” (2013, p. 239). technology used in design and other creative practices are often distinct from people engaging in According to Ritter, Baxter and Churchill (2014), activities with more well-defined outcomes (Cherry & UCD has a broader focus, greater emphasis on the Latulipe, 2014). Further, the design of software user, and lesser use of formal methods for application programming interfaces (APIs) presents requirements gathering and specification than other distinct challenges from other design processes in approaches, such as human factors and which user-centred methodologies are often used ergonomics, or socio-technical systems design. (e.g., the design of end-user-facing graphical They claim that adopting a user-centred approach interfaces) (Myers & Stylos, 2016). In RAPID-MIX, can help to address essential design problems and we have thus needed to carefully craft user-centred lead to systems that are more useful, usable and methodologies appropriate to the design of our new satisfying. It can help designers overcome errors or ML tools, within the additional constraints of a issues that arise from relying on their personal complex project with multiple academic and industry assumptions, experience or intuition. Ritter, Baxter partners. and Churchill argue that, on the one hand, UCD leads to financial savings, as iteratively refining The paper is structured as follows. In the following designs can lead to fewer problems in the final section, we review prior work in user-centred design product. On the other hand, though, UCD entails its and interactive machine learning. We then present own costs, and UCD does not guarantee the our main adopted research methodology and the success of a product: while “the lack of usability can CITARJ 2-26 Journal of Science and Technology of the Arts, Volume 10, No. 2 – Special Issue eNTERFACE’17 be a sufficient reason for failure”, “usability is neither model (e.g., showing that certain actions performed a necessary nor sufficient condition for success” (p. with a sensor should result in certain sounds or 14). The value of UCD may be assessed by game commands). Users can iteratively steer the estimating its return on investment (ROI)—the extent behaviour of the trained model by modifying these to which the time and effort spent doing user training examples. research provide worthwhile benefits; ROI is highly valued but also often difficult to measure and to Previous research on IML has mostly focused on manage (Ritter et al., 2014). Holtzblatt, Wendell and designing graphical user interfaces for end-user Wood (2004) also refer to the ROI as an important interaction with learning systems. This includes criterion for the adoption of UCD practices in interfaces that enable domain expert users (who organisational contexts. may have little or no machine learning expertise) to steer models by providing new training examples To avoid complicated, time-consuming and (Fails & Olsen, 2003; Fiebrink et al., 2011), adjusting expensive techniques for ensuring usability of a new misclassification costs (Kapoor et al., 2010),