AI-KIT • OCI Interacting with AI in Smart Home environments Bart van Dijk Final Master Project January 2020 PREFACE This work reports the result of my Final Master Project (FMP), part of the curriculum of the Industrial Design Master program of the Eindhoven University of Technology. The project has been executed over the course of two semesters: the Preparation for Final Master Project (Pre-FMP), and the Final Master Project. In this work, the Pre-FMP will be referred to as ‘Phase 1’, and consequently the FMP as ‘Phase 2’. The results of the first phase are described in a separate report, of which a summary will be presented in this work. While the initial direction of the project has originated from my personal interest and vision on design, the design brief got clearly defined after the project’s client (Bureau Moeilijke Dingen) had been acquired during the first phase of the project. 2 SUMMARY As the amount of interactive devices in our living environment is endlessly growing, other means to interact with these devices need to be explored. This work reports the design process in which the challenges are explored that emerge after implementing a learning system in Smart Home environments. The learning system allows users to design and develop their own models and functionality to automate tasks in their living environment. This approach to controlling connected devices asks for a new set of interactions which users are unacquainted with. First, the required interactions and design space are explored during the first phase of this project. Second, a test setup has been designed to explore and evaluate several interface elements with potential users and knowledgeables in the field. The findings of this study have been integrated in a new iteration, of which an interactive prototype has been developed. Concluding to this design process, the prototype has been evaluated on interface elements and usability. The findings of this work are discussed amongst potential directions for future work. 3 CONTENTS Preface 2 Summary 3 1 Introduction 5 2 Theoretical Background 6 2.1 Interactions 6 2.2 Machine Learning 6 3 Related Work 9 4 Design Brief 11 4.1 Client and Project 11 4.2 Competitor Analysis 13 4.3 Specific design challenge 13 5 Process Overview 15 6 First Project Phase 16 7 Second Project Phase 18 7.1 Label Monitoring explorations 18 7.2 Output Control features 22 7.3 Output Control exploration 23 7.4 Expert Panel study 27 7.5 Exploring future directions 34 7.6 Conceptualizing the final design 35 7.7 Concept prototyping 39 7.8 Evaluation study 43 8 Discussion 45 8.1 Concept value proposition 45 8.2 Interactions 45 8.3 Financial viability 46 8.4 Unsupervised learning 46 9 Conclusion 47 9.1 Acknowledgements 47 Terminology 48 References 49 Appendices 53 A - Personal Reflection 54 B - Competitor Analysis 56 B - Participant Booklet 57 C - Ethical Review Form 61 D - System Usability Scale results 63 E - Device/Feature Comparison 64 F - Concept Functionality and Interactions 65 G - Wiring Diagram 67 H - State Machine Diagram 68 4 1 - INTRODUCTION For centuries, technologic advancements have In order to oppose this interaction overload, new significantly contributed to the overall wellbeing of ways to interact with our devices in the home humans through major improvements in a broad environment are being explored. One approach, is range of sectors and fields (e.g. health, mobility, by integrating a Machine Learning based system in and comfortability). A long time ago, the rate of this context. While the system does not focus on advancements was relatively slow and could be achieving full automation (where no interactions observed on a nationwide scale (e.g. sewage, are required and all devices are continuously set steam powered trains, or power grids). Now, new to the correct state), the interaction burden is consumer products are rushed to the market on decreased by allowing users to envision their own a daily basis convincing us of new features that scenario, develop their model, and allocate output we need. Being convinced of the functionality, we states. This approach is differs from traditional, crowd our environments with devices demanding rule-based, devices where full-accuracy is directly an ever increasing amount of interactions. As achieved. As a result, the system requires a new a result, technology has absorbed a significant set of interactions and information providence that part of our available time and mental resources users are not traditionally acquainted with. This required to interact with these devices. work describes the design process of exploring the implications of integrating such a system in the As the era of ubiquitous computing unfolds, home environment. more and more connected products invade our personalized home environments. They allow us to personalize this environment and accurately match our specific preferences. As an example, a traditional light bulb allows to be set in to two states: 1 (On) - 0 (Off). New, often connected, light bulbs allow to be set in to a vast amount of states: 0-255 (Hue), 0-255 (Saturation), 0-255 (Brightness). While this allows users to specifically set the bulb to their preferences, it also requires a more complicated interaction to achieve the desired state. Moreover, as this trend can be observed for most devices (ranging from TV’s to Microwaves) a problem can be foreseen as the required interactions accumulate. 5 2 - THEORETICAL BACKGROUND 2.1 Interactions In 1997, Weiser and Brown predicted the on Smartphones after they were made popular by importance of designing calm technology in order Apple with Siri in 2011 (Aron, 2011). Initially, their to ensure a proper integration of technology in focus of use might have been on informing users in our daily lives (Weiser & Brown, 1997). As the era a diverse set of domains (e.g. weather, restaurants, of ubiquitous computing unfolds and the amount and navigation) while being on the go. Now, of computers start to outnumber people, we need Virtual Assistants are being integrated into our to ensure that humans remain in control. As this home environments and allow us to control most ubiquitous approach to computing is radically connected devices. While this approach allows a different from the approaches we are used to (e.g. large set of parameters to be controlled, the VUIs Personal Computing), they stress the importance come with several design challenges (Schnelle & of designing for calm technology. Many have Lyardet, 2006). Most prominently, as the medium continued upon their vision, resulting in a broad of VUIs (sound) is invisible, users are required set of approaches to increase the calmness of our to remember the correct phrases to control the everyday devices. desired parameters. As a result, the systems are not used to their full capabilities due to a lack of One approach explores the effect of interactions feedforward. This shows similarity with the need on required mental resources. For example, by for GUIs in 1982, when the traditionally used including the divided attention theory (a theory on Command Line Interfaces (CLIs) were the standard how mental resources are divided on the tasks that (Smith, Irby, Kimball, & Verplank, 1983). need to be simultaneously executed), interactions can be distributed on a continuum from focused- Due to the diversity in functionality that we aim (intentional, conscious, and direct precise control) to interact with, none of these approaches could to implicit (subconscious, unintentional, no direct be universally applied for all interactions with our control) interactions (Bakker, van den Hoven, devices. As a result it is no surprise we live in a world & Eggen, 2010). In the space between, we find where different types of interactions exist and take peripheral interactions (intentional, subconscious, place. This stresses the importance of exploring and direct imprecise control). This space is the contexts and functionalities the interaction considered fruitful for calm technologies, as it types are most suitable for. This work explores the allows users to control interactive devices outside implications of utilizing a Machine Learning based the center of their attentional field (Bakker & system to replace these required interactions. Niemantsverdriet, 2016). Another approach focusses ensuring a proper 2.2 Machine Learning integration of ubiquitous computing into our environments. In the work of Ishii and Ulmer Machine Learning based systems sprout in various (Ishii & Ullmer, 1997), they stress the importance context every day. Application areas include spam of exploring other means of interactions over filters, music/video recommendations, and weather the widespread use of Graphical User Interfaces forecasts. The cause of this rise is grounded in (GUIs). They introduce Tangible User Interfaces computing advancements and newly emerging (TUIs), where we use our world as interface by technologies (i.e. the internet, data storage, data augmenting real world objects with a coupling processing). Traditional rule-(IF-THEN statement) to the digital world we aim to interact with. This based systems have an immediate 100% accuracy allows us to integrate our computational devices in for the scenario’s they have been programmed for our environment while using natural affordances in a deterministic approach. However, as the home (Gaver, 1991) to facilitate the interactions. is a hyper-personalized environment in which scenario’s differ and adapt over time, a rule-based Recently, a third approach has been widely system is hard to maintain as the set of rules needs introduced into our home environments: Voice User to be altered continuously. On the other hand, the Interfaces (VUIs) (Schnelle & Lyardet, 2006). Virtual probabilistic approach of Machine Learning allows (VUI based) Assistants have been consistently used for these rules to be continuously developed by the 6 2 - Theoretical Background system without the need of explicit programming. 3.