AI-KIT • OCI Interacting with AI in Smart Home environments
Bart van Dijk Final Master Project January 2020 PREFACE
This work reports the result of my Final Master Project (FMP), part of the curriculum of the Industrial Design Master program of the Eindhoven University of Technology. The project has been executed over the course of two semesters: the Preparation for Final Master Project (Pre-FMP), and the Final Master Project. In this work, the Pre-FMP will be referred to as ‘Phase 1’, and consequently the FMP as ‘Phase 2’. The results of the first phase are described in a separate report, of which a summary will be presented in this work. While the initial direction of the project has originated from my personal interest and vision on design, the design brief got clearly defined after the project’s client (Bureau Moeilijke Dingen) had been acquired during the first phase of the project.
2 SUMMARY
As the amount of interactive devices in our living environment is endlessly growing, other means to interact with these devices need to be explored. This work reports the design process in which the challenges are explored that emerge after implementing a learning system in Smart Home environments. The learning system allows users to design and develop their own models and functionality to automate tasks in their living environment. This approach to controlling connected devices asks for a new set of interactions which users are unacquainted with. First, the required interactions and design space are explored during the first phase of this project. Second, a test setup has been designed to explore and evaluate several interface elements with potential users and knowledgeables in the field. The findings of this study have been integrated in a new iteration, of which an interactive prototype has been developed. Concluding to this design process, the prototype has been evaluated on interface elements and usability. The findings of this work are discussed amongst potential directions for future work.
3 CONTENTS
Preface 2 Summary 3 1 Introduction 5 2 Theoretical Background 6 2.1 Interactions 6 2.2 Machine Learning 6 3 Related Work 9 4 Design Brief 11 4.1 Client and Project 11 4.2 Competitor Analysis 13 4.3 Specific design challenge 13 5 Process Overview 15 6 First Project Phase 16 7 Second Project Phase 18 7.1 Label Monitoring explorations 18 7.2 Output Control features 22 7.3 Output Control exploration 23 7.4 Expert Panel study 27 7.5 Exploring future directions 34 7.6 Conceptualizing the final design 35 7.7 Concept prototyping 39 7.8 Evaluation study 43 8 Discussion 45 8.1 Concept value proposition 45 8.2 Interactions 45 8.3 Financial viability 46 8.4 Unsupervised learning 46 9 Conclusion 47 9.1 Acknowledgements 47 Terminology 48 References 49 Appendices 53 A - Personal Reflection 54 B - Competitor Analysis 56 B - Participant Booklet 57 C - Ethical Review Form 61 D - System Usability Scale results 63 E - Device/Feature Comparison 64 F - Concept Functionality and Interactions 65 G - Wiring Diagram 67 H - State Machine Diagram 68
4 1 - INTRODUCTION
For centuries, technologic advancements have In order to oppose this interaction overload, new significantly contributed to the overall wellbeing of ways to interact with our devices in the home humans through major improvements in a broad environment are being explored. One approach, is range of sectors and fields (e.g. health, mobility, by integrating a Machine Learning based system in and comfortability). A long time ago, the rate of this context. While the system does not focus on advancements was relatively slow and could be achieving full automation (where no interactions observed on a nationwide scale (e.g. sewage, are required and all devices are continuously set steam powered trains, or power grids). Now, new to the correct state), the interaction burden is consumer products are rushed to the market on decreased by allowing users to envision their own a daily basis convincing us of new features that scenario, develop their model, and allocate output we need. Being convinced of the functionality, we states. This approach is differs from traditional, crowd our environments with devices demanding rule-based, devices where full-accuracy is directly an ever increasing amount of interactions. As achieved. As a result, the system requires a new a result, technology has absorbed a significant set of interactions and information providence that part of our available time and mental resources users are not traditionally acquainted with. This required to interact with these devices. work describes the design process of exploring the implications of integrating such a system in the As the era of ubiquitous computing unfolds, home environment. more and more connected products invade our personalized home environments. They allow us to personalize this environment and accurately match our specific preferences. As an example, a traditional light bulb allows to be set in to two states: 1 (On) - 0 (Off). New, often connected, light bulbs allow to be set in to a vast amount of states: 0-255 (Hue), 0-255 (Saturation), 0-255 (Brightness). While this allows users to specifically set the bulb to their preferences, it also requires a more complicated interaction to achieve the desired state. Moreover, as this trend can be observed for most devices (ranging from TV’s to Microwaves) a problem can be foreseen as the required interactions accumulate.
5 2 - THEORETICAL BACKGROUND
2.1 Interactions
In 1997, Weiser and Brown predicted the on Smartphones after they were made popular by importance of designing calm technology in order Apple with Siri in 2011 (Aron, 2011). Initially, their to ensure a proper integration of technology in focus of use might have been on informing users in our daily lives (Weiser & Brown, 1997). As the era a diverse set of domains (e.g. weather, restaurants, of ubiquitous computing unfolds and the amount and navigation) while being on the go. Now, of computers start to outnumber people, we need Virtual Assistants are being integrated into our to ensure that humans remain in control. As this home environments and allow us to control most ubiquitous approach to computing is radically connected devices. While this approach allows a different from the approaches we are used to (e.g. large set of parameters to be controlled, the VUIs Personal Computing), they stress the importance come with several design challenges (Schnelle & of designing for calm technology. Many have Lyardet, 2006). Most prominently, as the medium continued upon their vision, resulting in a broad of VUIs (sound) is invisible, users are required set of approaches to increase the calmness of our to remember the correct phrases to control the everyday devices. desired parameters. As a result, the systems are not used to their full capabilities due to a lack of One approach explores the effect of interactions feedforward. This shows similarity with the need on required mental resources. For example, by for GUIs in 1982, when the traditionally used including the divided attention theory (a theory on Command Line Interfaces (CLIs) were the standard how mental resources are divided on the tasks that (Smith, Irby, Kimball, & Verplank, 1983). need to be simultaneously executed), interactions can be distributed on a continuum from focused- Due to the diversity in functionality that we aim (intentional, conscious, and direct precise control) to interact with, none of these approaches could to implicit (subconscious, unintentional, no direct be universally applied for all interactions with our control) interactions (Bakker, van den Hoven, devices. As a result it is no surprise we live in a world & Eggen, 2010). In the space between, we find where different types of interactions exist and take peripheral interactions (intentional, subconscious, place. This stresses the importance of exploring and direct imprecise control). This space is the contexts and functionalities the interaction considered fruitful for calm technologies, as it types are most suitable for. This work explores the allows users to control interactive devices outside implications of utilizing a Machine Learning based the center of their attentional field (Bakker & system to replace these required interactions. Niemantsverdriet, 2016).
Another approach focusses ensuring a proper 2.2 Machine Learning integration of ubiquitous computing into our environments. In the work of Ishii and Ulmer Machine Learning based systems sprout in various (Ishii & Ullmer, 1997), they stress the importance context every day. Application areas include spam of exploring other means of interactions over filters, music/video recommendations, and weather the widespread use of Graphical User Interfaces forecasts. The cause of this rise is grounded in (GUIs). They introduce Tangible User Interfaces computing advancements and newly emerging (TUIs), where we use our world as interface by technologies (i.e. the internet, data storage, data augmenting real world objects with a coupling processing). Traditional rule-(IF-THEN statement) to the digital world we aim to interact with. This based systems have an immediate 100% accuracy allows us to integrate our computational devices in for the scenario’s they have been programmed for our environment while using natural affordances in a deterministic approach. However, as the home (Gaver, 1991) to facilitate the interactions. is a hyper-personalized environment in which scenario’s differ and adapt over time, a rule-based Recently, a third approach has been widely system is hard to maintain as the set of rules needs introduced into our home environments: Voice User to be altered continuously. On the other hand, the Interfaces (VUIs) (Schnelle & Lyardet, 2006). Virtual probabilistic approach of Machine Learning allows (VUI based) Assistants have been consistently used for these rules to be continuously developed by the
6 2 - Theoretical Background
system without the need of explicit programming. 3. Model Selection But as this approach is probabilistic full accuracy As Machine Learning is widely implemented, will never be achieved, and the system will require models have been pre-created by developers for a training in order to increase its accuracy for a diverse set of purposes. While some models implementation (Ethem Alpaydin, 2004). Different focus on the recognition of images, others focus types (i.e. Supervised Learning, Unsupervised on the discovery of patterns in numerical data. This Learning, and Reinforcement Learning) of Machine stresses the importance of defining the model’s Learning are currently implemented in everyday goal prior to the start of this process as it allows for products. The different types vary upon their the right model to be selected in this step. application areas, learning process (dataset or experience based), and algorithm’s used. This 4. Model Training work implements Supervised-, Reinforcement-, As the dataset used for this step is labeled, and (limited) Unsupervised Learning principles as the model is able to iterate and independently described below. improve its accuracy over training steps. The different features of the dataset are associated Supervised Learning with individual weights and biases to define In summary, this approach allows a model to their relationship to the datapoint’s label. During make a prediction on which predefined label fits the training phase, different values for these a datapoint best based on previously analyzed weights and biases are explored to incrementally datapoints. It requires the model to be trained increase the model’s capability to allocate a label using a labeled dataset in order to be able to classify to a datapoint based on the selected features. unlabeled data after the model has been trained. This type of learning requires a comprehensive 5. Model Evaluation dataset that includes features (aspects of data used After the training has been completed, the during the analysis for classification) and the labels remaining (evaluation) dataset is used to evaluate (desired label(s) to be recognized) that is associated the accuracy of the model. A new datapoint is fed with that particular data point and set of features. to the model after which the model will determine A typical Supervised Machine Learning process can the most probable label based on the included be described in seven steps: features and previously trained weights and biases.
1. Data Collection 6. (Hyper-) Parameter Tuning During the first phase of the process, data to train Prior to the training phase, several parameters (e.g. and evaluate the model needs to be collected. As Epochs (times to go through the training dataset), the model will be trained based on this dataset in Batch Size (amount of datapoints per training step), later steps, any biases included in the dataset will and Learning Rate (difference of weights and biases influence the accuracy of the trained model. As between training steps)) were set that influence previously described, the data is required to be the accuracy of the model. Tuning parameters labeled and should include the features relevant is an iterative process where the parameters are for training. manually tweaked (6), the model is re-trained (4), and evaluated (5). How the tuning of these 2. Data Preparation parameters influences the accuracy of the model Before the dataset can be used, several steps need is highly dependent of the dataset and model to be undertaken. For example, the dataset needs specifics. to randomized (order of entries) and checked for imbalances. An imbalanced dataset can result in 7. Making Predictions a biased and inaccurate model as the model will In this step, the model is ready for use and can express its preference towards a certain outcome. be deployed. A piece of unlabeled data is fed On top of that, in order to train the model and into the model after which the model will make a later evaluate its accuracy, the data needs to be prediction and allocate the most probable label. split in training- (often: ~80%) and evaluation (often: ~20%) data which will be used in later steps.
7 2 - Theoretical Background
Unsupervised Learning Models and Algorithms Where Supervised Machine Learning makes use As mentioned in step three of the seven steps of of labeled data to train its model, Unsupervised Supervised Machine Learning, the model type Machine Learning does not require labeled influences the accuracy of the overall model when data. As less information about the datapoints implemented. As a model is created through is known, the algorithm’s used in Unsupervised training (using an algorithm and dataset), the Machine Learning are often more complicated selected algorithm (a set of actions/rules) is of and the models require a more extensive training. similar importance for the trained model’s accuracy. One of the most frequently used approach to Over the years, many Machine Learning suitable Unsupervised Machine Learning is clustering (Hu algorithms have been developed. In this work, & Hao, 2012; Roman, 2019), where datapoints Supervised Machine Learning will be used to detect are grouped (clustered) based on the similarity of household activities based on sensory data. The different features. In this approach, the parameters sensors generate continuous sensor data over time, used for the clustering process are unknown, and which results in time-series data (datapoints over the model is optimized using an error-margin and time, which as a result should not be randomized). cluster-quantity. Different algorithms can be used to train the model using labeled intervals of the dataset (e.g. decision Reinforcement Learning trees (Hu & Hao, 2012; Lin, Williamson, Born, & Both Supervised and Unsupervised Machine Debarr, 2012)). However, this would require an Learning make use of initial training datasets to interval to pass before a change can be noticed. This improve the accuracy of their model prior to further work implements Long Short-Term Memory (LSTM) implementation. In Reinforcement learning, how as it is able to recognized changes immediately due the data is collected and processed for the model to its state-aware characteristics. After training, differs. A set of rules is provided to the system, the model will become able to recognize pattern’s and it is asked to explore the consequences of in the time-series data that are associated with actions in different states, as a result the model is the trained label. This allows for events to be trained through its experiences (Case, 2019). The recognized without the need of explicit rule-based experiences are added to the dataset, improving programming (i.e. based on thresholds). the model while expanding the dataset over time.
8 3 - RELATED WORK
Interacting with IoT approach allows for parameters to be set to As connected devices invade our home detail, the lack of feedforward acts as a bottleneck environment and the Internet of Things (IoT) slowly during the interaction. Virtual Assistants make use becomes reality, different approaches to facilitate of Machine Learning to convert speech-to-text the required interactions are explored and (after which it can be interpreted and responded brought to the market. As most of these devices to) and improve its skillset (Polyakov et al., 2018). are accompanied with individual smartphone Other devices, such as the Google Nest Learning applications (e.g. Philips Hue (Philips, 2019b), Belkin Thermostat (Google, 2019), use Machine Learning Wemo (Belkin, 2019), and Ikea FYRTUR (IKEA, n.d.)), with the aim to make interactions implicit (Ju & smartphones have become remotes to control the Leifer, 2008). Out of the box, the thermostat act’s home environment. While these applications allow passively and uses user input to detect patterns in device parameters to be specifically controlled, the desired temperatures. Over time, it will become the large amount of mental recourses required able to take over the user’s actions by automatically to operate this focused interaction (Bakker & controlling the indoor climate while contributing Niemantsverdriet, 2016; Bakker, van den Hoven, towards energy savings. As the thermostat can be & Eggen, 2015) can be considered as burdensome. considered as one of the pioneers of smart systems As a result, manufacturers of these products in the home environment, the public’s opinion and introduce remotes to decrease the threshold to experiences are of great value when designing new interact with the devices. For example, the Philips intelligent systems. Useable takeaways include: Hue Tap (Philips, 2019a) allows the user to quickly incidental intelligibility (the variable rate of interest set the light to customizable scenes. While this that users have to understand the intelligent interaction requires significantly lower focus of the system), constrained engagement (engaging user on the interaction, the functionality of the IoT users to interact with intelligent systems without device is restricted as well. overwhelming them), and exception flagging (allowing users to flag inputs that should not be As this approach crowds the environment with considered for learning) (Yang & Newman, 2013). remotes and phones with applications, some companies aim to mediate and unite the separate Knowledge Economy devices. The Logitech Harmony (Logitech, 2020) Due to the hyper-personalized nature of the allows different remotes (thus devices) to be home environment, one of the main challenges in combined into a single one. The devices act as a this scope is to design a product that suits these hub, and differentiate in how they are positioned individual contexts. Not only does the home differ in the home context. Where the Logitech Harmony in shape and layout, the specific set of IoT devices is a mobile device, others (e.g. 4Control (Control4 and its users (the people that live in in the house) Corporation, 2016) and Atmos (Atmos, 2019)) use differ as well. As a result, the needs vary and users a central place in the home to facilitate the control tend to express their creativity as they pursue of IoT devices. While these interfaces decrease their own vision of interactions and functionality the amount of interactions required to get to the of the home. This trend, part of the knowledge desired scene through presets, it can be disputed economy (Brand & Rocchi, 2011), has allowed for whether this graphical approach is sufficiently the introduction of a new set of products. This has scalable for the growing system of IoT devices. resulted in people participating in the development of their own functionality to match their individual Virtual Assistants use case through several platforms. For example Another approach to facilitate these interactions IFTTT (IFTTT, n.d.), a platform that enables users to has become increasingly more available in recent create their own ‘applets’ in which external inputs years. VUIs (Schnelle & Lyardet, 2006) in the form (e.g. Gmail (Google, n.d.-a)) can be connected to of Virtual Assistants have made their entry into outputs (e.g. Philips Hue (Philips, 2019b)). For the home environment and allow users to ask example, users are able to blink their bulbs when questions and control their devices. The three main ey receive an email. competitors in this space are the Google Assistant (Google, n.d.-b), Amazon Alexa (Amazon, 2018), and Siri by Apple (Apple Inc, 2019). While this scalable
9 3 - Related Work
Reality Editor In practice, the applet-based approach proves to be unscalable as users lose oversight in the set of connections that have been made over time. In the MIT Media lab, researchers are exploring novel methods to allow for these connections between Internet of Things (IoT) devices to be made and altered. With their Reality Editor, users are able to continuously reprogram the functionality and connections of their devices using a Augmented Reality enabled smartphone application (Heun, Hobin, & Maes, 2013). While this approach allows users to playfully reconnect their devices without requiring them to remember the allocated ID, it requires a focused interaction to make alterations which might be less suitable for everyday use.
10 4 - DESIGN BRIEF
4.1 Client and Project
This work is in collaboration with Bureau Moeilijke Target group Dingen, a design studio located in Eindhoven, the As the creation of a labeled dataset and training Netherlands. They design and develop technical of a model requires extensive effort before of the solutions for a broad range of challenges, in which system to create value by controlling output, AI-Kit they focus on modularity, scalability, and quality of is not positioned towards users that are exclusively interaction. Their work includes (interaction) design, interested in home automation. Instead, the user front-/backend development, and prototyping is expected to be interested in Machine Learning (Bureau Moeilijke Dingen, 2019). and to be motivated to learn more about it in a practical manner. As a result, AI-Kit is targeted AI-Kit towards tech-savvy individuals that enjoy playing Machine Learning based systems are increasingly with services like IFTTT (IFTTT, n.d.) and Internet more integrated into our daily lives. Applications of Things devices. As AI-Kit will (possibly) be used include spam filters, video/audio recommendation in a multi-user environment, other users that are systems, and virtual assistants. Despite this less acquainted or interested in technology should increasing acquaintance, the ability to explore be considered as well. This does solely include this the capabilities of this technology has remained group as users, as others could make use of the exclusive to the hands of Data Scientists and deployment phase of the system. For example, a Software Engineers. With AI-Kit, in line with the tech-savvy individual could set up and go through previously described knowledge economy (Brand the training phase and allow for the individuals & Rocchi, 2011), users are able to design and parents to reap the harvest during the deployment develop their own Machine Learning functionality. phase where a burdensome action is automated. By connecting various sensors (e.g. humidity, While AI-Kit has initially been targeted at usage temperature, movement (PIR)) that are integrated contexts where the value is worth the effort in the environment and the desired outputs that is required during the training phase (for (e.g. Philips Hue lighting (Philips, 2019b), Nest example, local shops that want to increase their thermostat (Google, 2019), or Sonos speakers sales). This work focusses on the opportunities (Sonos Inc., 2020)), users are able to detect an and challenges that arise when such system event (label) and allocate output states. A global is implemented in smart home environments overview of the different components of AI-Kit can where the effort limit is expected to be lower. be seen in figure 4.1. The different usages phases and required steps are described under workflow on the next page.
Figure 4.1 AI-Kit overview
11 4 - Design Brief
Workflow Example usage scenario 1. Positioning and connecting sensors Pete is cleaning his living room. While doing so, The various sensors that are relevant for the he enjoys listening to jazz music on his home detection of a certain event are installed on relevant entertainment system. He returns a cup to the positions. The sensors are connected to AI-Kit in a kitchen while singing along to the music, after which web-app environment. he grabs the vacuum cleaner out of the cabinet. He turns the vacuum cleaner on and is disappointed 2. Data collection that he cannot hear the music anymore. While he The sensors will generate time-series data. The could turn up the volume manually, he feels like it data is logged and can be reviewed in de web-app would be too much of work as he needs to search environment. for the remote control. Instead, he could use AI- Kit to recognize whether the vacuum cleaner is 3. Data labeling turned on and music is playing to automatically In order to generate a labeled dataset, the user is increase the volume of the home entertainment required to label chunks (timeframes) of data. As system using a centrally placed microphone. The this labeling process is done in retrospect, users trained model in AI-Kit is able to make a distinction can make use of reference (or contextual) probes. between the sound of a vacuum cleaner and loud These reference probes are sensors with the ability chatter of people (which should not result in a raise to record scenario’s (e.g. a camera or microphone). in volume), which would not be possible using a The recording of these probes can be used to traditional rule-based sensor with a threshold. review the events of a certain timeframe, and label the data accordingly. Potential revenue models As AI-Kit is in its exploratory stage, the distinct 4. Model training and optimization revenue model is yet to be defined. Nevertheless, After a sufficient labeled dataset has been several approaches are explored as this work generated, the model can be trained using will be complementary to AI-Kit. One approach Supervised Machine Learning to recognize pattern’s consists of making the deployment phase of the in the time-series data. After the model has been model a paid, premium feature (possibly through trained, its accuracy can be evaluated and the user a monthly subscription). By allowing users to can determine whether the accuracy is sufficient develop their own model, they might be more for the selected use case. To some extent, the tempted to see their work in action in exchange model is automatically optimized as a second layer for a fee. As continuous server costs will originate of Machine Learning is applied to tune the (hyper-) from data storage space and processing power, a parameters. By doing so, the optimal parameter subscription based revenue model matches the values can be found resulting in an increased cost. Another approach that is currently being accuracy of the model. explored, is excluding third party IoT sensors. By doing so, users are restricted to use the sensors 5. Deployment that are compatible with AI-Kit. This requires them After the model has been sufficiently trained, it to purchase additional or new sensors types when has the ability to recognize the trained labels (at they want to train new models. As these sensors a sufficient rate). The user allocates output states are a one-time purchase, the (lifetime) server that should be activated when a label has been cost should be included in the price. A different recognized. approach to the server costs, would be to allow users to locally store and process their data locally 6. Re-training in some sort of AI-Kit hub. This might be appealing Over time, as habits (thus scenario’s and data to users that are concerned about their privacy, patterns) change, the accuracy of the model might as it allows them to utilize the functionality while deteriorate. The user is able to review and retrain keeping the data in-doors. the model at all times in the web-app environment. On top of that, new output states can be allocated when desired.
12 4 - Design Brief
4.2 Competitor Analysis
AI-Kit is not the only competitor in the market of Machine Learning applets. Moreover, a significant empowering users with the ability to design and part of the competitors focus on teaching children develop their own applets. Moreover, there are code-like thinking while allowing them to play with other services available that allow users to play electronics (e.g. SAM Labs (SAM Labs, 2020)). AI-Kit around with Machine Learning. In order to obtain a on the other hand addresses a higher skill level clear view on how AI-Kit is positioned in the market providing more control and depth when desired by and how the platform distinguishes from others, a skilled user. a (feature focused) competitor analysis (Levy, 2015) has been executed (Appendix B) of which a summary can be seen in figure 4.2. 4.3 Specific design challenge
A trend can be observed where some competitors In this work, the opportunities and design focus on making Machine Learning easily accessible challenges of implementing AI-Kit in the home is (e.g. Google’s Teachable Machine 2.0 (Google, explored. In order to make this system suitable 2017)), and others focus on easy control of third for the home environment several aspects need party devices (e.g. Flic Button’s (Flic Corp., 2019)). to be explored. First, as the home might be less AI-Kit differentiates itself by combining the two, suitable for the use of contextual sensors (e.g. allowing output to be controlled with the use of a camera due to privacy issues), others means
Data monitoring
3
2
DIY projects ML based
1
0
Output activation Skill level
AI-Kit Teachable Machine (Google, n.d.-c) Anki Vector (Anki, 2019) Flic (Flic Corp., 2019) Easy Output Control
Figure 4.2 Competitor Radar
13 4 - Design Brief
to create and review the labeled dataset need to be explored. On top of that, the user need to be enabled to provide feedback to the system as it will detect false positives and negatives, this will imply the introduction new interactions. As AI-Kit consists of various components (figure 4.1), the required functionality and suitable location is explored in the first phase of this project. During the second phase, there is an emphasis on exploring and designing the specific form and interaction for the found functionality. As AI-Kit is currently its conceptual phase, only usage scenarios, sensor probes, and conceptual (web-app) interface elements have been explored. Consequently, there is no working version available in which models can be developed and interacted with.
14 5 - PROCESS OVERVIEW
The limited level of AI-Kit’s development has Develop complicated the exploration of the design The defined requirements were integrated in challenges, required features, and their evaluation. a tangible interface over the course of several To overcome this challenge, several instances of iterations. First, methods to easily provide feedback requirement definitions, that concluded phases to AI-Kit were explored (Ch. 7.1). Later, novel of converging and diverging, were used to approaches to interact with output’s have been provide structure and ensure independence. To developed (Ch. 7.3) based on the previously defined further depict this process structure, the double requirements. In the Expert Panel study, the taken diamond design process model (Design Council, approach upon the design challenge was evaluated 2011; Groeger & Schweitzer, 2015) has been used after iteratively developing the interaction and as point of reference. The different activities of other interface elements (Ch. 7.4). Over the course this design process can be roughly categorized of several iterations, the design requirements were amongst the different design phases. The iterative translated into a design (Ch. 7.7). approach to this design challenge is shown by the continuous jumps in both directions between the Deliver defined phases. At the end of the first phase, a prototype has been developed to communicate the defined set of Discover requirements (Ch. 6). Furthermore, a prototype has The design space, in terms of theoretical background been developed to conclude the second phase of (Ch. 2) and related work (Ch. 3), has been explored this project (Ch. 7.7). The prototype has been used in twofold during both phases of the project. As to evaluate the implementation of the concept’s the targeted users are likely to be unacquainted features and their interactions (Ch. 7.8). The work with the capabilities and requirements of the is concluded in a discussion, in which the project’s technology, they were expected to have a hard findings and potential directions for future work is time imagining their needs which would emerge discussed (Ch. 8). when interacting with such a system. To overcome this challenge, their potential needs were explored in the first phase of this project (Ch. 6). Different devices to interact with AI-Kit were explored, as they would provide a potential solution space to facilitate the required interactions (Ch. 6). Later, the requirements for controlling the output of AI-Kit were explored (Ch. 7.2) as well. After the Expert Panel study, the future directions were briefly explored by reconsidering the roles of other devices that interact with AI-Kit (Ch. 7.5).
Define Using the outcomes of the explorations, the found design opportunities and required features were defined multiple times. First, concluding to the first phase of this project where the requirements for the second phase were defined (Ch. 6). Moreover, during the kick-off of the second phase, the design challenge was further refined to ensure this project’s independence (introduction to Ch. 7). Moreover, after the needs upon controlling output parameters were explored, the additional requirements for the interface have been defined (Ch. 7.2). Based on the study findings and the exploration on the roles of other devices, the concluding concept’s features were defined (Ch. 7.6).
15 6 - FIRST PROJECT PHASE
During the first phase of this project, the Context opportunities and challenges of integrating AI-Kit In the second exploration, new ways to provide AI- in Smart Home environments were explored. As Kit with contextual data have been explored. For contextual probes seemed inappropriate for the example, by using connected buttons to indicate home environment (e.g. due to privacy issues), that a certain label is currently active. This approach it became evident that new methods to interact was thought to be promising due to the connection with AI-Kit were required. A descriptive model between the real world and AI-Kit, allowing users (figure 4.1) of AI-Kit was developed to depict the to label data without the need for a data labeling different devices that could be used to facilitate the session or camera (as contextual probe). interaction. Three areas of interest were selected: the input, the contextual probes, and output. Output In a third exploration, the output of AI-Kit was Input further explored as the activation of outputs is a TThe different types and attachment mechanisms significant part of the expected value for users. By of sensors that would be required to sense a using environmental sensors, the system would broad set of labels were hands-on explored (figure slowly become able to imitate user actions as 6.1 – 6.6). The created mock-ups were used to the scenario that requires an output interactions explore their potential locations in the home- is registered. This approach would require AI- environment. Concluding to this exploration, it Kit to individually detect patterns in data, which was found that the sensors would need a universal would require unsupervised Machine Learning of mounting system to allow for custom placement. which the feasibility for this usage scenario can be Moreover, the sensors should allow for re-location disputed. Moreover, as AI-Kit will interact with IoT to stimulate explorative behavior of users. It was devices (that often don’t have a static interaction), found that the placed location of the sensors might the collection of output interactions might be not be optimal to facilitate daily interactions (e.g. troublesome. a PIR motion sensor in the top corner of a room). This exploration was discontinued as generic IoT sensors were considered as a strong competitor to compete against with custom made sensors.
Figure 6.1 - Adjustable sensor Figure 6.2 - Door sensor Figure 6.3 - Sensor mock-up
Figure 6.4 - Multi-side sensor Figure 6.5 - Velcro mechanism Figure 6.6 - Pressure sensor
16 6 - First Project Phase
Conclusion Short-term preferences The contextual approach seemed promising The translation from recognized labels to activated (due to its ability to label data in-the-moment) in outputs should be easy to adjust as user needs combination with the described output approach should be considered as dynamic. This is especially (imitating user actions). Concluding to this process, necessary as preferences, habits, and devices an interactive prototype (figure 6.7) was made to continuously change in the home environment communicate the findings of the process. The (Funk, Chen, Yang, & Chen, 2018). prototype consisted of a light dimmer (output interaction) that would slowly learn the state it Exception flagging should be in, imitating user actions. From these By allowing users to flag interactions as exceptional explorations, a list of design criteria was defined: (and exclude them from the training) their understanding of the underlying model can be Label based control increased. Moreover, excluding these outliers A set of outputs can be controlled as group by could decrease the required training time (Yang & making use of scenes that get activated upon label Newman, 2013). recognition. Fluent bootstrapping process Direct output control The ability of the system to slowly imitate user As control through scenes might become complex actions was discussed as promising as it would over time, simple and direct control of individual decrease the emphasis on the training phase of AI- outputs should be accessible at all times. Kit’s usage while creating immediate value to the user.
Figure 6.7 - Final prototype of project phase one
17 7 - SECOND PROJECT PHASE
At the start of the second phase it became clear that AI-Kit would not be developed to an experienceable stage prior to the project’s deadline. As a result, no interactions between the designed artifacts of this project and AI-Kit could take place. In order to allow this project to be developed regardless of the development of AI-Kit, a handover point was selected: the detected label. This point was found suitable while reviewing the previously introduced model (figure 4.1) in consultation with Bureau Moeilijke Dingen.
As a result of this handover point, new contextual data elements were introduced into the project. As the detected label (input for this project) can Figure 7.1.1 - Sketched interface exploration include false negatives and positives, a method to provide feedback to AI-Kit should be included. By doing so, the AI-Kit can improve the model that Fast forward Add scene has been trained for that specific label through * Reinforcement Learning. Moreover, as outputs will be automatically activated when a trained label has been recognized, it should be easy to review and monitor AI-Kit’s status and detected labels. If this would be excluded from the interface, users * would be required to address the web-app to see why, for example, a light has turned on. In these first explorations, there has been a focus on label * monitoring and feedback providence to AI-Kit which scene[1] ** scene[1] will be further referred to as the ‘Label Monitoring’ selector selector mode of the design. The ability to control outputs % % will be addressed in a consecutive exploration, and will be referred to as the ‘Output Control’ mode. Figure 7.1.2 - Digital interface explorations * scene[2] selector 7.1 Label Monitoring explorations % * Early Interface Explorations history labeling To kick-off this process, a hands-on approach was used in which first versions of the interface were explored. With a rapid increase in fidelity (from sketches (figure 7.1.1), to computer visuals * history (figure 7.1.2), to interactive (Adobe XD (Adobe, labeling 2019)) mockups (figure 7.1.3)), a large amount (199 artboards) of interfaces (and their states) were explored. Storyboards were used to explore how the interfaces would be used in different scenarios. These storyboards were made on a custom made template, that allows for the characters and scenario’s to be easily defined (figure 7.1.5). To further illustrate how different features were integrated in these exploration, one interface is Figure 7.1.3 - Adobe XD exploration explained as an example on the next page.
18 7 - Second Project Phase
Idle (figure 7.1.4) In the device’s idle state, a timeline is shown. This allows the user to monitor the state of AI-Kit by reviewing an overview of the labels that have been detected over a period of time. The top of the dial represents ‘now’, where just recognized labels would appear. The labels are color-coded, and the thickness represents the duration that a label has been active. The length of the label represents the certainty of AI-Kit on the recognition of a label. If AI-Kit is uncertain of the recognition of a label, the label will be displayed smaller.
Removing a recognized label (figure 7.1.5) When a user want’s to remove a false positive, Figure 7.1.4 - Idle the following steps need to be undertaken. First, the central dial should be rotated to the label that should be removed. Second, the central dial has to be pressed and dragged over the label (erasing the label from the timeline). Third, the central dial should be released and after a (timout) delay the dial will return to its idle state.
Adding a (false negative) label (figure 7.1.6) If AI-Kit did not recognize the occurrence of a label, it can be added manually. If the central dial is pressed in its idle state, the label selection interface would appear (figure 7.1.6). Using the central dial, the desired label can be selected. In this menu, the color coding is consistent with the colors used in the timeline. The size represents the certainty of AI-Kit that this label is currently occurring, similar Figure 7.1.5 - Removing a label to a pie chart. As this menu should support a large amount of labels (as every label in training can be selected), labels are categorized and distributed over several layers. To increase the detail of a label, one should press twice on the central dial. By doing so, the label detail (e.g. household tasks to vacuum cleaning) would increase linearly with the users need for detailed control and the time required for the interaction. After a label has been selected (by a single press and timeout), it would be added to the interface.
Exception flagging To add a label as exception, a similar interaction could be used. However, instead of pressing the central dial a pull would be required. After this pull, the label selection menu along with an ‘exception flagging’ indicator would appear. Figure 7.1.6 - Adding a label
19 7 - Second Project Phase
Limitations Conclusion As these, Adobe XD based, mock-ups were limited The explorative nature of this iteration allowed to computer-mouse interactions they proved to be for the introduction of several new features in the difficult to evaluate. Especially as the central dial project. But as the 3D Mouse could only supports a is designed to be tangible button, the interaction small rotation (a couple of degrees), the interfaces significantly differed from its designed form. could not be explored to its full extend. Moreover, In order to overcome this limitation, MDF dial the interfaces had a narrow view on the usage mockups were made first (figure 7.1.7). Then, a timeline of AI-Kit and only integrated a segment processing script was developed to connect a of the defined features. In the next iteration, there Spacemouse (3Dconnexion, 2018) to the interface was a focus on exploring the distinct usage phases explorations. By placing the 3D mouse on top of of AI-Kit and the corresponding required features the mock-up, the interactions could be explored and interactions. tangibly (figure 7.1.8).
Form As this interface would be placed in a central place, the look and feel has been explored from the start. Using a moodboard, an aesthetic language was selected which was thought to be suitable for the home environment as it allowed for the continuous presence of the technology without being blatant. Being inspired by the RT20 Radio of Dieter Rams (Luis, 2018), the form of the interface was explored of which several renders made (next page, figure 7.1.9 – 7.1.12) with the use of Autodesk Fusion (Autodesk, 2019) and KeyShot (KeyShot, 2019) (all other renders shown in this report were exclusively made with this software).
Figure 7.1.7 - MDF Knobs Figure 7.1.8 - Interactive Spacemouse setup
20 7 - Second Project Phase
Figure 7.1.9 - Render exploration (1) Figure 7.1.11 - Render exploration (3)
Figure 7.1.10 - Render exploration (2) Figure 7.1.12 - Render exploration (4)
21 7 - Second Project Phase
7.2 Output Control features
As the ability of AI-Kit to recognize a label increases Deployment over time, different usage phases can be defined. After a certainty-threshold has been reached, the The timeline in figure 7.2.1 shows the development allocated output states can be activated upon of a label and usage phases, in which the recognition label recognition. This threshold is thought to be certainty (y-axis) is shown over time (x-axis). As in required as the amount of false positives (caused the previous iteration the focus has been the Label by a low recognition certainty) would result in Monitoring mode, this iteration primarily explores frequent unexpected output behavior. the possibilities of easy output control, as stressed as importance to for dynamic output translation. Dynamic output translation Consistent with the first iteration, storyboards were As the needs of users continuously change, the used to explore the usage phases and required translation between recognized labels and the functionality. During this exploration, the following activated output should be changeable as well. phases and complementary features were found: For example, as one might physically move a light from the kitchen to the living room, it will become Label initialization irrelevant for a label that is recognized based When a label is initialized, AI-Kit will performance on a kitchen activity. As a consequence, a user poorly on the recognition of that label due to the should be able to alter the connection between limited available data. As a result, there is low the label and the output. This is not limited to the certainty whether a label is actually occurring. connection between labels and outputs or on/off- state changes, as users might want to change other Bootstrapping parameters (e.g. hue) within labels as well. If the While the user is training AI-Kit to recognize a label, design facilitates an easy interaction to facilitate the recognition certainty slowly increases. The these changes, users are potentially more likely rate of improvement is based on the difficulty of to explore (and use) the capabilities of their IoT the time-series data pattern to be recognized, and output devices. the amount of feedback given to AI-Kit. As habits change over time, the data pattern’s that should be associated with a label change as well. As a result, this phase will last indefinitely as users are required to update the model to allow labels to be continuously recognized.
Figure 7.2.1 - Usage phases
22 7 - Second Project Phase
7.3 Output Control exploration
Using the previously described exploration as input, four directions were proposed that included the required features. As the goal of this exploration was to explore methods to easily control and adjust output states, the label feedback does not differ between the explorations.
1 Trigger, 1 output (figure 7.3.1) In this first exploration, each connected output Figure 7.3.1 - 1 Trigger, 1 output would have a corresponding trigger integrated in the device. When a label has been recognized, the outputs are set to their desired states and the corresponding triggers would pop-up: indicating their activation. By changing the knob on the trigger, the output’s state could be easily configured. While this approach allows for the direct control of outputs, it has significant limitations in terms of scalability. As only a predefined set of triggers can be integrated in the device (limited to the available space), the outputs can potentially outnumber the triggers. Moreover, as the interaction is dependent of the output type (e.g. a slider for a dimming light versus a toggle for an I/O-light), a second scalability issue might occur where the interactions become unintuitive.
1 Trigger, 1 variable output (figure 7.3.2) Figure 7.3.2 - 1 Trigger, 1 variable output In a consecutive exploration, an attempt was made to resolve the scalability limitation was by creating a dynamic mapping between triggers and outputs. For example, when label A has been detected, a trigger would be in control of Light A. But if Label B would be detected, the trigger would be in control of Light B. But as labels are automatically recognized, the mapping of the trigger could change out of the blue, resulting in a mapping issue between the trigger and output. Moreover, a scalability issue would still occur if the number of outputs (for a label) exceeds the amount of triggers. Moreover, it would be challenging to maintain interaction consistency as triggers change of functionality multiple times a day.
1 Trigger, 1 output category (figure 7.3.3) In another attempt to overcome the scalability issues, triggers would be statically allocated to an output category. In the interface, there would be different types of triggers (i.e. light, temperature, or ventilation) which would pop-up if a label has been activated containing outputs of that category. Figure 7.3.3 - 1 Trigger, 1 output category
23 7 - Second Project Phase
The triggers are interchangeable, allowing users to preferably at locations where an easy mental select a custom set of output trigger-types. While connection between the sticker and the output can this approach solves the scalability by allowing be made. For example, the sticker associated with a large number of outputs to be connected to a a light could be placed on the lampshade. As the single trigger, it does not resolve the mapping stickers don’t need a physical connection with the issue as described under the ‘1 trigger, 1 variable output, the user is free to determine the optimal output’ exploration. Moreover, as the outputs are position for the sticker. The sticker for connected controlled as a group, the feature requirement window blinds, for example, does not have to be of direct output control has been diluted in this placed high near the actual device. Instead, the exploration. sticker can be placed at eye-height to allow for an easy interaction. The versatility of the stickers allow 1 Trigger, all outputs (figure 7.3.4) for the interface to be customized to the individual In the final exploration, the possibility to control home, accurately matching the user’s needs. all potential outputs with one trigger has been explored. In order to achieve this, a linking Link mechanism (usage) mechanism was designed to easily change the If the output parameters for a label have to be mapping between the trigger and the desired adjusted, the trigger can be detached from the output. This would allow the trigger solution to interface. Using the mobile trigger, an RFID sticker be scalable to a large amount of outputs while can be scanned. After doing so, the user has direct decreasing the amount of required triggers. control over that specific output. The knob located on the trigger allows for various parameters to be Link mechanism (setup) adjusted as one can cycle through parameters with Radio-frequency identification (RFID) was thought a single press. After the scene is set, the trigger can to be a suitable technology, as it allows for a large be returned to the interface and the configuration number of tags to be quickly identified. During will be saved to the label. the output initialization phase (where an output would be configured and connected to AI-Kit using This solution exists in a similar design space as the the manufacturer’s API), an RFID sticker would be previously described Reality Editor (Heun et al., allocated to the just initialized output using an RFID 2013). One of the main challenges they depict, is scanner integrated in a detachable trigger. The the creation of mental image in which an ID number sticker can be placed at any convenient location, is allocated to a certain output. As the list of IoT
Figure 7.3.4 - 1 Trigger, all outputs
24 7 - Second Project Phase
devices in the home starts to grow, remembering is detected. In this approach, AI-Kit becomes pro- the output location and the allocated ID becomes active and proposes (unnamed) labels to the user. infeasible. The described RFID approach overcomes Moreover, in case of a false positive, the user is this challenge, as the RFID sticker facilitates the able to intuitively provide feedback to the model by mapping between an output (and its location) and setting the output to the desired state. the output’s ID in a tangible manner. An obvious limitation of this approach will, most Matching interactions probably, also recognize patterns in unrelated As the universal trigger should be able to control sensors and output’s. If one would always turn up a broad set of output types, the interaction should the heat when the neighbor’s dog is coincidentally dynamically match the different output types as barking in the garden outside (detected by a sound well. To further illustrate these differences, the sensor), a useless cause-effect relation could different types of outputs are depicted (figure be developed by the model. On top of that, the 7.3.5). As the trigger’s knob should facilitate all feasibility of the Unsupervised Machine Learning interactions with these different output types, based system to recognize the patterns in the a dynamic approach would be appropriate. The ambiguous sensor data using output interactions as Haptic Engine (Van Valkenhoef, 2017) seemed to reference and system output should be evaluated. allow for this functionality, as the dynamic haptics could match the individual output types. Depending Conclusion on the output type, the knob’s feedback could be This exploration was concluded with the creation of adjusted to feel like a toggle (dichotomous) , a multiple renders in which both the Label Monitoring mode-switch (nominal/ordinal), or a potentiometer and Output Control mode have been implemented (ratio). (next page, figure 7.3.6 - figure 7.3.8). In some of the initial explorations, the label-timeline was Unsupervised Machine Learning capabilities mapped along (half) a circle while in others a linear Another advantage of the RFID-based approach approach was used. As in the previous renders, lies in the potential for Unsupervised Machine there was an emphasis on circular timelines, the Learning. As briefly described during the first new iteration used a linear timeline to provide a phase of this project, AI-Kit would slowly become balance in the different approaches during later able to imitate user actions. As the output trigger evaluation. The feature of the Label Monitor were can be used to interact with IoT devices, AI-Kit can unchanged. The detachable output trigger obtained monitor when certain outputs need to be activated. a LED ring around the knob, to provide feedback to As an example, the scenario of a kitchen with a users whether a label has been properly scanned CO2 sensor, a smoke sensor, and a connected and what specific output parameter they are extraction hood can be used. As every interaction controlling. with the extraction hood is picked up by AI-Kit, patterns in the environmental (CO2 and smoke) sensors can be recognized over time. After a period of training, AI-Kit would become able to activate the extraction hood when a rise of CO2 and/or smoke
Output Feature Values Scale Type Interaction Generic light bulb On/Off 1/0 Nominal (dichotomous) Switch (IoT) Curtains Raised - Lowered 0-255 Ratio Potentiometer Media center Input source A, B, C, D, … Nominal Rotary encoder (little pulse / rotation) (Gas) Stove heat (gas flow) 1, 2, 3, 4, … Ordinal Rotary encoder (little pulse / rotation) Thermostat Temperature 17 - 23 °C Interval Rotary encoder (many pulses / rotation) Figure 7.3.5 - Output types and interaction examples
25 7 - Second Project Phase
Figure 7.3.6 - Seperated trigger Figure 7.3.7 - Backside of design
Figure 7.3.8 - The designed interface integrating the ‘Label Monitoring’ and ‘Output Control’ mode
26 7 - Second Project Phase
7.4 Expert Panel study
In order to overcome the interaction limitations of the previous iterations (as described in chapter 7.1) and evaluate the interface with knowledgeable in the field, an Expert Panel study was designed. The studies allowed the needs, interface, and technologic feasibility of the previously described iterations to be evaluated. First, an interactive prototype was developed, then the study was designed in detail, and the results were analyzed after its execution.
Prototype development A new level of fidelity had to be achieved in order to allow the interface to be evaluated. One Label Monitoring interface was selected and developed extensively in Adobe XD to explore the required interactions (figure 7.4.1). As the designed interface was fairly based on assumptions, it would be ignorant to immediately create a high-fidelity Figure 7.4.1 - Adobe XD exploration prototype. On the other hand, as the interface consisted of a GUI and a tangible knob, a certain level of fidelity would be required to allow the prototype to be experienced. To overcome this difficulty, other means to quickly iteration upon interfaces were explored. This resulted in a setup, in which a (mini-)beamer would project upon the back of a semi-translucent surface (e.g. paper or engraved Perspex). This would allow the exploration of different screen sizes, shapes, and positions later in the process. The beamer would be used as an external laptop monitor to project a Processing (“Processing.org,” n.d.) sketch containing the visual interface elements. A knob (or other interaction elements) could be placed on the surface as well, Figure 7.4.2 - Test setup (3D Model) of which the signal would be read out using an Arduino Uno (Arduino, 2014). The data would be fed back in the processing sketch by connecting the Arduino to the same laptop, resulting in an interactive interface that could easily adjusted.
As the location of the beamer had to be precisely figured out, an extensive 3D model of the setup was made (figure 7.4.2). The surface projections of the 3D model were extracted and converted to be compatible with the lasercutter. The setup box was cut out of 6mm MDF to ensure a rigid enclosure, after which the box was painted in a neutral color (figure 7.4.3) to limit its influence on the interface experiences. Figure 7.4.3 - Test setup (painted)
27 7 - Second Project Phase
Concluding, magnets (to hold the detachable top panel) and electronics were installed (figure 7.4.5). The front panel (on which the beamer is projecting) of this setup is interchangeable, allowing for different iterations to be developed. In total, seven different front panels (next page, figure 7.4.6 – 7.4.12) and a panel-holder (figure 7.4.4) were made. As the panels differed in orientation (portrait/landscape), interchangeable stands were made that could be attached to the interface to match the panel’s orientation.
As the standard processing environment proved to be insufficient for the scale of this project, an industry standard workspace was set up using Visual Studio Code (Microsoft, 2020) and Sourcetree (Git version control) (Atlassian, 2019). By equipping each front panel with a voltage divider with ratios Figure 7.4.4 - Front panels (and holder) between the resistors, and (analog) reading the divided voltage, the interface could automatically detect which front panel was located in the setup; automatically projecting the correct interface. Some front panels were finished using a standard knob while others were equipped with a custom 3D printed one (figure 7.3.13).
Figure 7.4.5 - Test Setup electronics
28 7 - Second Project Phase
Figure 7.4.6 - Front panel 1 Figure 7.4.10 - Front panel 5
Figure 7.4.7 - Front panel 2 Figure 7.4.11 - Front panel 6
Figure 7.4.8 - Front panel 3 Figure 7.4.12 - Front panel 7
Figure 7.4.9 - Front panel 4 Figure 7.4.13 - 3D Printed knobs
29 7 - Second Project Phase
Study setup and execution automatically. During the study, voice recordings Due to the broad set of goals defined in for were made to allow the results to be analyzed later. study, the lab study consisted of various phases. To accompany the participants (and researcher) First, the topic would be introduced in a scripted along the various study phases, a participant presentation in which the design challenge and (and researcher) booklet has been created required features would be explained. Second, the (figure 7.4.14, Appendix C). The booklet included interface would be presented to the participant study information, the informed consent form, a after he/she would be asked to execute various presentation hand-out, the tasks, the cheat-sheet, assignments on the interface. For example, they and contact information. were asked to remove and add labels (throughout various layers of detail). The participants could After the study had been accepted by the ethical make use of a provided interaction cheat-sheet, procedure as defined by the department (Appendix as they were asked to execute the assignments D), a participant pool was selected. Due to the without further instructions. This section was extensiveness of the study, the participant pool size inspired by the Thinking Aloud method (Krahmer had to be restricted to ensure a feasible analysis & Ummelen, 2004) in which participants are asked of the generated qualitative data. To ensure a high to talk out loud while interacting with the interface. Return of Investment, a diverse participant pool of 5 Third, the participants were asked to fill in a System (+ one pilot) has been selected (Nielsen, 1989). The Usability Scale (SUS) (Lewis, 2018). Concluding, participants had backgrounds in Industrial Design a semi-structured interview was held to obtain (Bachelor and Master students), and coming/former insights in their experiences with the prototypes, Artificial Intelligence (Master students). While all of and probe the feasibility of the concept. On top of the participants were considered as knowledgeable that, the output trigger was introduced (using the in the field of IoT, their knowledge upon AI (and render shown in figure 7.2.9), to further discuss the Machine Learning) differed considerably. feasibility of implementing Unsupervised Machine Learning to allow AI-Kit to detect and prose labels
Figure 7.4.14 - Participant Booklets (1)
30 7 - Second Project Phase
Study analysis method Second, the created notes were prepared. The notes The results of the study, along with the were printed on different colors, which allowed pseudonymisation table, were stored encrypted the different participants to be distinguished in and password protected on a local hard drive. The a glance. Moreover, this allows categories to be pseudonymisation process allowed participants to reviewed on whether they have been touched withdraw from the study at all times. Two sets of data upon by different participants. were obtained during the study: voice recordings and the completed System Usability Scales. The Third, a second researcher was acquired for this usability scale has been designed to be used on phase of the study in order to diminish the bias a large set of participants to evaluate whether a enforced in the categorization of the quotes and product is ready for market, which does not match avoid any further ownership issues. The researcher the scale and goal of this study. The results of the was first introduced to the study method and topic scale were analyzed to conclude in a percentile using the scripted presentation as provided to the grade (Lewis, 2018), usable for the comparison of participants. Then, an overview of the different successive iterations. The qualitative data has been study components was provided. This explanation analyzed following the work of Lucero (Lucero, proved to be necessary to allow the researcher to 2015) in which an Affinity Diagramming method is quickly interpret the generated notes. The notes described to evaluate interactive prototypes. This were clustered, sub-clustered, split and merged on method has been the foundation of the analysis, a large table. After a small amount of notes were but due to a different study scale some alterations left (<15%, as described in the paper), the clustering had to be made. was terminated.
First, the voice recordings were transcribed. The Fourth, together with the secondary researcher, questions and the interviewers remarks were all the quotes and clusters were read through to removed (to allow topics to naturally emerge), ensure that every quote has been placed in the and the salient remarks of the participants were correct cluster. Then, the results of the affinity highlighted and annotated. This proved to be wall were documented by taking high resolution necessary, as participant are likely to use quotes photos of the clusters and depicting the cluster similar to “This feature needs some improvements topics accompanied with 1-4 example quotes. In as…” The annotations were distinguishable from total, 332 notes were clustered amongst 36 groups the original quotes using bold and regular fonts, (figure 7.4.15). and described which feature they were mentioning in their answer, for example. An attempt was made to avoid ownership issues of the generated notes by using original sections of the data, instead of researcher notes of analyzed videos.
Figure 7.4.15 - Affinity Diagramming result
31 7 - Second Project Phase
Affinity Diagramming findings or dishwashing). Another participant proposed a A broad range of findings resulted from the method in which generic datasets can be shared qualitative data analysis of which some were used between users. While the dataset will have to be in further iterations. The presented interface was altered to the specific environment of the user, it received reasonable, as the participants supported could potentially decrease the required training several design decisions. The participants were duration. For this approach it is important that probed whether they understood the design users remain in control of their data at all times, elements (e.g. thickness, length, and color of allowing them to select which data is shared. labels), and all of them agreed upon the value of color-coding the labels. However, they stressed Overall, the universal trigger to control outputs the importance of allowing users to set the color was positively received. It was indicated that such a themselves, due to the ambiguity of color-coding. trigger could be used to play out labels. One could Most of the participants indicated to immediately for example act like arrives at home, and use the see the relation between the label thickness and trigger to turn on the light in the hallway. By doing duration. The length however was not perceived so, datapoints are created while providing a natural as the certainty of AI-Kit, but as the intensity (e.g. interaction to set the scene. Moreover, the tags washing hands vs. showering) of a label. After could be used to control groups of outputs (e.g. a brief explanation, some indicated to like the one tag for 3 kitchen lights) to further decrease the feature as it would allow users to see the label- amount of interactions required. Some skepticism certainty grow over time which was thought to be a was expressed about the detachable feature of motivating factor to get through the training phase. the trigger, as it might decrease the connection between the label and the edited output. This On the other hand, they thought the interface requires the trigger to clearly indicate the label lacked several features. For example, a return you are currently editing. The Output Control button in the label-selection menu, an ‘undo’ mode was also described as an intuitive method to button to undo actions, and a method to change provide feedback to the system, as it allows users the interval that is displayed on the interface. Some to be occupied with the result instead of training. alterations to the detailing of the interface had to be made as well. Participants indicated for example The Unsupervised Machine Learning approach that the pointer of the cursor moved too slow, was depicted as interesting but technologically deleting labels was too easy, and that the layout of challenging. It was stated that there would be a the interface needed some additional explorations. high risk of irrelevant patterns to be recognized and proposed by the system. A potential difference The feasibility of the concept was heavily discussed in users and use cases was characterized, as some as well. While they indicated that the home- users might be more passive and benefit from environment suits the personalization abilities the label proposal functionality. To match these of the learning technology, some concerns were individual needs, an adjustable threshold should expressed as well. Participants expressed their be used to indicate how quickly a new label should doubt upon the user experience of the project, as be proposed. they questioned whether users would be willing to go through the training phase. The duration of System Usability Scale findings the training phase highly influences the experience Using the previously described method, the results similarly, but as too many factors (e.g. feedback of the System Usability Scale were analyzed. The amount, size of dataset, location of sensors) had interface received a score of 63.5 (Appendix E), to be taken into account, they could not make a corresponding with a C- and a 35-40% percentile prediction upon the duration of this phase. They ranking (Lewis, 2018). The score shows significant did depict how a layered approach might be room for improvement, especially for user suitable, in which more complex labels are added confidence while interacting with the system. This over time. AI-Kit could, for example, first learn to score will be used during consecutive iterations as recognize a kitchen activity. Over time, after the input and for comparison purposes. generic kitchen activity label has been sufficiently trained, in-depth labels can be trained (e.g. cooking
32 7 - Second Project Phase
Study discussion Conclusion While the Lab Setup of this study allowed early The project’s extensiveness has been confirmed versions of the interface to be presented, its by this study as various usage cases and scenarios limitations should be considered as well. As the were discussed. Possibly, as the other products interface has been decontextualized, it should to interact with AI-Kit (e.g. the web-app and a be questioned whether the experiences of the potential smartphone interface) were withhold participants would hold in the home (Koskinen, from participants during the task execution, a large Zimmerman, Binder, Redstrom, & Wensveen, amount of additional features were requested. To 2011). This was observed while participants were further frame the position of the interface, with executing various tasks on the interface that did not regard to user needs and target group, the various match the current time and location. For example, devices within AI-Kit should be defined amongst participants were presented the following task: their targeted user and integrated features.
“After you got home, you decided to do The potential to utilize Unsupervised Machine some household tasks. Unfortunately, Learning to allow AI-Kit to propose labels was AI-KIT has not recognized this label. discussed with the project’s client, Bureau Moeilijke Communicate to AI-KIT that you are Dingen. While this approach seemed promising, currently performing household tasks the requirement of users to design their own labels and that this label should be active. underlies the designed workflow of AI-Kit. Moreover, After which the light will turn on and as the technologic feasibility was questioned, this music will start playing.” aspect was excluded from future iterations in this design process. This decision was made to prevent However, as the study was executed in the middle of the design process from stagnating over the the day on a University, the implausibility effected complicated nature of this feature. Moreover, the the experience of the participants. Moreover, in the feature could be used as interesting starting point study the participants sat in front of the prototyped in future processes, in which the feasibility can be interface when they were asked to execute the explored more extensively. tasks. As a result, the effort required to interact is implausible to reflect situated experiences as they would be required to walk up to the interface in a real-world scenario.
While a participant pool with varying technologic knowledge was selected, they might not reflect the diverse target group which the interface is aimed for. As the interface would be situated in a home context, users with less knowledge upon the system’s internals would be invited to interact with the interface as well. This group was only limitedly represented in the study.
33 7 - Second Project Phase
7.5 Exploring future directions
The expert panel study’s result has emphasized While the monitor and the trigger both individually the extensiveness of the AI-Kit project in terms of introduced novel functionality in the AI-Kit family, users, features, and usage scenario’s. As a result, they seemed to be separate products. While the it became clear that the involved devices and their iteration with the docked trigger (figure 7.2.9) roles should be explored and defined further. To integrated both features, there was a conflict in the facilitate these different use cases, a product family mobility of the devices. As the label monitor was might come in to play where different devices already considered to be semi-stationary, it was match the individual needs. Moreover, as Bureau questioned when you would detach the trigger Moeilijke Dingen is exploring the value of developing out of an already quite mobile device. Moreover, their own sensors, the fact that AI-Kit will consist of during the expert panel study, it was found that the a product family is self-evident. To further explore connection between output states and the label the different potential products in this family and should be emphasized. In the consecutive iteration, their uses, a comparative exploration has been an attempt has been made to integrate both of the performed. By doing so, the definition of the functionalities (monitoring and output control) in products would increase while allowing potential one device while considering their usage scenarios. gabs to be discovered simultaneously.
The devices that were explored consist of: the web- app (in development), custom sensor boxes (in development), a phone-app (potential, low-risk), the Label Monitor, and the Output Control trigger. The parameters used for comparison are: targeted users (tech-savvy or people living in the usage context), features (e.g. AI-Kit monitoring or output control), usage-scenario’s (e.g. context, interaction duration, or required focus). The comparison has been executed in two-fold, where first primarily the feature-focus, usage-scenario, and targeted user was compared (Appendix F - 1). Second, the features were compare more thoroughly (Appendix F - 2).
34 7 - Second Project Phase
7.6 Conceptualizing the final design
Dual functionality explorations In an attempt to integrate both functionalities (Label Monitoring and Output Control) in one product while considering its mobility, a mobile controller (figure 7.6.1 and 7.6.2) has been designed. For this exploration, there was a large focus on exploring the aesthetic form and the interactions to control output parameters. Unfortunately, the label monitoring functionality was under-considered. Moreover, one of the results from the expert panel study has been the request to clearly separate the interactions of the device as the overload on a single button was perceived as complicated. On Figure 7.6.1 - Mobile Interface Design 1 top of that, as the interface will live in a certain context for a while (for model training purposes), the exaggerated interface mobility would be in conflict its projected use.
To improve the integration of the two functionality modes, the switch between these modes has been selected as design element of interest. The switch between these two modes has been one of the main concerns in this project’s phase, as the individual usages had been broadly designed in previous iterations. Different approaches to this switch have been briefly explored: different interaction elements, orientation changes, triggering docking stations, and sliding/flip-over front panels (figure 7.6.3). Figure 7.6.2 - Mobile Interface design 2
While the flip-over exploration seemed the most promising to facilitate the two usage modes, this approach entails several complicated aspects. A flipping front cover implies a (potentially fragile) hinge mechanism, cover-detection electronics, and a re-usable interface which all required several more iterations. As at this stage of the project, the RFID-based output control had not been developed to an experienceable level this has been prioritized over the quality of interaction to switch between the two usage modes. The decision was made to define the flipping mechanism as future work and out of scope for this design process, after which a new, simplified, interaction was designed to switch between the two modes. Figure 7.6.3 - Flip-over front panel sketch
35 7 - Second Project Phase
Concept features and form During the expert panel study, it was found that Over the course of this design process, several the presented interface called for detailed control. features have been added and removed over the The interface however, has been designed for course of several iterations. In order to maintain label monitoring and brief interactions as detailed an overview of the technologic requirements, the control is provided in the complementary (web-) concept’s features were defined (detailed overview app. To decrease the call for detailed control, the in Appendix G). Being inspired by morphological resolution of the linear interface was decreased by analysis techniques (Álvarez & Ritchey, 2015), the using LEDs instead of a screen. Moreover, the LEDs layout of the potential components were hands- could allow the linear interface to become invisible on explored (figure 7.6.4). The rotating knob of when not in use (e.g. when no label has been previous iterations was replaced by a slider in order recognized) by using a semi-translucent cover. The to increase the natural action-function coupling in most suitable interactions have been selected an terms of direction, dynamics, and location with added to the feature definitions (Appendix G) after the linear interface (Wensveen, Djajadiningrat, & a brief exploration using interactive mock-ups. Overbeeke, 2004). The slider can be used to both select labels, and to control output parameters depending on the current mode.
Figure 7.6.4 - Morphologic exploration
36 7 - Second Project Phase
The designed linear interface can be used to show both the detected labels (figure 7.6.6 and figure 7.6.7) and to provide feedforward when controlling outputs (figure 7.6.5). A relatively small screen was added to provide additional information such as label- or output names and facilitate Label Monitor and Output Control interactions. A user is able to interact with the menu displayed on the screen using the capacitive buttons located around it. Matching the focus of the designed interface (Output Control and (daily life) Implementation) within AI-Kit, a name has been allocated to the device: Oci. Developing towards the final iteration of this design process, the materials have been selected (wood and matte black plastic (ABS)) based on the previously explored aesthetic language. A contextualized (including the shape, size, material, and potential usage context) render can be seen in figure 7.6.8 on the next page. Consecutively, a prototype has been developed to communicate the concept and compare it to the previously evaluated iterations. Figure 7.6.5 - Output Control
Figure 7.6.6 - Label Monitor (no label selected)
Figure 7.6.7 - Label Monitor (label selected)
37 7 - Second Project Phase
Figure 7.6.8 - Contextualized render
38 7 - Second Project Phase
7.7 Concept prototyping
Electronics selection and development Motorized slide potentiometer Simultaneous to the design of the interaction and The motorized slider (PSM01-082A-103B2) can interface elements, the electrical components provide haptic feedback to match the selected were explored to ensure the feasibility of the output parameter as previously described. prototype. Using multiple comparison charts the Moreover, the integrated DC motor allows the (dis-)advantages of several interaction and display knob to automatically move to the correct position components were explored. The modules and to match the current state of an output. complementary components were selected after which the connections (Appendix H) were soldered Motor Driver on two stackable layers of Prototype PCB board A DC Motor driver (DRV8833) was required to drive (figure 7.7.1). The multiple layers were required to the DC motor integrated in the slider. fit the prototype in the designed form. The included components and their roles are described below. Capacitive buttons The capacitive buttons next to the screen were Component overview made using copper tape (underneath the thin Array of led-strips wooden slice), and a capacitive sensor breakout A cheap LED matrix for the linear interface is made board (MPR121, to decrease the amount of by using multiple LED strips (WS2812B, 144 LEDs required pins on the microcontroller). Moreover, per meter). As the LEDs are capable of emitting the board can be connected to the capacitive bright light, they can be hidden underneath a thin sensing pin of the slider. By doing so, the controller (1mm) slice of wood making them invisible in the can sense when the knob is being touched. off state allowing them to appear out of the blue when a label has been detected. RFID Reader An RFID reader (PN532) reads RFID tags and 2.2” TFT Screen communicate the ID to the microcontroller over I2C. A small (ILI9341 based) screen that matches the height of the LED array has been selected. The screen can be controlled using the SPI interface of a microcontroller.
Figure 7.7.1 - Designed Protype PCB
39 7 - Second Project Phase
Microcontroller (main) Software A microcontroller with integrated Wi-Fi connectivity A new workplace has been set up to develop the was selected to allow the interface to connect to software required for the described electronics. An IoT devices. The controller was required to have an online (GIT) repository was created, allowing the extensive amount of pins and memory, as a lot of code to be accessible for multiple collaborators. modules (i.e. a screen, capacitive sensor board, DC This allowed the employees of Bureau Moeilijke motor driver, and battery) had to be connected. A Dingen to quickly watch along with my work if (ESP32 based) Wemos D32 Pro has been selected I would encounter any problems. The different as it was found to be capable of driving all the states (and their connections) of the interface have components. been designed using a state machine diagram (Appendix I), and the coding environment has been Microcontroller (Arduino Nano) set up accordingly (making use of state and module As the RFID reader proved to be incompatible controllers and distributed header and source code with the D32 Pro, a second microcontroller was files). The Philips Hue (Philips, 2019b) IoT Lighting added to the scheme. The Arduino Nano would system has been selected as controllable output, communicate the found ID to the D32 Pro. as it has an extended API that allows the connected bulbs to be controlled using (HTTP) requests. To Level Converter simplify the process of sending requests to the Hue The digital pin voltage of the Arduino Nano (5V) is Bridge, a library (facilitating these requests) has incompatible with the D32 Pro’s pins (3.3V). In order been customized to be compatible with the ESP32. to allow the microcontrollers to communicate, a Level Converter was required to convert the signal During the development of this interface, several to the correct voltage. complications had to be overcome. For example, as the RFID Reader proved to be incompatible Battery with the ESP32, an Arduino Nano was added to the As the design is required to be handheld (for the circuit. However, as software-serial communication output control mode), a Li-Po battery (LP654290, interferes with the Wi-Fi capabilities of the ESP32 (as 2600 mAh) has been selected. The battery can be required to control the Hue bulbs), a new method charged using the on-board charging module of to facilitate the communication between the two the D32 Pro (preventing the battery from over/ boards had to be developed. By writing an analog under charging). value (PWM Wave) from the Nano to the ESP, a large numbers of ID’s could be communicated. A Step-up Boost Converter second pin was required as reading the PWM Wave As the battery has to power modules that significantly delays the sketch on the ESP. When a require different voltages, multiple Step-Up high signal on the second pin has been detected, Boost Converters were used. The D32 Pro can be the ESP would start reading the PWM signal. By powered using the (nominal) voltage of the battery doing so, the sketch would only be delayed when a (3.7V). The first Step-Up (HW668) boosts the voltage new RFID tag has been scanned. up to 5V (powering the LED’s, the Arduino Nano, and several modules), and a consecutive Step-Up (MT3608) boosts the voltage up to 10V which is used for the DC motor (and driver).
Reed Relay In order to detect the change between modes, a Reed Relay was selected. The relay allows a present magnet to be detected. The magnet can be integrated in a docking station or in the front cover to detect the state change of possible future iterations.
40 7 - Second Project Phase
Casing The nut holes in the top cover allowed components While at first, some attempt was made to shape the (motorized slider, led matrix, and the TFT screen) prototype close to the proposed concept (render of to be firmly attached (figure 7.7.5 and figure 7.7.6). figure 7.6.8), some concessions were required to be The nuts were covered using a thin slice of balsa made over the course of several casing iterations. wood. The rest of the electronics (RFID reader, In a first attempt to develop the casing, an extended Battery, and the 2 layered PCB) were stuck to the 3D model has been made to fit all the electronic inside of the box. Concluding, a hole was milled components (figure 7.7.2). Due to the inefficiency to allow the battery to be charged using a regular of the model, the 3D printing process would (micro-USB) cable. An overview of the developed take too long. Expert knowledge was obtained to prototype can be seen in figure 7.7.6 and figure improve the model, after which the model has been 7.7.7 on the next page. optimized for the printing process. The new model consisted of three parts with less internal infill (figure 7.7.3). In this step, some concessions were made upon the form to ensure an efficient printing process. Some concerns were expressed upon the finish of the print, and alternative approaches were explored. This concluded with the purchase of a plastic (PLA) ‘General Purpose Enclosure’, which would house all the electronics in a smooth finish. A top cover for the enclosure was 3D modeled and printed, requiring far less printing material.
Figure 7.7.2 - First 3D Model Figure 7.7.3 - 3D Model (less infill)
Figure 7.7.4 - Top panel (top) Figure 7.7.5 - Top panel (bottom)
41 7 - Second Project Phase
Figure 7.7.6 - Final prototype (Output Control mode)
Figure 7.7.7 - Final prototype (Label Monitor mode)
42 7 - Second Project Phase
7.8 Evaluation study
Setup Execution and results The developed prototype has been evaluated in a In coherence with the Expert Panel study, brief evaluation study. The aim of this study was the Evaluation Study has been executed in a to identify which of the label-monitor alterations participant pool of five (+ one pilot) in a lab setting. were positively/negatively received. Moreover, as Three participants took part in the original study, the prototype integrated the first experienceable and allowed for a comparison of the results. As version of controlling outputs using RFID tags, these three participants are reintroduced into the initial responses were necessary to evaluate the topic, their background knowledge could influence feature. their SUS scoring. Due to the small amount of participants, no significant results could be The setup of the study had much resemblance (in obtained. The two new participants (consisting of terms of setup and data-handling) with the original students from different faculties) were recruited to Expert Panel study. First, information on the study briefly explore the influence of prior knowledge on was provided to the selected participants, after the usability scale’s result. which they were asked to sign a consent form. The scripted presentation upon AI-Kit and the Label Monitoring mode design challenge was given, with an additional Reviewing the SUS scores (Appendix E) of the introduction to the Output Control mode. Second, recurring participants, the new Label Monitoring the participants were asked to execute various interface was either rated equal (1) or higher (2) tasks on the Label Monitoring interface. As not than the previously presented iteration. The two all features were implemented in the prototype, newly recruited participants rated the interface an interactive mockup (made in Adobe XD, higher than the average rating of the previous figure 7.6.6 and 7.6.7) has been made to discuss interface. While no conclusions from this brief several undeveloped features. Subsequently, analysis can be drawn, the changes seem to have the participants were asked to fill in the System improved the design. This also came forward Usability Scale (SUS) based on their experiences during the interviews, where different participants with the Label Monitoring interface. Third, the mentioned: Output Control mode would be introduced. They were asked to set the staged environment “This is much better. It makes more to their desired setting, controlling the state of sense than rotating a knob*” three Philips Hue bulbs. The participants could walk around, scan RFID tags, and interact with the “The idea of the timeline makes much Hue’s instantly. After the lighting was set to the more sense now that it’s linear*” (as desired state, additional Output Control features opposed to the circular design of the were discussed using the mock-up (figure 7.6.5). previous interface) A separate Usability Scale was completed for this part of the interface. On the other hand, while discussing the label color and certainty, one (new) participant noted: As some participants were selected from the previous participant pool, the results of the “I would need some more time to SUS for the Label Monitoring interface could completely understand it*” be compared using a transparent overlay . This allowed the participants to be probed on what aspects they thought majorly improved or were better in the previously presented iteration in a short semi-structured interview. To combine all the different study elements, a new booklet (with updated assignments) had been created to accompany the participants throughout the study.
43 7 - Second Project Phase
Output Control mode The Output Control interface was well received. The average usability score of this interface scored the highest (in comparison to the Label Monitoring interfaces), and several positive remarks upon the interface were made. The distinction between the two usage modes by using the device’s orientation worked well. One participant indicated:
“First, the interface is horizontal. Then, if I want to do something else I grab it and it becomes vertical. This provides me with visual cues of what I am currently doing*”
Moreover, it was indicated that the ‘tap and control’ type of interaction was natural and invited for interactions. Different participants stated:
“I liked the 1:1 connection with the light and what you would like to change. We have Philips Hue’s at home, and I miss the direct interaction with my light. In some way, this brings that interaction back, which is something that I like*”.
“I liked it. It was intuitive to use and it worked really well*”
This method to interact with IoT Devices seems promising, and the results of this study could serve as input for future work upon this type of control.
*These quotes were recorded during the study. For this report, they have been translated from Dutch to English.
44 8 - DISCUSSION
8.1 Concept value proposition
The design challenges of implementing AI-Kit in and Medium-sized Enterprises (SME’s). As these Smart Home environments have been explored companies might not have the financial resources in this design process. While several use cases for to, for example, hire a revenue optimization analyst, a learning system in the home environment have they might find value in utilizing AI-Kit to explore been explored over the course of this project, the influence of certain aspects (e.g. temperature defining the unique value proposition for the right or humidity) on sales. As by doing so, AI-Kit brings target group has been troublesome. Most use cases more value to the user (in terms of an increase of implicated the automation of some type of human- potential sales), there might be a larger incentive to output interaction. Generally in home automation, go through the training phase. causal thinking is used to define the set of rules required to activate an output (with the desired parameters). If the defined rules are accurate 8.2 Interactions and include a sufficient amount of exceptions, the rule-based approach will be superior to the In this design process, an attempt has been learning-based competitor without the need of made to bring a complicated technology down a training phase. In other words, the fact that to its core. By making the functionality easily most automation functionality can be achieved accessible, other users are involved in the training using rule-based systems as long as the rules are process as well. Moreover, the designed type of extensive and elaborate enough, has complicated output control might aid in the exploration of the the value proposition of a learning system in this output’s potential, unveiling more functionality context. This proved to be the case while creating of IoT products. While the functionality has been storyboards and probing participants on potential extensively considered, the interaction and use cases. The main value of this system might lay implementation could benefit from an additional in scenarios that have a level of ambiguity (e.g. design iteration and consecutive field study. While chatter in the living room), but as this approach to the selection of labels and controlling of outputs interact with outputs is unconventional it would using the RFID stickers and slider was perceived require a level of adaptation of the user. To further positively, the other interactions (capacitive buttons explore the value of AI-Kit in this context, it will be and mode switching) should be further explored. crucial to develop the concept to an experienceable An initial attempt has been made to design novel stage (including model training). By doing so, users action possibilities to change the mode (using the will be able to hands-on explore the possibilities of front panel as briefly described) of the interface, the system which will allow new application areas with promising results. Despite this fact, the pick- to be revealed. up interaction to switch between modes has been selected to ensure the feasibility of the prototype. Another important aspect that will influence the In future work, the integration of these different overall experience of this system, is the duration functionalities could be improved by further of the training phase. In the Expert Panel study, considering the usage (in terms of frequency of use it was depicted how the amount of interactions and relocation) of the interface. To achieve this, the (feedback) would be far more important than the system should be developed to a higher level of timespan over which they occur. This amount fidelity, allowing all usage phases of the system to will highly influence the overall experience of the be evaluated in the desired context. system, as a too lengthy learning phase might cause users to lose interest. In this work, there has been Utilizing the capabilities of the motorized sliding a focus on the design implications assuming that potentiometer to match the different output the training phase would be contained within this parameters and its state seemed promising. As the limit. However, if this project would be continued, slider is able to directly move to the output’s state, these aspects should be considered and evaluated the coupling between the interface and output is to ensure a positive effort/value ratio for the user. increased. Moreover, the haptic feedback could Another target group that could be considered, are make the slider feel like a toggle, mode-switch, or owners (or companies collaborative with) of Small scale to match the output’s parameter to increase
45 8 - Discussion
this match even further. In this work, the interface 8.4 Unsupervised Learning has only been connected to a set of Philips Hue controlling a single parameter: brightness. In future While this approach to output control is radically work, it would be valuable to explore the capabilities different from the approach used in AI-Kit, the of this single slider to control other outputs (and its potential use case is promising and serves as parameters) as well. This would truly release the input for future work. As previously described, capabilities of this concept, allowing users to move the natural interactions with outputs (potentially around and naturally set the environment to their using the universal controller and RFID tags) could preferences. facilitate a type of Unsupervised Machine Learning. By keeping track of what actions take place at what moment, the system could slowly take over 8.3 Financial viability user actions as patterns are recognized in sensor data. As this allows users to interact with their While this work has primarily focused on the environment in a traditional manner, this approach exploration of the requirements while attempting allows for a natural decrease in the amount of to design the best possible interface, the market required user actions. This approach however, is feasibility of the design should be considered. This strongly dependent on the capability of the system approach seemed to be suitable for the design to detect practical patterns in the time-series data in process, as users needed to be probed on their the moment. Extensive future research is required interest towards the design while exploring the to explore the required sensors, their locations, demanded features. It should be noted that the and the technologic feasibility of live Unsupervised designed interface would be an expensive product. Machine Learning for this application. Moreover, As the target group forms a niche market, production it should be explored whether users enjoy the volumes will be low resulting in high production experience of a gradual increase in automation. costs per unit. Furthermore, the interface houses Potentially, users might feel out of control several modules which would require extensive of their environment resulting in unpleasant development and testing on safety and endurance. experiences which should be considered during Combined, the interface as presented in this the development of such system. report would be rather expensive. As a next step, the designed features should be evaluated on their value in an implemented scenario. By doing so, the features can be prioritized before making concessions to move to a market viable product.
If the physical products (sensor boxes and the presented interface) turnover would be used to finance the required server space and computing capabilities (to host the collected data and train models), the price would increase even further to cover the monthly cost. Over the course of this project, a new approach to finance these expenses was found. By making the system exclusively compatible with AI-Kit branded RFID tags, users are bound to purchasing additional tags to expand their functionality. The exclusive RFID tags could be sold with a high profit margin (due to low cost and a high value to users), potentially making AI-Kit more profitable as a whole.
46 9 - CONCLUSION
In this design process, the design challenges that 9.1 Acknowledgements emerge when implementing a Machine Learning system (AI-Kit) in Smart Home environments have I would like to thank my graduation mentor Joep been explored. During the initial phases of the Frens for his coaching over the course of this project. project, the features a potential user would require While I might have felt lost in our discussions to interact with such a system were defined. Over fueled by the project’s complexity, these dialogs the course of several iterations, the defined features have definitely benefit the design process and were integrated in an interactive prototype used its outcome. I want to thank the employees of for evaluation in an Expert Panel study. Concluding Bureau Moeilijke Dingen, providing me with expert from the iterations and the findings of the study, knowledge and continuous feedback. In particular, two distinct feature modes were defined as crucial I want to thank Joep Elderman for his contributions to allow the functionality of AI-Kit to be integrated to this project in terms of coaching, support, and in daily life. In the Label Monitoring mode, a user is technological advice. Concluding, I would like to able to provide feedback to AI-Kit while monitoring thank the participants of the studies for their the overall state of the system. In this mode, the participation, and peer-students for their feedback labels detected by AI-Kit are communicated to the and advice. user including details on duration and detection certainty. Moreover, a user is able to confirm, flag, edit, and remove (in case of a false positive) labels using the interface. On top of that, users can select a non-detected label (false negative) through various layers, providing more detailed control as the interaction progresses. The Output Control mode enables users to easily set their output parameters to their preferences. By allocating an RFID sticker to a certain output, users can control their IoT device by tapping the sticker with the designed interface. The functionalities have been integrated into one design (Oci), housing all the required functionalities for a daily life integration of AI-Kit in the home environment. An interactive prototype of the concept has been developed for communication and evaluation. In a brief study, the concept features and their integration have been evaluated hinting towards an improvement in comparison to the previous version (as evaluated during the Expert Panel study). The proposed concept is a significant step in the search of facilitating an easy interaction with AI-Kit, allowing users to design and develop their own Machine Learning Functionality. Future work is required to evaluate the presented features in a contextual deployment, allowing features to be prioritized while moving towards a viable product.
47 TERMINOLOGY
The system Contextual data and probes Throughout this report, ‘The system’ is used to As it might be challenging to recall why a sensor refer to AI-Kit as a whole. This includes the Machine value has changed hours or days ago, contextual Learning model, data collection/processing (or reference) data is used to simplify the data procedures, and the activation of outputs. labeling process. Contextual data can, for example consist of an audio or video feed that is in sync with Outputs the time-series data. This allows users to review the Connected devices serve as the output of AI- feed while examining a chunk of data that needs Kit. Examples include connected light bulbs, to be labeled. Contextual probes are sensor boxes thermostats, curtains, and many more. that collect the contextual data, this could for example be a connected camera or microphone. Output parameters Connected devices have several parameters that Output interaction can be controlled. For example, some light bulbs The interaction that a user has with an output. This can be set to a specific hue, brightness, and could imply a connected output through AI-Kit or a saturation level. These settings are considered as traditional output using a switch. parameters of the output. Bootstrapping (phase) Exception flagging After a label training has been initialized by a An interaction that should be considered as an user, the label detection certainty needs to grow. exception can be flagged by a user. This allows the To achieve this, feedback should be provided user to communicate to the system that a scenario to the system to increase the size and quality of or action is rare, and should not be added to the the dataset used for training. As habits of users training dataset as it will potentially decrease the inevitably change over time, some feedback will model’s accuracy (Yang & Newman, 2013). always be required to maintain a labels detection certainty. Detection certainty As the recognition of labels by AI-Kit is probabilistic, Deployment (phase) full detection accuracy will never be reached. The After a certain level of detection certainty is detection certainty refers to the certainty that AI-Kit reached, the trained model can be deployed. In the has on the recognition of a label. deployment phase, outputs will be activated in the desired setting after a label has been recognized. Label Monitoring mode The first of the two usage modes integrated in the Output Control mode design is the Label Monitoring mode. This mode The second usage mode allows for easy allows users to monitor the state of AI-Kit and output interactions. By scanning an RFID tag interact with detected labels. In the final design of that is allocated to an output during a one- this report, this mode allows labels to be flagged, time setup, users are in direct control of a edited, confirmed, or removed in case of a false specific output. Different output parameters positive. When AI-Kit did not detect the occurrence can be adjusted when connected to an output. of a label (false negative), users can add the label throughout various layers of control. Proposed labels (by AI-Kit) Using Unsupervised Machine Learning, AI-Kit Output trigger could proactively propose (unnamed) labels by During Output Control explorations, the term recognizing patterns in the data. (output) trigger has been used to describe a small device that would consist of an enclose and a Data labeling simple interaction. The trigger would be mapped to In the data labeling process, labels are allocated a certain output (parameter). Some of the explored to timeframes of time-series data. By doing so, triggers were attached to the main interface, while a labeled dataset is created that can be used for others were mobile. Supervised Machine Learning.
48 REFERENCES
3Dconnexion. (2018). 3Dconnexion - SpaceMouse Autodesk. (2019). Cloud Powered 3D CAD/CAM Compact. Retrieved January 6, 2020, from Software for Product Design | Fusion 360. https://www.3dconnexion.eu/spacemouse_ Retrieved January 8, 2020, from https://www. compact/eu/ autodesk.com/products/fusion-360/overview
Adobe. (2019). Adobe XD | UX/UI design and col- Autosen. (n.d.). io-key: Remote monitoring, made laboration tool. Retrieved January 5, 2020, easy. Retrieved January 21, 2020, from from https://www.adobe.com/products/ https://autosen.com/en/io-key xd.html Bakker, S., & Niemantsverdriet, K. (2016). The in- Álvarez, A., & Ritchey, T. (2015). Applications of teraction-attention continuum: Considering General Morphological Analysis. Acta Mor- various levels of human attention in inter- phologica Generalis AMG, 4(1). action design. International Journal of Design, 10(2), 1–14. Amazon. (2018). Echo Dot (3rd Generation) | Al- exa-enabled Bluetooth Speaker. Retrieved Bakker, S., van den Hoven, E., & Eggen, B. (2010). March 11, 2019, from https://www.amazon. Design for the Periphery. EuroHaptics 2010, com/gp/product/B0792K2BK6/ref=s9_ 71. acsd_al_bw_c_x_1_w?pf_rd_m=ATVPDKIKX- 0DER&pf_rd_s=merchandised-search-4&pf_ Bakker, S., van den Hoven, E., & Eggen, B. (2015). rd_r=G5HXRKHT6Q0NFTW64GQG&pf_rd_ Peripheral interaction: characteristics and t=101&pf_rd_p=69c777a9-30f3-4687-b3d5- considerations. Personal and Ubiquitous 72ea2fa4610a&pf_rd_i=9818047011 Computing, 19(1), 239–254. https://doi. org/10.1007/s00779-014-0775-2 Anki. (2019). Anki Vector | The Home Robot With Interactive AI Technology | Anki USA. Re- Belkin. (2019). Wemo Plug. Retrieved January 3, trieved January 21, 2020, from https://anki. 2020, from https://www.belkin.com/us/p/P- com/en-us/vector.html F7C063/
Apple Inc. (2019). HomePod - Apple. Retrieved Brand, R., & Rocchi, S. (2011). Rethinking value in March 11, 2019, from https://www.apple. a changing landscape. A Philips Design Paper. com/homepod/ Philips Design, Koninklijke Philips Electronics NV. Arduino. (2014). Arduino UNO Rev3. Retrieved January 8, 2020, from https://store.arduino. Bureau Moeilijke Dingen. (2019). BMD Studio. Re- cc/arduino-uno-rev3 trieved January 4, 2020, from https://moeili- jkedingen.nl/ Aron, J. (2011). How innovative is Apple’s new voice assistant, Siri? New Scientist. https://doi. Case, R. (2019). Machine Learning, Part 4: Rein- org/10.1016/s0262-4079(11)62647-x forcement Learning. Retrieved January 2, 2020, from https://towardsdatascience. Atlassian. (2019). Sourcetree | Free Git GUI for com/machine-learning-part-4-reinforce- Mac and Windows. Retrieved January 8, ment-learning-43070cbd83ab 2020, from https://www.sourcetreeapp.com/
Atmos. (2019). Atmos: Simplify Your Smart Home. Retrieved March 11, 2019, from https://at- moshome.com/
49 References
Control4 Corporation. (2016). Home Automation Groeger, L., & Schweitzer, J. (2015). Transforma- and Smart Home Systems | Control4. Re- tional leadership, design thinking and the trieved May 27, 2017, from https://www. innovative organization. In EGOS paper pro- control4.com/ posal for Sub-theme 09 : (SWG) Leadership in Art , Design and Organization . (pp. 1–19). Design Council. (2011). Design methods for develop- Retrieved from https://www.researchgate. ing services. net/publication/279853991_Transformation- al_leadership_design_thinking_and_the_inno- Ethem Alpaydin. (2004). Introduction to Machine vative_organization Learning Second Edition Adaptive Computation and Machine Learning. Heun, V., Hobin, J., & Maes, P. (2013). Reality editor: Programming smarter objects. Flic Corp. (2019). Flic - Smart Bluetooth Button UbiComp 2013 Adjunct - Adjunct Publica- - Pure simplicity at the click of a button. Re- tion of the 2013 ACM Conference on Ubiq- trieved June 12, 2019, from https://flic.io/ uitous Computing, 307–310. https://doi. org/10.1145/2494091.2494185 FROLIC. (2019). Smartians | FROLIC studio |. Re- trieved March 11, 2019, from http://frolicstu- Hu, F., & Hao, Q. (2012). Intelligent Sensor Net- dio.com/portfolio/smartians/ works. In Intelligent Sensor Networks (pp. 3–29). CRC Press. https://doi.org/10.1201/ Funk, M., Chen, L.-L., Yang, S.-W., & Chen, Y.-K. b14300-21 (2018). Addressing the need to capture sce- narios, intentions and preferences: Interac- IFTTT. (n.d.). My services - IFTTT. Retrieved January tive intentional programming in the smart 4, 2020, from https://ifttt.com/ home. International Journal of Design, 12(1), 53–66. IKEA. (n.d.). FYRTUR - IKEA. Retrieved January 3, 2020, from https://www.ikea.com/gb/en/p/ Gaver, W. W. (1991). Technology affordances. In fyrtur-block-out-roller-blind-wireless-battery- Conference on Human Factors in Computing operated-grey-30408173/ Systems - Proceedings (pp. 79–84). Associa- tion for Computing Machinery. https://doi. Ishii, H., & Ullmer, B. (1997). Tangible bits: towards org/10.1145/108844.108856 seamless interfaces between people, bits, and atoms. Proceedings of the SIGCHI Confer- Google. (n.d.-a). Gmail: gratis opslag en e-mail ence on Human Factors in Computing Systems van Google. Retrieved January 4, 2020, from CHI 97, 234–241. https://doi.org/http://doi. https://www.google.com/gmail/about/ acm.org/10.1145/604045.604048
Google. (n.d.-b). Google Home - Google Store. Re- Ju, W., & Leifer, L. (2008). The Design of Implicit trieved March 11, 2019, from https://store. Interactions: Making Interactive Systems google.com/product/google_home Less Obnoxious. Design Issues, 24(3), 72–84. https://doi.org/10.1162/desi.2008.24.3.72 Google. (n.d.-c). Teachable Machine. Retrieved January 4, 2020, from https://teachablema- Junkbot Robotics LLC. (2019). Junkbot - Make your chine.withgoogle.com/ own robots. Retrieved January 21, 2020, from https://www.junkbot.co/en Google. (2019). Google Nest Learning Thermostat. Retrieved January 3, 2020, from https://store. google.com/us/product/nest_learning_ther- mostat_3rd_gen?hl=en-US
50 References
KeyShot. (2019). KeyShot | 3D Rendering software LUXROBO Co. (2018). LUXROBO. Retrieved January and Animation software for SolidWorks, 21, 2020, from http://modi.luxrobo.com/ Rhino, Creo, SketchUp and More. Retrieved January 8, 2020, from https://www.keyshot. Microsoft. (2020). Visual Studio Code - Code Ed- com/ iting. Redefined. Retrieved January 8, 2020, from https://code.visualstudio.com/ Koskinen, I., Zimmerman, J., Binder, T., Redstrom, J., & Wensveen, S. (2011). Design research Modrobotics. (2017). Cubelets Robot Construction through practice: From the lab, field, and show- for Kids | Modular Robotics. Retrieved Janu- room. Elsevier. ary 21, 2020, from https://www.modrobotics. com/Cubelets/ Krahmer, E., & Ummelen, N. (2004). Thinking about thinking aloud: A comparison of two Nielsen, J. (1989). Usability Engineering at a Dis- verbal protocols for usability testing. IEEE count. In Proceedings of the Third Internation- Transactions on Professional Communication, al Conference on Human-Computer Interaction 47(2), 105–117. https://doi.org/10.1109/ on Designing and Using Human-Computer TPC.2004.828205 Interfaces and Knowledge Based Systems (2nd Ed.) (pp. 394–401). USA: Elsevier Science Inc. Levy, J. (2015). UX strategy: How to devise innovative digital products that people want. “ O’Reilly Philips. (2019a). Hue Tap switch. Retrieved January Media, Inc.” 3, 2020, from https://www2.meethue.com/ en-us/p/hue-tap-switch/046677473365 Lewis, J. R. (2018). The System Usability Scale: Past, Present, and Future. International Journal of Philips. (2019b). Philips Hue. Retrieved January 4, Human–Computer Interaction, 34(7), 577–590. 2020, from http://www2.meethue.com/nl-nl/ https://doi.org/10.1080/10447318.2018.145 5307 Polyakov, E. V., Mazhanov, M. S., Rolich, A. Y., Voskov, L. S., Kachalova, M. V., & Polyakov, S. Lin, J., Williamson, S., Born, K., & Debarr, A. (2012). V. (2018). Investigation and development of Pattern Recognition in Time Series. https:// the intelligent voice assistant for the Internet doi.org/10.1201/b11822-36 of Things using machine learning. In Mos- cow Workshop on Electronic and Networking Logitech. (2020). Harmony Remotes, Universal Technologies, MWENT 2018 - Proceedings (Vol. Remote Controls | Logitech. Retrieved Janu- 2018-March, pp. 1–5). Institute of Electrical ary 4, 2020, from https://www.logitech.com/ and Electronics Engineers Inc. https://doi. en-us/harmony-universal-remotes org/10.1109/MWENT.2018.8337236
Lucero, A. (2015). Using affinity diagrams to evalu- Processing.org. (n.d.). Retrieved June 20, 2019, ate interactive prototypes. In Lecture Notes in from https://processing.org/ Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes Roman, V. (2019). Unsupervised Machine Learning: in Bioinformatics) (Vol. 9297, pp. 231–248). Clustering Analysis -- Towards Data Science. Springer Verlag. https://doi.org/10.1007/978- Towards Data Science. Retrieved from https:// 3-319-22668-2_19 towardsdatascience.com/unsupervised-ma- chine-learning-clustering-analysis-d40f- Luis. (2018). Dieter Rams 10 Principles of “Good 2b34ae7e Design.” Retrieved January 6, 2020, from https://uxdesign.cc/dieter-rams-10-princi- SAM Labs. (2020). All Products – SAM Labs. Re- ples-of-good-design-applied-to-ux-design- trieved from https://samlabs.com/collec- a4a45daedebb tions/all-products
51 References
Schnelle, D., & Lyardet, F. (2006). Voice user inter- face design patterns. EuroPLoP 2006 - 11th European Conference on Pattern Languages of Programs.
Smith, D. C., Irby, C., Kimball, R., & Verplank, B. (1983). Designing the Star User Interface. Retrieved from https://www.researchgate. net/publication/242448464
Sonos Inc. (2020). One: The Smart Speaker for Music Lovers | Sonos. Retrieved January 4, 2020, from https://www.sonos.com/en/ shop/one.html
StatisticsSolutions. (2015). Data Levels of Measure- ment - Statistics Solutions. Retrieved January 7, 2020, from https://www.statisticssolutions. com/data-levels-of-measurement/
Van Valkenhoef, F. M. B. (2017). Haptic engine.
Weiser, M., & Brown, J. S. (1997). The coming age of calm technology. In Beyond calculation: The next fifty years of computing. Retrieved from https://link.springer.com/content/ pdf/10.1007/978-1-4612-0685-9_6.pdf
Wensveen, S. A. G., Djajadiningrat, J. P., & Over- beeke, C. J. (2004). Interaction frogger. In Pro- ceedings of the 2004 conference on Designing interactive systems processes, practices, meth- ods, and techniques - DIS ’04 (p. 177). New York, New York, USA: ACM Press. https://doi. org/10.1145/1013115.1013140
Yang, R., & Newman, M. W. (2013). Learning from a learning thermostat: Lessons for intel- ligent systems for the home. In UbiComp 2013 - Proceedings of the 2013 ACM Inter- national Joint Conference on Pervasive and Ubiquitous Computing (pp. 93–102). New York, New York, USA: ACM Press. https://doi. org/10.1145/2493432.2493489
52 APPENDICES
A Personal Reflection 54
B Competitor Analysis 56
E Participant Booklet 57
D Ethical Review Form 61
E System Usability Scale results 63
F Device/Feature Comparison 64
G Concept Functionality and Interactions 65
H Wiring Diagram 67
I State Machine Diagram 68
53 Appendices
B - Competitor Analysis
Features AI-Kit 2019) (Anki, Vecotr Anki Teachable Machine (Google,n.d.-c) 2019) Flic (Flic Corp., 2019) (FROLIC, Smartians 2019) LLC., Robotics (Junkbot Junkbot IFTTT (IFTTT, n.d.) IO-key (Autosen, n.d.) 2018) Co., (LUXROBO MODI 2017) (Modrobotics, Cubelets 2020) Labs, (SAM Samslabs Data monitoring 3021?013000 ML based 32300000000 Reinforcement Learning 2 1 1 //////// pricing ? 2 0 2 ? 20323-2 Skill level required 13212113101 Output activation 32031031101 Easy output control 12031021100 Children focus 00100300333 DIY projects 30333331333
56 Appendices
C - Participant Booklet
AI-KIT INTERACTIONS
Expert Panel Study November 2019
Participant Booklet
BART VAN DIJK M2.2 - 0896105 Eindhoven University of Technology
TABLE OF CONTENTS
STUDY INFORMATION 4
INFORMED CONSENT FORM 5
DESIGN CHALLENGE PRESENTATION 7
BOOTSTRAPPING 9
INTERACTION CHEAT SHEET 10
SYSTEM USABILITY SCALE 12
QUESTION TIME 13
CONTACT INFORMATION 14
3
Bart van Dijk Eindhoven University of Technology
57 Appendices | C - Participant Booklet
Consent Form for AI-KIT INTERACTIONS study YOU WILL BE GIVEN A COPY OF THIS INFORMED CONSENT FORM
Please tick the appropriate boxes Yes No Taking part in the study I have read and understood the study information dated [28/10/19], or it has been read to me. I have been able to ask questions about the study and my questions have been answered to my satisfaction. STUDY INFORMATION I consent voluntarily to be a participant in this study and understand that I can refuse to answer questions and I can withdraw from the study at any time, without having to give a reason. I understand that taking part in the study involves [the collection of written notes and voice [28/10/2019] recordings. These recordings can be used for transcription, after which they will be removed.]
In this study, an interface to interact with a democratized Machine Learning system will be Use of the information in the study evaluated and explored. The study aims to obtain insights in the opinion of experts in the field of Smart Homes, Internet of Things (IoT), and Machine Learning systems. I understand that information I provide will be used for [educational purposes. The anonymized information might be included in a graduation thesis that will be reviewed by The study consists of various phases. First, the topic will be introduced in a short various academic]. presentation. Then, using a prototype, several interface (elements) will be evaluated in a I understand that personal information collected about me that can identify me, such as [e.g. usability study. In a third phase, the prototypes will act as a medium to discuss the various my name], will not be shared beyond the study team. interface aspects in a semi-structured interview. The prototypes that you will interact with are safe, and do not include any high voltage components. Consent to be Audio Recorded During the study, voice recordings may be made. Parts of these recordings will be transcribed for data recollection. The data collected in this study (e.g., voice recordings) will I agree to be audio recorded. be pseudonymized. Future use and reuse of the information by others Participation to this study is voluntary. If you wish to withdraw from this study, you are free I give permission for the [voice recordings] that I provide to be archived on [a local hard drive] to say so and terminate the study at any time without any consequences. On request, the so it can be used for future research and learning. data collected in this study will be removed and/or rectified. Removing pseudonymized data is achieved using an anonymization table, stored encrypted on a local hard drive. The I give permission for the [audio transcriptions] that I provide to be archived on [a local hard pseudonymized data collected during this research will be stored locally (on hard drives) at drive] so it can be used for future research and learning. all times. Results of this study can be used published for research purposes.
For any other questions, Please contact the researcher at: [email protected] Signatures
______Name of participant Signature Date
I have accurately read out the information sheet to the potential participant and, to the best of my ability, ensured that the participant understands to what they are freely consenting.
______Researcher name Signature Date
Study contact details for further information: [email protected] 5 4 Bart van Dijk | Eindhoven University of Technology Bart van Dijk Eindhoven University of Technology Bart van Dijk Eindhoven University of Technology
DESIGN CHALLENGE PRESENTATION BOOTSTRAPPING
An interface aimed to facilitate the bootstrapping process is presented. Please use the interface to complete the tasks as described on this page. If you are stuck, feel free to use the Interaction Cheat Sheet on the next page.
Reviewing a Label You just parked your car. Please confirm whether AI-Kit has recognised this label. If the label has not been recognized, add the label car parked.
Figure 1 Descriptive model of AI-Kit. The input and output of the system is expandable to a large (undefined) number of items (sensors and connected devices).
Providing feedback to AI-Kit The interface has detected that you were cooking a couple of hours ago. You just got home from work, so the recognised label is incorrect. Remove the label.
Setting the Scene After you got home, you decided to do some household tasks. Unfortunatly, AI-KIT has not recognised this label. Communicate to AI-KIT that you are currently performing household tasks and that this label should be active. After which the light will turn on and music will start playing. Figure 2 Descriptive model of the different usage phases of AI-Kit for a specific Label. The design challenges for different usage phases are annotated.
Key Design Challenges Setting the Scene (2) Label Initialisation As you start vacuum cleaning, the current label settings are not detailed enough as the Set label name AI-KIT Environment music volume is too low. Change the lable to vacuum cleaning. As a result, the music Set output(s) volume will increase.
Bootstrapping Remove False Positives Data labeling Add False Negatives Direct control
Output Translation Direct Control Preferences Dynamic translation 7 6 Flag Exceptions
Bart van Dijk Eindhoven University of Technology Bart van Dijk Eindhoven University of Technology
58 Appendices | C - Participant Booklet
INTERACTION CHEAT SHEET INTERACTION CHEAT SHEET
General Add False Negatives Newly recognised labels appear on the right. In 12 hours, Press the knob to enter the Label Selector. the recognized labels rotate to left after which they will disappear.
General Interaction Add False Negatives - Label Selector The interaction cursor can be moved by rotating the knob. Single press to select and add the label. Double press to Extra info can appear on the surface below the knob. increase label detail level, after which a new set of labels will appear.
AI-KIT Label Recognition Automatically recognised labels appear on the right. AI-KIT’s certainty of recognition and label-duration can be seen in a glance for each label.
Removing False Positives Remove False Positives by pressing the knob when the cursor is above a label, erasing the recognised label; providing feedback to AI-KIT.
9 8
Bart van Dijk Eindhoven University of Technology Bart van Dijk Eindhoven University of Technology
SYSTEM USABILITY SCALE
Please rate the presented interface using the System Usability Score as presented below. Please keep the execution of the previous tasks in mind.
System Usability Scale
© Digital Equipment Corporation, 1986.
Strongly Strongly disagree agree
1. I think that I would like to use this system frequently
2. I found the system unnecessarily complex
3. I thought the system was easy to use
QUESTION TIME 4. I think that I would need the support of a technical person to be able to use this system
5. I found the various functions in this system were well integrated
6. I thought there was too much inconsistency in this system
7. I would imagine that most people would learn to use this system very quickly
8. I found the system very cumbersome to use
9. I felt very confident using the system
10. I needed to learn a lot of things before I could get going with this system
11 10
Bart van Dijk Eindhoven University of Technology Bart van Dijk Eindhoven University of Technology
59 Appendices | C - Participant Booklet
CONTACT INFORMATION
Researcher - Bart van Dijk
M2.2 - 0896105 Eindhoven University of Technology [email protected]
Coach - Dr. Ir. Joep Frens
Future Everyday Group Eindhoven University of Technology [email protected]
Client - Joep Elderman
Bureau Moeilijke Dingen www.moeilijkedingen.nl [email protected]
12
60 Appendices
D - Ethical Review Form
Ethical Review Form (Version 27.06.2019)
This Ethical Review Form should be completed for every research study that involves human
participants or personally identifiable data and should be submitted before potential participants are
approached to take part in the research study.
Part 1: General Study Information
1 Project title AI-Kit Interactions
2 Researcher B. van Dijk
3 Email researcher [email protected]
4 Supervisor(s) dr. ir. Joep Frens
5 Faculty/department Industrial Design
6 Research location Atlas, 5612 AZ Eindhoven
7 Research period (start/end date) 28 October 2019 / 4 November 2019
8 Funding agency -
9 [If Applicable] Study is part of an educational DFR220 course with code:
10 [If Applicable] Proposal already approved by - external Ethical Review Board: Add name, date of approval, and contact details of the ERB
11 Short description of the research question Exploring key interface features for interacting with a (Machine Learning based) home automation system. 12 Description of the research method Lab Study, consisting of usability testing and semi- structured interviews.
13 Description of the research population, exclusion Experts in the domain of Smart Homes and IoT, consisting criteria of: staff of the faculty Industrial Design (TU/e); Smart Home Developers (company) 14 Description of the measurements and/or Participants will be asked to perform different tasks for stimuli/treatments which they will be required to interact with a designed interface. Optionally, the interface on which the tasks will be performed will be swapped between tasks. The interfaces will be evaluated using the SUS (System Usability Scale). In the next study phase, different aspects (feasibility; value) of the system will be discussed. 15 Number of participants 5
16 Explain why the research is socially important. The study aims to explore the interaction with a Machine What benefits and harm to society may result Learning system, contributing to the overall development from the study? of how we interact with AI.
17 Provide a brief statement of the risks you expect The participants will be interacting with a low-voltage for the participants or others involved in the interface, resulting in no risk while interacting with the research or educational activity and explain. Take prototype. The collected data will be pseudonymized. 1
Ethical Review Form Ethical Review Form
into consideration any personal data you may Important: gather and privacy issues. If you answered all questions with ‘’no’’, you can skip parts 3 - 4 and go directly to part 5. Check which documents you need to enclose and continue with signature and submission.
Part 2: Checklist for Minimal Risk If you answered one or more questions with “yes”, please continue with parts 3 – 5. Yes No 1 Does the study involve participants who are particularly vulnerable or unable to give informed X Part 3: Study Procedures and Sample Size Justification consent? (e.g. children, people with learning difficulties, patients, people receiving counselling, people living in care or nursing homes, people recruited through self-help 1 Elaborate on all boxes answered 11. A low-voltage prototype will be used to evaluate the different groups) with “yes” in part 2. Describe how interfaces. you safeguard any potential risk 2 Are the participants, outside the context of the research, in a dependent or subordinate X for the research participant. position to the investigator (such as own children or own students)? 2 Describe and justify the number As the study is explorative, multiple participants results in a greater 3 Will it be necessary for participants to take part in the study without their knowledge and X of participants you need for this chance of obtaining valuable insights. Moreover, as the participants have consent at the time? (e.g. covert observation of people in non-public places) research or educational activity. different areas of expertise, different aspects will be brought to the Also justify the number of surface. Also, by recruiting multiple participants, the development of the
observations you need, taking interface can be evaluated over time as well. 4 Will the study involve actively deceiving the participants? (e.g. will participants be deliberately X into account the risks and benefits falsely informed, will information be withheld from them or will they be misled in such a way that they are likely to object or show unease when debriefed about the study)
5 Will the study involve discussion or collection of personal data? (e.g. name, address, phone X number, email address, IP address, BSN number, location data) or will the study collect and store videos, pictures, or other identifiable data of human subjects?1. Please check the FAQ’s Part 4: Data and Privacy Statement on the intranet. If yes: please follow the procedure. Make sure you perform a Data Protection Impact Assessment (DPIA) and make a Data Management Plan if necessary and let the data steward check it. 1 Explain whether your data are The data will be pseudonymized form the point of data collection to completely anonymous, or if they processing. The questionnaire forms will be numbered, allocating a
will be de-identified participant number to the test and participant. This number will also be 6 X Will participants be asked to discuss or report sexual experiences, religion, alcohol or drug (pseudonymized or anonymized) used to store the recordings of the interviews. use, or suicidal thoughts, or other topics that are highly personal or intimate? and explain how
7 Will participating in the research be burdensome? (e.g. requiring participants to wear a X 2 Who will have access to the data? The data will be stored on a local hard drive, only accessible with the TU/e device 24/7 for several weeks, to fill in questionnaires for hours, to travel long distances to a password of the researcher. research location, to be interviewed multiple times)? 3 Will you store personal ☒ No information that will allow ☐ Yes, and I declare I will follow the general data protection regulation X 8 May the research procedure cause harm or discomfort to the participant in any way? (e.g. participants to be identified from (GDPR). causing pain or more than mild discomfort, stress, anxiety or by administering drinks, foods, their data? See VSNU draft. drugs) ☒ 4 Will you share de-identified data No (e.g., upon publication in a public ☐ 9 Will blood or other (bio)samples be obtained from participants (e.g. also external imaging of X Yes, and I will inform participants about how their data will be shared, repository)? and ask consent to share their data. I will, to the best of my knowledge the body)? and ability, make sure the data do not contain information that can identify participants. 10 Will financial inducement (other than reasonable expenses and compensation for time) be X offered to participants?
11 Will the experiment involve the use of physical devices that are not ‘CE’ certified? X
2 3
61 Appendices | D - Ethical Review Form
Ethical Review Form
Part 5: Closures and Signatures
1 Enclosures (tick if applicable):
☒ Informed consent form; ☐ Informed consent form for other agencies when the research is conducted at a location (such as a school); ☐ Text used for ads (to find participants); ☐ Text used for debriefings; ☐ Approval other research ethics committee; ☐ Any other information which might be relevant for decision making by ERB; ☐ Data Protection Impact Assessment checked by the privacy officer ☐ Data Management Plan checked by a data steward
2 Signature(s)
Signature(s) of researcher(s) Date:
Signature research supervisor (if applicable) Date:
4
62 Appendices
E - System Usability Scale results
# Question (Expert Panel Study) p1(p) p2 p3 p4 p5 p6 AVE 1 I think that I would like to use this system frequently. 424242 2.8 2 I found the system unnecessarily complex. 242423 3.0 3 I thought the system was easy to use. 424443 3.4 4 I think that I would need the support of a technical person to be able to use this 121112 1.4 5 I found the various functions in this system were well integrated. 434244 3.4 6 I thought there was too much inconsistency in this system. 121221 1.6 7 I would imagine that most people would learn to use this system very quic 435434 3.8 8 I found the system very cumbersome to use. 242222 2.4 9 I felt very confident using the system. 423323 2.6 10 I needed to learn a lot of things before I could get going with this system. 432231 2.2 Total 75 43 80 60 68 68 64
# Question (Evaluation Study - Label Monitor) p1 (p) p2 p3 p5 p7 p8 AVE 1 I think that I would like to use this system frequently. 4 3 4 4 2 3 3.2 2 I found the system unnecessarily complex. 2 2 2 1 1 1 1.4 3 I thought the system was easy to use. 455 4 344.2 4 I think that I would need the support of a technical person to be able to use this 2 2 1 1 2 2 1.6 5 I found the various functions in this system were well integrated. 443 4 343.6 6 I thought there was too much inconsistency in this system. 222 1 232.0 7 I would imagine that most people would learn to use this system very quic 543 3 443.6 8 I found the system very cumbersome to use. 221 1 121.4 9 I felt very confident using the system. 444 4 433.8 10 I needed to learn a lot of things before I could get going with this system. 311 2 221.6 Total 75 78 80 83 70 70 76
# Question (Evaluation Study - Output Control) p1 (p) p2 p3 p5 p7 p8 AVE 1 I think that I would like to use this system frequently. 5 5 3 5 2 3 3.6 2 I found the system unnecessarily complex. 2 1 1 2 1 2 1.4 3 I thought the system was easy to use. 544 4 444.0 4 I think that I would need the support of a technical person to be able to use this 2 1 1 1 2 2 1.4 5 I found the various functions in this system were well integrated. 444 5 434.0 6 I thought there was too much inconsistency in this system. 211 2 221.6 7 I would imagine that most people would learn to use this system very quic 544 4 444.0 8 I found the system very cumbersome to use. 212 1 231.8 9 I felt very confident using the system. 544 4 444.0 10 I needed to learn a lot of things before I could get going with this system. 222 2 322.2 Total 85 88 80 85 70 68 78
63 Appendices
F - Device/Feature Comparison
1 - Device comparison
AI-Kit Product Core-features Usage User The main focus of the web-app is to train the As the web-app is exclusively accessable on a As the focus of this product will be to label data model. The focus of the web-app is to allow computer, the interaction will be focused. On and train the model, a tech-savvy user will be users to label timeframes of data, which is top of that, the user is stationary, and as a the main target. used for the training phase. On top of that, the result it will be hard to set the output online environment can be used to allocate parameters in another room. Web-app output's on label recognition and to set up new sensors.
On the phone app, users are able to quickly Users are able to quickly provide feedback to As the focus off the mobile application is provide feedback to AI-Kit. The interface AI-Kit. On top of that, the mobility of the similar to the web-app, the targeted user is potentially offers a simplified replica of the interface allows people to 'walk around' and similar as well: tech savvy users. While the web-app interface. It allows some, recent, data set the output's to their desired scene. threshold might be a bit lower to interact with to be labeled. On top of that, users are able to AI-Kit (in comparison with a web-app), the user Phone app set their output parameters due to the mobility is still required to have enough interest in the of hte interface. Potentially, the phone app product to provide feedback and/or allocate could also make use of the RFID technology to output states. easliy remap the desired output parameters upon label recognition. Without the sensors, users might be able to The sensors will probably live stationary in the As primarely the tech-savvy users will be explore the capabilities of AI-Kit with the use of home. While sensors can be adjusted or occupied with the design and development of the sensors integrated in their phones. If their moved entirely (e.g. when allocated to a new new labels, they will be the user that installs interest is triggered, additional sensors can be label), they are expected to remain in the same and interacts (e.g. setup) with the sensors. bought to extend AI-Kit's capabilities. While the position for a while as required for the trianing Sensors main feature of the sensor's is to collect data, phase prior to it's deployment. limited label monitoring could be integrated on the sensors as well (e.g. using color-coded LED's).
The monitor provides a simplified interface of The device is expected to live semi-stationary As the threshold to review the state of AI-Kit, the web-app. It allows a user to quickly review in the home. As a label in training will require other users are expected to use this inteface as the state of AI-Kit, and why a certain output feedback for a while, the monitorring interface well. Not only to use the labels to achieve a (parameter set) might be activated. It offers is expected to be placed at a relevant location. certain scenery, but also to help the tech-savvy simplified control, allowing users to provide The position should allow users to quickly in the development of labels. Label Monitor quick feedback on false positives and review the monitor and, if necessary, provide negatives. Moreover, the interface allows feedback. exceptions to be flagged. This inteface allows the functionality of AI-Kit to be integrated in the daily lives of users. The trigger allows output parameters to be The trigger is a mobile device. Living as a As the people that live with a tech-savvy family quickly set to a desired setting. The set remote in the home envirionmnet, users are member will be output-oriented, they will find parameters can be saved to a label. This allows able to set their devices to the desired state. the most value in the trigger. It allows them to users to simply explore the capabilities of The trigger is aimed to be used exclusively to directly interact towards their goal and value: Output Control connected devices without the need of an app set the desired output states for certian labels. the states of outputs. Potentially, if this device Trigger or the remembrance of output ID's. The As a result, the trigger is not used on a daily might be capable of facilitating the input for trigger can also be used to set up initial output basis but only when a label's output setting Unsupervised Machine Learning, it will become parameters for labels. should be changed. This approach would be interesting for the tech-savvy user as well. very suitable for the Unsupervised Machine Learning functionality, if this becomes an area
2 - Feature comparison
Group Feature Web-app app Phone Label Monitor Trigger Output Docked trigger boxes Sensor Label Monitor Change labels (correct/incorrect) ++ + + - + - Label Monitor Change label timeframe ++ + ---- Label Monitor Add label detail through layers ++ ++ + - + - Label Monitor Review Contextual datastream ++ + ---- Output control Change output's (rule based) + + - + + - Output control Direct output control - - - + + - Other features Flag exceptions -- - - + + + Other features Continuously monitor AI-Kit operations - + + - + - Other features physical presence (peripheral monitoring) - - + - + + Cognitive resources Amount of cognitive resources required --- -- + ++ + ++
64 Appendices
G - Concept Functionality and Interactions
1 - Concept features
Label Monitor mode Output Control mode
Feature Functionality Designed interaction Feature Functionality Designed interaction
The RFID reader can be used to allocate By providing a continuous overview of the In the idle state of the interface, a linear RFID-ID’s to output’s in the (web-)app (for A un-allocated RFID tag can be scanned after Monitor AI-Kit’s detected labels, the cause of output timeline is displayed. Labels are color-coded Setup of output’s phones without an integrated RFID reader). which it will be prompted in the web-app to state behavior can be determined. and move from right to left. This approach allows users to ‘play’ the allow for further allocation. labels and set the scene accordingly.
An RFID tag, placed at a (user selected) A detected label is selected by sliding a An RFID tag (allocated to an output) can be convenient location can be scanned by On the label monitor, a label can be selected Situated output Selecting label sliding knob underneath the linear interface scanned, after which the interface is tapping the front of the device against the for further interactions. control (timeline). connected to the designated output. tag. Feedback is provided on the screen, as the name of the scanned output is shown.
The slider can be just to adjust an output The interface is in direct connection (over Wi- parameter. The output parameter’s name is After a label has been selected, additional Fi) with the IoT output. A default parameter Review detected More information upon a recognized label displayed on the screen. On the location of information (detection time, certainty, menu) Output control is selected as parameter to interact with. labels can be viewed. the time-line interface (in monitor mode), will be displayed on the screen. Users can adjust the parameter’s value after feedforward is provided to indicate the which it saved to a label. changes made with the slider.
A ‘remove’ button will be displayed on the The capacitive buttons can be used to cycle screen after a label has been selected. Users can cycle through a set of parameters Remove False Remove labels that have been falsely Output control – through different parameters. After a button Capacitive buttons allow users to select the that are associated with an output (e.g. Hue, Positive labels recognized by AI kit. other parameters is pressed, the screen text and feedforward button. A confirmation window verifies the Saturation, and Brightness). (next to the slider) updates. action.
An ‘add’ button is visible on the screen. The button can be selected using a capacitive The group name is displayed on the screen. RFID-tags can be allocated to groups of Add False Add labels that have not been recognized by button. The slider can be used to select a Grouped output Other than that, the interaction is equal to output to quickly change output parameters Negative labels AI-Kit. color-coded label in the label selector, which control ‘output control’ and ‘output control – other of multiple outputs at once. can be confirmed with one of the capacitive parameters’. buttons.
When a label has been selected, a ‘flag’ Exception The outputs can also be set to the desired A capacitive button can be pressed to flag an Labels can be flagged so they can be Flag labels option appears in the menu which can be flagging of state without being allocated to a certain exception. Feedforward is provided on the reviewed later (e.g. in the web-app). selected using a capacitive button. output control label. screen.
Labels are distributed over several layers of Layered detail, providing users with more control as In the label selector, users can increase the The haptic feedback matches the different A motorized slide potentiometer facilitates approach of label Haptic feedback they take more time to interact with the label detail using a capacitive button. controllable output types. haptic feedback. selection device.
Labels can be added as exception, this Labels can be added and removed as The knob will automatically move to the The motorized slider will automatically move Exception requires an additional capacitive button to Automatic output exception, allowing users to quickly set the position that reflects the current state of the to the correct position based on the selected flagging of labels be pressed after the desired label has been state reflection scene to the desired state. output parameter. output and output parameter. added.
The certainty of label detection of AI-Kit is The label detection certainty is reflected in The set output parameters can be saved to a A capacitive button can be used to confirm Monitor label- shown, allowing users to monitor the the timeline as the labels vary in height (0- Save to label label. If no label has been selected, the user the adjustment of output parameters for the certainty growth growth of labels over time. 100%). will be asked to select one. selected label.
Monitor label The duration of a detected label can be The thickness of recognized labels on the duration reviewed by a user. timeline reflect their duration.
Confirm the detection of labels to increase A selected label can be confirmed using a Confirm labels the accuracy of the model. capacitive button and a menu-item.
A selected label’s output parameters can be Users can edit the set output parameters for adjusted using a capacitive button and a Edit labels a label. menu-item, after which the Output Control interface will appear.
65 Appendices | G - Concept Functionality and Interactions
2 - Miscellaneous features
Miscellaneous
Feature Functionality Designed interaction
The RFID reader can be used to allocate RFID-ID’s to output’s in the (web-)app (for A un-allocated RFID tag can be scanned after Setup of phones without an integrated RFID reader). which it will be prompted in the web-app to output’s This approach allows users to ‘play’ the allow for further allocation. labels and set the scene accordingly.
An RFID tag, placed at a (user selected) Situated An RFID tag (allocated to an output) can be convenient location can be scanned by output scanned, after which the interface is tapping the front of the device against the control connected to the designated output. tag. Feedback is provided on the screen, as the name of the scanned output is shown.
66 Appendices
H - Wiring Diagram
67 Appendices
I - State Machine Diagram
Boot Start
MPR121 found Failed Connect to PN532 Wi-Fi error
PN532 found
Connect to Wi‐Fi
Wi-Fi connected
Failed Error receiving Hue states
Boot completed
1 (Page 2)
(Pa
0 “Confirm” Monitor Labels
“Cancel” User slides on label “Remove” “Add” Confirm? Label Selected
“Flag” “Confirm” “Edit”
Remove Label Flag Label Confirm Label Edit Label Add Label
1 1 Show selector Undo flag 0 0 Wait for reed Increase detail Select Label Flag
“Cancel”
Timeout Confirm?
“Confirm”
68 Appendices | I - State Machine Diagram
0 Show selector
1 Waiting for RFID Select Label Increase detail
“Cancel”
“Confirm” Confirm? 0
Connected to Error receiving output state 1
Obtained ID
Got output state
(Page 1) 0 1 “Cancel”
Show state Canceled edit
User moves slider App prompt Updated slider Prev Feature Next Feature
“<<” “>>” “Save” “X” Label updated Update label? Confirm cancel? “Cancel” “Confirm” “Exception”
Exception
”
69 Bart van Dijk Final Master Project January 2020
Eindhoven University of Technology Department of Industrial Design
Mentor Dr. Ir. J.W. Frens
Client J. Elderman Bureau Moeilijke Dingen