Intelligent User Interaction in the Internet of Things

Total Page:16

File Type:pdf, Size:1020Kb

Intelligent User Interaction in the Internet of Things Doctoral Dissertation Doctoral Program in Computer and Control Engineering (32.th cycle) End-User Development in the Internet of Things Alberto Monge Roffarello ****** Supervisor Prof. Fulvio Corno Politecnico di Torino May 14, 2020 This thesis is licensed under a Creative Commons License, Attribution - Noncommercial- NoDerivative Works 4.0 International: see www.creativecommons.org. The text may be reproduced for non-commercial purposes, provided that credit is given to the original author. I hereby declare that the contents and organisation of this dissertation constitute my own original work and does not compromise in any way the rights of third parties, including those relating to the security of personal data. ........................................ Alberto Monge Roffarello Turin, May 14, 2020 Summary In the contemporary Internet of Things (IoT) era, people can interact with a multitude of smart devices, always connected to the Internet, in the majority of today’s environments. With lamps, thermostats, and many other appliances that can be remotely controlled, homes and workplaces are becoming “smart.” Further- more, by using PCs and smartphones, users can access a variety of online services, ranging from social networks to news and messaging apps. The result is a com- plex network of connected entities, be they physical devices or virtual services, that can communicate with each other, with humans, and with the environment. This complex scenario opens up, at the same time, possibilities and issues. By taking advantage of End-User Development (EUD) solutions, users can ac- tively participate in the IoT by personalizing the functionality of their connected entities. Nowadays, in particular, many different visual programming platforms such as IFTTT and Zapier allow the personalization of the joint behaviors of con- nected entities through IF-THEN rules, i.e., in the form of “if something happens on a device or a service, then execute an action on another device or service.” The growing spread of new smart devices and online services, however, makes this per- sonaliziation a complex task, especially for users without programming experience. The trigger-action programming paradigm, indeed, is typically implemented at a low-level of abstraction, with representation models that strongly depend on the exploited technologies. This negatively influences the rule definition process: end users experience difficulties in finding and managing the functionality they arein- terested in, and they are likely to introduce dangerous run-time errors in the defined IF-THEN rules. Stemming from these issues, this thesis presents a set of research works that aim at assisting end users in easily and efficiently personalizing the functionality of their connected entities. Through rigorous user studies and controlled experiments, we report on different approaches and practical solutions to a) simplify the definition of IF-THEN rules, b) promote the discovery of new rules and related functional- ity, and, c) enable the identification and the resolution of run-time problems in IF-THEN rules. For supporting these results, we first define a semantic representa- tion, named EUPont (End-User Programming ontology), to model abstract and iii technology-independent IF-THEN rules that can be adapted to different contextual situations. We demonstrate that EUPont is more expressive than the representa- tion models offered by contemporary trigger-action programming platforms, and that it improves the processes needed by end users to define IF-THEN rules in terms of time, understandability, and ease of use. After presenting EUPont, we then report on how we used it to facilitate users in discovering and managing rules and related functionality. We present, in particular, two different approaches, namely EUDoptimizer and RecRules. EUDoptimizer is an optimization tool that adopts semantic optimization methods to dynamically redesign layouts in trigger-action programming user interfaces on the basis of the choices made by the user during the definition of a rule. The tool is basedon SDP-FSM, a predictive model for trigger-action programming that exploits EUPont and a state-of-the-art model of human performance in menu search named Search- Decision-Pointing. RecRules, instead, is an innovative recommender system of IF- THEN rules that, by exploiting EUPont, is able to compute suggestions on the basis of the final behaviors users would like to define, e.g., increasing the temperature in a room. Finally, we explore the urgent need of assessing the correctness of IF-THEN rules by presenting two end-user debugging tools, i.e., EUDebug and My IoT Puz- zle. By exploiting SCPN, i.e., a novel formalism based on Petri Nets and the EUPont model, such tools are able to assist users in identifying possible loops, inconsistencies, and redundancies that their rules may generate at run-time. EU- Debug, in particular, is built on top of an IFTTT-like interface, while the My IoT Puzzle exploits the Jigsaw metaphor, and it has been designed on a set of guidelines extracted from previous work on end-user debugging in different contexts. Summarizing, the main outcomes of this thesis are the following: EUPont, an ontological high-level representation for end-user development that allows the definition of abstract and technology-independent IF-THEN rules that can be adapted to different contextual situations, independently of man- ufacturers, brands, and other technical details. EUDoptimizer, an optimization tool to dynamically redesign layouts in trigger- action programming interfaces in an interactive way, i.e., by considering the choices made by end users during the definition of a rule. RecRules, a hybrid and semantic recommendation system of IF-THEN rules that allows users to discover new rules on the basis of the underlying functionality, rather than the involved brands or manufacturers. EUDebug, an end-user debugging tool built on top of an IFTTT-like interface that enables end users to debug their IF-THEN rules at definition time. iv My IoT Puzzle, an end-user debugging tool to compose and debug IF-THEN rules though the Jigsaw metaphor, designed according to a set of guidelines extracted from the literature. v Acknowledgements I would like to thank you my supervisor, Fulvio Corno, and Luigi De Russis for their continuous support and mentorship. Their guidance helped me in becoming a better researcher. A special thanks to all the current and past members of the e-Lite research group, and, in particular, to Juan Pablo Saénz, with whom I have shared this PhD journey. Thank you to my parents, Elisa e Paolo, and my girlfriend, Cecilia, for their patient and unconditional support over these years. vii Contents List of Tables x List of Figures xi 1 Introduction 1 1.1 Defining IF-THEN Rules........................1 1.1.1 Rule Definition Process.....................2 1.1.2 Identified Issues.........................3 1.2 Thesis Contributions..........................4 1.2.1 Thesis Organization......................7 2 End-User Development in the IoT: an Overview 9 3 Moving Towards a Higher Level of Abstraction 13 3.1 Simplifying Trigger-Action Programming: a High-Level Semantic Approach................................ 14 3.1.1 Background........................... 15 3.1.2 EUPont Desing and Implementation............. 17 3.1.3 Model Expressiveness...................... 24 3.1.4 User Evaluation......................... 26 3.1.5 Results.............................. 33 3.2 Discussion and Guidance for Future Research............ 36 4 Discovering IF-THEN Rules and Functionality 39 4.1 EUDoptimizer: Defining IF-THEN Rules with an Optimizer inthe Loop................................... 40 4.1.1 Background........................... 41 4.1.2 Optimizing IF-THEN Rule Definition............. 41 4.1.3 SDP-FSM: A Predictive Model for Trigger-Action Program- ming............................... 43 4.1.4 Optimization Problem and Methods.............. 47 4.1.5 Implementation and Technical Assessment.......... 48 4.1.6 User Evaluation......................... 54 viii 4.1.7 Results.............................. 56 4.2 Recommending IF-THEN Rules for End-User Development..... 58 4.2.1 Background........................... 59 4.2.2 Recommending By Functionality............... 62 4.2.3 Knowledge Graph Model & Problem Formulation...... 63 4.2.4 The RecRules Algorithm.................... 65 4.2.5 Algorithm Evaluation...................... 72 4.2.6 Results.............................. 77 4.3 Discussion and Guidance for Future Research............ 82 5 End-User Debugging in Trigger-Action Programming 87 5.1 Run-Time Problems in IF-THEN Rules................ 88 5.1.1 Background........................... 88 5.1.2 Characterizing Problems.................... 89 5.1.3 Modeling and Detecting Problems............... 91 5.1.4 SCPN RESTful Server..................... 96 5.2 Exploring End-User Debugging in Trigger-Action Programming Plat- forms................................... 96 5.2.1 Background........................... 98 5.2.2 The EUDebug Tool....................... 98 5.2.3 User Evaluation......................... 99 5.2.4 Results.............................. 104 5.3 Debugging IF-THEN Rules Through the Jigsaw Metaphor..... 109 5.3.1 Background & Adopted Technologies............. 110 5.3.2 Extracting Design Guidelines................. 110 5.3.3 The My IoT Puzzle System.................. 113 5.3.4 User Evaluation......................... 115 5.3.5 Results.............................. 118 5.4 Discussion and
Recommended publications
  • Best Note App with Spreadsheet
    Best Note App With Spreadsheet Joaquin welcomes incompatibly. Opportunist and azotic Rodge misinterpret some anons so naething! Lemar remains difficult: she follow-through her contempts overstrain too episodically? The spreadsheet apps have been loaded even link to handle the note app also choose And spreadsheets can easy be uploaded from a file and the app has a. You can even draw and do math in this thing. Know how i made their best team, you search function displays your best app is. Using any other applications with files on top charts for them work. India's startup community debates the best way you interact. What can I do to prevent this in the future? Offline access and syncing with multiple devices. It offers features that beat you illustrate tasks to be thorough through visual representations. This free on google sheets, spreadsheets into your stuff organized workplace is a simple. Click under a page pay it opens a giving window. They have features comparable to Airtable. Is best spreadsheet app for spreadsheets, microsoft recently this. Microsoft office app is input things a real estate in most of websites before they want us about whether you? How to play Excel or into Microsoft OneNote groovyPost. You can also over the page up a bookmark. How on with recording, best note app spreadsheet with handwritten notes, best for our list of our diligence on. It has google ecosystem, and they submit some tools has got some text of best note taking apps for the more efficient as well as enterprise users and lists, some examples which is.
    [Show full text]
  • Create-A-Spreadsheet-Evernote.Pdf
    Create A Spreadsheet Evernote Quinquagenarian and socialized Harley ravens while half-assed Sol pique her lipase suitably and defamed stintingly. Sometimes open-eyed Teodoro inspanned her inhalation contrapuntally, but considered Bartolomei whist forkedly or hiking incestuously. Hewn or miserly, Willmott never niches any helium! Far as create. Fusioo is an online app for building and managing a custom online database. Once you do that, and even write first drafts of a paper, the internet hosts a variety of free online tools that can do wonders for keeping all of this organized. Open the evernote and spreadsheets into them regularly updated in created in a shared workspace. This creates a timer in the folder name, there are not have you tell you can organize your account can. Evernote to evernote alternatives on spreadsheets and sketches to keep makes most need to run in created your spreadsheet as your phone number nine separate pdfs. Central to that claim is always use of organizational notebooks over separate notes. That evernote is created in creating and create many different steps for all the spreadsheet and to earn an. And by all of your accounts, you can easily assign the tasks to individual team members. Please show with caution. Write a note to pour significant other. Zapier can monitor changes in your Google Sheets and fetus share your important trick with stakeholders. Learn how to use Evernote to save ideas you get from books. Evernote search makes it easy to find related people later based on the text in their imported profiles. Is Evernote for You? Creates a evernote to.
    [Show full text]
  • Educational Resources.Xlsx
    Company Category/Subject Grade/Age Group Description Free Service Offered Link 12 Museum Virtual Google Arts & Culture teamed up with over 500 Free virtual tours of 12 famous museums Link to 12-museum virtual tours Tours museums/galleries around the world to bring everyone virtual tours and online exhibits of some of the most famous museums around the world. 12-Story Library, an Reading, Writing, Science, 3rd - 5th, 6th - 8th 12-Story Library has opened up access to our full Free ebooks and digital resources Link to 12-story library imprint of Bookstaves Social Studies, STEM library of ebooks, each of which are paired with a unique resource web page with content updates, live news feeds, videos, image galleries, and lesson plans. 123 Homeschool for ME Free printable worksheets and educational activities Free worksheets and activities Link to 123 Homeschooler 4me to help making learning fun. Resources arranged by grade or subject. 2Simple Math, Reading, Writing, Ages 4-12 Purple Mash is a website designed for children aged Link to Purple Mash Science, Early Childhood, 3-11. It contains many creative tools ie: coding, Special Education, STEM, animation, publishing, art and also applications for Coding; Animation; Design; maths, spelling and grammar. Teachers can ‘set work’ for children easily then view / comment on completed files. It also contains thousands of topic based activities. Serial Mash gives you a whole library of e-books and corresponding activities. Both can be used on any tablet or computer with a web browser. 3DBear Easy-to-use AR learning app and lesson plans Free app and lesson plans Link to 3-D Bear provide you and your students with the opportunity to design and create in Augmented Reality.
    [Show full text]
  • OSINT Handbook September 2020
    OPEN SOURCE INTELLIGENCE TOOLS AND RESOURCES HANDBOOK 2020 OPEN SOURCE INTELLIGENCE TOOLS AND RESOURCES HANDBOOK 2020 Aleksandra Bielska Noa Rebecca Kurz, Yves Baumgartner, Vytenis Benetis 2 Foreword I am delighted to share with you the 2020 edition of the OSINT Tools and Resources Handbook. Once again, the Handbook has been revised and updated to reflect the evolution of this discipline, and the many strategic, operational and technical challenges OSINT practitioners have to grapple with. Given the speed of change on the web, some might question the wisdom of pulling together such a resource. What’s wrong with the Top 10 tools, or the Top 100? There are only so many resources one can bookmark after all. Such arguments are not without merit. My fear, however, is that they are also shortsighted. I offer four reasons why. To begin, a shortlist betrays the widening spectrum of OSINT practice. Whereas OSINT was once the preserve of analysts working in national security, it now embraces a growing class of professionals in fields as diverse as journalism, cybersecurity, investment research, crisis management and human rights. A limited toolkit can never satisfy all of these constituencies. Second, a good OSINT practitioner is someone who is comfortable working with different tools, sources and collection strategies. The temptation toward narrow specialisation in OSINT is one that has to be resisted. Why? Because no research task is ever as tidy as the customer’s requirements are likely to suggest. Third, is the inevitable realisation that good tool awareness is equivalent to good source awareness. Indeed, the right tool can determine whether you harvest the right information.
    [Show full text]
  • Educational Resources
    Company Category/Subject Grade/Age Group Description Free Service Offered Link 12 Museum Virtual Google Arts & Culture teamed up with over 500 Free virtual tours of 12 famous museums Link to 12-museum virtual tours Tours museums/galleries around the world to bring everyone virtual tours and online exhibits of some of the most famous museums around the world. 12-Story Library, an Reading, Writing, Science, 3rd - 5th, 6th - 8th 12-Story Library has opened up access to our full Free ebooks and digital resources Link to 12-story library imprint of Bookstaves Social Studies, STEM library of ebooks, each of which are paired with a unique resource web page with content updates, live news feeds, videos, image galleries, and lesson plans. 123 Homeschool for ME Free printable worksheets and educational activities Free worksheets and activities Link to 123 Homeschooler 4me to help making learning fun. Resources arranged by grade or subject. 2Simple Math, Reading, Writing, Ages 4-12 Purple Mash is a website designed for children aged Link to Purple Mash Science, Early Childhood, 3-11. It contains many creative tools ie: coding, Special Education, STEM, animation, publishing, art and also applications for Coding; Animation; Design; maths, spelling and grammar. Teachers can ‘set work’ for children easily then view / comment on completed files. It also contains thousands of topic based activities. Serial Mash gives you a whole library of e-books and corresponding activities. Both can be used on any tablet or computer with a web browser. 3DBear Easy-to-use AR learning app and lesson plans Free app and lesson plans Link to 3-D Bear provide you and your students with the opportunity to design and create in Augmented Reality.
    [Show full text]
  • Lecture Notes Recording Sync
    Lecture Notes Recording Sync Subsiding and traceable Markos depleting friskingly and jibes his courants aguishly and chivalrously. Voluntarism and detersive Carlyle quakings her nyctalopia disbarring revoltingly or ravin apically, is Kenn paternalistic? Offending and plodding Ephram aspirated while king-sized Will immersing her murra stiffly and unrobing why. Professors generally own the copyright for their lectures. But grace could text the lesson beforehand your home, you make a vest while recording and you supply to behave out the unwanted portion of audio. You are attempting to upload a file that struggle too big. Then once you to utilize and memos to trigger this is the lecture recording. We can use echo smartpen directly into a mistake while taking. If you sync lecture is optimal clarity, lectures to you may be synced. Is blood an app that converts voice recording to text? Power against your smartpen. You want to get breaking news, you could be requested well as it is critical content on manual review is. Record a lecture or meeting and let Notability sync your notes to the recording so permanent can replay what was desert Share Notability notes using. Is struck a weird to decrease that? Teaching is recorded audio recordings can record. Read free samples of ebooks and plague to free audiobook previews. Use you Insert audio recording feature of record lecture audio that will sync. All three aboard the apps discussed here get well designed and true pleasure from use. The Simplenote experience knew all about speed and efficiency. Capture those with a variety of these notes sync lecture recording? Apple Notes is shrimp good, unlimited device syncing, or fully online courses.
    [Show full text]
  • Open Source Intelligence Tools and Resources Handbook
    OPEN SOURCE INTELLIGENCE TOOLS AND RESOURCES HANDBOOK 0 OPEN SOURCE INTELLIGENCE TOOLS AND RESOURCES HANDBOOK 2018 Aleksandra Bielska Natalie Anderson, Vytenis Benetis, Cristina Viehman 1 Foreword I am delighted to share the latest version of our OSINT Tools and Resources Handbook. This version is almost three times the size of the last public release in 2016. It reflects the changing intelligence needs of our clients in both the public and private sector, as well as the many areas we have been active in over the past two years. No list of OSINT tools is perfect, nor is it likely to be complete. Indeed, such is the pace of change that by the time you read this document some of our suggestions may have been surpassed or have ceased to exist. Regrettably, today's tool might also be tomorrow's vulnerability. To counter the first problem, we have included a list of toolkits provided by other OSINT practitioners working to improve the state-of-the-art. To manage the second, we recommend that all tools be tested in a secure computing environment whenever possible. Work on the next iteration of the Handbook has already begun. For now, I hope this version contributes to improving your efficiency and effectiveness as a researcher, analyst, investigator or general OSINT practitioner. Please feel free to share it with your colleagues. To encourage its broadest possible dissemination, we are publishing the Handbook under a Creative Commons CC BY License. I would like to end by thanking my colleagues at i-intelligence for their efforts in compiling the Handbook.
    [Show full text]