ONTOLOGIES FOR INTERACTION gerrit niezen Enabling serendipitous interoperability in smart environments This document was typeset using LATEX and the typographical look-and-feel classicthesis developed by André Miede. The style was inspired by Robert Bringhurst’s seminal book on typography “The Elements of Typographic Style”. Cover design by Paul Verspaget, Verspaget & Bruinink Printed by Eindhoven University Press “iPhone”, “Alarm Clock”, “Computer” and “Wireless” symbols by The Noun Project, from The Noun Project collection. “A graphic language for touch”, Timo Arnall. “Desk Lamp” symbol by Edward Boatman, “Ra- dio” symbol by National Park Service, “User” symbol by Denis Chenu, “Music” symbol by Ryan Oksenhorn, from The Noun Project collection. Gerrit Niezen: Ontologies for interaction, Enabling serendipitous interoperability in smart environments, © June 2012 A catalogue record is available from the Eindhoven University of Technology Library. ISBN: 978-90-386-3234-6 Proefontwerp Technische Universiteit Eindhoven Trefw.: Ontologies, Semantic Web, Software Architecture, Inter- action Design Ontologies for interaction Enabling serendipitous interoperability in smart environments PROEFONTWERP ter verkrijging van de graad van doctor aan de Technische Universiteit Eindhoven, op gezag van de rector magnificus, prof.dr.ir. C.J. van Duijn, voor een commissie aangewezen door het College voor Promoties in het openbaar te verdedigen op dinsdag 9 oktober 2012 om 16.00 uur door Gerrit Niezen geboren te Pretoria, Zuid-Afrika De documentatie van het proefontwerp is goedgekeurd door de promotoren: prof.dr.ir. L.M.G. Feijs en prof.dr. P. Markopoulos Copromotor: dr. J. Hu PDEng MEng CONTENTS iframingtheproblemandcurrentstate-of-the-art 1 1introduction 3 1.1 Background 4 1.1.1 Multi-device user interaction 4 1.1.2 Configuring connections between devices 5 1.2 Context of the work and research questions 6 1.2.1 The SOFIA project 7 1.2.2 Ubiquitous computing 7 1.2.3 Affordances 8 1.2.4 Ontologies 9 1.2.5 Research questions 10 1.3 Methodology 11 1.4 Outline of the thesis 13 2relatedwork 15 2.1 Related projects and frameworks 15 2.1.1 SpeakEasy (circa 2000-2003) 16 2.1.2 EventHeap (circa 2000-2005) 17 2.1.3 The XWeb architecture (circa 2001-2003) 18 2.1.4 AutoHAN (circa 2001) 19 2.1.5 e-Gadgets (circa 2004) 22 2.2 Ubicomp ontologies 23 2.2.1 SOUPA (circa 2004) 24 2.2.2 BDI and the MoGATU ontology 24 2.2.3 Gaia (circa 2004-2007) 25 2.2.4 CAMUS (circa 2004-2005) 26 2.3 User interface software architectures 26 2.3.1 Ullmer & Ishii’s MCRpd model 27 2.4 Modelling input devices 31 2.4.1 Foley’s taxonomy and its extensions 31 2.5 Semantic models 33 2.5.1 The Frogger framework 33 2.5.2 Models of intentionality 34 2.6 Outlook 36 ii design iterations and constructing a theory 37 3designiterationi 39 v vi contents 3.1 Requirements 39 3.2 Ontology Design 41 3.3 Device design 45 3.3.1 Interaction Tile 46 3.3.2 Lamp 48 3.3.3 Mobile phones 48 3.3.4 RFID reader used in interaction tile 48 3.4 Implementation 49 3.4.1 Interaction Tile KP 50 3.4.2 Music Player KP 52 3.4.3 Light KP 52 3.4.4 SIB 53 3.5 Discussion & Conclusion 54 4designiterationii 57 4.1 Requirements 57 4.2 Ontology Design 59 4.2.1 Semantic Media ontology 59 4.2.2 Semantic Interaction ontology 60 4.3 Device Design 62 4.3.1 Wall-wash lighting and presence sensors 62 4.3.2 Connector object 63 4.3.3 Spotlight Navigation 65 4.3.4 Lighting Device 68 4.4 Implementation 69 4.4.1 ADK-SIB 70 4.4.2 Semantic matching of media types 72 4.4.3 Device states 74 4.5 Evaluation 75 4.6 Discussion & Conclusion 76 5designiterationiii 79 5.1 Requirements 79 5.2 Ontology design 81 5.3 Device modifications 82 5.3.1 Squeezebox radio 82 5.3.2 Android mobile devices 83 5.3.3 Wakeup experience service 85 5.3.4 Zeo 86 5.4 Implementation 88 5.4.1 Feedback and Feedforward 88 5.5 Discussion & Conclusion 94 6semanticconnectionsframework 97 6.1 User interaction model 97 6.2 Smart Objects 98 contents vii 6.2.1 Identification 99 6.2.2 Interaction primitives 99 6.3 Semantic Connections 101 6.3.1 Directionality 102 6.3.2 Transitivity 102 6.3.3 Permanent and temporary connections 103 6.3.4 Connections connect different entities 103 6.4 Semantic Transformers 103 6.5 Finite state machine examples 104 6.6 Feedback and feedforward 109 6.6.1 Feedback of objects 109 6.6.2 Feedback of connections 110 6.6.3 Feedforward 110 6.7 Discussion & Conclusion 112 iii generalised models, software architecture and evaluation 115 7devicecapabilitymodelling 117 7.1 GUI-based techniques 117 7.2 Non-GUI techniques 119 7.2.1 UAProf 119 7.2.2 Universal Plug and Play (UPnP) 119 7.2.3 SPICE DCS 121 7.3 Registering devices on startup 121 7.3.1 Identifying devices 122 7.3.2 Registering a device’s functionality 125 7.4 Reasoning with device capabilities 125 7.4.1 Representing functionalities as predicates 126 8eventmodelling 131 8.1 Related work 132 8.1.1 The Event Ontology 133 8.1.2 DUL 133 8.1.3 Event-Model-F 133 8.1.4 Linked Open Descriptions of Events (LODE) 134 8.1.5 Ontologies for temporal reasoning 134 8.2 Interaction events 135 8.2.1 System events 137 8.2.2 Feedback 137 8.2.3 Discussion & Conclusion 138 9ontologyengineering 141 9.1 Layers of ontologies 143 9.1.1 Foundational ontologies 143 9.1.2 Core ontologies 143 viii contents 9.1.3 Domain ontologies 143 9.1.4 Application ontologies 144 9.2 Our approach 144 9.3 Reasoning with OWL 144 9.3.1 Consistency checking 146 9.3.2 Necessary versus necessary and sufficient 146 9.3.3 Inverse properties 147 9.3.4 Property chains 147 9.3.5 Using cardinality restrictions 147 9.4 Reasoning with SPIN 148 9.4.1 Integrity constraints 148 9.4.2 SPARQL Rules 149 9.4.3 Built-in SPARQL Functions 149 9.4.4 Custom functions 150 9.4.5 Magic properties 151 9.5 Ontology design patterns 151 9.5.1 The Role pattern 152 9.5.2 Descriptions and Situations (DnS) pattern 153 9.5.3 Defining n-ary relations 155 9.5.4 Naming interaction events 156 9.5.5 Using local reflexivity in property chains 157 9.5.6 Semantic matching with property chains 158 9.5.7 Inferring new individuals 160 9.5.8 Removing inferred triples 162 9.5.9 Inferring subclass relationships using prop- erties 162 9.5.10 Inferring connections between smart ob- jects and semantic transformers 164 9.6 Discussion 166 10 software architecture 169 10.1 Characteristics of ubicomp middleware 169 10.2 Publish/subscribe paradigm and the blackboard pattern 170 10.3 Smart Space Access Protocol (SSAP) 172 10.4 Smart-M3 architecture 173 10.5 ADK-SIB 173 11 evaluation 175 11.1 Evaluating the system performance 175 11.1.1 Introduction 175 11.1.2 Experimental setup 176 11.1.3 Experimental Results 180 11.1.4 Discussion 185 11.2 Evaluating the ontology 187 contents ix 11.2.1 Introduction 187 11.2.2 Validating the work using Cognitive Di- mensions 188 11.2.3 Method 190 11.2.4 Results 190 11.2.5 Non-CD related questions 195 11.2.6 Discussion & Conclusion 197 12 conclusion 199 12.1 Achievements and observations 199 12.2 Providing affordances and feedback for smart ob- jects 201 12.3 Software architecture 201 12.4 Ontologies 202 12.5 Low cost, high tech 203 12.6 Future work 204 iv appendix 207 bibliography 221 LIST OF FIGURES Figure 1 Iterative design methodology 12 Figure 2 The research approach used in the the- sis 12 Figure 3 The MCRpd model of Ullmer & Ishii for Tangible User Interfaces (TUIs) 27 Figure 4 The continuum of intentionality 34 Figure 5 Ontology indicating subclass relationships 41 Figure 6 Individuals that were instantiated based on the ontology 44 Figure 7 The interaction tile and mobile phone 46 Figure 8 A laser-cut version of the interaction tile prototype 48 Figure 9 The interaction and cubes, with the lamp in the background 49 Figure 10 The Nokia 5800 XpressMusic mobile phone with the lamp and some cubes 50 Figure 11 An overview of the demonstrator 51 Figure 12 System architecture of demonstrator 51 Figure 13 Semantic Media Ontology 59 Figure 14 Semantic Interaction Ontology 60 Figure 15 Wall-wash lighting developed by TU/e SAN 63 Figure 16 The Connector prototype and a smart phone used as a media player 63 Figure 17 Spotlight Navigation prototype 66 Figure 18 Projection of the Spotlight Navigation when connecting two devices together 67 Figure 19 Image showing the Connector scanning the lighting device. 68 Figure 20 The devices and their connections as used in the system 69 Figure 21 Technical details of the Smart Home pi- lot 71 Figure 22 Inferring the media path 73 Figure 23 Sub-domains of well-being 80 Figure 24 Logitech Squeezebox Radio 82 Figure 25 Playing music from the phone on the Squeeze- box radio 84 x List of Figures xi Figure 26 The sleep use case scenario, with the Zeo sleep monitor on the left, the dimmable light and the Connector object in the mid- dle, and the Squeezebox on the right 86 Figure 27 The Zeo headband 87 Figure 28 An overview of the sleep use case 88 Figure 29 Alarm functionality of the phone shared with the radio and the lamp 89 Figure 30 Temporary connections for a PreviewEvent when source and sink are directly con- nected 92 Figure 31 Temporary connections for a PreviewEvent when source and sink are connected via a semantic transformer 92 Figure 32 Semantic connections user interaction model 98 Figure 33 Nokia Play 360◦ speaker system and N9 mobile phone 101 Figure 34 FSMs for a simple light with a switch and a light with a labelled switch 105 Figure 35 Light and light switch as two separate smart objects with a semantic connection 105 Figure 36 Light connected to light switch with aug- mented feedback 106 Figure 37 FSM showing semantic connection with