UNIVERSIDAD POLITÉCNICA DE MADRID ESCUELA TÉCNICA SUPERIOR DE INGENIEROS DE TELECOMUNICACIÓN A Model for Mobile-Instrumented Interaction and Object Orchestration in Smart Environments Tesis Doctoral LUCA BERGESIO Ingeniero en Informática Máster en Tecnologías y Sistémas de Comunicaciones 2017 Departamento de Señales, Sistemas y Radiocomunicaciones Escuela Técnica Superior de Ingenieros de Telecomunicación Universidad Politécnica de Madrid A Model for Mobile-Instrumented Interaction and Object Orchestration in Smart Environments Tesis Doctoral Autor: Luca Bergesio Ingeniero en Informática Máster en Tecnologías y Sistémas de Comunicaciones Directores: Ana María Bernardos Barbolla Doctora Ingeniera de Telecomunicación José Ramón Casar Corredera Doctor Ingeniero de Telecomunicación June 28, 2017 “The jump from zero to whatever baud rate is the most important jump you can make. After that everyone always wants to go straight to the speed of light” J. T. Tengdin’s First (and Only) Law of Telecommunications “When you see a good move, look for a better one” Emanuel Lasker, World Chess Champion from 1894 to 1921 Abstract The proliferation of the smartphones has given a considerable boost to the spread of the smart objects and the consequent creation of smart spaces. Smart objects are electronic devices that are able to work interactively and au- tonomously, usually preserving the interaction metaphor of their non-electronic counterpart. Through a network interface, they can cooperate with other objects: their strengths do not lie in their hardware, but in the capabilities to manage in- teractions among them and in the resulting orchestrated behaviour. Smart spaces are environments composed of smart devices, where they work together, producing some behaviour of benefit to the dwellers. The current workflow requires that the user downloads an application from a digital distribution platform for each smart object he wants to use. This model converts the smartphone in a control centre, but it limits the potentialities of the objects. Devices are connected in a one-to-one network with the smartphone, a configuration that prevents the direct communication among the objects and that puts the responsibility to coordinate the objects in the hands of the smartphone. Moreover, there are only a few frameworks that permit the integration of several applications and the creation of complex behaviours that involve many objects from different manufacturers. The first challenge considered in this thesis is to propose a new workflow that permits to integrate any kind of smart device in any behaviour of the smart space. The workflow will include the discovery of the new objects and their configuration, without the need of downloading a new standalone application for every object. It will provide to the user a simple configuration tool to create personalized be- haviours (scenes), based on the event-condition-action paradigm. Finally, it will automatically orchestrate the smart devices to produce the desired behaviours of the environment. Smart spaces are thought to behave in a personalized way, adapting to the par- ticularities of their inhabitants. “Personalization” is about understanding the needs of each individual and helping satisfy a goal that efficiently and knowledgeably ad- dresses each individual’s need in a given context. Thus, the second challenge tackled in this thesis is to move forward on how to evolve smart spaces from customizable v to personalized environments. The third open issue considered in this research is how to make portable the personalized configuration, i.e. how to enable a user to commute among different smart spaces preserving and adapting his personalized settings to each of them. Both personalization and portability will be included in the tool to automatically aid the user to gain a full and transparent control over the environment. Solutions to tackle device fragmentation, interoperability, seamless discovery, scene modelling, orchestration and reasoning are needed to achieve these goals. In this context, the contributions of the thesis can be summarized as follows: • The definition of a workflow that permits to personalize and control a smart space using the event-condition-action paradigm. • The design of a model to describe any kind of smart object and its capabilities. • The extension of the previous model to describe a smart space as a composition of its smart objects. • The application of the model to the workflow, reinterpreting the main Object- Oriented Programming features and using them to describe the interactions between objects and the recommendation process. • The proposal of an architecture that implements each step of the workflow and its relations with the model. • The proposal, development and evaluation of service concepts in real smart space settings. vi Contents Abstract v List of Figures x List of Tables xii 1Introduction 1 1.1 Context and Motivation ......................... 3 1.2 Contributions ............................... 6 1.3 Document Structure ........................... 7 2StateoftheArt 9 2.1 Smart Spaces and Smart Objects .................... 9 2.2 Personalization in Smart Spaces ..................... 12 2.3 Smart Spaces Modelling ......................... 14 2.4 Architectures for Smart Spaces ..................... 16 2.4.1 Interoperability .......................... 18 2.4.2 Discovery ............................. 20 2.5 Mobile-Mediated Interaction in Smart Spaces ............. 22 2.6 Summary ................................. 25 3OperationalConceptsandWorkflow 29 3.0.1 A Scenario ............................. 29 3.1 Operational Concepts ........................... 31 3.1.1 Identification and Location .................... 34 3.2 System Workflow ............................. 37 3.3 Technology Agnostic Architecture .................... 41 3.4 Summary ................................. 44 4SmartObjectModelling 45 4.1 Using OOP to Model and Network Smart Objects ........... 47 vii 4.2 Smart Object Model Proposal ...................... 48 4.2.1 Modelling a Smart Object .................... 53 4.3 Formalization of the Generic Object Model ............... 59 4.3.1 Modelling Sensors ......................... 64 4.3.2 Modelling Actuators ....................... 67 4.3.3 Events, Conditions and Actions in the Model .......... 70 4.3.4 Modelling a Smart Object .................... 72 4.4 Summary ................................. 80 5AnArchitectureforSmartSpacesUsingtheOOPSmartObject Model 83 5.1 Architecture Overview .......................... 83 5.1.1 Scene Description ......................... 85 5.2 Module Content .............................. 89 5.3 Discovery of Smart Objects ....................... 92 5.4 Personalization of the Smart Space ................... 96 5.4.1 Manual Scene Creation ...................... 97 5.4.2 Recommendation ......................... 99 5.4.3 Generalization and Specialization ................100 5.4.4 Case-Based Reasoner .......................107 5.4.5 Adaptation of previous scenes ..................111 5.4.6 Scene Compatibility Analysis and Object Configuration ....113 5.5 Smart Space Orchestration ........................114 5.6 Summary .................................116 6Service-OrientedValidation 119 6.1 Experimental Scenario ..........................119 6.2 MECCANO ................................120 6.2.1 Smart Objects ...........................121 6.2.2 Service Discussion .........................123 6.3 PERSEO .................................128 6.3.1 Smart Objects ...........................130 6.3.2 Service Discussion .........................131 6.4 Follow-Me .................................135 6.4.1 Smart Objects ...........................137 6.4.2 Service Discussion .........................137 6.5 Air Traffic Control Station ........................139 6.5.1 Service Discussion .........................140 6.6 Summary .................................142 viii 7ConclusionsandFutureResearchDirections 145 7.1 Contribution Summary ..........................145 7.2 Future Work ................................147 7.2.1 Model ...............................147 7.2.2 Standardization ..........................147 7.2.3 Case-Based Reasoning ......................147 7.2.4 Security ..............................148 7.2.5 Shared Objects ..........................148 7.2.6 Scalability .............................149 7.2.7 Interaction Issues .........................149 7.2.8 Coding ...............................149 AArchitectureoftheOrchestrator 151 A.1 Scene Description .............................154 BCreationofaSmartObject 157 B.1 Model ...................................157 B.2 Hardware .................................158 B.3 Arduino Code ...............................158 B.4 Notification Server ............................161 B.5 Module ...................................162 B.6 Discovery .................................167 References 169 ix List of Figures 2.1 Some commercial smart objects ..................... 11 2.2 Some smart home projects ........................ 12 2.3 Three kinds of semi-automatic discovery used in commercial devices . 22 3.1 System workflow ............................. 37 3.2 Technology agnostic architecture .................... 42 4.1 Components of a generic smart object .................. 52 4.2 Model of a video wall ........................... 54 4.3 Model of a smart door .......................... 56 4.4 Model of a drone with camera ...................... 58 4.5 Generic smart object described with a class diagram .......... 60 4.6 Extension