Model and Tools for Integrating IoT into Mixed Reality Environments: Towards a Virtual-Real Seamless Continuum Jérémy Lacoche, Morgan Le Chenechal, E Villain, A Foulonneau To cite this version: Jérémy Lacoche, Morgan Le Chenechal, E Villain, A Foulonneau. Model and Tools for Integrating IoT into Mixed Reality Environments: Towards a Virtual-Real Seamless Continuum. ICAT-EGVE 2019 - International Conference on Artificial Reality and Telexistence and Eurographics Symposium on Virtual Environments, Sep 2019, Tokyo, Japan. hal-02332096 HAL Id: hal-02332096 https://hal.archives-ouvertes.fr/hal-02332096 Submitted on 24 Oct 2019 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. International Conference on Artificial Reality and Telexistence Eurographics Symposium on Virtual Environments (2019) Y. Kakehi and A. Hiyama (Editors) Model and Tools for Integrating IoT into Mixed Reality Environments: Towards a Virtual-Real Seamless Continuum J. Lacoche1 , M. Le Chenechal2 , E. Villain1 and A. Foulonneau1 1Orange Labs Rennes 2OpenMind Innovation Abstract This paper introduces a new software model and new tools for managing indoor smart environments (smart home, smart build- ing, smart factories, etc.) thanks to MR technologies. Our fully-integrated solution is mainly based on a software modelization of connected objects used to manage them independently from their actual nature: these objects can be simulated or real. Based on this model our goal is to create a continuum between a real smart environment and its 3D digital twin in order to simulate and manipulate it. Therefore, two kinds of tools are introduced to leverage this model. First, we introduce two complementary tools, an AR and a VR one, for the creation of the digital twin of a given smart environment. Secondly, we propose 3D interac- tions and dedicated metaphors for the creation of automation scenarios in the same VR application. These scenarios are then converted to a Petri-net based model that can be edited later by expert users. Adjusting the parameters of our model allows to navigate on the continuum in order to use the digital twin for simulation, deployment and real/virtual synchronization purposes. These different contributions and their benefits are illustrated thanks to the automation configuration of a room in our lab. CCS Concepts • Human-centered computing ! Interactive systems and tools; HCI theory, concepts and models; Virtual reality; 1. Introduction achieve as it requests a deep understanding of the system in order to create desired behaviors. Nowadays, only desktop applications are The emergence of Mixed Reality (MR) technologies, which include available to achieve this goal, which increases the cognitive load Augmented Reality (AR) and Virtual Reality (VR), provides more of users. Indeed, they have to map identifiers, without any spatial affordable hardware and improves the acceptability of such devices references, to real connected objects located in their smart environ- to the consumer market. This growing market leads to a dissemina- ment. We state that MR is able to solve this issue by providing an tion of these technologies into companies of any size as well as in immersive way to manage IoTs behaviors in smart environments. the users’ own homes. Meanwhile, the Internet of Things (IoT) is a pretty dense network of physical objects connected over the In- ternet that are able to sense the real world and to act on it [Kop11]. Our goal is to create a continuum between a real smart environ- As these objects and their features are ramping up while becoming ment and its digital twin using MR technologies. The digital twin cheaper [JJH∗15], they tend to transform our world into a compo- aims to act as a mediation tool between a user and the real smart sition of smart environments: smart-homes, smart-buildings, smart environment he/she wants to manage. We propose to use the digi- cities. A smart environment is defined as an environment able to tal twin of a real environment for the configuration of its behaviors acquire and apply knowledge about the environment and its inhabi- and for their validation and deployment. Our contributions focus on tants in order to improve their experience in it [CD07]. Particularly, automation behaviors but they could also target the creation of in- connected objects can offer services to automatize some users’ re- teraction behaviors. Automation behaviors differ from interaction current behaviors in order to save them time and efforts, and assist behaviors as their progress does not require an explicit user com- them in different tasks in such smart environments. mand such as a gesture or a voice command. To do so, we introduce a software model based on the multi-agent model Presentation- MR and IoT fields share a common aspect that is to propose Abstraction-Control (PAC) [Cou87] that allows a seamless man- a way to build a connection between the real world and a virtual agement of simulated and real connected object. This model is version of it: its digital twin [GS12]. While MR focuses more on leveraged by two kinds of tools in order to provide a full pipeline interactions and rendering, IoT allows the automation of routines for managing a smart environment. For now, these tools focus on that act seamlessly from the user’s point of view. However, the indoor smart environments but they could be extended later to sup- IoT setup process in smart environments remains pretty complex to port the outdoor ones as well. These tools address application de- c 2019 The Author(s) Eurographics Proceedings c 2019 The Eurographics Association. J. Lacoche et al. / Model and Tools for Integrating IoT into Mixed RealityEnvironments: Towards a Virtual-Real Seamless Continuum Real World Virtual World lating connected objects behaviors. The goal of such a simulation is Digital twin capture Digital twin Interactions to validate these behaviors before deployment and to generate data. Geometry capture Inject Layout completion IoTs layout Behaviors configuration Prototyping Smart-Environments AVIot [JJH∗15] is a web-based interactive authoring and visual- ization tool of indoor smart environments. It proposes the user to import the model of his/her environment or to select in a database Sensing and Acting Sensing and Acting a 3D model close to it. Automation behaviors can be configured with a scripting language or with visual programming thanks to a Behaviors Real World Merging Real and Behaviors Deployment Monitoring Virtual Behaviors Simulation combination of boolean events and actions. For now, it does not (No synchronization ) (Weak synchronization) (Strong synchronization) (No synchronization) propose any solution for linking the prototyped smart environment Virtual-Real Continuum with the real world. Similarly, Belluci et al. [BZDA17] propose Figure 1: A digital twin of a smart environment is captured in AR X-Reality toolkit for the rapid prototyping of smart environments and configured in VR. Our software model described in this pa- dedicated to novice users based on the configuration of a 3D digital per allows to navigate on the virtual-real continuum to exploit this twin of the real environment. It includes a virtual world 2D editor, digital twin for multiple purposes. a state chart editor to program behaviors of virtual and physical el- ements and a runtime execution server for deployment. X-Reality toolkit allows the communication between virtual and physical ob- velopers and designers as well as architects and building managers. jects opening perspectives for real-time monitoring and simulation. First, we propose a Google Tango based application to capture the To continue, Pfeiffer et al. [PPL18] introduce a pipeline for the cre- geometry and the positions of the connected objects of a given envi- ation of MR interactions with IoTs. An application designer has ronment. Then, a VR editor allows to load the result of this capture to capture a 360 panorama of the real environment first and then (or another geometry file) in order to complete its configuration configure it in VR thanks to pre-developed visual augmentations with additional connected objects and furniture. Then, within the and behaviors. Automation behaviors are programmed thanks to same VR application, automation scenarios can be created thanks Node-RED (https://nodered.org/) and interaction events to 3D interactions and to a dedicated visual metaphor based on 3D are published using MQTT [MQT14]. With this process, they can jacks connectors and sockets. This VR editor generates a Petri-net target both simulated and real connected objects. It allows a user based scenario that can be edited later by expert users. Once the to prototype its MR application in a 360 panorama then deploy it capture and configuration steps are performed, our software model in the real world. Finally, Reality Editor [HHM13] is a a handheld allows to navigate on the real virtual continuum in order to define AR interface that allows users to create automation behaviors by how much the real world and its digital twin are linked. Indeed, this linking objects between them directly in the real world. property of our model can be used to move from purely simulations to real deployments, through real and virtual worlds synchroniza- Simulating Smart-Environments tion. This continuum and our pipeline for the creation of a digital Simulation of smart environments can be used to validate pre- twin are detailed in Figure1. To illustrate these different contribu- defined services and behaviors before deploying them in the real tions, the different examples that we give in this paper are based world.