Eventing in the Hue system
Citation for published version (APA): Samadi Khah, P. (2018). Eventing in the Hue system. Technische Universiteit Eindhoven.
Document status and date: Published: 24/10/2018
Document Version: Publisher’s PDF, also known as Version of Record (includes final page, issue and volume numbers)
Please check the document version of this publication:
• A submitted manuscript is the version of the article upon submission and before peer-review. There can be important differences between the submitted version and the official published version of record. People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website. • The final author version and the galley proof are versions of the publication after peer review. • The final published version features the final layout of the paper including the volume, issue and page numbers. Link to publication
General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.
• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal.
If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above, please follow below link for the End User Agreement: www.tue.nl/taverne
Take down policy If you believe that this document breaches copyright please contact us at: [email protected] providing details and we will investigate your claim.
Download date: 02. Oct. 2021
/ Department of Mathematics and Computer Science / PDEng Software Technology
Eventing in the Hue
System
Pouya Samadi Khah
Where innovation starts
Eventing in the Hue System
Pouya Samadi Khah October 2018 Eventing in the Hue System
Eindhoven University of Technology Stan Ackermans Institute / Software Technology
Partners
Signify Eindhoven University of Technology
Steering Group Daniel Goergen (Signify) Walter Slegers (Signify) George Yianni (Signify) Tanir Ozcelebi (TU/e) Yanja Dajsuren (TU/e)
Date October 2018
Document Status Public
SAI report no. 2018/093
The design described in this report has been carried out in accordance with the TU/e Code of Scientific Conduct.
ii
Contact Eindhoven University of Technology Address Department of Mathematics and Computer Science MF 5.080A, P.O. Box 513, NL-5600 MB, Eindhoven, The Netherlands +31402474334
Published by Eindhoven University of Technology Stan Ackermans Institute
Printed by Eindhoven University of Technology UniversiteitsDrukkerij
SAI report no. 2018/093
Abstract The brain of the Hue system is an embedded device called Hue Bridge. This device controls and monitors Hue lamps, sensors, and switches; it acts as a local home lighting controller. The bridge communicates both with IP and ZigBee networks and facilitates the message translation from one to another. In this system, apps and services poll to get information. Polling in the Hue system generates lots of load, consumes a lot of bandwidth, and forces the client to compare and detect the changes if there is a change. To address these challenges, the eventing approach is suggested. Any change from one state to another is an event. Eventing is hugely beneficial compared to polling in sending the delta messages instead of the whole state to the client, lowering the latency with reducing load on the server, and scaling. Moreover, the removal of polling enables more flexibility and efficiency in their application development.
Keywords Eventing, Application eventing, Bridge Broker, Cloud Broker, Hue system, Hue bridge, Hue cloud, gRPC, Nchan, fine-grained subscription, dynamic group subscription, Server Sent Events, wildcard, Signify, TU/e, PDEng, Software Technology
Preferred Eventing in The Hue System, SAI Technical Report, October 2018. (2018/093) reference
Partnership This project was supported by Eindhoven University of Technology and Signify.
Disclaimer Reference herein to any specific commercial products, process, or service by trade name, Endorsement trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the Eindhoven University of Technology or Signify. The views and opinions of authors expressed herein do not necessarily state or reflect those of the Eindhoven University of Technology or Signify and shall not be used for advertising or product endorsement purposes.
Disclaimer While every effort will be made to ensure that the information contained within this report Liability is accurate and up to date, Eindhoven University of Technology makes no warranty, representation or undertaking whether expressed or implied, nor does it assume any legal liability, whether direct or indirect, or responsibility for the accuracy, completeness, or usefulness of any information.
Trademarks Product and company names mentioned herein may be trademarks and/or service marks of their respective owners. We use these names without any particular endorsement or with the intent to infringe the copyright of the respective owners.
Copyright Copyright © 2018. Eindhoven University of Technology. All rights reserved. No part of the material protected by this copyright notice may be reproduced, modified, or redistributed in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the Eindhoven University of Technology and Signify.
iii
4
Technische Universiteit Eindhoven
Preface
This report offers a detailed account of the graduation project for the PDEng (Professional Doctorate in Engineering) Software Technology program on behalf of the Eindhoven University of Technology and the Stan Ackermans Institute. This project was carried out in Signify over a period of ten months from January until October 2018. This company designs, develops, and produces lighting solutions and applications both for professional and consumer markets. The project's goal is to evaluate the author as a software designer, while providing Signify with an eventing system for local users and suggesting different design solutions. The original need was to find out about eventing design alter- natives in order to support business use cases. This report contains insights, the design as well as the description of the process that led there. Therefore, in addition to the new design, the domain, risk management, project management, conclusions are explained in corresponding chapters. This report is primarily intended for readers with a technical background. However, certain chapters may be interesting for non-technical readers, such as project managers and IoT enthusiasts.
Pouya Samadi Khah October 2018
i Eventing in the Hue System ii Technische Universiteit Eindhoven
Acknowledgement
A number of people have contributed to the successful completion of this project. First, I would like to express my appreciation to Daniel Goergen and Walter Slegers, my supervisors from Signify. In the context of our weekly synchronization meetings they provided guidance and support throughout all the phases of this project. Furthermore, I would also like to thank Tanir Ozcelebi, my TU/e supervisor, for his valuable contribution and meaningful feedback. This project would not have been a success without the support from domain experts, and there were many. The best example of such people is Kevin Worm and I worked closely with him. I would like to also thank the Partnership and Cloud teams who despite their really busy agendas were willing to answer every single one of my questions. I also would like express my gratitude to George Yianni and Leon Bouwmeester who helped me a lot with organizational situations. I thank them for their patience, critical thinking, advice, and continuous support during this project. Additionally, I would also like to thank everyone involved in the PDEng program, especially our program director, Yanja Dajsuren. Thank you for our conversations, your understanding, your advice and everything; this program and my experience would not be the same without you. Last but not least, I would like to express my gratitude, from the bottom of my heart, to four people for their continuous support during this program and for believing in me. First and foremost, a special thank you goes to my parents for their incredible patience and support. Second, a special thank you goes to my brother, Kaveh for everything he has done for me, and finally a special thank you goes to my girlfriend, Aida, for being my biggest support.
iii Eventing in the Hue System iv Technische Universiteit Eindhoven
Executive Summary
Philips Hue is a wireless lighting system designed to transform how users experience light in their homes. It is one of the leading Internet of Things products in the world. Philips Hue changes the way users interact with light by enabling color adjustable lights controlled from smartphones, web services or other control logic devices running in the system. Furthermore, it is an open system, i.e., via standardized or published interfaces, which partners and third party developers can offer several different functionalities. In the Hue system, lights, switches, sensors and other resources are represented as web resources and can be addressed using REpresentational State Transfer (REST) web services. The brain of the Hue system is an embedded device called Hue bridge. This device controls and monitors ZigBee lights (Hue lamps), sensors, and switches; it acts as a local home lighting controller. The bridge communicates both with IP and ZigBee networks and facilitates the message translation from one to another. In this system, sensors and switches event/trigger actions to communicate. On the other hand, apps and services poll to get information. Polling in the Hue system generates lots of load, consumes a lot of bandwidth, and forces the client to compare and detect the changes if there is a change. To address these challenges, the eventing approach is suggested. Eventing or Event driven architecture changes the way Application Programming Interface (API) consumers interact with the APIs. Any change from one state to another is an event. For instance, when the light goes On, it results in a state event. Eventing is hugely beneficial compared to polling in sending the delta messages instead of the whole state to the client, lowering the latency with reducing load on the server, and scaling. Moreover, the removal of polling enables more flexibility and efficiency in the company’s application development. This project demonstrates different eventing architectures for the Hue system and specifically fo- cuses on the application eventing solution. The application eventing refers to the eventing between the bridge and applications in local network. Therefore, one of the project deliverables is the proof of concept of eventing with prototypes for light state changes. Another outcome is the design and implementation of fine-grained subscription and dynamic group subscription support in application eventing. The final result is giving suggestions in order to improve the Hue system architecture for optimizing resource usage in application eventing.
v Eventing in the Hue System Technische Universiteit Eindhoven
List of Tables
3.1 Eventing Technologies ...... 16
4.1 Risk management chart ...... 19
5.1 High-level Functional User Stories ...... 21 5.2 Low-level Functional User Stories ...... 22 5.3 Efficiency ...... 22 5.4 Reliability ...... 23 5.5 Resource Usage ...... 23
7.1 Nchan Upgraded Feature Summary Table ...... 39 7.2 Bridge Improvements Comparison Table ...... 46
9.1 Benchmark results for the SSE approach ...... 57
vi Eventing in the Hue System Technische Universiteit Eindhoven
List of Figures
2.1 Stakeholder's power interest grid ...... 7
3.1 Hue System overview ...... 9 3.2 gRPC Architecture ...... 10 3.3 Polling ...... 11 3.4 MQTT Sequence Diagram ...... 13 3.5 SSE Sequence Diagram ...... 14 3.6 WebSub Sequence Diagram ...... 15 3.7 Webhook Sequence Diagram ...... 15
6.1 Eventing with the Bridge Broker Component Diagram ...... 25 6.2 Remote Connection Setup for Eventing with the Bridge Broker ...... 25 6.3 Third Party Cloud Connection Setup for Eventing with the Bridge Broker ...... 26 6.4 Local Connection Setup ...... 26 6.5 Eventing with the Bridge Broker Happy Flow ...... 26 6.6 Eventing with the Cloud Broker Component Diagram ...... 27 6.7 Remote Connection Setup for Eventing with the Cloud Broker ...... 28 6.8 Third Party Cloud Connection Setup for Eventing with the Cloud Broker ...... 28 6.9 Eventing with the Cloud Broker Happy Flow ...... 29 6.10 Hybrid-Parallel Eventing Component Diagram ...... 30 6.11 Hybrid-Parallel Eventing Happy Flow ...... 30 6.12 Hybrid-Integrated Eventing Component Diagram ...... 31 6.13 Hybrid-Integrated Eventing - Variant 1 Diagram ...... 32 6.14 Hybrid-Integrated Eventing - Variant 2 Diagram ...... 32 6.15 Design Alternatives Comparison Table ...... 33
7.1 Application Eventing Component Diagram ...... 35 7.2 Tree Structure of the Topics ...... 37 vii Eventing in the Hue System Technische Universiteit Eindhoven
7.3 Graph Structure of the Resource Paths ...... 37 7.4 Nchan with storing queue of events - Component Diagram ...... 38 7.5 Nchan with storing queue of events - sequence diagram ...... 40 7.6 Nchan with storing latest stater per attribute - Component Diagram ...... 41 7.7 Nchan with storing latest stater per attribute - Sequence Diagram ...... 41 7.8 gRPC Adapter / Push down subscriptions - Component Diagram ...... 42 7.9 gRPC Adapter / Push down subscriptions - Sequence Diagram ...... 43 7.10 gRPC Adapter with event filters - Component Diagram ...... 44 7.11 gRPC Adapter with event filters - Sequence Diagram ...... 45
8.1 Eventing in the Bridge with Nchan - Demo Architecture ...... 47 8.2 Eventing in the Bridge with Nchan - Sequence Diagram ...... 48 8.3 Eventing in the Bridge with Nchan - Console snippet ...... 49 8.4 Fine-grained Subscription - Demo Architecture ...... 50 8.5 Fine-grained Subscription - Sequence Diagram ...... 51 8.6 Fine-grained Subscription - Client App Snippet ...... 52 8.7 Dynamic Group Subscription - Demo Architecture ...... 53 8.8 Dynamic Group Subscription - Sequence Diagram ...... 54 8.9 Dynamic Group Subscription for the Demo App ...... 54 8.10 Dynamic Group Subscription - Client App Snippet ...... 55
9.1 Benchmark Architecture Diagram ...... 56
A.1 Snapshot of project's time-line ...... 65
viii Eventing in the Hue System Technische Universiteit Eindhoven
Contents
Preface i
Acknowledgement iii
Executive Summary v
List of Tables vi
List of Figures viii
1 Introduction 1 1.1 Context of the PDEng Program ...... 1 1.2 Context of the Project ...... 1 1.3 High-level Project Goals ...... 2 1.4 Low-level Project Goals ...... 2 1.5 Outline ...... 2
2 Stakeholder Analysis 4 2.1 Eindhoven University of Technology (TU/e) ...... 4 2.2 Software Technology Program (ST) ...... 4 2.3 Signify ...... 4 2.4 Main Stakeholders ...... 5 2.4.1 TU/e ...... 5 2.4.2 Signify ...... 5 2.5 Stakeholder Analysis Overview ...... 6
3 Domain Analysis 8 3.1 Philips ...... 8 3.2 Signify ...... 8 3.3 Philips Hue System ...... 9 ix Eventing in the Hue System Technische Universiteit Eindhoven
3.4 Polling ...... 10 3.5 Eventing ...... 11 3.5.1 Eventing Technologies ...... 12
4 Feasibility Analysis 17 4.1 Challenges and Risks ...... 17 4.2 Risk Management ...... 18
5 System Requirements 20 5.1 Requirements Gathering Process ...... 20 5.2 Project Scope ...... 20 5.3 Functional Requirements ...... 21 5.3.1 Fine-grained Subscription ...... 21 5.3.2 Dynamic Group Subscription ...... 21 5.4 Non-functional Requirements ...... 22 5.4.1 Efficiency ...... 22 5.4.2 Reliability ...... 22 5.4.3 Resource Usage ...... 23
6 Eventing Design Alternatives 24 6.1 Eventing with the Bridge Broker ...... 24 6.2 Eventing with the Cloud Broker ...... 27 6.3 Hybrid - Parallel Eventing ...... 29 6.4 Hybrid - Integrated Eventing ...... 30 6.5 Comparison of the Design Alternatives ...... 32
7 System Design 34 7.1 Hybrid - Integrated Eventing ...... 34 7.1.1 Application Eventing Architecture ...... 34 7.1.2 Upgrading the Bridge Broker ...... 36 7.1.3 Possible Improvements of the Bridge Architecture ...... 38
8 Implementation 47 8.1 Eventing in the Bridge with Nchan ...... 47 8.2 Fine-grained Subscription with Wildcard Support ...... 50 8.3 Dynamic Group Subscription with Graph Structure ...... 53 x Eventing in the Hue System Technische Universiteit Eindhoven
9 Validation and Verification 56 9.1 Validation ...... 56 9.2 Verification ...... 58
10 Conclusion 59 10.1 Results ...... 59 10.2 Future Work ...... 60
Glossary 61
A Project Management 64 A.1 Introduction ...... 64 A.2 The Process of Management ...... 64 A.3 Project Planning and Scheduling ...... 64 A.4 Evaluation and Execution ...... 66
B About the Authors 67
xi Eventing in the Hue System Technische Universiteit Eindhoven
1 Introduction
In this chapter, the context of the PDEng program and the project is presented along with a brief reference to the project goals, followed by the outline of the project per chapter.
1.1 Context of the PDEng Program
The assignment described in this report is part of a ten-month collaboration between Eindhoven Uni- versity of Technology and Signify under the auspices of the Software Technology designer program. This Professional Doctorate in Engineering (PDEng) program is offered by Stan Ackermans Institute 4TU.School for Technological Design. Stan Ackermans Institute (SAI) is a federation of four leading Dutch technical universities: TU Delft, TU Eindhoven, University of Twente, and Wageningen University. The federation aims at maximiz- ing innovation by concentrating the strengths in research, education, and knowledge transfer of all technical universities in the Netherlands. The SAI manages more than twenty post-graduate technical designer programs across the four technical universities. Each designer program is intended to teach the skills needed to design the complex systems needed in the high tech industry to new master’s graduates who are starting their careers. The goal of a PDEng program is to provide an additional dimension to a full master’s program by extending it and adding new elements to it. A PDEng trainee further develops skills for synthesis and interdisciplinary work, acquiring the competencies to create innovative technological solutions for products, processes, and systems. The solutions are based on functional requirements as well as on business and market requirements, within the context of society as a whole. The technological designer program takes two years to complete. During the first year, extensive knowledge and ex- perience of the latest design methods and their applications gained through in-house team projects. The second year of the program is spent in industry where the PDEng trainee works on an individual assignment.
1.2 Context of the Project
In the IP domain of the Hue system, state changes of lights, switches, sensors, and other resources are available through REST web services [1]. The Hue Bridge for example regularly polls every connected Zigbee light [2] to be able to represent it as RESTful [3] resources in the IP world. Even though the polling approach is simple and reliable (state information will eventually be correct), it is highly inefficient and slow. Because of this it can take up to a few minutes until state information from a Zigbee device is received by an application. In larger systems the user is faced with outdated
1 Eventing in the Hue System Technische Universiteit Eindhoven user interfaces and it is a challenge to integrate the Hue system with other home automation systems. Besides that, in the Hue system, every time that the client polls the current state, it should compare the current one with the previous state to identify the changes. This process will add a lot of complexity to the API users, including smartphone applications or cloud services. Furthermore, in order to achieve a lower latency with polling, the frequency of it should increase which leads to extra load on the system. To deal with these complications, eventing approach is proposed.
1.3 High-level Project Goals
The high level goals are to evaluate technologies and approaches for eventing in the Hue system covering both “cloud services” and “application eventing” . For instance, one of the main eventing feature for the Hue system is proactively getting informed of light state changes. To develop this feature, the Hue API should integrate with third party cloud services and applications to get the light state changes within the Hue system. Information which needs to be shared proactively are: configuration changes and resource state changes:
1.4 Low-level Project Goals
In addition to contributing to the eventing project, which is mentioned above, the primary goal of this project is to investigate the feasibility of the fine-grained subscription and dynamic group subscrip- tion in application eventing in order to satisfy the requirements of the main stakeholders. Moreover, researching on different eventing solutions for application eventing and come up with the suitable technologies to implement and integrate with the Hue system.
1.5 Outline
This report is organized in the following chapters:
• Chapter 2 - Stakeholder Analysis: presents the identified stakeholders. The main stakeholders for this project were identified in the early phase of this project based on different points of interest of the partners.
• Chapter 3 - Domain Analysis: describes the context of eventing around the Hue system. This information aims to provide the terms that will give the reader a better understanding of the rest of the report.
• Chapter 4 - Feasibility Analysis: presents the feasibility analysis of the problem at hand. It shows the relation to the challenges and risks that were identified in the early stages of the project.
• Chapter 5 - Requirement Analysis: shows the process used to gather the project’s require- ments and how they get prioritized.
2 Eventing in the Hue System Technische Universiteit Eindhoven
• Chapter 6 - Design Alternatives: presents the investigation of alternative eventing tech- nologies along with its outcome. Additionally, this chapter discusses some important design decisions. These decisions are the foundations of the eventing architecture.
• Chapter 7 - System Design: focuses on the final design approaches.
• Chapter 8- Implementation: focuses on the implantation of different prototypes.
• Chapter 9- Validation and Verification: describes the process of validation and verification together with the techniques used.
• Chapter 10 - Conclusion: concludes the report. It gives a brief explanation about the achieve- ments and propose suggestions for the future work.
3 Eventing in the Hue System Technische Universiteit Eindhoven
2 Stakeholder Analysis
In this chapter, the identified stakeholders are mentioned. A detailed description of the involved parties is given and a list of the main stakeholders per party follows accompanied by a table of all affiliates and a visual representation of the analysis.
2.1 Eindhoven University of Technology (TU/e)
The Eindhoven University of Technology is responsible for the educational aspect of this project and fulfilling the requirements for a project of this type. That means certain standards need to be met. The TU/e is concerned with the design process, project management, and implementation.
2.2 Software Technology Program (ST)
The PDEng degree program in Software Technology is provided by the Department of Mathematics and Computer Science of Eindhoven University of Technology in the context of the 4TU.School for Technological Design, Stan Ackermans Institute.
This PDEng is an accredited and challenging two-year, third-cycle (doctorate-level) engineering de- gree program during which its trainees focus on strengthening their technical and non-technical com- petencies related to the effective and efficient design and development of software for resource– constrained software-intensive systems, such as real-time embedded or distributed systems, in an industrial setting. In particular, the focus is large-scale project based design and development of this kind of software.[4]
2.3 Signify
Signify, previously known as Philips Lighting Solutions B.V., is the leading provider of lighting solu- tions and applications for both professional and consumer markets, pioneering in how lighting is used to enhance the human experience in the places where people live and work.
Signify is creating lighting solutions that transform environments, create experiences, and help shape identities.This company serves its customers through a market segment approach, which encompasses homes, office and outdoor, industry, retail, hospitality, entertainment, health care and automotive.
4 Eventing in the Hue System Technische Universiteit Eindhoven
One of the main focuses of the company is connected lighting. In the context of this domain, Signify has many projects following the interest and needs of the market.
2.4 Main Stakeholders
The main group of people who were involved in the steering process of the project are presented below:
2.4.1 TU/e
Yanja Dajsuren (ST Program Director)
She is the program director of Software Technology PDEng program since 2016 and thus she is responsible for supervising the collaboration of the two parties of each design project.
Tanir Ozcelebi (TU/e Supervisor)
He is an Assistant Professor in the Department of Mathematics and Computer Science at TU/e. His main research interests are life-cycle management for resource-constrained embedded devices, archi- tecture development for smart spaces and Internet of Things, as well as resource and QoS management and data analytics for networked services. Besides his mentor-ship, his role included making sure that the design and documentation met the standard of a PDEng project.
Pouya Samadi Khah (PDEng Signify Trainee)
He is a PDEng candidate responsible for the implementation of the project.
2.4.2 Signify
George Yianni (Project Owner)
He is the head of technology and creator of Philips Hue, responsible for the technology choices made in the connected lighting business of Signify, which includes the Hue product. This ranges from design of new features, architectures and products to choices of which technology standards and platforms to adopt. He is the initiator and owner of the project in question.
Daniel Georgen (Project Manager)
He is a System Architect in the Philips Hue department.
Walter Slegers (Project Mentor)
He is a Software Architect in the Philips Hue department.
5 Eventing in the Hue System Technische Universiteit Eindhoven
Kevin Worm
He is a Embedded Software engineer who is leading the design of the eventing project.
Cloud Team
This team focuses on the cloud infrastructure of the Hue system.
Bridge Application Team
This team works on development of the Bridge.
SDK/App Team
SDK and App teams are responsible for the Hue application and also Hue API for third party devel- opers.
2.5 Stakeholder Analysis Overview
Figure 2.1 shows how different people were involved in different phases of the project and influenced its outcome in various ways. An effort of visualizing their contribution is presented in the following diagram.
• High power, interested people: these are the people that needs to make the greatest efforts with, e.g., the direct supervisors of the project who are actively steering the process.
• High power, less interested people: provide sufficient information to these people to ensure that they are up to date, but not overwhelmed with data, e.g., the head of the department who initiated the project.
• Low power, interested people: keep these people adequately informed, talk to them to ensure that no major issues arise. These people can help with the detail of the project, e.g., End Users, other Project Managers, and Business Community.
• Low power, less interested people: provide these people with minimal communication to pre- vent boredom, e.g., other departmental members, teams unaffected by the change.
6 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 2.1: Stakeholder's power interest grid
7 Eventing in the Hue System Technische Universiteit Eindhoven
3 Domain Analysis
In this chapter, we focus our interest inside the structure of Signify. The domain around the Hue system and eventing is described and analyzed focusing incrementally on the important constituents. As the possibilities for digital, connected lighting develop and expand, Signify's smart lighting in- frastructure can become the glue that connects the physical world to the digital realm, creating a true “Internet of Lights”. In this brave new world of connected intelligence, lighting can become an integral and responsive part of our everyday environments. [5]
3.1 Philips
In 1891, Gerard Philips, together with his father Frederik Philips, founded the firm Philips & Co. The company was established in empty business premises in Eindhoven. There was already considerable competition in the lamp market at that time. Gerard’s distinctive approach was to concentrate fully on the mass production of incandescent lamps. In 1895 Gerard’s brother Anton Philips joined the firm to look after sales, a move that proves successful. In 1895, 200,000 incandescent lamps were sold, and three years later more than 1,000,000. By the end of the 1890s Philips & Co. was one of the largest producers in the Netherlands and, with 1,000 employees, the country’s largest industrial employer [6]. Today, Koninklijke Philips N.V. (Royal Philips or the ‘Company’) is the parent company of the Philips Group (‘Philips’ or the ‘Group’). The Company is managed by the members of the Board of Man- agement and Executive Committee under the supervision of the Supervisory Board. The Executive Committee operates under the chairmanship of the Chief Executive Officer and shares responsibility for the deployment of Philips’ strategy and policies, and the achievement of its objectives and results. In September 2014, Philips announced its plan to sharpen its strategic focus by establishing two stan- dalone companies focused on the HealthTech and Lighting opportunities respectively. A stand-alone structure for Philips Lighting has been established within the Philips Group, effective February 1, 2016.
3.2 Signify
Since early 2018, Philips Lighting is fully separated from Philips Royal and rebranded itself to Signify. The Signify’s main business is divided into four business groups: LED, Lamps, Professional, and Home [7]. By serving professional and consumer markets, the company is making people’s lives safer, more productive, and more comfortable; businesses and cities more efficient and livable; and the world more sustainable.. The company employed approximately 32,000 people worldwide with sales of EUR 7 billion in 2017.
8 Eventing in the Hue System Technische Universiteit Eindhoven
3.3 Philips Hue System
In 2012, Signify launched the Philips Hue system. Hue is a connected home lighting system of linked bulbs that can be controlled by a smartphone or tablet via a ZigBee bridge that connects to the home router through the Ethernet. Philips Hue is one of the leading and most installed smart home / Internet of Things (IoT) products in the world. Philips Hue transforms enables color tunable lights to be controlled from smartphones, web services, or other control logic and devices running in the system. Furthermore, It is an open system, i.e., via standardized or published interfaces other suppliers can add components like smartphone apps, services, light switches, and lamps.[8] Figure 3.1 shows a high-level overview of the Hue system and its main components. Hue lamps communicate via a standardized ZigBee protocol allowing integration with ZigBee based devices like sensors and light switches. The Hue bridge handles the home automation and via the Hue bridge the ZigBee network is connected to the home IP network and the Internet. On the IP network side there are smartphones, web browsers, third party services (like IFTTT) and a Hue portal. All these components have software and use Hue interfaces or SDKs provided for Hue application development.
Figure 3.1: Hue System overview
Hue is integrated with various other ecosystems. Most integrations use the standard or open interfaces. Hue offers published interfaces to third party smartphone or tablet apps, third party services, as well as browsers to access a Hue system. Other ZigBee lights or nodes can also be added, but an invalidated device might give a less than ideal behavior. Basically the Hue services, apps, lights, sensors, and switches marked as Hue or friends of Hue use the same mechanisms as third party apps, third party services, and standard ZigBee nodes; possibly with some private Hue interfaces or private extensions to Hue published interfaces.
Hue API
The Hue API interface allows developers to make use of the functionality of the Philips Hue system. Using this interface, they can find information about the available devices in their local network,
9 Eventing in the Hue System Technische Universiteit Eindhoven control these devices, and do much more. The Hue API is a RESTful JSON interface in which clients interact with resources in the Philips Hue system. What this means is that every resource such as devices, groups and lights in the Philips Hue system is represented by a unique URI that is interacted with. gRPC gRPC is an open source remote procedure call framework by Google[9]. gRPC is designed to be extensible to support multiple content types. The initial release contains support for Protobuf and with external support for other content types such as FlatBuffers and Thrift, at varying levels of maturity. Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler [10]. The Hue system uses Protobuf specs to generate gRPC codes. As shown in Figure 3.2 on the server side, the server implements this interface and runs a gRPC server to handle client calls. Furthermore, the generated code on client-side acts as a proxy for the functions implemented in the server.
Figure 3.2: gRPC Architecture
3.4 Polling
Polling is a way to get data from a data source that produces the stream of events and updates. The client makes requests periodically, and the server sends data back to the client based on the requests [11]. In the general polling approach, if there is no data to be sent by the server, an empty response is returned. However, in the Hue system, the peer always sends the full/requested current state instead of empty response. The Figure 3.3 shows how polling works in the Hue system. Basically the client continuously asks from server about the current state in order to compare it with the previous state. Signify’s polling mechanism where a peer asks for the “state” results in:
• generating lots of load
• consuming bandwidth
• forcing the client to compare and detect nothing changed
10 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 3.3: Polling
• keep comparing the previous state with current state to find changes
Polling with lower frequencies will result in the client missing the updates close to the time the updates happen, and polling too frequently results in waste of resources as well as facing the rate limitation imposed by the server. In the Hue system, caching is used to reduce the waiting time for the response. However, polling introduces latency (the poll period) and useless load at both sides.
3.5 Eventing
In principle, an event is any change from one state to another, such as the change from having no messages to have a new message [12]. Events are appealing to API developers, because they function very well in asynchronous environments. By crafting APIs that trigger certain functions on new event delivery, API systems do not have to inherently wait for synchronous delivery or real time communication. Eventing is hugely beneficial compared to polling in:
• sending the delta messages instead of the whole state to the client
• lowering the latency compare to polling with less load on the server
• scaling. Adding more subscribers to a publisher needs less resources that adding more polling connection per client to the server.
• integrating with other IoT systems.
11 Eventing in the Hue System Technische Universiteit Eindhoven
3.5.1 Eventing Technologies
MQTT
MQTT is described on [13] as a machine-to-machine (M2M) / IoT connectivity protocol. This pro- tocol is so lightweight that it can be supported by some of the smallest measuring and monitoring devices, and it can transmit data over far reaching, sometimes intermittent networks. MQTT is a pub/sub messaging transport protocol that is optimized to connect physical devices and events with servers and other consumers. In other words, MQTT is developed to solve the connectivity challenges of the rapidly expanding sensors, actuators, phones, and tablets with established software processing technologies. These fundamentals also turn out to make this protocol suitable for the M2M or IoT world of connected devices where bandwidth and battery power are at a premium. The following are the four aspects to know about MQTT protocol:
• Broker The broker is primarily responsible for receiving all messages, filtering the messages, decide who is interested in it and then publishing the message to all subscribed clients [14].
• Topics Another important concept is topics. Topics are the way you register interest for in- coming messages or how you specify where you want to publish your message. Topics are represented with strings separated by slashes. The slashes indicate the topic level. Here’s an example on how you would create a topic for a lamp in your home office.
• Quality of Service (QoS) levels determine how each MQTT message is delivered and must be specified for every message sent through MQTT. It is important to choose the proper QoS [15] value for every message, because this value determines how the client and the server commu- nicate to deliver the message. Three QoS levels for message delivery could be achieved using MQTT:
– At most once (0) * Messages are delivered according to the best efforts of the operating environment. Message loss can occur. – At least once (1) * Messages are assured to arrive but duplicates can occur. – Exactly once (2) * Messages are assured to arrive exactly once. QoS is a major feature of MQTT. It makes communication in unreliable networks a lot easier because the protocol handles re-transmission and guarantees the delivery of the message, re- gardless how unreliable the underlying transport is. Also it empowers a client to choose the QoS level depending on its network reliability and application logic.
• MQTT Connection The MQTT protocol is based on top of TCP/IP and both client and broker need to have a TCP/IP stack. As it is shown in Figure 3.4, the MQTT connection itself is always between one client, which can be Publisher/Subscriber, and the broker, no client is connected to another client directly. The connection is initiated through a client sending a Connect message to the broker. The broker responds with a Connect Ack and a status code. Once the connection is established, the
12 Eventing in the Hue System Technische Universiteit Eindhoven
broker will keep it open as long as the client does not send a disconnect command or it loses the connection. Then the subscriber can subscribe to a topic and get events based on that topic. The publisher continually send updates to the broker and the subscriber receives the updates via broker.
Figure 3.4: MQTT Sequence Diagram
Server Sent Events (SSE)
Server Sent Events is a technology that enables the server to send automatic updates to clients e.g., real-time notifications or updates generated on the server. [16] SSE is not bidirectional in its communications. The server is issuing the events in a steady, predictable method. This is hugely useful for applications which do not need the two-way communications de- veloped into WebSocket protocols or other such solutions, as this means lower bandwidth and an allowance for the connection to be more temporary rather than always on during the duration of data transfer. In essence, the data is being transferred one way, and therefore there is no need to wait for data to be returned. In Figure 3.5, it is demonstrated that a connection is kept open as long as possible by the client/server pair when the connection is closed (i.e., by a router, proxy, firewall) the client reconnects to the server. SSE is similar to the long polling strategy, except that the connection is kept open as long as possible.
WebSub/ PubSubHubBub
WebSub formerly known as PubSubHubbubb is an API technology used to publish information on the Internet [17]. The information can take any form: HTML, text, pictures, audio, or any other kind of content. The idea behind WebSub is to push content rather than force clients to poll for it, which is typically how API implementations are designed to work. A setup includes these three characteristics:
13 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 3.5: SSE Sequence Diagram
• It is loosely coupled. Therefore, it is extremely scalable and flexible. The event provider’s task is generating the content and nothing more.
• Each other step is done through a separated middleman, and so the content is easily scaled and modulated to the architecture and design of the solution.
• WebSub lends itself very well to testing. A subscriber is narrowly limited to a set of events that they have requested under a class, so if a failure occurs, this natural segmentation informs the provider as to where the fault is and which class of users is experiencing the fault.
As is shown in Figure 3.6, subscribers discover the hub of a topic URL and send a POST request to one or more of the advertised hubs in order to receive updates when the topic changes. Publishers notify their hub(s) URLs when their topic(s) change. When the hub identifies a change in the topic, it sends a content distribution notification to all registered subscribers.
WebHooks
Webhooks are user-defined HTTP callbacks. They are triggered by some event in a web application and can facilitate integrating different applications or third-party APIs [18]. Webhooks can be handled by serverless frameworks such as AWS Lambda and Azure Functions. As shown in Figure 3.7, when the event occurs, subscriber makes an HTTP request (usually a POST or a GET) to the URL which is configured for the Webhooks. Subscriber’s request includes details of the event. The Server sends a callback URL to the subscriber. Then, the subscriber sends an response to the server.
14 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 3.6: WebSub Sequence Diagram
Figure 3.7: Webhook Sequence Diagram
Comparing Eventing Technologies
According to the points mentioned above, Table 3.1 gives a comparison in order to evaluate which technology is more compatible with the current architecture of the Hue system. In conclusion, SSE is suitable to integrate with the Hue system because it is based on HTTP, fast, and can be compatible with the Hue API itself.
15 Eventing in the Hue System Technische Universiteit Eindhoven Table 3.1: Eventing Technologies
16 Eventing in the Hue System Technische Universiteit Eindhoven
4 Feasibility Analysis
In this chapter, the feasibility analysis of the problem in question is presented. The issues and the risks identified in the early stages of the project are discussed below. Furthermore, a supportive table is depicted combining the hierarchy of the risks and challenges (i.e., the impact) with some mitigation strategies.
4.1 Challenges and Risks
Risks are ubiquitous in any domain or discipline. A crucial part of every project is to identify them as early as possible and try to mitigate their impact. Even from the initial project description, one could identify some possible challenges and risks and of course, many more were expected. This section is focused on the most important risks and challenges. On the one hand, the goal is to give enough context for the reader to follow and on the other hand, to use this as future improvement reference.
• The technical complexity of the system was based on its size but also on the lack of supportive documents. There were different repositories and very few documents. For a new developer, walking through guides, architectural documents, diagrams, and configuration and installation manuals can be invaluable especially as a starting point. Unfortunately, the existing documen- tation was scarce and outdated in many cases.
• Arranging meetings with all important stakeholders of this project are challenging. The dis- cussions with architects, Cloud team, Bridge team and Partnership team are a crucial input to the problem analysis and further definition of the use cases and requirements of the project. Given their busy everyday schedules, arranging discussions for exacting key requirements and evaluating the results are a challenging task that requires stakeholder management and time management skills.
• Due to the lack of my domain knowledge, the need for domain experts was imperative from the beginning. Many brainstorming meetings with various people from different departments were organized in order to acquire useful information and tips. The focus of the project was a module inside the main embedded device (i.e., the bridge) but in order to understand how the module works one should have sufficient understanding of the whole system. The knowledge about the specific module was distributed among many people and even more people were involved in the embedded device as a whole. Some of them were not even working for Signify at the time of research and some were working for different modules during the years. All in all, it was quite difficult to gather all the information needed from all these people and merge it in one non-contradictory, useful knowledge stream.
17 Eventing in the Hue System Technische Universiteit Eindhoven
• The author participated in more than 30 meetings during the process of requirements elicitation. The document that was produced (including mainly non-functional, functional requirements and use-cases) was quite extensive resulting in an increased risk of conflicting requirements and user-stories. Prioritization had to be made and that procedure was not straightforward introducing an obvious time overhead.
4.2 Risk Management
The above mentioned risks and challenges and more are gathered in Figure 4.1 in combination with their potential impact, probabilities and mitigation strategies.
18 Eventing in the Hue System Technische Universiteit Eindhoven Table 4.1: Risk management chart
19 Eventing in the Hue System Technische Universiteit Eindhoven
5 System Requirements
This chapter presents the system requirements from the project “Eventing in the Hue system.” After analysis of the domain and its problems, a set of requirements is extracted and formulated for this project.
5.1 Requirements Gathering Process
Both the problem and domain analysis reveal the core of the problem and its position in the bigger ecosystem. Still, a simple problem statement is not enough to proceed with solving the problem itself. What is necessary is a problem decomposition into small and traceable sub-problems that address a specific issue. This decomposition leads to the formalized requirements of the project. The problem decomposition process is conducted by repeated discussions with relevant stakeholders and thorough analysis of the problem and domain. These repeated discussions reveal the fine details of the problem and its sub-components, which further help to differentiate specific requirements for the project. Two sets of requirements are identified: functional requirements and non-functional requirements. Each requirement has a priority assigned to it, which indicates the importance of that requirement. The following three categories are defined for priority, based on the descriptions defined by S. Brander[19]:
• Must - absolute requirement of the specification.
• Should - there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.
• Optional - the requirement is truly optional.
5.2 Project Scope
Due to the time limitation of the 10 months final PDEng project, it is not possible to fulfill and touch all the parts of the project goal. Thus, the scope is narrowed to work on the evening architectural design decisions on the Hue system and focus on the design details of the application eventing and develop prototypes accordingly. The use cases are explained in the next section as functional requirements with the different levels of priority.
20 Eventing in the Hue System Technische Universiteit Eindhoven
5.3 Functional Requirements
This section includes the requirements that specify all the fundamental actions of the system. Each requirement is structured as user story with a brief description and an assigned priority. These user stories are divided in two groups: High-level user stories which are common for the general eventing approach (for cloud, remote, and local users) in the Hue system (Table 5.1), and Low-level user stories focuses on the specific features containing fine-grained and dynamic group subscriptions. Application eventing is for the users who access the Hue Bridge with LAN. In the user stories, a resource is referred to lights, sensors, switches, scenes, groups, rooms, or any other component that is accessible through Hue API. A local user refers to a user who interacts with the bridge through a local network setup which can be home Wi-Fi network. Table 5.1: High-level Functional User Stories
Story ID User Story Priority As a user, I want to receive a notification when a resource is added to US01 Must a bridge, so I know that I can interact with this resource. As a user, I want to receive a notification when a resource is removed US02 Must from a bridge, so I know that I can no longer interact with this resource. As a user, I want to receive a notification when the state attribute of US03 Must a specific resource is updated so that I can react to this change. As a user, I want to receive a notification when the configuration of a US04 Should specific resource is updated so that I can react to this change.
The table 5.2 demonstrates the more specific ones which are for fine-grained and dynamic group subscriptions. The following sections explain what exactly this feature can bring to the Hue System.
5.3.1 Fine-grained Subscription
Fine-grained subscription [20] gives the user the ability to subscribe exactly the part of the data that is interested in. For instance, if a user wants to subscriber to light A, but just for the brightness attribute of it, then the system needs to have a hierarchical infrastructure in order to let the user to subscribe to a specific proportion of it.
5.3.2 Dynamic Group Subscription
The feature means the user can subscribe to a group and get all the updates from that specific group without knowing the resources in it. For example, a user wants to get notifications about the resources (lamps, sensors) in his bedroom. Then he subscribes to the bedroom and gets all the notification about the resources. If a resource gets detached, the user will not get any notification since it is not in the group anymore.
21 Eventing in the Hue System Technische Universiteit Eindhoven
Table 5.2: Low-level Functional User Stories
Story ID User Story Priority As a local user, I want to receive all the events from the group of resources US05 Must that I am interested in, so I can react to it. (Fine-grained subscription) As a local user, I want to receive all the events from the specific resources US06 Must that I am interested in, so I can react to it. (Fine-grained subscription) As a local user, I want to receive events from the specific attribute of a resource US07 Must that I am interested in, so I can react to it. (Fine-grained subscription) As a local user, I want to receive events from the specific attribute of a group of US08 Must resources that I am interested in, so I can react to it. (Fine-grained subscription) As a local user, I want have a single subscription to all the events US09 Must of all the resources of a group. (Dynamic group subscription) As a local user, I want to receive events from a resource that is added US10 Should to an already subscribed group. (Dynamic group subscription) As a local user, I do not want to receive events from a resource that is removed US11 Should from an already subscribed group. (Dynamic group subscription)
5.4 Non-functional Requirements
5.4.1 Efficiency
The latency requirements defines the efficiency of the system. With eventing the goal to reach lesser latency than current state of the Hue system and make sure that more than 95% of the events have the latency less that half a second.
Table 5.3: Efficiency
ID Category Title Blocker Target Stretch 95% >2s 95% <= 0.5s 95% <= 0.1s NF01 Remote latency / Discrete sensor state events 99.5% >5s 99.5% <= 1s 99.5% <= 0.2s Connection-less 95% >3s 95% <= 0.5s 95% <= 0.1s NF02 (3rd party cloud) Light state events 99.5%>5s 99.5% <= 1s 99.5% <= 0.2s 95% >3s 95% <= 0.5s 95% <= 0.1s NF03 Remote latency / Discrete sensor state events 99.5%>5s 99.5% <= 1s 99.5% <= 0.2s Connection oriented 95% >3s 95% <= 0.5s 95% <= 0.1s NF04 (application) Light state events 99.5%>5s 99.5% <= 1s 99.5% <= 0.2s
5.4.2 Reliability
In this context, reliability is that the ratio of successful guaranteed delivery be at least the same as the Hue system.
22 Eventing in the Hue System Technische Universiteit Eindhoven
Table 5.4: Reliability
ID Category Title Blocker Target Stretch NF05 Event delivery/ Configuration events >98% >99.5% >99.9% NF06 (success rate) Light state events >98% >99.5% >99.9%
5.4.3 Resource Usage
For the resource usage, the space complexity of the memory used by eventing should be linear in O(n). Otherwise, the bridge can run out of memory and lead to malfunction or shutting down the system. Therefore, the maximum amount of stored events should be equal to the current full state of the system. In Table 5.5, “k” is the number of subscriptions. and the “n” is current state of the bridge with all the attributes. Table 5.5: Resource Usage
ID Category Title Blocker Target Stretch Space Complexity of the NF07 Eventing Resource Usage O(k*n) O(n) O(n) Eventing memory usage in the bridge
23 Eventing in the Hue System Technische Universiteit Eindhoven
6 Eventing Design Alternatives
In this chapter, some important design decisions are discussed. The decisions are based on the previ- ously mentioned requirements for the three types of end-users; Cloud, Remote and Local users. To come up with these design alternatives, three development teams were actively involved; the Bridge team, the Cloud team and also App/SDK teams. These designs could serve as the foundations of the event driven architecture for local/remote users and third-party cloud providers. In the Hue system, four different design alternatives are explained in the following sections:
• Eventing with the Bridge Broker
• Eventing with the Cloud Broker
• Hybrid-Parallel Eventing
• Hybrid-Integrated Eventing
6.1 Eventing with the Bridge Broker
In this approach, events are distributed through a broker in the bridge. The Hue cloud acts as a transparent proxy to make the bridge accessible from the Internet. In principle, the remote users and third party clouds subscribe directly to the bridge via the Hue cloud. As is shown in Figure 6.1, for the application and the third party cloud components there is one interface from the Hue cloud that can send events to both of them. The key point in this design is that there is only one bridge broker and in the Hue cloud there is a proxy, which handles the events and maps them to the external world. For the remote users connection setup (Figure 6.2), the user opens the Hue app, and when the app recognizes it is outside of home via the Internet, sends the authorization request to the Hue cloud. Then, after getting the token, the app subscribes to the events which send the request to the Hue cloud and from there it propagates to Bridge broker. For the third party clouds, it is a similar approach. For instance, user talks to third party cloud. Then, the third party cloud gets authorized via Hue cloud and it subscribes to events which is then forwarded to the Bridge broker.(Figure 6.3). However, for the local users, the authorization happens via the Bridge as shown in Figure 6.4 The eventing flow in this approach is shown in Figure 6.5. The events are sent from lights to the Bridge broker and from there to the end parties.
24 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 6.1: Eventing with the Bridge Broker Component Diagram
Figure 6.2: Remote Connection Setup for Eventing with the Bridge Broker
25 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 6.3: Third Party Cloud Connection Setup for Eventing with the Bridge Broker
Figure 6.4: Local Connection Setup
Figure 6.5: Eventing with the Bridge Broker Happy Flow
26 Eventing in the Hue System Technische Universiteit Eindhoven
6.2 Eventing with the Cloud Broker
In this approach the broker is installed inside the Hue cloud. Therefore, the bridge publishes all the events that it gets from the devices (lights, switches, sensors) to the cloud broker. Then, the cloud broker sends the events to the subscribers as is shown in Figure 6.6. The cloud provides a persistent connection for user applications and a non-persistent connection to cloud services.
Figure 6.6: Eventing with the Cloud Broker Component Diagram
Since the broker is in the cloud, there is no scalability issue, but the cost of cloud services might increase as the number of events increases. For the security part, the authentication and authorization are implemented in the cloud. One of the main drawbacks of this approach is that the offline eventing (without Internet connection) is not possible. Thus, the applications always access remotely via the Hue cloud in order to capture the events. For the remote users connection setup (Figure 6.7), the user opens the Hue app, and when the app recognizes it is outside of home via the Internet, sends the authorization request to the Hue cloud. Then, after getting the token, the app subscribes to the events which send the request to the Hue cloud broker. For the third party clouds, it is similar to previous design alternative. For example, user talks to the third party cloud. Then, the third party cloud gets authorized via Hue cloud and it subscribes to events.(Figure 6.8). In Figure 6.9, the eventing flow is demonstrated for the eventing with the cloud broker. When the events are detected by the bridge, it forwards all the events to the cloud broker. Then, the cloud broker
27 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 6.7: Remote Connection Setup for Eventing with the Cloud Broker
Figure 6.8: Third Party Cloud Connection Setup for Eventing with the Cloud Broker decides where to send the events based on the subscribers. The drawback of this approach is that , it creates a lot of traffic between the bridge and the Hue cloud due to the event forwarding.
28 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 6.9: Eventing with the Cloud Broker Happy Flow
6.3 Hybrid - Parallel Eventing
In this design alternative, events are distributed through a broker on the bridge and a broker in the cloud independent from each other (Figure 6.10). Basically, for the local users the Bridge broker is used and for the remote user and third party cloud the Hue cloud broker is used. The authentication and authorization are implemented in both brokers. The cloud provides a persistent connection for user applications and a non-persistent connection to cloud services. For the scalability, the maximum number of local clients is determined by the available resources on a bridge. Moreover, remote scaling depends on the costs of cloud services. This approach is also capable of offline eventing between application and bridge. The main drawback of this approach is the synchronization problems. Since the brokers are working in parallel, the events can easily get out of sync and also the recovery mechanism is a very difficult task to develop. Since this approach is a combination of the eventing with the bridge broker and the cloud broker, the remote connection setup for third party clouds and remote users are exactly same as second design alternative and the local connection setup is identical to the first design approach setup. In addition to that, the eventing flow is the combination of first two approaches (Figure 6.11)
29 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 6.10: Hybrid-Parallel Eventing Component Diagram
Figure 6.11: Hybrid-Parallel Eventing Happy Flow
6.4 Hybrid - Integrated Eventing
The Hybrid - Integrated approach, which is similar to the previous one, has a bridge and cloud broker. The main difference is that the Hue cloud subscribes to the bridge broker when it comes online. In this case, the bridge does not need to know any info from the cloud, since the Hue cloud is only a subscriber to the bridge. Figure 6.12 demonstrates the relations between each component. Events
30 Eventing in the Hue System Technische Universiteit Eindhoven are distributed through a broker on the bridge and then republished to a cloud broker. Authentication and authorization are implemented in both brokers. Same as the Hybrid-Parallel approach, the cloud creates a persistent connection for user applications and a non-persistent connection to cloud services. Moreover, the connection setup and eventing happy flow is the same as previous approach.
Figure 6.12: Hybrid-Integrated Eventing Component Diagram
This approach has two different architectures for the remote users. However, for the cloud and LAN subscribers the approach is the same:
• Single bridge-cloud subscribe channel for the cloud/remote App users: In this alternative form, as it is shown in Figure 6.13 all the Cloud/ Remote app subscribers go through a single channel between the Hue cloud and the bridge. This approach is optimized because there is single connection to send the data over. However, it adds complexity in the Hue cloud to create different interfaces for the App users and the Cloud providers. Moreover, these interfaces adds synchronization complexity and failure points. Therefore, maintaining the Bridge-Cloud eventing channel as well as interface components are pretty crucial. • Single bridge-cloud subscribe channel for the Cloud providers and virtual subscribe channel per remote application user: This variant, has the same architecture for the cloud providers. On the other hand, for the remote app user, it creates a connection per app (Figure 6.14 ). This approach bring lots of scalability and also simplicity since the app users are directly connected to the bridge via a transparent proxy. Therefore, the applications are responsible to maintain the connection between app and the cloud. On of the drawbacks of this approach is that the events are sent twice to the Hue
31 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 6.13: Hybrid-Integrated Eventing - Variant 1 Diagram
cloud from the bridge. Thus, duplication of sending data adds more load to the bridge cloud connection and more complexity to maintain it.
Figure 6.14: Hybrid-Integrated Eventing - Variant 2 Diagram
6.5 Comparison of the Design Alternatives
Table 6.15 shows the summary of points mentioned above and detailed comparisons among the men- tioned approaches.
32 Eventing in the Hue System Technische Universiteit Eindhoven Figure 6.15: Design Alternatives Comparison Table
33 Eventing in the Hue System Technische Universiteit Eindhoven
7 System Design
This chapter focuses on architectural reasoning and design decisions, which resulted in a design of the prototype. The purpose of designing an architecture for a system is solving a problem statement formalized with system requirements. In the later part of this chapter, the detailed implementation is described.
7.1 Hybrid - Integrated Eventing
Based on this comparison in the previous chapter, both of the hybrid approaches can satisfy the re- quirements for eventing with bridge and cloud brokers. The Hybrid-Integrated approach is chosen to be implemented, because:
• in the Hybrid-Integrated approach, the cloud subscriber decides through the subscription mech- anism if it wants to receive events or not. However, in the parallel version, the eventing daemon always sends all the events to the Hue cloud even if there are no cloud subscribers. Therefore, there is lots of unnecessary load on the bridge-cloud channel.
• in the Hybrid-Parallel, there are two independent brokers. Therefore, synchronizing these two, brings problems and complexity compared to the Hybrid-Integrated one in which the cloud broker is just another subscriber to the bridge broker. One of the main problems is if there is a connectivity issue on the path of one of these brokers, they get easily out-of-sync. However, in the Hybrid-Integrated version, the cloud broker is depending on the bridge broker, and if the bridge broker gets out-of-sync, the cloud broker automatically gets out-of-sync too.
7.1.1 Application Eventing Architecture
The focus of this report is on the application eventing. Therefore, in the following section, there is a detailed discussion about the design choices for the bridge architecture based on the application eventing. As it is shown in the Figure 7.1, the gRPC broker publishes the events to the Eventing and Homekit components. From the Eventing component the events are restructured and sent to the Nchan broker which then they publish to both local app users and remote/cloud subscribers. There are two new components that are going to be added to the current bridge system, Nchan and Eventing com- ponent. In addition to that, Websocket Client Daemon (WSCD) component, Nginx, and gRPC broker should be updated. The green color indicates the new components that is needed to be implemented in the bridge, and the yellow components show the components that needed to be modified.
34 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 7.1: Application Eventing Component Diagram
Eventing Component
Eventing component is a software package that will handle the events from gRPC broker and update them based on internal CLIP APIs, and publish them over Nchan. As shown in Figure 7.1 Eventing component is connected to gRPC broker and Nchan broker.
Nchan
Nchan is a scalable, flexible pub/sub server, which is a module for an Nginx web server. It has a message queue that can buffer messages in memory, on-disk, or via Redis. This default storage method uses a segment of shared memory to store messages and channel data. In Nchan, connections are handled asynchronously and distributed among any number of worker processes. One of the main advantages of Nchan is that it is highly scalable. Like Nginx, Nchan can easily handle as much traffic as it is thrown at it [21]. Messages are published to channels with HTTP POST requests or Web-socket and subscribed also through WebSocket, long-polling, SSE, old-fashioned interval polling, and more. As it is explained in the previous chapters, Nchan uses SSE as one of the communication protocols. For the proposed solution, the message queue buffers in the shared memory and it is going to be built in the bridge’s Nginx. Besides than the chosen communication protocol for the applications is SSE.
Nchan Limits
Although that Nchan is a very robust, scalable, and reliable broker, it needs a couple of components to be well adjusted in order to support fine-grained and dynamic group subscription modules. Fortu- nately, this software is open source and it can be easily upgraded to the systems need.
35 Eventing in the Hue System Technische Universiteit Eindhoven
In Nchan, each channel has its own message queue and the broker has no knowledge about the content inside. If a user subscribes to multiple topics the broker receives all the topics as it is and create a separate channel and message queue for them. However, if these topics have some common content, then the messages are stored redundantly inside Nchan shared memory. Since the bridge has limited storage, it can run out of memory. In summary, messages are treated as opaque objects. Therefore, there is no direct way to prevent messages from duplication for an arbitrary set of multiplexed chan- nels. Therefore, in order to support fine-grained subscription, there should a separate channel for every level of the subscribed topic. As an example, in the shared memory, there can be a channel for the topic “lights/1” and “lights/1/state/on” which results in duplication and memory misusage. Another significant fact which should be taken into consideration is to support dynamic group sub- scription. In order to implement that, the broker needs to know about the resource relations real-time. For instance, if a user creates a new group and wants to subscribe to it, or if a light that belongs to a group gets detached, the broker should update the subscribed channel in order to send the correct data to the users. However, Nchan does not support these features. The next section illustrates on how to address these challenges.
7.1.2 Upgrading the Bridge Broker
Current application eventing architecture is reliable for independent and static subscription channels which meets the requirements from the Table 5.1. However, in order to satisfy the requirements in the Table 5.2, there should be some modifications to Nchan as it is mentioned in previous section. The following sections explains the different approaches to mitigate these problems.
Fine-grained and Dynamic Group Subscription Support
Based on the points mentioned above, two main alternatives are proposed: wildcard feature and graph structure which maintains the relations between the resources.
Wildcard (Tree Structure Implementation) The concept of wildcard subscription, which is also known as hierarchical subscription is a term that is used in MQTT protocols. MQTT Topics are structured in a hierarchy similar to folders and files in a file system using the forward slash ( / ) as a delimiter. A client can subscribe to individual or multiple topics [22]. This is approach is to address the fine-grained subscription feature. When subscribing to multiple topics two wildcard characters can be used. They are:
• + (plus character) - single level wildcard. As the name already suggests, a single level wildcard is a substitute for one topic level.
• # (hash character) - multi-level wildcard. While the single level wildcard only covers one topic level, the multi-level wildcard covers an arbitrary number of topic levels. In order to determine the matching topics it is required that the multi-level wildcard is always the last character in the topic and it is preceded by a forward slash.
The wildcard approach has several different implementations. One of the main executions of the wildcard is based on the tree data structure. As it is demonstrated in Figure 7.2 each level of the
36 Eventing in the Hue System Technische Universiteit Eindhoven tree shows the URL depth. If the bridge broker supports the wildcard approach, it can detect the hierarchical level of topics and prevent the duplication of storing the redundant data in the memory.
Figure 7.2: Tree Structure of the Topics
Dynamic group subscription with graph structure In this approach, the broker has the knowledge about the structure of the resource paths. Moreover, it constantly checks them with the subscription topics to prevent sending duplicate packets to the same device. For instance, as it is demonstrated in Figure 7.3 node A has access node D in three different paths. If the module already knows the resource path structure, it is realizable to prevent storing redundant data four times. Additionally, the structure should get updated and notify the broker about the changes.
Figure 7.3: Graph Structure of the Resource Paths
For this concept there are two types of implementations; coupled and decoupled. For the coupled one, the broker has the knowledge of the message content and in the decoupled one there is an independent module that can be between the publisher and broker or installed on top of the publisher. Although
37 Eventing in the Hue System Technische Universiteit Eindhoven implementing the module inside the broker is easier, it violates the concept of pub/sub broker which emphasized on the decoupled nature of broker from publisher and subscriber. For the implementation, it is suitable to use regular expression. In regular expression, developers can choose any topic name that matches the regular expression,."/topics/[a-zA-Z0-9-_. %]+". With regex, it is applicable to configure when to subscribe, when to send messages, and how to handle the notification when it reaches the client app. The regex also covers the wildcarding as described above. Therefore, it is possible to keep track of the multi-level graph structures. As a result, this approach covers all the features that indicated to handle the dynamic topics and prevents the data deduplication, this approach is chosen for the PoC. These potential solutions to support fine-grained and dynamic group subscriptions are presented as a summary in the Table 7.1.
7.1.3 Possible Improvements of the Bridge Architecture
As it is denoted in the previous section, adding wildcard capability and dynamic group subscription support are significant for satisfying the mentioned use cases in the beginning of this chapter. More- over, there is a lot of room for improvement and optimization in the whole architecture. In this section, the different architectural possibilities for the bridge is discussed. The aim of this discussions and comparisons is to find a way to improve the recoverability and memory optimization of the bridge system.
Option 1 - Eventing with Queues in Nchan
Figure 7.4 illustrates that the Nchan broker has a small database that saves a queue of events for each channel. The advantage of this solution is that, in case of disconnection, the broker can re-emit the series of events from the queue. On the other hand, because of the limited capacity of the bridge, it can run out of memory and cause performance damage to the bridge.
Figure 7.4: Nchan with storing queue of events - Component Diagram
38 Eventing in the Hue System Technische Universiteit Eindhoven
Table 7.1: Nchan Upgraded Feature Summary Table
To have fine-grained and dynamic level subscription support, the system requires including wildcard matching and graph matching modules. The sequence diagram 7.5 shows the flow of the eventing. First the clients subscribe to the events on Nchan and then the generated events from lights and sensors transfers to gRPC and from there to the Data Model parser which translates them to the proper data model. Data model parser publishes these translated events to the Nchan (bridge broker). At the same time in Nchan, the events are stored in queues in a small database per channel. Additionally, the events are matched with subscribed events and sent over the SSE protocol.
39 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 7.5: Nchan with storing queue of events - sequence diagram
Option 2 - Eventing with Latest Event State Stored in Nchan
This design is similar to Option 1 with the difference that the bridge broker stores only the last event and re - emits that in case of disconnection or any other problem in the chain of eventing (Figure 7.6 ). The main advantage of this option over the previous one is that the bridge does not run out of memory, since it stores only the latest state per attribute. The sequence diagram 7.7 shows the flow of the eventing. Similar to the previous design, the clients subscribe to events on Nchan and then events come from the lights to the gRPC broker and from there to the Data Model parser. After translation, the events are sent to the Nchan (bridge broker). The latest stats per attribute are stored. Moreover, Inside Nchan, the events are matched with subscribed events and sent over the SSE protocol.
Option 3 - GRPC Adapter / Push Down Subscriptions
As it is described in the previous two options, the main reason to store the events is to recover them when the system goes out of sync or is disconnected. It is worth mentioning that gRPC as an interface natively supports some of the eventing use cases including out-of-sync recovery and wildcard hierarchy capability. Thus, the idea is to reuse these features in the eventing mechanism.
40 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 7.6: Nchan with storing latest stater per attribute - Component Diagram
Figure 7.7: Nchan with storing latest stater per attribute - Sequence Diagram
41 Eventing in the Hue System Technische Universiteit Eindhoven
For instance, with the capability of gRPC, when a new client connects to the Bridge App, it receives events covering the full system state. This function becomes hidden when an upstream broker (bridge or cloud broker) is getting placed in front of it. As a consequence, the bridge broker buries these additional subscribers from the Bridge App. If the brokers are configured as simple protocol adapters, it is possible to retain these native capabilities, without changing the external interfaces. Moreover, the fan-out provided by the brokers optimizes the amount of events that need to be transmitted (load / cost). Figure 7.8 presents the gRPC as the main broker and Nchan as a simple protocol adapter.
Figure 7.8: gRPC Adapter / Push down subscriptions - Component Diagram
The advantages of this approach are:
• During the recovery process, the infrastructure guarantees that the client is able to catch up.
• Only clients that are not in sync receive the additional events that are needed to recover.
• This approach does not require modifications in the Bridge App.
• The proposed architecture simplifies the functionality of bridge broker.
• The events are stored once in gRPC.
Despite the fact that this approach has plenty of advantages, the events are duplicated, resulting in an additional (unnecessary) load on the system. Especially in the cloud this could result in additional costs, as it multiplies the number of events whenever another partner integration is enabled. The sequence diagram 7.9 shows the flow of the eventing. Different clients subscribe directly to gRPC. Afterwards, gRPC parses the events and matches them with the topics and sends them per client to the Data Model Parser to change the data format. From there, the events are sent to Nchan which works as a protocol adapter to SSE.
Option 4 - GRPC Adapter with Event Filters
In this technique, the gRPC is also the main broker, but the subscribers are not subscribing directly to it. The subscription goes via a main channel from Eventing component to the gRPC (7.10). Without a large architectural change, it is not possible to directly target individual clients to get up- dated with the current state of the system. Therefore, application level filters (content-based) need to
42 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 7.9: gRPC Adapter / Push down subscriptions - Sequence Diagram be implemented between the brokers and the clients. By adding these filters, the current publishing infrastructure is retained. Nevertheless, adding only a filter is not enough. There should be a module in the filter which can differentiate the events. As an example, if we want to filter out events that a client has already received, we cannot do it based on the data alone. It would be perfectly valid for the same system state to occur multiple times, but it means that clients who have received the first occurrence, need to know about all the future occurrences of the same state.
Timestamp For filtering, if the time of the events is known and if it is possible to record for each client what the timestamp was of the last event they have received, it is possible to discard any events
43 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 7.10: gRPC Adapter with event filters - Component Diagram that are older than that. Currently, this is not possible because the timestamp that is included in the gRPC container format is not based on the occurrence of the event, but instead on its creation. When the Eventing component reconnects to the gRPC stream and it receives the current state, it does not know when these changes have occurred and it will add a new timestamp. To make this feature feasible, it is required to add timestamps of the event’s occurrence as an extension of the gRPC interface. The advantages of this approach are:
• The client is agnostic of the recovery process and the infrastructure guarantees that the client is able to catch up.
• Only clients that are not in sync receive the additional events that are needed to recover.
The disadvantages of this approach are:
• Bridge data store needs to be extended with timestamps per attribute which are exposed through bridge app /gRPC.
• Requires the brokers to be coupled to the event data format (The broker needs to be coupled anyway for dynamic level subscription).
• Requires a separate API for the multiplexed local/cloud subscription.
The flow is explained in Figure 7.11. Different clients subscribe to Nchan which notifies the Eventing component to subscribe to gRPC. Then gRPC parses the events and send them through the Data Model Parser to Nchan which works as an protocol adapter to SSE. Moreover, there are some filtering processes to handle the events to be sent to the related clients.
Overall Comparison between Potential Bridge Improvements
Table 7.2 shows the comparison table among four options. It is realizable from the Table that Option2 has a built-in recovery mechanism in gRPC and also wildcard matching module. Additionally, it needs to have the dynamic subscription capability. Therefore, Option 3 has better memory optimization and recovery mechanisms for the application eventing compared to the other ones.
44 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 7.11: gRPC Adapter with event filters - Sequence Diagram
45 Eventing in the Hue System Technische Universiteit Eindhoven Table 7.2: Bridge Improvements Comparison Table
46 Eventing in the Hue System Technische Universiteit Eindhoven
8 Implementation
The previous two chapters analyze the system architecture and design. This chapter explains the realization of the design, specifically the proof of concept. The implementation is based on the archi- tecture and design, elaborated in the corresponding chapters, and satisfies the functional requirements defined in the “System requirements” chapter. The implementation provided a demonstration of the feasibility of the project. Three products are implemented to prove the three parts of the design alternatives:
• Eventing in the bridge with Nchan
• Fine-grained subscription
• Dynamic group subscription
8.1 Eventing in the Bridge with Nchan
This prototype is for examining the feasibility of eventing within the bridge. For this part as is shown in the Figure 8.1, the Eventing Daemon and Nchan are the new components that are added to the bridge to make this prototype work.
Figure 8.1: Eventing in the Bridge with Nchan - Demo Architecture
To see the flow (8.2), the client, which here is the web app, subscribes to the general endpoint. The events come from lights to the gRPC and from there they propagate to Eventing Daemon. This module publishes these events to the Nchan. In Nchan, since it already has the Subscriber channel, it emits the events over SSE.
47 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 8.2: Eventing in the Bridge with Nchan - Sequence Diagram
Snippet 8.3 shows the data is logged in the console as a result of eventing. There are light-state attributes such as brightness, x-y, ct and also configuration attributes including name and service IDs.
48 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 8.3: Eventing in the Bridge with Nchan - Console snippet
49 Eventing in the Hue System Technische Universiteit Eindhoven
8.2 Fine-grained Subscription with Wildcard Support
As a consequence of the limited thesis scope, the investigated feature is implemented outside of the bridge device in order to prove functionality. Figure 8.4 shows the overview architecture of the prototype. The reverse proxy is used to forward the data from the bridge to the desktop broker which is implemented with Node.js. The desktop broker has the important functionalities of Nchan to simulate it’s behavior. In addition to that, there is a wildcard module which has been implemented inside to parse the topics and map the relevant event for it. For testing purposes, the wildcard module is being tested independently with unit tests and the whole prototype is being tested by scenario based testing.
Figure 8.4: Fine-grained Subscription - Demo Architecture
The sequence diagram 8.5 illustrates the flow of the eventing. The clients subscribe to events on the Desktop broker.Then events come from lights to gRPC and from there to the Data Model parser which translates the data to the proper data model. From there, the events are published to the Nginx which works as a reverse proxy. Nginx propagates the events to the Desktop broker. In the broker, the topics are parsed and the events are sent to the proper channels of the clients. Besides that, the latest states of the attributes are stored in the Desktop broker. Here is the snippet of the Client App 8.6. For demo purposes three lights are used. As it is demon- strated the app is subscribed to all the attributes of light 1, one/off state for all lights, brightness of light 3 and color changes of light2. In this example the top two are the wildcard subscription and the bottom two are per attributes subscriptions. The moment that there is an event from the lights, they will pop up to the relevant text boxes in the client application.
50 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 8.5: Fine-grained Subscription - Sequence Diagram
51 Eventing in the Hue System Technische Universiteit Eindhoven Figure 8.6: Fine-grained Subscription - Client App Snippet
52 Eventing in the Hue System Technische Universiteit Eindhoven
8.3 Dynamic Group Subscription with Graph Structure
It is mentioned in the previous chapter that dynamic group subscription gives the ability to subscribe by creating different level relations for topics. For instance, the user can create a group 1 which contains light 1 and light 2 and then create another group and assign light 3 to this new group and group 1 at the same time. This example proves the possibility to have a graph structure within different resource. To create this demo, because of the limited time, the whole experience is out of the bridge with a desktop broker and a publisher. Publisher is a server that keeps sending data with different topics to the desktop broker. Figure 8.7 demonstrates the components that are used for this PoC. In the Data Model Structure component the relations are kept.
Figure 8.7: Dynamic Group Subscription - Demo Architecture
The sequence diagram 8.8 shows the flow of the eventing. The clients subscribe to events on Desktop broker.Then generated events come from the Publisher to the Desktop broker. In the broker, the topics are parsed and the events are sent to the proper channels of the clients. Besides that, the latest states of the attributes are stored in the Desktop broker. The Data Model Structure which stores the relation between different resource levels sends an update in case of any change in the resources. Here is the snippet of the Client App 8.10. For demo purposes generates events for three lights and three sensors. As it is demonstrated the app is subscribed to all the lights of group 1 and group 2, sensor 3’s name and all names of all the sensors in group 2. Figure 8.9 demonstrates the relations between different resources. As it is displayed, the light 2 is part of group 1 and group 2. Therefore, it is shown in the snippet witch highlighted color that light 2 is appeared in both groups.
53 Eventing in the Hue System Technische Universiteit Eindhoven
Figure 8.8: Dynamic Group Subscription - Sequence Diagram
Figure 8.9: Dynamic Group Subscription for the Demo App
54 Eventing in the Hue System Technische Universiteit Eindhoven Figure 8.10: Dynamic Group Subscription - Client App Snippet
55 Eventing in the Hue System Technische Universiteit Eindhoven
9 Validation and Verification
The previous chapter elucidated the project’s implementation. To ensure that the proposed system has met the non-functional requirements, the process of verification and validation should be put in place. This chapter describes the process together with the techniques used.
9.1 Validation
Validation of a software product is an activity that ensures that an end product stakeholder’s true needs and expectations are met. [23]. Two aspects need to be taken into consideration in the validation process. First, does the product built addresses the functional requirements; does it do what it says in the functional requirements? Second, does the product satisfy the non-functional requirements? For the functional requirements, the validation should address whether the product offers the features defined. For the non-functional requirements, the validation should address how they are achieved and satisfied. Following an iterative development approach, the results were continuously validated by the stake- holders. Once per week, a meeting was scheduled with the main stakeholders, which included a progress report and a demo of the current version of the product. During the demo, feedback was provided on the completeness and correctness of the product under development.
Figure 9.1: Benchmark Architecture Diagram
56 Eventing in the Hue System Technische Universiteit Eindhoven
Regarding the non-functional requirements, the efficiency and reliability are identified as the most important ones. To evaluate them, a benchmark 9.1 is created to examine the different eventing technologies and validate them with the fact that they have the latency that is met with non-functional requirements. This benchmark is generated based on different payloads, throughput, number of events and endpoint regions. The simulation environment is a laptop which sends events to the different endpoints and calculates the latency . As it is indicated, the average latency is approximately three times lower than the target latency men- tioned in non-functional requirements. Although most of the results are promising, some of them are anomalies (written in red). This is happening because of the naive implementation of the eventing approach or the small number of experiments.
Table 9.1: Benchmark results for the SSE approach
57 Eventing in the Hue System Technische Universiteit Eindhoven
9.2 Verification
Verification of a software product is a test of a system to prove that it meets all its specified require- ments at a particular stage of its development [23]. Besides that, it is the process of evaluating whether the system is engineered properly. This means that verification should check how well the product is developed, tested, and documented. To ensure the quality of the product, verification is performed by software testing and code quality measurements. Considering the test plan framework is split into two parts, namely web application – presentation layer (subscriber) and business logic layer (broker) – different testing is applied to them. In the web application, the user interface needs to be tested, as well as the code, and in the business logic layer the code needs to be tested. The code in the broker is mainly tested manually, by running specific scenarios, as well as by perform- ing unit tests on specific functions in the code. The testing environment consists of one server running all layers of the system. The Operating System for this server was Linux Ubuntu 16.04 LTS[24]. On the user interface, manual testing was conducted. The user interface was tested with the following browsers: Google Chrome (version 60.0.3112.90) [25], Mozilla Firefox (version 54.0.1) [26]. To check if the prototypes meet the requirements from Table 5.1 and 5.2, lots of scenario based testing was performed to verify the functionalities and it is demonstrated in snippets in the Chapter 8.
58 Eventing in the Hue System Technische Universiteit Eindhoven
10 Conclusion
This chapter analyses the results achieved by this project as well as the added value to the stakeholders. It also gives recommendations for the future work.
10.1 Results
The main part of the Hue system is the Hue bridge which controls and monitors Hue lamps, sensors, and switches. In order to communicate, the Hue bridge uses polling to trigger actions. Essentially, the pattern of having a client routinely call an endpoint for new data, called “polling”, is wasteful in terms of:
• resources committed to the action from the developer
• the traffic seen by the vendors
• the actual result to effort
In the Hue system, polling generates loads of traffic, consumes bandwidth and adds an extra process to discover changes by comparing the current state with the previous one. To address these drawbacks, Eventing approach is suggested. Eventing helps the system with scaling, lowering the latency, and optimizing the process by sending the delta messages. This project has several outputs:
• indicate a clear overview of different eventing design alternatives, compare them based on sig- nificant factors, including latency, recoverability, reliability, memory optimization and ease of implementation. It also demonstrates architectural diagrams to design eventing in the bridge and the Hue cloud.
• prove the concept of the eventing by showing the main features of the it. Moreover, satisfy the high-level functional requirements by getting events from state/ configuration changes in lights. This is very beneficial for IoT industry and smart home ecosystem.
• prove the concept of fine-grained subscription with wildcard support and feasibility of it. This feature can give the user a tremendous amount of flexibility to control their lights.
• prove the concept of dynamic group subscription with multi-paths, which gives the eventing system to be adjusted in real-time based on group changes and also provides the system to have graph structure relations between resources.
59 Eventing in the Hue System Technische Universiteit Eindhoven
10.2 Future Work
The last part of the outcomes discusses future possibilities or recommendations for the eventing prod- uct that were identified along the way as well as the implementations that was not performed due to various constraints, such as scope and time.
• The main recommendation is mentioned in the section which describes the possible improve- ments in eventing architecture in the bridge in order to optimize the memory and lower the latency. The gRPC adapter with push down subscription and the gRPC adapter with various eventing filters are two suggested architectures. The main idea is to reuse the functionalities of the gRPC as the main broker and prevent the events from getting duplicated or stored multiple times.
• Add state synchronization. Upgrade the bridge architecture to handle recovery mechanism and synchronizations.
• Add eventing throttling. To manage the expected load, the system needs to manage the amount of events that a bridge can send to the cloud. This approach would not have a negative impact on the eventing latency when the system is in regular use. Besides that, the system should stay in sync after throttling kicks in. Moreover, it should preservers the order of the events
• Integrate all the Hue API resources to integrate with eventing. For instance, scenes, light groups, and rooms.
• Design and implement the security mechanisms for eventing. Currently, there is no mechanism to authorize or authenticates the clients to subscribe for the events. Besides that, there can be lots of new use cases for securing the fine-grained subscription feature. This can be done by designing a fine-grained access control on fine-grained subscriptions. For instance, a user wants to give the ability to guests to just subscribe to lights of a specific room.
60 Eventing in the Hue System Technische Universiteit Eindhoven
Glossary
API Application Programming Interface CLIP Connected Lighting Interface Protocol GRPC Google’s Remote Procedure Call HTML Hyper Text Markup Language IoT The Internet of Things IP Internet Protocol JSON JavaScript Object Notation KPI Key Performance Indicator LAN Local Area Network M2M Machine to Machine MQTT Message Queuing Telemetry Transport OOTI Onwerpersopleiding Technische Informatica PDEng Professional Doctorate in Engineering PoC Proof of Concept PSG Project Steering Group QoS Quality of Service REST Representational State Transfer (architectural style) RESTful APIs Interfaces that adhere to the REST style SAI Stan Ackermans Institute SDK Software Development Kit SSE Server-sent events ST Software Technology TU/e Eindhoven University of Technology URL Uniform Resource Locator WSCD Websocket Client Daemon
61 Eventing in the Hue System Technische Universiteit Eindhoven
Bibliography
[1] Roy T Fielding and Richard N Taylor. Architectural styles and the design of network-based software architectures, volume 7. University of California, Irvine Doctoral dissertation, 2000. [2] Jianfeng Wang. Zigbee light link and its applicationss. IEEE Wireless Communications, 20(4):6– 7, 2013. [3] Leonard Richardson and Sam Ruby. RESTful web services. " O’Reilly Media, Inc.", 2008. [4] TU/e. Software technology pdeng programme. https://www.tue.nl/en/education/ graduate-school/pdeng-software-technology/, 2018. [Online; accessed 2018- 02-30]. [5] Miguel Castro, Antonio J Jara, and Antonio FG Skarmeta. Smart lighting solutions for smart cities. In Advanced information networking and applications workshops (WAINA), 2013 27th international conference on, pages 1374–1379. IEEE, 2013. [6] Philips Company. Philips. https://www.philips.com/global, 2018. [Online; ac- cessed 2018-08-01]. [7] Signify. Signify. https://www.signify.com/nl-nl, 2018. [Online; accessed 2018-05- 30]. [8] Signify. Philips hue system. https://www2.meethue.com/en-us, 2018. [Online; ac- cessed 2018-03-30]. [9] Google. grpc. https://grpc.io/, 2018. [Online; accessed 2018-03-30]. [10] Kenton Varda. Protocol buffers: Google’s data interchange format. Google Open Source Blog, Available at least as early as Jul, 72, 2008. [11] Zhi Guo Gao, Yun Zhang Pei, Chen Wang, and Bo Yang. Information polling method, apparatus and system, June 19 2012. US Patent 8,204,032. [12] Brenda M Michelson. Event-driven architecture overview. Patricia Seybold Group, 2(12):10– 1571, 2006. [13] Urs Hunkeler, Hong Linh Truong, and Andy Stanford-Clark. Mqtt-s—a publish/subscribe pro- tocol for wireless sensor networks. In Communication systems software and middleware and workshops, 2008. comsware 2008. 3rd international conference on, pages 791–798. IEEE, 2008. [14] Brian D Goodman, Frank Jania, Konrad C Lagarde, Chen Shu, and Michael Van Der Meulen. Pub/sub message invoking a subscribers client application program, February 15 2011. US Patent 7,890,572.
62 Eventing in the Hue System Technische Universiteit Eindhoven
[15] Zheng Wang and Jon Crowcroft. Quality-of-service routing for supporting multimedia applica- tions. IEEE Journal on selected areas in communications, 14(7):1228–1234, 1996.
[16] Server sent events. https://html.spec.whatwg.org/multipage/ server-sent-events.html#server-sent-events, 2018. [Online; accessed 2018-03-30].
[17] World Wide Web Consortium. Websub. https://www.w3.org/TR/websub/, 2018. [On- line; accessed 2018-04-01].
[18] Stephen Fink, Hoang Anh Le, Vinod Muthusamy, Rodric Rabbah, and Jeremias Werner. Man- aging external feeds in an event-based computing system, August 24 2017. US Patent App. 15/374,201.
[19] S. Brander. Key words for use in rfcs to indicate requirement levels, 1997. Harvard University, Network Working Group.
[20] Jean Bacon, David M Eyers, Jatinder Singh, and Peter R Pietzuch. Access control in pub- lish/subscribe systems. In Proceedings of the second international conference on Distributed event-based systems, pages 23–34. ACM, 2008.
[21] Leo Ponomarev. Nchan. https://nchan.io/, 2018. [Online; accessed 2018-05-30].
[22] Urs Hunkeler, Hong Linh Truong, and Andy Stanford-Clark. Mqtt-s—a publish/subscribe pro- tocol for wireless sensor networks. In Communication systems software and middleware and workshops, 2008. comsware 2008. 3rd international conference on, pages 791–798. IEEE, 2008.
[23] Janet M Twomey and Alice E Smith. Validation and verification, 1997.
[24] Abraham Silberschatz, Peter Baer Galvin, and Greg Gagne. Operating system concepts essen- tials. John Wiley & Sons, Inc., 2014.
[25] Google. Chrome browser. https://www.google.com/chrome/, 2018. [Online; ac- cessed 2018-06-26].
[26] Mozilla. Firefox browser. https://www.mozilla.org/en-US/firefox/, 2018. [On- line; accessed 2018-07-01].
63 Eventing in the Hue System Technische Universiteit Eindhoven
A Project Management
A.1 Introduction
As for every project, its success is tightly connected with its management. Project management is an absolutely essential process of every project and the challenge it aims to tackle is the achievement of all the project's goals within the given constraints. Such constraints can refer to the available budget, time, and people. Projects can be seen as time-variant systems, as their parameters change during their execution. There- fore, it is important to monitor the project's progress constantly and make readjustments to the initial parameters. This task is part of the project's management.
A.2 The Process of Management
An important task for the project's trainee was to organize the project's management. The project management included the creation of an execution plan, the monitoring of the project's progress, and the briefing of the project's stakeholders regarding the progress. As a result, the trainee worked proactively regarding the project's management by creating specific proposals. These proposals were presented to the Project Steering Group, which in the end accepted or declined each proposal.
A.3 Project Planning and Scheduling
The method is adjusted to the needs of the project having:
• Weekly progress meetings with the two company supervisors.
• Monthly Progress Steering Group (PSG) meetings with the two company supervisors and the TU/e mentor
• Performance Evaluations every three months by the PSG members.
• Biweekly meetings with the Cloud and Partnership teams to give an update about ongoing development
• Working with Kanban framework for incremental and agile part
• Working with TeamGantt framework for the overall project time-line
64 Eventing in the Hue System Technische Universiteit Eindhoven 1 24 17 9/18 10 3 27 20 8/18 13 6 30 23 16 7/18 9 2 25 18 6/18 11 4 28 21 14 5/18 7 30 23 16 4/18 9 2 26 19 3/18 12 5 26 19 12 2/18 5 29 22 Figure A.1: Snapshot of project's time-line 15 1/18 8 3 0% 0% 0% 90% 90% 70% 90% 80% 60% 70% 80% 86% 94% 71% 84% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0h 0h 0h 0h 0h 0h 0h 0h 0h Eventing Project Inititation Kickoff meeting Technology Exploration Meeting with Stakeholders Architecture and Design Design Alternatives System Architecture Prototyping Setting up Hue System environment ... PoC Testing and Validation Validating the prototype (Benchmark... Final Deliverables Documentation Stakeholder Management Problem Analysis Risk Analysis Requirement Document Domain Analysis Design Alternatives System Architecture Solution and Results Benchmarking Results Presentation Final Edit based on Feedback Powered by TCPDF (www.tcpdf.org)
65 Eventing in the Hue System Technische Universiteit Eindhoven
A.4 Evaluation and Execution
The execution of the project followed a structured path based on the project planning. The first two months were mainly meetings with stakeholders and reading literature. They also included getting to know the high-level requirements of the stakeholders and refining them. For the next three months, the project was focused on different design alternatives for eventing architecture in the Hue system. After that, the project refocused on the application eventing for local users. Finally, the last two months, was wrapping up the project and finalizing the documentation. Following the agile approach of iterative software development, every month a Project Steering Group (PSG) meeting was held where the progress of the project was presented by the trainee and the direc- tion of the project was redefined. During these meetings, the trainee presented the current status as well as parts of the final deliverable. Apart from the evaluation of the project's goals, the PSG (excluding the trainee) evaluated the trainee's performance in three phases. Each phase of evaluation was performed on a three-month time frame. The trainee was evaluated against specific criteria, which were predefined by TU/e.
66 Eventing in the Hue System Technische Universiteit Eindhoven
B About the Authors
Pouya Samadikhah received his Bache- lor’s degree from the University of Tehran, Iran in 2013 in Computer Science. Then, he joined the EIT Digital master program and received his double degree master of science in distributed systems and services from Aalto University, Finland, and KTH Royal Institute of Technology, Sweden in 2016. During his masters, he worked as a re- search assistant in Ericsson Research where he has done his final thesis project in the cloud-computing department. His thesis was about the performance modeling of the vari- ous components of OpenStack in function of different operating conditions. The result of this project has been presented at OpenStack Summit 2015 in Toronto. He also worked as a Full-Stack software intern for a year in a startup company in the San Francisco bay area, United States. He worked with Python as back-end part and AngularJS as front-end part. From October 2016 until Oc- tober 2018, he worked at the Eindhoven Uni- versity of Technology, as PDEng trainee in the Software Technology program from the 4TU.Stan Ackermans Institute.
67 Eventing in the Hue System
Where innovation starts