An Open Geospatial Internet of Things Cloud Service Architecture Based on the Big Data Lambda Architecture
Total Page:16
File Type:pdf, Size:1020Kb
University of Calgary PRISM: University of Calgary's Digital Repository Graduate Studies The Vault: Electronic Theses and Dissertations 2018-12-20 An Open Geospatial Internet of Things Cloud Service Architecture Based on the Big Data Lambda Architecture Khalafbeigi, Tania Khalafbeigi, T. (2018). An Open Geospatial Internet of Things Cloud Service Architecture Based on the Big Data Lambda Architecture (Unpublished doctoral thesis). University of Calgary, Calgary, AB. http://hdl.handle.net/1880/109398 doctoral thesis University of Calgary graduate students retain copyright ownership and moral rights for their thesis. You may use this material in any way that is permitted by the Copyright Act or through licensing that has been assigned to the document. For uses that are not allowable under copyright legislation or licensing, you are required to seek permission. Downloaded from PRISM: https://prism.ucalgary.ca UNIVERSITY OF CALGARY An Open Geospatial Internet of Things Cloud Service Architecture Based on the Big Data Lambda Architecture by Tania Khalafbeigi A THESIS SUBMITTED TO THE FACULTY OF GRADUATE STUDIES IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY GRADUATE PROGRAM IN GEOMATICS ENGINEERING CALGARY, ALBERTA DECEMBER, 2018 © Tania Khalafbeigi 2018 Abstract The Internet of Things (IoT) consists of sensors and actuators embedded in everyday devices interconnecting and communicating through interoperable information and communication technologies. The real potential of IoT is in creating innovative applications by integrating and repurposing IoT sensing and controlling capabilities from different sources. However, proprietary IoT systems now create silos that make the IoT goal almost unreachable as the applications need to deal with heterogeneous data from different systems. In addition to the problem of heterogeneity, big data is a challenge for all technologies in the modern world. As predicted by CISCO and IDC, the number of internet-connected objects will reach at least 50 billion by 2020. As a result, IoT is facing heterogeneity and big data challenges including volume and velocity. We have proposed an architecture for IoT with the focus on data management challenges in this dissertation. The proposed architecture merges the Lambda architecture with the SensorThings API. The SensorThings API is used as a solution for the heterogeneity problem. One of the solutions for data heterogeneity or so-called interoperability in IoT is the use of a standard API. SensorThings API has been proven to be a mature, open geospatial standard for IoT by various literature, implementations, and its widespread adoption. Moreover, the Lambda architecture addresses big data volume and velocity challenges through the use of three layers architecture: batch, serving, and speed. We implemented a case study of our proposed architecture with real air quality data. For our implementation, we used Hadoop and Azure technologies. Our case study showed that our proposed architecture significantly improves the performance of IoT service on real-world big open data. Keywords: Internet of Things, Big Data, SensorThings API, Lambda Architecture ii Acknowledgements I would like to thank the following individuals who supported me and did not leave me alone in all aspects of my research: • Dr Steve Liang for his guidance, financial support, and excellent supervision. • Dr Xin Wang, Dr Reda Alhajj, Dr Mea Wang, and Dr Kyle O'Keefe for providing useful comments about my research during my candidacy exam. • Charlene K for helping me polishing the English of this dissertation • Mahdi, my husband, and also my parents for their spiritual support. I would also want to thank my colleagues in GeoSensorWeb lab for all their help when we worked on SensorThings API design, especially Alec Huang and Mohammad Jazayeri. Last, but not least, I would like to thank SensorUp Inc. for sponsoring my implementation on Microsoft Azure. iii Table of Contents Abstract ............................................................................................................................... ii Acknowledgements ............................................................................................................ iii Table of Contents ............................................................................................................... iv List of Tables .................................................................................................................... vii List of Figures and Illustrations ....................................................................................... viii List of Symbols, Abbreviations and Nomenclature .............................................................x INTRODUCTION ...............................................................................................................1 Challenges of Data Management in IoT ..........................................................................3 IoT Big Data Management .........................................................................................3 Data Volume ........................................................................................................4 Data Velocity .......................................................................................................5 Data Variety .........................................................................................................5 Objectives and Proposed Solutions ..................................................................................6 Objective 1: To Propose a Solution for the Data Variety Challenge for IoT .............6 Objective 2: To Propose a Solution for the Data Volume Challenge for IoT ............9 Objective 3: To Propose a Solution for the Data Velocity Challenge for IoT ...........9 SENSORTHINGS API, DETAILS AND DESIGN DECISIONS ....................................12 Introduction ....................................................................................................................12 OData and SensorThings ...............................................................................................16 IoT Reference Model and Place of SensorThings .........................................................16 Related Work .................................................................................................................18 IoT Interoperability Challenge .................................................................................19 Adaptor/Translator Solutions .............................................................................19 IoT Platforms and Standards ..............................................................................21 SensorThings API in Literature ................................................................................25 Data Model ....................................................................................................................29 SensorThings Application Interface ..............................................................................39 Data Retrieval ...........................................................................................................39 MQTT ................................................................................................................42 DataArray ...........................................................................................................43 Data Insertion and Modification ...............................................................................43 MQTT ................................................................................................................45 DataArray ...........................................................................................................45 Batch Requests Extension ..............................................................................................45 MultiDatastream Extension ...........................................................................................46 Summary ........................................................................................................................47 Future Work ...................................................................................................................48 DATA MANAGEMENT FOR INTERNET OF THINGS AND LAMBDA ARCHITECTURE ....................................................................................................50 Big Data and Internet of Things ....................................................................................50 Related Work .................................................................................................................54 Related Literature .....................................................................................................55 iv Big Data Platforms and Technologies ......................................................................64 Hadoop ...............................................................................................................65 Hortonworks ......................................................................................................66 Pivotal Big Data Suite ........................................................................................67 Cloudera Enterprise Data Hub ...........................................................................68 MapR .................................................................................................................69 Database