OGC Standards for Raster Data Management
Total Page:16
File Type:pdf, Size:1020Kb
A Prototype Infrastructure for Sentinel Earth Observation Data Relative to Portugal Supporting OGC Standards for Raster Data Management Diogo Filipe Pimenta Andrade Thesis to obtain the Master of Science Degree in Information Systems and Computer Engineering Supervisor(s): Prof. Bruno Emanuel da Grac¸a Martins Prof. Mario´ Jorge Costa Gaspar da Silva Examinatiom Committee Chairperson: Prof. Daniel Jorge Viegas Gonc¸alves Supervisor: Prof. Prof. Bruno Emanuel da Grac¸a Martins Members of the Committe: Prof. Armanda Rodrigues May 2017 The only way to achieve the impossible is to believe it is possible. Charles Lutwidge Dodgson Acknowledgments I would like to thank those who have been present and have in a way or another contributed to this thesis. First, I would like to thank my advisors, Prof. Bruno Martins and Prof. Mario´ Gaspar, for all the guidance, patience, and support. I would also like to thank Eng. Bruno Anjos and Eng. Marco Silva from Instituto Portuguesˆ do Mar e da Atmosfera (IPMA) for the technical support. I also thank the future mother of my children, Barbara´ Agua,´ for always being kind and supportive. I can not thank her enough for all the encouragement and dedication. I thank my good friend Joao˜ Vieira, for the inspiration and motivation that brought me to pursue higher education. I would like to thank my brother, Joao˜ Andrade, for being the father that I never had. Finally, I would like to thank the taxpayers of Portugal for providing me with financial support, through a scholarship from Governo Regional da Madeira, through the SASUL scholarship, and through the researh grant from the Fundac¸ao˜ para a Cienciaˆ e Tecnologia (FCT) within project DataS- torm - EXCL/EEI-ESS/0257/2012. I also would like to express my gratitude to the Republica´ ”A Desordem dos Engenheiros”, for all the moments, for helping me to grow as a person, for being my second family, and for facilitating my life in an economic way. I will never forget and will be always grateful. iii Abstract This document describes an architecture for the IPSentinel infrastructure for managing Earth ob- servation data together with its implementation. This infrastructure was developed using the DHuS software from the European Space Agency together with the RasDaMan array database manage- ment system, to catalogue, disseminate and process Sentinel Earth observation products for the Portuguese territory. RasDaMan implements standards from the Open Geospatial Consortium, such as Web Coverage Service and Web Coverage Processing Service, which provide access and pro- cessing of the rasters encoding Earth observation data, through the Internet. The reported experiments show that the prototype system meets the functional requirements. This dissertation also provides measurements of the used computational resources, in terms of storage space and response times. Keywords Remote Sensing Products, Earth Observation Data, Raster Data, OGC Standards, Geospatial Data Infrastructures, Big Data Management v Resumo Este documento descreve a arquitectura e a implementac¸ao˜ de um prototipo´ da infraestrutura IP Sentinel para a gestao˜ de produtos de observac¸ao˜ da Terra. Esta infraestrutura foi desenvolvida usando o software DHuS da Agenciaˆ Espacial Europeia, juntamente com o sistema de gestao˜ de base de dados de arrays RasDaMan. Esta infraestrutura tem como objectivo catalogar, disseminar e processar produtos de observac¸ao˜ da Terra Sentinel relativos ao territorio´ Portugues.ˆ O RasDaMan implementa normas do Open Geospatial Consortium, tais como as normas Web Coverage Service and Web Coverage Processing Service, que disponibilizam o acesso e o processamento de rasters que codificam os dados de obvservac¸ao˜ da Terra, atraves´ da Internet. As experienciasˆ relizadas mostram que o prototipo´ cumpre os requisitos funcionais. Esta disserta- c¸ao˜ fornece tambem´ algumas medic¸oes˜ dos recursos computacionais utilizados, em termos do espac¸o de armazenamento e do tempo de resposta. Palavras Chave Detecc¸ao˜ Remota, Dados de Observac¸ao˜ da Terra, Dados Raster, Especificac¸oes˜ OGC, Infras- truturas para Dados Geoespaciais, Gestao˜ de Megadados vii Contents 1 Introduction 1 1.1 Thesis Proposal . .2 1.2 Summary of Contributions . .3 1.3 Document Organization . .3 2 Concepts and Related Work 5 2.1 Concepts . .6 2.1.1 The Sentinel Programme . .6 2.1.2 Computational Storage of EO Rasters . .7 2.1.3 Array Database Managing Services . .9 2.1.3.A Array Algebra . 10 2.1.4 Open Geospatial Consortium Standards . 11 2.1.4.A OGC Web Coverage Service . 13 2.1.4.B OGC Web Coverage Processing Service . 14 2.1.4.C OGC Web Map Service . 15 2.2 Related Work . 16 2.2.1 The ESA Data Hub Service . 17 2.2.2 The RasDaMan Array Database Management System . 21 2.2.3 The SciDB Array Database Management System . 23 2.2.4 Handing Raster data in MonetDB and the TELEIOS Infrastructure . 26 2.3 Summary and Discussion . 28 3 Prototype Infrastructure 31 3.1 Overview . 32 3.2 System Assumptions and Functional Requirements . 33 3.3 Architecture . 34 3.4 Implementation . 39 3.4.1 Integration of Petascope with the Data Hub Service (DHuS) software . 39 3.4.2 Modification of Petascope and DHuS Web Clients . 39 3.4.3 Modification of the DHuS Core to Automate the Process of Ingesting Coverages in RasDaMan . 41 3.5 Summary . 42 ix 4 Validation 45 4.1 Requirements Compliance . 46 4.2 Measurement of Required Computational Resources . 48 4.2.1 Storage Space . 48 4.2.2 Response Time . 49 4.3 Summary . 54 5 Conclusions and Future Work 57 5.1 Conclusions . 58 5.2 Future Work . 58 Bibliography 61 x List of Figures 2.1 The two GeoTIFF raster coordinate systems [Ritter and Ruth, 1997]. .7 2.2 The GeoTIFF raster, map, and world space [Ritter and Ruth, 1997]. .8 2.3 Example of GeoTIFF metadata parameter set [Ritter and Ruth, 1997]. .9 2.4 Constituents of an array. 10 2.5 Examples of 2D rectified (1 and 2) and referenceable (3 and 4) grids. 12 2.6 Dissemination of Earth Observation (EO) data from a centralised Data Hub. 17 2.7 Related view on the DHuS architecture centered around the Spring Framework [Fred´ eric´ Pi- dancier and Mbaye, 2014]. 18 2.8 The DHuS graphical user interface. 20 2.9 The DHuS service architecture [Fred´ eric´ Pidancier and Mbaye, 2014]. 21 2.10 RasDaMan infrastructure and its Petascope and SECORE components for exposing arrays through OGC web services. 23 2.11 Some SciQL array operations. 26 2.12 TELEIOS Infrascture Overview. 28 3.1 Major functionalities of the IPSentinel prototype. 32 3.2 Overview of interactions of the prototype with ESA SciHub and users. 34 3.3 Overview of the modified DHuS project. 35 3.4 Available APIs in the DHuS. 35 3.5 Component & connector allocated-to files view over the Tomcat and web applications. 36 3.6 IPSentinel context diagram. 36 3.7 Component & connector view from the DHuS core. 37 3.8 Component & connector view with pipe and filter style from the Rasdaman Feeder. 37 3.9 Petascope web client with GetMap tab open. 40 3.10 A product listed with Open Geospatial Consortium (OGC) services availability checked. 40 3.11 Product details screen displaying OGC service buttons. 41 3.12 Convention structure for the Sentinel1 products name . 42 3.13 Diagram of involved java classes in the implementation. 43 4.1 Result of executing a getCoverage of Sentinel 1 image with only VV band selected. 47 4.2 Result of executing a false colouring processing through the Web Coverage Processing Service (WCPS) query language. 47 xi 4.3 Result of executing multiples getMap requests. 47 4.4 Scenario 2: Regions selected by the 4 clients. 50 4.5 Results produced by the queries in the Table 4.8. 53 4.6 Result produced by the query in the Listing 4.7. 54 xii List of Tables 2.1 The parameters of a GetCapabilities request. 14 2.2 The parameters of a DescribeCoverage request. 15 2.3 The parameters of a GetCoverage request. 15 2.4 The parameters of a GetCapabilities request. 15 2.5 The parameters of a GetMap request. 16 2.6 Examples of operations supported by RasQL . 22 2.7 Two dimensional SciDB array. 24 2.8 The matrix stored as three BATs. 27 4.1 Space occupied by Sentinel 1 products on disk. 48 4.2 Space occupied by GRD type products in RasDaMan. 49 4.3 Storage space required for different rolling archive plans. 49 4.4 Scenario 1: Response time . 50 4.5 Scenario 2: Response time . 50 4.6 Scenario 3: Response time . 50 4.7 Resume of response times of the WCS operations. 51 4.8 Summary of response time from different queries. 52 xiii Abbreviations AFL Array Functional Language AQL Array Query Language BAT Binary Association Table DAOs Data Access Objects DBMS Database Management System DGT Direcc¸ao˜ Geral do Territorio´ DHuS Data Hub Service CRS Coordinate Reference Systems EO Earth Observation EPSG European Petroleum Survey Group ESA European Space Agency EW Extra Wide swath GIS Geographic Information Systems GML Geography Markup Language GRD Ground Range Detected GUI graphical user interface HSQLDB HyperSQL Database IPMA Instituto Portuguesˆ do Mar e da Atmosfera IW Interferometric Wide swath MDD multidimensional discrete data OData Open Data Protocol OGC Open Geospatial Consortium xv POM Project Object Model SLC Single Look Complex URL Uniform Resource Locator UTM Universal Transverse Mercator WAR Web application ARchive WCPS Web Coverage Processing Service WCS Web Coverage Service WMS Web Map Service xvi 1 Introduction Contents 1.1 Thesis Proposal . .2 1.2 Summary of Contributions . .3 1.3 Document Organization . ..