INSPIRE Coverages Publishing, Retrieving, Processing

Barcelona, Spain, 30-sep-2016 Alex Dumitru Jacobs University | rasdaman GmbH [gamingfeeds.com] Research supported by EU through EarthServer project rasdaman :: Barcelona 2016 :: ©2016 rasdaman Data Homogenization With OGC Standards

sensor feeds

coverage server

rasdaman :: Barcelona 2016 :: ©2016 rasdaman2 Ingestion / Publishing . How do I publish my data?

. Some common myths • “I have to write the GML files manually” • “The coverages should reflect the physical structure on disk” • “Data needs to be in GML format”

. Standardized ingestion through WCS-T • Extension to the WCS standard (OGC approved) • Takes care of metadata + data

rasdaman :: Barcelona 2016 :: ©2016 rasdaman Ingestion: standardized ingestion . WCS-T based tool for automatic ingestion & incremental coverage assembly, ex: • Set of 2-D images  single large 2-D coverage • Set of 2-D images  3-D timeseries datacube • Set of 3-D images  3-D timeseries datacube

. Flexible & convenient • reproducible ingestion procedures, easy to share across machines / institutions • extensible: developer tools provided, python support • solutions for incomplete metadata, misalignment, etc. • Most file types supported: GeoTIFF, NetCDF, GRIB, ASCII DEM, HDF etc

rasdaman :: Barcelona 2016 :: ©2016 rasdaman Web Coverage Service (WCS) . Coverages designed to work with WFS, WMS, WCS, WCPS, WPS, SOS, … • WCS specialized  most functionality specific to coverages

. WCS Core: access to spatio-temporal coverages & subsets • format encoding on the fly • subset = trim | slice Large, growing implementation basis: rasdaman, GDAL, QGIS, OpenLayers, OPeNDAP, MapServer, GeoServer, NASA WorldWind, EOx- Server; Pyxis, ERDAS, . WCS Extensions: optional functionality facets ArcGIS, ...

• Scaling, CRS transformation, … rasdaman :: Barcelona 2016 :: ©2016 rasdaman OGC Web Coverage Processing Service (WCPS) . = high-level spatio-temporal geo analytics language

[JacobsU, Fraunhofer; data courtesy BGS, ESA] . "From MODIS scenes M1, M2, M3: difference between red & nir, as TIFF" •…but only those where nir exceeds 127 somewhere for $c in ( M1, M2, M3 ) where some( $c.nir > 127 ) return (tiff , encode( A $c.red - $c.nir, tiffC) “image/tiff“ )

rasdaman :: Barcelona 2016 :: ©2016 rasdaman6 Q: W...whatever...S ? . from data-intensive to processing-intensive services

WMS WCS WCPS data visualization data access ad-hoc analytics

WCS Dogma: Never force the users to download more data than what they need rasdaman :: Barcelona 2016 :: ©2016 rasdaman rasdaman: Agile Array Analytics

. „raster data manager“: n-D arrays in SQL • [VLDB 1994, VLDB 1997, SIGMOD 1998, VLDB 2003, …]

. Array Algebra [NGITS 1998]

. Declarative, optimizable QL

. Scalable, parallel architecture • „tile streaming“

. Implemented & in use

rasdaman :: Barcelona 2016 :: ©2016 rasdaman Tiling

. Goal: faster tile loading by adapting storage units to access patterns

. Approach: partition n-D array into n-D partitions („tiles“)

. Tiling classification based on degree of alignment [ICDE 1999]

regular irregular partially aligned totally nonaligned

aligned nonaligned

rasdaman :: Barcelona 2016 :: ©2016 rasdaman Conclusion . OGC coverages as unifying concept for space/time grids & beyond

. OGC WCS: Functionality & flexibility • from simple extraction (WCS Core) to complex analytics (WCPS)

. Consensus across OGC, ISO, INSPIRE

. rasdaman: mature, scalability-proven, in use

[rasdaman screenshots] rasdaman :: Barcelona 2016 :: ©2016 rasdaman EarthServer: Datacubes at Your Fingertips . Operational Agile Analytics on 1+ Petabyte space/time datacubes • Earth & Planetary Science

. Based on & extending rasdaman

. Intercontinental initiative: EU + US + AUS

Phase 1 reviewers: "proven evidence" that rasdaman will “significantly transform […] access and use data“ …and "with no doubt has been shaping the Big Earth Data landscape” …

www.earthserver.eu INSPIRErasdaman WCS :: Barcelona :: ©2015 2016 P. :: Baumann ©2016 rasdaman