Spice Me Up: The Extended OGC Coverage Implementation Schema Peter Baumann Jacobs University | rasdaman GmbH [gamingfeeds.com] OGC CIS :: © 2017 Peter Baumann Features & Coverages . The basis of all: geographic feature . Special kind of feature: coverage - aka space-time varying phenomenon - regular & irregular grids, point clouds, meshes • Usually, coverages are Big Geo Data OGC CIS :: © 2017 Peter Baumann Coverages in INSPIRE . All raster data are coverages: Orthoimagery, elevation, thematic raster layers, weather, … - More generally: sensor, image [timeseries], simulation, statistics data - Can have space and/or time coordinates . Used in many INSPIRE themes: AC-MF, OF, ER, EL, NZ, OI, LU, LC, GE, SO - Data shall be provided using (mainly) the raster data spatial representation type – coverages - INSPIRE: „WCS is the natural way to provide coverages“ OGC CIS :: © 2017 Peter Baumann Big Datacube Standards . Open Geospatial Consortium (OGC) : - WCS „Big Geo Data“ standards suite - rasdaman Reference Implementation . ISO: - TC211: coverages & WCS • ISO 19123 19123-1 • OGC CIS 19123-2 • OGC WCS ISO WCS - SC32: SQL/MDA („Multi-Dimensional Arrays“) . INSPIRE: - coverages & WCS - rasdaman Reference Implementation OGC CIS :: © 2017 Peter Baumann Coverage Types - abstract: OGC Abstract Topic 6 = ISO 19123 - concrete, interoperable: Coverage Implementation Schema (CIS) ISO 19123-2 • aka GMLCOV • CIS = CIS 1.0 + CIS 1.1 «FeatureType» Abstract Coverage MultiSolid Grid Coverage CIS 1.1 Coverage MultiSurface Rectified Coverage GeneralGridCoverage Grid Coverage MultiCurve Referenceable Coverage GridCoverage MultiPoint Coverage OGC CIS :: © 2017 Peter Baumann Coverage Definition Coverage Implementation Schema «Feature Type» GML::Feature «Feature Type» Coverage 0..1 metadata domainSet rangeType rangeSet «Data Type» «Data Type» «Data Type» DomainSet SWE Common::DataRecord RangeSet [OGC 09-146r6] OGC CIS :: © 2017 Peter Baumann A Simple Coverage, in GML OGC CIS :: © 2017 Peter Baumann A Simple Coverage, in JSON { "type": "CoverageByDomainAndRangeType", "domainSet":{ "type": "DomainSetType", "generalGrid":{ "type": "GeneralGridCoverageType", "srsName": "http://www.opengis.net/def/crs/OGC/0/Index2D", "axisLabels": ["i", "j"], "axis": [{ "type": "IndexAxisType“, "axisLabel": "i", "lowerBound": 0, "upperBound": 2 },{ "type": "IndexAxisType“, "axisLabel": "j", "lowerBound": 0, "upperBound": 2 }] } }, "rangeSet": { "type": "RangeSetType", "dataBlock": { "type": "VDataBlockType", "values": [1,2,3,4,5,6,7,8,9] } }, "rangeType": { "type": "DataRecordType", "field":[{ "type": "QuantityType", "definition": "ogcType:unsignedInt", "uom": { "type": "UnitReference", "code": "10^0“ } }] } } OGC CIS :: © 2017 Peter Baumann A Simple Coverage, in RDF <http://www.opengis.net/cis/1.1/examples/CIS_05_2D> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.opengis.net/cis/1.1/CoverageByDomainAndRangeType> . <http://www.opengis.net/cis/1.1/examples/CIS_05_2D> <http://www.opengis.net/cis/1.1/domainSet> <http://www.opengis.net/cis/1.1/examples/CIS_DS_05_2D> . <http://www.opengis.net/cis/1.1/examples/CIS_DS_05_2D> <http://www.opengis.net/cis/1.1/generalGrid> <http://www.opengis.net/cis/1.1/examples/CIS_DS_GG_05_2D> . <http://www.opengis.net/cis/1.1/examples/CIS_DS_05_2D> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.opengis.net/cis/1.1/DomainSetType> . <http://www.opengis.net/cis/1.1/examples/CIS_DS_GG_05_2D> <http://www.opengis.net/cis/1.1/axis> <http://www.opengis.net/cis/1.1/examples/CIS_DS_GG_I_05_2D> . <http://www.opengis.net/cis/1.1/examples/CIS_DS_GG_05_2D> <http://www.opengis.net/cis/1.1/axis> <http://www.opengis.net/cis/1.1/examples/CIS_DS_GG_J_05_2D> . <http://www.opengis.net/cis/1.1/examples/CIS_DS_GG_05_2D> <http://www.opengis.net/cis/1.1/axisLabels> <http://www.opengis.net/cis/1.1/axisLabels0> . <http://www.opengis.net/cis/1.1/axisLabels0> <http://www.w3.org/1999/02/22-rdf-syntax-ns#first> "i" . <http://www.opengis.net/cis/1.1/axisLabels0> <http://www.w3.org/1999/02/22-rdf-syntax-ns#rest> <http://www.opengis.net/cis/1.1/axisLa <http://www.opengis.net/cis/1.1/axisLabels1> <http://www.w3.org/1999/02/22-rdf-syntax-ns#first> "j" . <http://www.opengis.net/cis/1.1/axisLabels1> <http://www.w3.org/1999/02/22-rdf-syntax-nsOGC#rest> CIS <http://www.w3.org/1999/02/22-rd :: © 2017 Peter Baumann f-syn Encoding Coverages Coverage Domain set Range type . Single file encoding: Range set - Informationally complete: GML, JSON, RDF, … App Metadata - Further formats: GeoTIFF, NetCDF, JPEG2000, GRIB, … . Multipart: container( “header” + file1 + file2 + … ) Coverage - Multipart/MIME, zip, GMLJP2, SAFE, GeoPackage, ... Domain set - Collections, tiling, streaming Range type xlink App Metadata NetCDF OGC CIS :: © 2017 Peter Baumann Classical Approach vs Datacubes t WaterML 2.0: time slices - Good performance in x/y - Disastrous performance on timeseries analysis - same with TimeseriesML OGC Coverages: Implementation can choose efficient layout - Same performance in all directions, space & time OGC CIS :: © 2017 Peter Baumann OGC Web Coverage Service (WCS) . WCS Core: access to spatio-temporal coverages & subsets - Encoding on the fly - subset = trim | slice Large, growing implementation basis: rasdaman, GDAL, QGIS, OpenLayers, OPeNDAP, MapServer, GeoServer, GMU, NASA WorldWind, EOx- Server; Pyxis, ERDAS, ArcGIS, ... WCS Extensions: optional functionality facets - from extraction up to flexible analytics . WCS Application Profiles: domain-oriented bundling OGC CIS :: © 2017 Peter Baumann WCS Suite Big Picture OGC CIS :: © 2017 Peter Baumann WCS Core GetCoverage . Download a coverage (or a subset thereof), values guaranteed unchanged . Ex: „download coverage c001“ http://www.acme.com/wcs ? SERVICE=WCS & VERSION=2.0 & REQUEST=GetCoverage & COVERAGEID=c001 . Ex: „coverage c001, lat/long cutout, time slice t=2009-11-06T23:20:52” http://www.acme.com/wcs ? SERVICE=WCS & VERSION=2.0 & REQUEST=GetCoverage & COVERAGEID=c001 & SUBSET=Long(100,120) & SUBSET=Lat(50,60) & SUBSET=time("2009-11-06T23:20:52") . Ex: “coverage c001, in GeoTIFF” http://www.acme.com/wcs ? SERVICE=WCS & VERSION=2.0 & REQUEST=GetCoverage & COVERAGEID=c001 & FORMAT=“image/tiff“ OGC CIS :: © 2017 Peter Baumann WCS Transaction Extension [OGC 13-057] . = WCS-T: InsertCoverage + DeleteCoverage + UpdateCoverage . Ex: http://www.acme.com/wcs ? SERVICE=WCS & VERSION = 2.0 & REQUEST=InsertCoverage & COVERAGEREF=http://bcme.com/archive/hurricane.nc & USEID=new . In rasdaman, enhanced with JSON recipe & ingredients layer - Data sources, defaults, missing data, continuous ingest, ... OGC CIS :: © 2017 Peter Baumann WCPS: Datacube Analytics . Web Coverage Processing Service = spatio-temporal datacube analytics language • "From MODIS scenes M1, M2, M3: difference red & nir, as TIFF" • “…but only those where nir exceeds 127 somewhere” for $c in ( M1, M2, M3 ) where some( $c.nir > 127 ) return encode( $c.red - $c.nir, “image/tiff“ ) • s/t regular & irregular grids OGC CIS :: © 2017 Peter Baumann ...But That‘s Not What You Want to See . Let users remain in comfort zone of well-known tools - Map navigation: OpenLayers, Leaflet, ... - Virtual globe: NASA WorldWind, Cesium, ... - Web GIS: MapServer, GeoServer, QGIS, ArcGIS, ... - Analysis: GDAL, R, python (OWSLIB, Jupyter notebooks), ... ...via WCS / WCPS / WMS as standard client/server APIs [screenshots: diverse clients accessing rasdaman] OGC CIS :: © 2017 Peter Baumann WCS Testing: Down to Single Pixel Level Credits: Jinsongdi Yu, Stephan Meissl OGC CIS :: © 2017 Peter Baumann rasdaman: Agile Array Analytics = „raster data manager“: SQL + n-D arrays - Scalable parallel “tile streaming” architecture - Spatio-temporal regular & irregular grids . Mature, in operational use - blueprint for Big Datacube standards: ISO, OGC, INSPIRE - OGC & INSPIRE WCS Reference Implementation OGC CIS :: © 2017 Peter Baumann Architecture [SSTD 2013] Web clients (m2m, browser) Internet rasdaman geo services distributed query rasserver processing No single point of failure external file system database alternative archives storage optional compression [SSTD 2013] OGC CIS :: © 2017 Peter Baumann Parallel, Distributed Processing select max((A.nir - A.red) / (A.nir + A.red)) - max((B.nir - B.red) / (B.nir + B.red)) Dataset D - max((C.nir - C.red) / (C.nir + C.red)) - max((D.nir - D.red) / (D.nir + D.red)) from A, B, C, D 1 query 1,000+ cloud nodes Dataset C [ACM SIGMOD DanaC 2014] [VLDB BOSS 2016] Dataset A Dataset B OGC CIS :: © 2017 Peter Baumann EarthServer: Datacubes At Your Fingertips . Agile Analytics on x/y/t + x/y/z/t Earth & Planetary datacubes - Rigorously standards: OGC WMS + WCS + WCPS - EU rasdaman + US NASA WorldWind - 700+ TB 1+ PB . Intercontinental initiative, 3+3 years: EU + US + AUS . www.earthserver.eu, www.planetserver.eu INSPIREOGC CIS WCS :: ::© 2017© 2017 Peter Peter Baumann Baumann MEA: Sample Datacubes Coverages OGC CIS :: © 2017 Peter Baumann MEA: Daily Hydro Estimator OGC CIS :: © 2017 Peter Baumann ECMWF: River Discharge OGC CIS :: © 2017 Peter Baumann MEA: Land Surface Temperature, Cloudfree OGC CIS :: © 2017 Peter Baumann Server-Side Processing: Federation OGC CIS :: © 2017 Peter Baumann ESA Tech Harm Strat Meetg, 2017-feb-21 OGC CIS :: © 2017 Peter Baumann Conclusion . Coverage Suite: data & service model for spatio-temporal data - CIS + WCS - All raster data: regular & irregular sensor, image (timeseries), simulation, statistics data - from simple access (WCS Core) to datacube analytics (WCPS) - Conformance tests down to pixel level, interoperable