Dados Raster Na Terralib
Total Page:16
File Type:pdf, Size:1020Kb
RASTER DATA HANDLING IN SPATIAL DATABASES: The Case for Images Lúbia Vinhas Ricardo Cartaxo Gilberto Camara Karine Ferreira Antonio Miguel Vieira Monteiro DPI/INPE 9th Workshop on Meteorological operational systems ECMWF AnAn OutlineOutline ofof thisthis TalkTalk INPE’s Motivation The Rationality for Having Images Stored in DBMS The Challenges Our Solution and Where We Are at this Stage Algorithm Development: API for Images Spatial Operations Conclusion and Future Works INPE’sINPE’s MotivationMotivation SatelliteSatellite AcquiredAcquired DataData isis EverywhereEverywhere !!!! SatelliteSatellite DerivedDerived ObservationalObservational DataData •• LargeLarge MassMass ofof HighlyHighly DimensionalDimensional SpatioSpatio-- TemporalTemporal DataData 3030 YearsYears ofof lessonslessons learnedlearned fromfrom dealingdealing withwith HighHigh DimensionalDimensional SpatioSpatio--TemporalTemporal ImageImage DataData fromfrom EarthEarth RemoteRemote SensingSensing SatellitesSatellites andand AirborneAirborne Sensors.Sensors. INPE’sINPE’s ImageImage DataData CentreCentre ProjectProject TThehe RationalityRationality forfor HavinHavingg ImaImaggeses StoredStored inin DBMSDBMS AA NewNew GenerationGeneration ofof SpatiallySpatially EnabledEnabled DBMS;DBMS; HugeHuge AmountAmount ofof DataData thatthat mustmust bebe DealtDealt with,with, comingcoming fromfrom aa VarietyVariety ofof SensorsSensors overover aa varietyvariety ofof plataformsplataforms;; MakeMake DataData recoveryrecovery andand IntegrationIntegration aa moremore easyeasy Task;Task; TheThe ChallengesChallenges Technological Challenges: • Efficient Spatially Enabled DBMS • Provide spatial operations on spatial data types stored in different DBMS Scientific&Technological Challenges: • Methods and Techniques for Parameter/Pattern/Information-Content Extraction from High Dimensional Integrated Spatio-Temporal Datasets TheThe Challenges:Challenges: TheThe ApplicationsApplications NeedsNeeds drivingdriving thethe TechnologyTechnology NeedsNeeds •• RunRun inin aa corporativecorporative environmentenvironment •• AccessAccess datadata byby internetinternet andand intranetintranet •• TypicalTypical useuse ofof imageimage datadata isis visualizationvisualization •• IntegratesIntegrates descriptivedescriptive datadata storedstored inin aa conventionalconventional objectobject--relationalrelational DBMSDBMS •• IntegratesIntegrates vectorvector datadata TheThe Challenge:Challenge: TheThe BasicBasic RequirementsRequirements TheThe ImageImage DataData shouldshould bebe storedstored inin thethe existingexisting objectobject--relationalrelational databasedatabase managementmanagement systemsystem •• DataData integrityintegrity andand consistencyconsistency •• IndependentIndependent andand effectiveeffective accessaccess byby usersusers ofof multiplemultiple applicationsapplications TheThe Challenges:Challenges: TheThe ResearchResearch NeedsNeeds drivingdriving thethe ScientificScientific NeedsNeeds Parameter/Pattern/Parameter/Pattern/ InformationInformation--ContentContent Extraction:Extraction: AnotherAnother TypicalTypical useuse ofof imageimage datadata isis gettinggetting informationinformation outout ofof it:it: Needs:Needs: NewNew MethodsMethods andand AlgorithmsAlgorithms OurOur AimAim …… ProvideProvide aa ResearchResearch TestbedTestbed forfor DealingDealing withwith LargeLarge RasterRaster DatasetsDatasets thatthat cancan helphelp in:in: • Enabling Data Integration. Grid Data, Image Data, Observations Data and other Geographic Data types could be used together; • Enabling easy new algorithms development for parameter extraction from Satellite Image Datasets; • Enabling the test of new spatial-temporal statistics methods for “mining” high dimensional datasets …… andand WhereWhere wewe areare atat thisthis StageStage AdvancesAdvances inin databasedatabase technologytechnology provideprovide supportsupport forfor majormajor advancesadvances inin nonnon--conventionalconventional databasedatabase applicationsapplications SpatialSpatial DataData inin RelationalRelational DatabasesDatabases •Integration of spatial data types in object-relational database management systems •Efficient handling of spatial data types • vector: polygons, lines and points • RasterRaster DataData Structures:Structures: ImagesImages oror anyany otherother GriddedGridded datadata •Tools for query and manipulation of spatial data ItIt isis TimeTime forfor Images…Images… AA specialspecial interestinterest inin thethe spatialspatial databasesdatabases communitycommunity isis thethe efficientefficient handlinghandling ofof rasterraster datadata AnAn approachapproach isis toto developdevelop specializedspecialized imageimage datadata serversservers •• MainMain advantageadvantage:: thethe capacitycapacity ofof performanceperformance improvementsimprovements OurOur ApproachApproach IncludeInclude BuildingBuilding RasterRaster DataData ManagementManagement capabilitiescapabilities intointo ObjectObject--RelationalRelational DatabaseDatabase ManagementManagement SystemsSystems •• MainMain advantagesadvantages:: •• eeasyasy interfaceinterface withwith existingexisting useruser environmentsenvironments •• ToTo accommodateaccommodate notnot onlyonly typicaltypical ImageImage Data,Data, butbut alsoalso RasterRaster DataData inin generalgeneral OurOur SolutionSolution …… OurOur TechnologicalTechnological Solution:Solution: TerraLibTerraLib (http://www.terralib.org) ThisThis workwork isis partpart ofof thethe developmendevelopmentt ofof TerraLibTerraLib isis anan OpenOpen SourceSource LicencedLicenced (LGPL)(LGPL) GeographicGeographic LibraryLibrary forfor providingproviding supportsupport forfor thethe developmentdevelopment ofof GeographicGeographic ApplicationsApplications poweredpowered byby SpatiallySpatially enabledenabled DBMSDBMS MainMain featuresfeatures:: GeometryGeometry isis storestoredd andand managedmanaged inin thethe DBMSDBMS FacilitiesFacilities supportedsupported inin differentsdifferents DBMSDBMS asas ORACLE,ORACLE, PostgreSQLPostgreSQL,, MySQLMySQL,, ORACLEORACLE Spatial,Spatial, PostgreSQLPostgreSQL//PostGISPostGIS,, MSMS DatabasesDatabases throughthrough ADOADO TerraLib Interface with DBMS TerraLib TeDatabase ADO Driver MySQL Driver OracleSpatial Driver PostgreSQL Driver Access MySQL Oracle PostgreSQL Spatial SQLServer ImageImage ((RasterRaster)) DataData NeedsNeeds efficientefficient storagestorage andand indexingindexing mechanismsmechanisms decodingdecoding ofof thethe differentdifferent imageimage datadata formatsformats basicbasic datadata manipulationmanipulation functionsfunctions convenientconvenient waysways ofof accessingaccessing thethe imageimage datadata byby algorithmsalgorithms Two Main Aspects 1.1. AA DBMSDBMS DataData ModelModel TablesTables schemaschema SpatialSpatial indexingindexing SupportSupport toto compressioncompression 2.2. AA setset ofof C++C++ classesclasses toto allowallow applicationsapplications toto dealdeal withwith RasterRaster DataData EfficiencyEfficiency andand flexibilityflexibility toto accessaccess thethe datadata DBMS Data Model Defines,Defines, atat aa physicalphysical level,level, howhow toto storestore rasterraster datadata inin aa objectobject--relationalrelational databasedatabase AnAn ineffectiveineffective approach:approach: Store each point of the image in a row of a table [x,y,z] AnotherAnother approach:approach: The entire image is written to a blob and stored in a field of a table AA variationvariation ofof thethe secondsecond approachapproach waswas adopted:adopted: Tiles of image are written to a blob and stored in a field of a table Tiling Specific parts of the image can be retrieved and processing independently User control over the size of the tiles Example: zooming operation Tiling → DBMS Data Model Each raster data is stored in a table Each row stores a tile of a particular band tile_id band blob T1 1 ... T1 2 ... T1 3 ... Multi-resolution Large image Small canvas Image is shown with a degraded resolution Much of the information retrieved is not used Multi-resolution Lower resolution versions of the image are also stored in the database Application decides the best resolution level to be retrieved User control of the number of resolution levels Level 2 → Resolution R * 22 Level 1 → Resolution R * 21 Level 0 → Resolution R * 20 (original) Multi-resolution To store an image in a lower resolution less tiles are needed 30 m 60 m 120 m 240 m Multi-resolution Each row of a Raster table contains information about the level of resolution of the tile tile_id band resolution_factor blob T1 1 0 ... T1 1 1 ... T1 2 0 ... T1 2 1 ... T1 3 0 ... T1 3 1 ... Spatial Indexing For each tile the coordinates of its bounding box are stored Using a SQL statement an application can select the tiles that intercept a given area in a given resolution level tile_id band resolution_factor lower _x lower_y upper_x upper_y blob T1 1 0 ... T1 1 1 ... T1 2 0 ... T1 2 1 ... T1 3 0 ... T1 3 1 ... SELECT * FROM raster_table WHERE NOT (lower_x > 10 OR upper_x < 20 OR lower_y > 10 OR upperY < 20 ) AND resolution_factor = 0 Accessing Pixels Individually Typical image processing algorithm: for i=0 to num rows for j=0 to num cols process Image(i,j) To query the database for each pixel of image can be costly Solution: keep a cache of tiles in memory Virtual Memory Optimize the access of pixels of an image Tiles in memory have the same identification of the database Application Cache (3 tiles) (i,j) ? Tile XW (i,j) ? Tile