Effective Spatial Data Partitioning for Scalable Query Processing Ablimit Aji Hoang Vo Fusheng Wang HP Labs Emory University Stony Brook University Palo Alto, California, USA Atlanta, Georgia, USA Stony Brook,New York, USA
[email protected] [email protected] [email protected] ABSTRACT amounts of spatial data in a way that was never before pos- Recently, MapReduce based spatial query systems have emerged sible. The volume and velocity of data only increase signifi- as a cost effective and scalable solution to large scale spa- cantly as we shift towards the Internet of Things paradigm tial data processing and analytics. MapReduce based sys- in which devices have spatial awareness and produce data tems achieve massive scalability by partitioning the data and while interacting with each other. As science and businesses running query tasks on those partitions in parallel. There- are becoming increasingly data-driven, timely analysis and fore, effective data partitioning is critical for task paral- management of such data is of utmost importance to data lelization, load balancing, and directly affects system perfor- owners. A wide spectrum of applications and scientific disci- mance. However, several pitfalls of spatial data partitioning plines such as GIS, Location Based Social Networks (LBSN), make this task particularly challenging. First, data skew neuroscience [4], medical imaging [38] and astronomy [27], is very common in spatial applications. To achieve best can benefit from an efficient spatial query system to cope query performance, data skew need to be reduced to the with the challenges of Spatial Big Data. minimum. Second, spatial partitioning approaches generate To effectively store, manage and process such large amounts boundary objects that cross multiple partitions, and add of spatial data, a scalable distributed data management sys- extra query processing overhead.