Parallel Space-Time Kernel Density Estimation Erik Saule†, Dinesh Panchananam†, Alexander Hohl‡, Wenwu Tang‡, Eric Delmelle‡ † Dept. of Computer Science, ‡Dept. of Geography and Earth Sciences UNC Charlotte Charlotte, NC, USA Email: {esaule,dpanchan,ahohl,wtang4,eric.delmelle}@uncc.edu Abstract—The exponential growth of available data has these can be constructed, data scientists need to interactively increased the need for interactive exploratory analysis. Dataset visualize and explore the data to understand its structure. can no longer be understood through manual crawling and In this paper, we present the space-time kernel density simple statistics. In Geographical Information Systems (GIS), the dataset is often composed of events localized in space and estimation application which essentially builds a 3D density time; and visualizing such a dataset involves building a map map of events located in space and time. This problem is of where the events occurred. computationally expensive using existing algorithms. That We focus in this paper on events that are localized among is why we developed better sequential algorithms for this three dimensions (latitude, longitude, and time), and on com- problem that reduced the complexity by orders of magnitude. puting the first step of the visualization pipeline, space-time kernel density estimation (STKDE), which is most computa- And to bring the runtime in the realm of near real-time, we tionally expensive. Starting from a gold standard implementa- designed parallel strategies for shared memory machines. tion, we show how algorithm design and engineering, parallel Section II presents the STKDE application along with a decomposition, and scheduling can be applied to bring near reference implementation that uses a voxel-based algorithm real-time computing to space-time kernel density estimation. VB to compute STKDE. In Section III, we investigate the We validate our techniques on real world datasets extracted from infectious disease, social media, and ornithology. sequential problem and develop a point-based algorithm PB that significantly reduce the complexity compared to VB. Keywords-space-time kernel density; performance; spatial We also identify invariants linked to the structure of the algorithm; shared-memory parallelism; scheduling kernel density estimate function that allows to reduce the computational cost by an additional factor and build the PB- I. INTRODUCTION SYM algorithm based on that observation. We develop two The rapid propagation of infectious diseases (e.g. parallelization strategies (PB-SYM-DR and PB-SYM-DD) zika, Ebola, H1N1, dengue fever) is conducive to in Section IV that are designed to make the computation serious, epidemic outbreaks, posing a threat to vulnerable pleasingly parallel. However both strategies are not work- populations. Such diseases have complex transmission efficient and can cause significant work overhead in some cycles, and effective public health responses require the cases. That is why we introduce PB-SYM-PD in Section V ability to monitor outbreaks in a timely manner [1]. Space- that provide a work-efficient decomposition of the points. time statistics facilitate the discovery of disease dynamics However that algorithm has internal dependencies which including rate of spread, seasonal cyclic patterns, direction, can induce a long critical path and cause load imbalance; intensity (i.e. clusters), and risk of diffusion to new regions. and we develop PB-SYM-PD-SCHED to leverage alternative However, obtaining accurate results from space-time ordering of the work to reduce the critical path. To reduce statistics is computationally very demanding, which is the critical path further, we propose the PB-SYM-PD-REP problematic when public health interventions are promptly algorithm which introduces some work overhead only where needed. The issues of computational efforts are exacerbated is needed to achieve a short critical path. Section VI presents with spatiotemporal datasets of increasing size, diversity experimental results on 4 different datasets and 21 instances and availability [2]. High-performance computing reduces extracted from these datasets. The experiments highlights the effort required to identify these patterns, however that the sequential strategies are efficient. The parallel strate- heterogeneity in the data must be accounted for [3]. gies have their pros and cons and different strategies appear Epidemiology is only one of the application domains to be best in different scenarios. Section VII discusses the where it is important to understand how events occurring at applicability of related problems and techniques. different locations and different times form clusters. Political II. SPACE-TIME KERNEL DENSITY ESTIMATION analysis, social media analysis, or the study of animal migra- tion also require the understanding of spatial and temporal A. Description locality of events. The massive amount of data we see nowa- Space-time kernel density (STKDE) is used for identi- days is often analyzed using complex models. But before fying spatiotemporal patterns in datasets. It is a temporal (a) h = 2500m, h = 14days (b) h = 500m, h = 7days s t s t Figure 2. The computation of STKDE happens in a domain space of size Figure 1. Visualization of Dengue fever cases in Cali, Colombia in 2010 gx, gy, gt. Each point impacts the neighboring space at a distance hs in and 2011 for different spatial bandwidth and temporal bandwidth. space and ht in time; forming a cylinder of diameter 2hs and of height 2ht. n Number of points extension of the traditional 2D kernel density estimation [4] s = (x, y, t) A voxel and sampling coordinate which generates density surface (“heatmap”) from a set (xi, yi, ti) Coordinate of point i of n points located in a geographic space. The resulting hs, ht Spatial and temporal bandwidth gx, gy, gt Real size of the domain (in meters) density estimates are visualized within the space-time cube sres, tres Resolution (in meters) framework using two spatial (x, y) and a temporal dimension s = (X, Y, T ) A voxel in voxel space (t) [5]. STKDE creates a discretized 3D volume where (Xi,Yi,Ti) Voxel of point i each voxel (3D equivalent of a pixel) is assigned a density Gx,Gy,Gt Size of the domain (in voxels) Hs,Ht Bandwidth (in voxels) estimate based on the surrounding points. The space-time density is estimated using (following the notations of [3]): Table I NOTATIONS 1 x − xi y − yi t − ti B. Gold Standard Implementation fˆ x, y, t k , k ( ) = 2 X s( ) t( ) nhsht hs hs ht i|di<hs,ti<ht The gold standard implementation (for instance from [3]) follows the exact definition of STKDE as it is given above. It Density fˆ(x, y, t) of each voxel is determined by number is a voxel-based algorithm we call VB. For each voxel, VB and distance of events (points) (x , y , t ) within its vicinity, i i i finds the points within the temporal and spatial bandwidths which is conceptualized by a cylinder. The spatial bandwidth and calculates the contribution to the density of this voxel. h forms a circle which, due to the orthogonal relationship s The pseudo code is given in Algorithm 1. between space and time, is extended to a cylinder by temporal bandwidth h . For every event inside the cylinder, t Algorithm 1 VB the spatial (di) and temporal (ti) distances are smaller than for all voxels s = (x, y, t) do hs and ht. Therefore, the event receives a weight based on sum = 0 the kernel functions ks and kt (distance decay): for all points i at xi, yi, ti do 2 2 if (xi − x) + (yi − y) < hs and |ti − t| ≤ ht then − − − p x xi y yi t ti sum+ = ks( , )kt( ) π 2 2 hs hs ht ks(u, v) = (1 − u) (1 − v) sum stkde[X][Y ][T ] = 2 2 nhsht 3 k (w) = (1 − w)2 t 4 The algorithm performs θ(GxGyGtn) distance tests and Figure 1 illustrates how varying the bandwidths used in 2 computes θ(nHs Ht) densities. Since Hs is smaller than Gx STKDE helps focusing the graphical visualization of Dengue and Gy and since Ht is smaller than Gt, the complexity of fever cases in Cali, Colombia [6]. Figure 2 shows the impact the algorithm is θ(GxGyGtn) and it requires θ(GxGyGt) of a single point on the neighboring space. memory. Computationally, the domain of size gx, gy, gt is dis- cretized in voxels using a spatial resolution sres and a tem- III. ALGORITHM DESIGN AND ENGINEERING poral resolution tres. Therefore, the domain is represented gx gy gt A. Point-based Algorithm by a grid of size Gx = sres , Gy = sres , Gt = tres . Each point causes a density increase in the voxels that are The voxel-based algorithm suffers from a massive cost within a cylinder centered on the point, of radius equal to of computing distances. The gold implementation reduces hs the spatial bandwidth in voxels Hs = sres , and of half the number of distance computation by decomposing the ht height equal to the temporal bandwidth Ht = tres . domain, but that still incurs many distance calculations. Most All the notations are summarized in Table I. As a conven- of these calculations are unnecessary since we know where tion all uppercase notations denote quantities in voxels and each point will radiate density. The point-based algorithm all lowercase notations denote quantities in domain space. PB, given in Algorithm 2, leverages that property. Algorithm 2 PB the two invariants. (See pseudocode in Algorithm 3.) Notice for all voxels s = (x, y, t) do that leveraging the symmetries in the calculation does not stkde[X][Y ][T ] = 0 change the complexity (PB-SYM still has a complexity of for each points i at xi, yi, ti do 2 for Xi − Hs ≤ X ≤ Xi + Hs do Θ(GxGyGt + nHs Ht)) but reduces the number of flops.