The Process of Parallelizing the Conjunction Prediction Algorithm of ESA’S SSA Conjunction Prediction Service Using GPGPU
Total Page：16
File Type：pdf, Size：1020Kb
The Process of Parallelizing the Conjunction Prediction Algorithm of ESA’s SSA Conjunction Prediction Service using GPGPU ESAC Trainee Project September 2012 – March 2013 Marius Fehr Mentors: Vicente Navarro and Luis Martin 1 Space Situational Awareness SWE – Space Weather NEO – Near Earth Objects SST – Space Surveillance and Tracking 2 Space Surveillance and Tracking Catalog (JSpOC, US Air Force): 16’000 Objects > 10cm Estimates: 600’000 Objects > 1 cm 3 The Conjunction Prediction System 4 All vs. All Conjunction Analysis 1 2 3 4 5 6 7 [1,2] [1,3] [1,4] [1,5] ... [5,6] [5,7] [6,7] 1 2 3 4 5 6 ... 7 > Number of pairs grows quadratically with the number of objects > The analyses of all objects pairs are independent → Huge potential for parallelism > 10k objects could theoretically be analyzed in millions of parallel threads > CPUs usually launch not more than up to a dozen of threads > How can we exploit that ? 5 GPU – NVIDIAs Fermi Architecture http://benchmarkreviews.com/images/reviews/processor/NVIDIA_Fermi/nvidia-fermi-gf100-gpu-shader-model-block-diagram-full.png 6 http://benchmarkreviews.com/images/reviews/processor/NVIDIA_Fermi/nvidia-fermi-gf100-gpu-block-diagram-benchmarkreviews.png CUDA – Grid, Blocks and Threads Example: Matrix Multiplication 4 x 4 Matrices A and B 4 blocks with 4 threads each, 16 threads in total Grid Blocks Threads > Abstraction of Multiprocessors and Cores > GPU distributes blocks to idle multiprocessors > Idle/waiting threads are swapped out instantly > Up to 65k x 65k x 65k blocks > Up to 1024 threads per block 7 CUDA – Program Flow CPU GPU Kernel Kernel Grid Blocks Threads Memory GPU Memory Data Data Result Result 8 CAN – Conjunction Analysis 10k Objects List of Objects 50M pairs Prediction Period Apogee-Perigee Filter 50M → 20M Loop over Epochs Load Ephemeris Data to Ephemeris Memory if necessary Data ( Files/DB ) Smart Sieve 20M → 40k Linear Search Find Time and Distance of Closest Approach Calculate Collision Risk and write Conjunctions Conjunctions to Files/DB. ( Files/DB ) 9 Identifying Parallelizable Functions 10'000 Objects - 8 Days Apogee-Perigee Filter Loading Ephemerides OpenMP Interpolating in Smart Sieve Smart Sieve Interpolating in Linear Search Linear Search Find TCA Conjunction Definition Penetration Factor Remaining Operations 0 100 200 300 400 500 600 700 800 900 Runtime [s] 10 Smart Sieve Kernel Object pairs that passed the Apogee-Perigee Filter [1,2] [1,3] [1,4] [1,5] [1,6] [1,7] [2,3] [2,4] ... [5,6] [5,7] [6,7] ... Filter 1 ... Filter 2 ... Filter 3 ... Filter N ... Increase potential pair counter with atomicAdd() Critical Section > In general: Having a large number of threads compete for a resource is expensive > BUT: only about 0.2% of all threads actually reach the critical section 11 Linear Search Epoch ... -1 -1 -1 -1 Interpolation Search for Sign Change > Kernel 1: 1 Thread = 1 object : > Interpolating the state vectors needed for current time step > Kernel 2: 1 Thread = 1 potential pair : > Searching for a sign change in the relative velocity 12 Linear Search Epoch ... -1 -1 -1 -1 -1 Interpolation Search for Sign Change > Kernel 1: 1 Thread = 1 object : > Interpolating the state vectors needed for current time step > Kernel 2: 1 Thread = 1 potential pair : > Searching for a sign change in the relative velocity 13 Linear Search Epoch ... -1 -1 -1 -1 -1 +1 Interpolation Search for Sign Change > Kernel 1: 1 Thread = 1 object : > Interpolating the state vectors needed for current time step > Kernel 2: 1 Thread = 1 potential pair : > Searching for a sign change in the relative velocity 14 Find TCA Kernel Potential Pairs ... Sign Change ? Zero Finder ... > Kernel: 1 Thread = 1 potential pair : > Check if Linear Search found a time step with a sign change > Start zero finder (Regula Falsi) > Requires state vector interpolation for every intermediate step http://commons.wikimedia.org/wiki/File:Regula_falsi.gif 15 GPU Timeline > Goal: Minimize memory transfers and (re)allocations I Smart Sieve I LS I LS I LS Find TCA I Smart Sieve I LS I GPU Load ephemeris, pairs and constants Resize pot. pairs allocation if necessary Apogee-Perigee Load Retrieve results Filter Ephemeris of this epoch CPU Epoch 1 Epoch 2 Time 16 Test Environment > CPU: Intel Xenon E5620 4 Cores @ 2.4 GHz > Memory: 6 GB > GPU: NVIDIA Geforce GTX 580 > 1.5 GB of Memory > 512 CUDA Cores > Compute Capability 2.0 > Faster double precision calculations with the Geforce 500 Series than the newer 600 Series 17 Computation Time - Results I 1400 Fortran C CUDA 1200 OpenMP Algorithm 1000 Optimization R u n 800 t i m e [ s ] 600 GPU 400 - 88 % 200 0 313 625 1250 2500 5000 10000 Number of Objects 18 Computation Time - Results II 10'000 Objects - 8 Days CUDA Kernel Interpolating in Smart Sieve CUDA Kernel Interpolating in Linear Search CUDA Kernel Smart Sieve Algorithm Optimization CUDA Kernel Find TCA Algorithm Optimization Conjunction Definition Fortran C C CUDA CUDA Kernel Linear Search Algorithm Optimization Penetration Factor Apogee-Perigee Filter Remaining Operations CUDA Memory operations 0 50 100 150 200 250 300 350 400 450 500 Runtime [s] 19 Conclusion > Considerable improvement of the 8 days, 10’000 objects, computation time all vs. all > Parallelization with CUDA Algorithm Opt. -41% > Other optimizations CUDA -88% > Bottleneck: I/O > Reading ephemeris data from file/DB > Writing conjunctions to file/DB > Future Work > Parallelize other parts of the CPS > Computation of Conjunction Risk, Orbit Propagation, ... > Recompute ephemeris instead of loading from file/DB 20 What about your program? > Can your program be divided into thousands of parallel (and equal) computations? > Is there any communication or cooperation necessary between threads? > Only efficient between threads of the same block (< 1024) > Is the computational effort required huge compared to the size of the data? > Does your program use libraries like BLAS or FFT? > Try cuBLAS and cuFFT > Be aware: The GPU has a very flat memory hierarchy / small caches > 64 KB L1 cache, 0 – 768 KB L2 cache 21 Questions ? > Marius Fehr [email protected] > Vicente Navarro [email protected] > Luis Martin [email protected] 22 The CUDA C Extension int vector_addition ( ) { > All threads execute the same int U = 1000; int V = 256; piece of code, the kernel int N = U * V; int a[N], b[N], c[N]; > Kernel replaces loop // fill a and b > int *cuda_a, *cuda_b, *cuda_c; Index of each thread computed from block and thread id and block cudaMalloc( (void**) &cuda_a, N * sizeof(int) ); cudaMalloc( (void**) &cuda_b, N * sizeof(int) ); dimension cudaMalloc( (void**) &cuda_c, N * sizeof(int) ); > Global memory can be accessed cudaMemcpyHtoD( cuda_a, a, N * sizeof(int) ); cudaMemcpyHtoD( cuda_b, b, N * sizeof(int) ); by every thread add<<< U , V >>>( cuda_a, cuda_b, cuda_c ); > Make sure there are no race cudaMemcpyDtoH( c, cuda_c, N * sizeof(int) ); conditions cudaFree( cuda_a ); cudaFree( cuda_b ); __global__ void add( int *a, int *b, int *c ){ cudaFree( cuda_c ); int idx = blockIdx.x * blockDim.x return 0; + threadIdx.x; } c [ idx ] = a [ idx ] + b [ idx ]; } 23 CUDA GPUs 3500 7 3000 6 2500 5 2000 4 1500 3 1000 2 500 1 0 0 Quadro 2000 Quadro 4000 Quadro 6000 Geforce GTX 580 Geforce GTX 680 Tesla C2075 GPU FP32 [Gflops] FP64 [Gflops] Price [$] Memory [GB] 24 > Creates a GPU-enabled version of the Fortran code > Works either using directives (like OpenMP) or automatic analysis and optimization > 30-day trial available Warps and Threads Threads are grouped in Warps 1 Warp = 32 Threads 1 Core: 1 instruction in 1 cycle BUT all cores in the same group Group 1 execute the same instruction at the Group 2 same time Single Instruction Multiple Thread Example: Fermi Architecture 2 groups x 16 Cores: 1 Warp in 1 cycle BUT not every core has it’s own Special Function Unit Shared Memory GPU Grid Data Data Blocks ... Threads ● Copy from global to Data shared memory and back . in . ● Global Typically 16 KB per block GPU Memory ● Very low latency Data Data ● Shared memory is only visible for threads inside ... the block ● CUDA provides tools for synchronization Texture Memory GPU Dedicated Hardware Grid Global Blocks GPU Memory Threads ● Global memory accessed by dedicated hardware, used for textures ● Read cache; ~ 6 – 8 KB per multiprocessor, optimized for spatial locality in texture coordinates ● Serves as read-through cache and supports multiple simultaneous reads through hardware accelerated filtering .