View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Computer Science Technical Reports @Virginia Tech CampProf: A Visual Performance Analysis Tool for Memory Bound GPU Kernels Ashwin M. Aji, Mayank Daga, Wu-chun Feng Dept. of Computer Science Virginia Tech Blacksburg, USA faaji, mdaga, [email protected] Abstract—Current GPU tools and performance models provide which may severely affect the performance of memory-bound some common architectural insights that guide the programmers CUDA kernels [6], [7]. Our studies show that the performance to write optimal code. We challenge these performance models, can degrade by up to seven-times because of partition camp- by modeling and analyzing a lesser known, but very severe performance pitfall, called ‘Partition Camping’, in NVIDIA ing (Figure 3). While common optimization techniques for GPUs. Partition Camping is caused by memory accesses that NVIDIA GPUs have been widely studied, and many tools are skewed towards a subset of the available memory partitions, and models guide programmers to perform common intra- which may degrade the performance of memory-bound CUDA block optimizations, neither the current performance models kernels by up to seven-times. No existing tool can detect the nor existing tools can discover the partition camping effect in partition camping effect in CUDA kernels. We complement the existing tools by developing ‘CampProf’, a CUDA kernels. spreadsheet based, visual analysis tool, that detects the degree to We develop CampProf, which is a new, easy-to-use, and a which any memory-bound kernel suffers from partition camping. spreadsheet based visual analysis tool for detecting partition In addition, CampProf also predicts the kernel’s performance camping effects in memory-bound CUDA kernels. The tool at all execution configurations, if its performance parameters takes the kernel execution time and the number of memory are known at any one of them. To demonstrate the utility of CampProf, we analyze three different applications using our transactions of any one kernel configuration as the input, and tool, and demonstrate how it can be used to discover partition displays a range of execution times for all the other kernel camping. We also demonstrate how CampProf can be used to configurations (Figure 1). The upper and lower bounds indicate monitor the performance improvements in the kernels, as the the performance levels with and without the partition camping partition camping effect is being removed. problem respectively. The relative position of the actual execu- The performance model that drives CampProf was developed by applying multiple linear regression techniques over a set of tion time with respect to the performance range will show the specific micro-benchmarks that simulated the partition camping degree to which the partition camping problem exists in the behavior. Our results show that the geometric mean of errors in kernel. In addition, CampProf also predicts the exact execution our prediction model is within 12% of the actual execution times. times of the kernel at all the other execution configurations. In summary, CampProf is a new, accurate, and easy-to-use tool We recommend CampProf to be used in conjunction with the that can be used in conjunction with the existing tools to analyze and improve the overall performance of memory-bound CUDA other existing tools, like CUDA Occupancy Calculator [8] and kernels. CUDA Visual Profiler (CudaProf) [9], to analyze the overall Keywords-CUDA; Partition Camping; Analysis; Optimization; performance of memory-bound CUDA applications. NVIDIA GPU’s CampProf uses a performance prediction model, which we developed by first creating several micro-benchmarks that cap- I. INTRODUCTION tured the performance of all the different memory transaction Graphics Processing Units (GPUs) are being increasingly types, with and without the partition camping behavior. Based adopted by the high-performance computing (HPC) com- on the execution times of the micro-benchmarks and the dif- munity due to their remarkable performance-price ratio, but ferent memory transactions, we used multiple linear regression a thorough understanding of the underlying architecture is to model the performance range of actual CUDA kernels. To still needed to optimize the GPU-accelerated applications [1]. demonstrate the utility of CampProf in real applications, we Several performance models have recently been developed analyze three very different memory-bound CUDA kernels, to study the organization of the GPU and accurately predict and show how our tool and model can be used to discover the performance of the GPU-kernels [2]–[5]. Our paper chal- the partition camping problem. We also demonstrate how the lenges and complements the existing performance models, by tool can be used to monitor the performance of the kernel, modeling and analyzing a lesser known but extremely severe where the execution time progresses towards the best case performance pitfall, called Partition Camping, in NVIDIA after the partition camping effect has been reduced. Also, we GPUs. show that our performance prediction model has a geometric Partition Camping is caused by memory accesses that are mean error of less than 12% when validated against the actual skewed towards a subset of the available memory partitions, kernel execution times. grammed via simple extensions to the C programming lan- guage. CUDA follows a code off-loading model, i.e. data- Actual Kernel Running Time parallel, compute-intensive portions of applications running on the host processor are typically off-loaded onto the device. The Extrapolated Kernel Time kernel is the portion of the program that is compiled to the instruction set of the device and then off-loaded to the device before execution. Execution Time Execution Execution Configuration of a CUDA Kernel: The threads Degree of Partition Camping in the kernel are hierarchically ordered as a logical grid of thread blocks, and the CUDA thread scheduler will schedule the blocks for execution on the SMs. When executing a block Active Warps per SM on the SM, CUDA splits the block into groups of 32-threads Worst Case (With Partition Camping) Extrapolated called warps, where the entire warp executes one common Best Case (Without Partition Camping) instruction at a time. CUDA schedules blocks (or warps) on the SMs in batches, and not all together, due to register and Fig. 1. Conceptual Output Chart of CampProf. shared memory resource constraints. The blocks (or warps) in the current batch are called the active blocks (or warps) per SM. The CUDA thread scheduler treats all the active blocks of The rest of this paper is organized as follows: Section II an SM as a unified set of active warps ready to be scheduled provides background on the NVIDIA GPU, the CUDA archi- for execution. In this way, CUDA hides the memory access tecture and the partition camping problem. Section III presents latency of one warp by scheduling another active warp for the related work. Section IV described the CampProf tool in execution [6], [7]. In short, a kernel with an arbitrary number detail, followed by performance modeling techniques using of blocks will perform only as good as the kernel with a micro-benchmarks and statistical analysis tools. Section V configuration equal to set of active warps. In this paper, we explains the experimental setup. Section VI discusses the have chosen ‘active warps per SM’ as the metric to describe experimental results. Section VII concludes the paper and the execution configuration of any kernel, because it is much proposes some future work. simpler to be represented in only a single dimension. There are some hardware restrictions imposed on the II. BACKGROUND ON THE NVIDIA GPUS NVIDIA GPUs with compute capability 1.3 that limits the In this section, we explain the basic architecture of the possible number of active warps that can be scheduled on each general NVIDIA GPU, the CUDA programming model, com- SM. The warp size for the current GPUs is 32 threads. The mon optimization techniques and the lesser known partition maximum number of active threads per multiprocessor can camping problem. be 1024, which means that the maximum number of active warps per SM is 32. Also, the maximum number of threads A. The NVIDIA GPUs and CUDA in a block is 512, and the maximum number of active blocks The NVIDIA GPU (or device) consists of a set of Single per multiprocessor is 8 [6]. Due to a combination of these Instruction Multiple Data (SIMD) streaming multiprocessors restrictions, the number of active warps per SM can range (SMs), where each SM consists of eight scalar processor anywhere from 1 to 16, followed by even-numbered warps (SP) cores, two special function units and a double precision from 18 to 32. processing unit with a multi-threaded instruction unit. The actual number of SMs vary depending on the different GPU B. The Partition Camping Problem models. Optimization techniques for NVIDIA GPUs have been The SMs on the GPU can simultaneously access the device widely studied, and many proprietary tools, like CUDA Visual memory, which consists of read-write global memory, 64 KB Profiler (CudaProf) and the CUDA Occupancy Calculator of read-only constant memory and read-only texture memory. spreadsheet tool, guide programmers to perform common However, all the device memory modules can be read or intra-block optimizations. These include optimizing arithmetic written to by the host processor. Each SM has on-chip memory, instruction throughput, efficiently accessing global memory, which can be accessed by all the SPs within the SM and will and avoiding bank conflicts in shared memory. In this pa- be one of the following four types: a set of registers; 16 KB per, we study a lesser known performance pitfall, which of ‘shared memory’, which is a software-managed data cache; NVIDIA calls ‘partition camping’, where memory requests a read-only constant memory cache; and a read-only texture across blocks get serialized by fewer memory controllers on memory cache.