Takustr. 7 Zuse Institute Berlin 14195 Berlin Germany
MATTHIAS NOACK1,FLORIAN WENDE1, GEORG ZITZLSBERGER2,MICHAEL KLEMM2 AND THOMAS STEINKE1 1 ZUSE INSTITUTE BERLIN
2 INTEL DEUTSCHLAND GMBH
KART – A Runtime Compilation Library for Improving HPC Application Performance
ZIB Report 16-48 (October 2016) Zuse Institute Berlin Takustr. 7 14195 Berlin Germany
Telephone: +49 30-84185-0 Telefax: +49 30-84185-125
E-mail: [email protected] URL: http://www.zib.de
ZIB-Report (Print) ISSN 1438-0064 ZIB-Report (Internet) ISSN 2192-7782 KART – A Runtime Compilation Library for Improving HPC Application Performance
Matthias Noack∗, Florian Wende∗, Georg Zitzlsberger†, Michael Klemm† and Thomas Steinke∗ ∗Zuse Institute Berlin, Takustraße 7, 14195 Berlin, Germany, Email: {noack, wende, steinke}@zib.de †Intel Deutschland GmbH, Email: {georg.zitzlsberger, michael.klemm}@intel.com
Abstract—The effectiveness of ahead-of-time compiler opti- timizations to enhance code generation. However, compiler mization heavily depends on the amount of available information optimizations for C, C++, and Fortran can only be applied at compile time. Input-specific information that is only available upon the provided algorithmic implementations and need to be at runtime cannot be used, although it often determines loop counts, branching predicates and paths, as well as memory-access general and adhere to language standards. Knowledge about patterns. It can also be crucial for generating efficient SIMD- the runtime context of a unit of code would allow to optimize vectorized code. This is especially relevant for the many-core for specific memory access strides, eliminate conditional code, architectures paving the way to exascale computing, which are or apply workload-dependent loop transformations. more sensitive to code-optimization. We explore the design-space A typical approach to remedy this is to apply multi- for using input-specific information at compile-time and present KART, a C++ library solution that allows developers to compile, versioning, that is, generating multiple specialized instantia- link, and execute code (e.g., C, C++, Fortran) at application tions of the same function, loop, or code fragment. This can be runtime. Besides mere runtime compilation of performance- achieved by a programmer using dedicated implementations, critical code, KART can be used to instantiate the same code like C++ template specializations, or preprocessor macros, for multiple times using different inputs, compilers, and options. example. Compilers can also emit versioned code to handle Other techniques like auto-tuning and code-generation can be integrated into a KART-enabled application instead of being aligned versus unaligned data, to create different code paths scripted around it. We evaluate runtimes and compilation costs for different instructions sets (e.g., Streaming SIMD Exten- for different synthetic kernels, and show the effectiveness for two sions and Advanced Vector Extensions), or to avoid SIMD real-world applications, HEOM and a WSM6 proxy. vectorization for too small loop trip counts, to just name a few. Because multi-versioning can dramatically increase I.INTRODUCTION the code size, compilers usually only generate a few code Many C, C++, and Fortran HPC applications are general- versions and provide a general fallback code path. For the ized solutions for their respective domains. They typically large combinatorial space spanned by the potential inputs of implement a wide range of algorithms that are meant to be an HPC application, multi-versioning becomes ineffective. applicable for many different workloads or combinations of Programmers can try to help the compiler by adding compi- input sets. Whilst the applications grow with adding more lation hints (e.g., pragmas/directives or attributes) to limit the methods and features there is also a growing demand for amount of code versions. But even if a programmer provides applying a subset of methods to restricted workloads. Such different implementations there are limits. Optimizations can use cases do not require the full complexity and flexibility of only be applied for a small set of categories of workloads, the original implementations. Hence users strive for optimiz- and also lead to code size increase which can make an ing their (limited) workloads by tuning the implementations. implementation harder to maintain. While it is quite simple to This starts with algorithmic optimizations, delivering different provide different optimizations for different dimensionality of implementations of the original algorithm, such as reducing input data sets, it is much harder to do so for different memory dimensions or replacing algorithms that work better for smaller access patterns, access strides, or loop trip counts. There are problem sizes. To give a few examples, for the WSM6 proxy far too many different goals to optimize for, and grouping code [1], the authors provided constants at compile time and them into categories for directed optimization is hard. R achieved a more than 40% performance gain on the Intel The main contributions of this work are the exploration of TM Xeon Phi coprocessor (former codename Knights Corner, the design space for exploiting runtime data for compiler op- KNC). For the DL MESO code it was recently demon- timization, a light-weight, flexible runtime-compilation frame- strated [2] that if increasingly more source code constants work (KART), and its evaluation. Our solution is to recompile are known at compile time a performance improvement of algorithms (kernels) during the runtime of an application, more than five times can be achieved by enabling better SIMD thereby optimizing within the current context of kernel execu- optimizations for the compiler’s auto-vectorizer. tion. This especially allows to optimize for values that manifest At some point in this process, implementations in source as constants during runtime but were not known at compile code are specific enough to be further optimized by compilers. time. KART is general enough to benefit a wide range of Compilers typically do not apply algorithmic optimizations applications without being limited to a certain back-end. The that change the semantics, but they have a rich set of op- approach is particularly relevant for many-core architectures like the Intel Xeon Phi (co)processor, whose microarchitecture An important question for this approach is: How to acquire is more sensitive to code-optimizations. the needed data from the input at build time? Typically, For frequently used kernels, we show that the improved reading and parsing of input data is done at runtime during optimization outweighs the runtime compilation overhead. the initialization phase. So some application code has to be We also regard this approach as a solution for OpenMP [3] executed to get the data required for building the application. applications to benefit from the same dynamic recompilation To break this circular dependency, an input parser would be TM advantages that OpenCL [4] and CUDA [5] provide. required to analyze the input data and to provide the necessary constant values before the build process can be started. II.RELATED WORK A concern for some developers, e.g., of commercial codes, In the following section we will exclude work which is might also be that binary releases of applications (or libraries) focused on just-in-time compilation for interpreter or scripting would still require to contain the runtime-compiled code languages (e.g., Python, Perl, Lua). We are aware that Java and sections as source. Microsoft’s .NET Framework rely on JIT compilation for high- speed code execution, but both have still a small representation B. Pre-instantiate Code for all Cases in the HPC domain. The LLVM compiler infrastructure [6] includes a JIT A different approach would be to prepare for a potential code generation system which supports various CPU archi- and classified set of inputs. Performance critical functions tectures, and provides a foundation for just-in-time compi- would be instantiated or specialized multiple times with dif- lation projects. For the QCD application domain, the QDP- ferent classes of compile-time constants when compiling the JIT/LLVM approach [7], which extends the ideas of QDP- application. This is similar to what some compilers already do when generating multiple versions of functions or loops, e.g. JIT/PTX [8], uses C++ expression templates to implement LLVM IR code generators that emit executable code via to provide both a scalar and a vectorized version. Such multi- the LLVM JIT component. In Julia, the language design is versioning is bound by combinatorial effects of the parameter combined with the high-performance LLVM-based JIT com- space and for all input variables their numerical domain needs piler [9], [10]. This enables code generation that comes close to be known at compile time. to the performance of a C implementation. One manual implementation approach is to use C++ tem- The LIBXSMM library [11] for small dense matrix-matrix plates with desired constants as template parameters, i.e. multiplications enables static and JIT code generation for classification of values. Those can be explicitly instantiated R R to collect the resulting function pointers in a map with their Intel Xeon processors and Intel Xeon Phi (co)processors. While the basic ideas of LIBXSMM are similar to our proposal classification as the key. Additionally, a fall-back implemen- in that it specifically compiles kernels for a particular target tation without compile-time constants but a set of additional architecture, it only supports sgemm and dgemm with restricted parameters could be provided. At runtime, the calling code alpha and beta inputs. uses actual values from the input to select the appropriate In the context of MPI communication, Schneider et. al. [12] function template specialization, or a fall-back if no suitable demonstrate that through runtime compilation of (un)pack instantiation was found. For dimensions of domains this might functions for non-contiguous data in MPI an order of magni- be applicable as those are typically limited. However, optimiz- tude better performance can be achieved over the interpretation ing for different sizes becomes infeasible if their values cannot scheme found in today’s MPI implementations. be restricted to a small amount of classes. In contrast to the specialized solutions above, our approach is more general, as it applies to arbitrary code and allows use C. Call a Compiler Library at Runtime of different C, C++, and Fortran compilers. Another solution that is already commonly used and in- spired this work is OpenCL that aims for portability across III.DESIGN SPACE heterogeneous compute devices, such as CPUs, GPGPUs, Intel A. Recompile Everything Xeon Phi (co)processors, or FPGAs. It divides the application The most straight-forward approach to using input-specific into host code for a host processor and device code that data as compile-time constants would be to simply recompile runs on a compute device and is compiled at runtime. This the whole application for each set of input data. It completely enables portability, but can also be used for passing runtime- avoids the complexity of additional compilation at runtime data from the host program into the compilation of the and enables the compiler to apply cross-module optimization device code. Previous work has shown that there can be a techniques like whole-program optimization. However, large performance benefit when compile-time constants are known parts of the code are typically not performance critical and to the OpenCL compiler [13]. Since recently, Nvidia’s CUDA recompiling them would introduce a prohibitive compilation GPGPU framework provides similar means [14]. overhead that grows with the size of the code base and Porting an existing (HPC) application to OpenCL is a major optimization levels. For large codes with compile times in the effort and requires rewriting code using OpenCL’s kernel range of hours, this is a significant impact on time-to-solution language, explicitly managing kernel invocations and memory and renders quality assurance procedures less feasible. transfers between host and compute device.
2 application dynamic KART offers a slim API to compile a code fragment given // kernel call site kernel foo = get kernel(. . . ); symbols either as a text string or a source file, and to call the result comp. 5 kernel foo(. . . ); app. ... 4 kernel foo after compiling the code. KART is implemented in C++ using input time consts kernel bar several Boost libraries [17] and provides APIs for C, C++, and ... Fortran, so far. KART The intended flexibility to use any compiler requires the kernel kernel 1 source loaded invocation of command-line compilers and linkers in separate sources files library 3 processes at runtime to create a shared library from a given 2 source code. Subsequently, the resulting library is linked to the running program via dlopen(), and the contained functions dynamic can be accessed and executed using their (symbolic) name. compiler linker library Compile time constants can be integrated into the runtime- compiled code by either generating the corresponding lines Fig. 1. Schematic of KART: 1) Compile-time constants derived from the before passing the code to KART, or can be specified using input, and kernel source code are passed to KART. 2) KART starts a system compiler and linker to create a dynamically linked library. 3) The library is compiler command line options (e.g., -DNAME=VALUE) for dynamically linked into the application. 4) The application queries KART which KART provides an interface, too. Figure 1 illustrates using the kernel’s function name to get a callable function pointer. 5) The the basic working principle of KART. kernel is invoked. Figure 2 shows a class diagram containing the C++ API of KART together with its abstractions. The API provides OpenMP supports heterogeneity but is limited to one code a toolset abstraction that encapsulates the specification of a version and does not provide runtime compilation. C, C++, compiler and linker command, as well as different sets of or Fortran compilers do not expose a library API that could options for these commands. Toolsets are defined in small be used in an OpenCL-like fashion. LLVM with its modular configuration files. A default toolset file can be set via an architecture would be a viable basis to implement such a mech- environment variable. Once a toolset object is created from a anism. The major drawback of an LLVM-based solution is configuration file, additional options can be specified, e.g., to the limitation to mere LLVM optimization abilities. OpenMP add constant definitions. support of LLVM is still not complete, although catching up The second main abstraction is the program, which is fast [15], [16]. constructed from either a string or a file containing the source code. The program is then built using the selected toolset D. Call Arbitrary Compilers at Runtime and once built, callable function pointers can be acquired by A more abstract and flexible solution is to provide an API specifying a name, and optionally a signature to ensure type- to use any installed command-line compiler from within the safety in C++. The application must know and use the correct application and then incorporate the resulting object code into signature of the called kernel. the running program. Being close to the OpenCL model, Already built programs can be rebuild with a different it yields the highest flexibility, as any compiler, even for toolset. This is more efficient than creating new program different languages, can be used. It can defer the compilation objects from the same source. The motivations behind this of performance critical-code sections until execution time functionality are auto-tuning and benchmark codes that apply and apply the best-optimizing compiler for a specific code many configurations to the same code. Another use-case is fragment and target architecture, or to multi-version a kernel applications with temporary run-time constants, for instance using different compilers and apply auto-tuning. Also different loop trip counts or sizes of data structures that may change (hand-written) implementations of the same kernel can be after a load-(re)balancing step between compute nodes. used. As such, optimization is not only defined by compiler A rebuild invalidates existing pointers, which is handled capabilities and varying compile-time constants, but also by by using smart function pointers (called kernel ptr). These the user at algorithmic level. Kernels can be supplied in lan- pointers are callable objects that are transparently used like guages different from the one chosen for the host application, a function with negligible overhead. Each kernel ptr is reg- which increases the flexibility, e.g., by using C SIMD intrinsics istered with its corresponding program on construction, and within a Fortran application. This is the approach we have unregistered on destruction. This allows the program object chosen for the solution presented in this paper. to update all pointers after a rebuild has been performed. When a program goes out of scope, all registered objects are IV. THE KART LIBRARY invalidated to prevent scoping errors. In order to cover the most relevant languages for HPC A. Design and Implementation applications, there is also an API for C and Fortran. The C API KART provides application developers with the means to is a wrapper around the C++ implementation that uses opaque compile and use pieces of code at runtime with a minimal handles and functions, instead of objects and methods. The amount of overhead and maximal flexibility. KART resembles Fortran interface is a set of bindings for the C API. a minimal build system with a library interface. Internally, KART uses the POSIX calls dlopen(),
3 toolset #include "kart/kart.hpp" ... Ret, Args. . . // signature type toolset(const string& config file name) using my_kernel_t = double(*)(double, double); kernel ptr get compiler options() : const string& const char my_kernel_src[] = R"kart_src( extern "C" { set compiler options(const string&) : void ... append compiler options(const string&) : void operator(Args. . . ) : Ret // original function get linker options() : const string& double my_kernel(double a, double b) set linker options(const string&) : void 0..* { return a * b * CONST; } append linker options(const string&) : void 1 })kart_src"; // close raw string literal 0..* 0..1 int main(int argc, char** argv) { program // create program ... kart::program my_prog(my_kernel_src); program(const string& src) // create default toolset create from file(const std::string& file name) : program kart::toolset ts; create from binary(const std::string& file name) : program // append a constant definiton (runtime value) ts.append_compiler_options(" DCONST=5.0"); build(const toolset& ts) : void // build program using toolset− rebuild(const toolset& ts) : void my_prog.build(ts) get build log() : const string& // get the kernel get kernel Sig T (const std::string& name) : Sig T auto my_kernel = h i get kernel ptr Sig T (const std::string& name) : kernel ptr ... my_prog.get_kernel
4 potential gain before restructuring the code. The runtime gain for example if loop counts for local partitions of an irregular determines an upper bound for the acceptable compilation grid are used. In cases where every node does the same work, overhead. The process is very similar to adapting an appli- there is potential for optimization. The situation can be used cation for offloading to an accelerator—without the need to for an auto-tuning step, where different compilation options or rewrite kernels in another language and optimize them for the kernel variants are built and timed, followed by an exchange accelerator’s architecture. of the timings and the best-performing kernels. Ranks can The intrusiveness of incorporating runtime kernel compila- exchange their compilation results via a distributed file system tion into an existing code base depends on the current code or by transferring the generated binaries as messages. If that is structure, as the build-time and run-time compiled source not applicable, factoring out the compilation step into a small needs to be separated. For a well-structured code base, this prologue job that runs on a single node, processes the input, means identifying the compilation units and adding the KART and pre-builds the kernel binaries for the big job might be an API calls into the application’s initialization phase. option. KART can also use wrapper compilers if the runtime- Most build systems provide a verbose mode that prints out compiled code uses MPI directly. the compile and link commands used. That is the natural A less technical but more legal problem is the licensing starting point to generate a toolset specification for KART issue. On large clusters and supercomputers, commercial com- that include necessary flags and dependencies. In a second pilers are often using a license server with a limited amount step, the dependencies can be minimized, and compile flags of licenses. Depending on the configuration, there is a chance further optimized to improve compilation and runtime of the for an unintended “denial-of-service attack” against a license kernel, respectively. server, by taking all licenses of a shared system hostage. So The following sections describe how KART can be used in it is either impossible, or at least not desirable, to check out different HPC application patterns. thousands of licenses. For these cases, compilation needs to 2) Coprocessors: Applications for the Intel Xeon Phi be limited to a few ranks, or be performed using the prologue- coprocessor often use an offload programming model like strategy. OpenMP or Intel’s LEO (Language Extensions for Offload). 5) Auto-tuning: KART can be used to easily implement In such a setting, KART can be called prior to the offload auto-tuning and also to complement it orthogonally. Within to cross-compile the coprocessor code leveraging the better an applications, the use-cases are slightly different: KART single-thread performance of the host CPU. For the first- can address given runtime-values, not subject to tuning, by generation of Intel Xeon Phi coprocessor (Knights Corner), making them a compile-time known value enabling better cross-compilation is even mandatory, as there is no highly optimization. For instance, given a tunable value like a block- optimizing compiler available running natively on the copro- size, KART could be used to auto-tune through a range of cessor. values, e.g., distributed across ranks of an MPI-job during KART allows the developer to specify the directory and initialization, while using the input-specific overall size of file name for the created binary, which in turn allows using a the blocked data-structure as a compile-time constant. Where shared file system for direct access. The offload only needs the auto-tuning typically generates multiple versions for a wide path to the binary as an argument, from which a KART pro- variety of options/inputs with later run-time selection, KART gram object can be created inside the offloaded code. Within generates the code for the current scenario at runtime. a more flexible offloading framework, like HAM Offload [18], Beside the described HPC-relevant patterns and use cases, a reverse offload from a native coprocessor application to the the availability of a general runtime compilation mechanism host can be used to offload the compilation, even remotely can be exploited in more sophisticated ways. There is no over a fabric. fixed pattern. For instance, instead of just using compile- Another possibility would be a symmetric MPI job with pro- time constants or different compilers, languages, and options cesses on both, the coprocessor and the host. The host rank(s) for existing code, KART can be used as a back-end for could then do the compilation work for the coprocessors in a code generators and domain-specific languages. In contrast to similar fashion as described above, communicating either the existing specialized solutions (see Section II), KART provides path on a shared filesystem, or the binary itself. a maximum of flexibility for a broad variety of use-cases. 3) OpenMP: OpenMP can be used within KART-compiled kernels or higher up in the calling tree. We have successfully V. EVALUATION tested both. Code that is built at runtime using KART is and A. Benchmarks behaves like a shared library. So as with other libraries that make use of OpenMP, the developer has to make sure that there To demonstrate the motivation for KART, we exemplify is no conflict between the OpenMP implementation used in the the potential of runtime kernel compilation with two simple application code and the one used inside the library. Other than and synthetic benchmarks. We also demonstrate the achieved that, there are no known limitations. speed-ups for two tested real world applications. Table I 4) MPI: For large distributed jobs, solely using KART on lists the hardware used for benchmarking. The used software the node level is the easiest way, but not always the best. It is versions are: Intel OpenCL Runtime 16.1.1, Intel C++ Compiler beneficial in cases where every node uses different constants, 16.0.3, GCC 6.1.0, and Clang 3.8.1.
5 TABLE I HEOM kernel compilation cost CHARACTERISTICSOFTHEPROCESSORPLATFORMSUSEDFOR empty kernel auto vect. kernel BENCHMARKING.THE XEONSYSTEMISADUALSOCKETNODE. 56 ms OpenCL, HSW 95 ms Intel Xeon processor Intel Xeon Phi 2438 ms KART+intel, HSW 2935 ms E5-2630 v3 (HSW) processor 7210 (KNL) KART+gcc, HSW 1524 ms Cores 8 64 1552 ms Threads 16 ( 8 2) 256 ( 64 4) KART+clang, HSW 942 ms × × 950 ms SIMD Width 256 Bit 512 Bit OpenCL, KNL 176 ms Frequency 2.4 GHz 1.3 GHz 500 ms 6634 ms MCDRAM — 16 GiB (ECC) KART+intel, KNL 10077 ms Memory BW 59 GiB/s 102 / 352 GiB/s 7844 ms KART+gcc, KNL 7910 ms TDP 85 W 215 W 6198 ms KART+clang, KNL 6508 ms
0 2000 4000 6000 8000 10000 compile time [ms] 1) Runtime Compilation: On principle, runtime compila- tion always introduces overhead. Hence, it is important to un- Fig. 4. Comparison of runtime compilation costs using OpenCL and KART derstand the trade-off between runtime compilation overhead with different compilers for the OpenMP-vectorized Hexciton kernel of the HEOM benchmark, and an empty kernel. There is a high constant cost for and kernel speed-up first. The overhead introduced by KART any compilation regardless of the kernel size. Only the Intel C++ Compiler is largely determined by the runtime of the invoked compiler adds a significant cost to the empty kernel, but also generates the fastest code and linker. The speed-up of the runtime-compiled kernel in case of auto vectorization. OpenCL’s compiler with a library ABI and also less indirectly compiled code from standard library headers is two orders of without incorporating the compile time over the reference magnitude faster. The Xeon Phi processor (KNL) compilation times suffer kernel is sb = tref/tkart , with sb > 1 for tref > tkart. from the lower single thread performance compared with a Xeon (HSW). It is an upper bound for the actual speed-up that includes n tref the compile time, which is s = · with n being n tkart + tcompile the number of kernel calls. Numerator· and denominator are per kernel invocation, the compilation cost, and the frequency two linear functions for the respective overall runtime cost of recompilation. The large constant cost of each compiler of the reference and KART-compiled kernel. For given run invocation can be distributed among multiple kernels by using a single program object that aggregates all sources. and compile times, there is an nc where both functions cross 2) Synthetic Kernels: Demonstrating the potential of run- and s = 1. For every n > nc, there is an actual application time kernel compilation highly depends on the kernel itself speed-up, asymptotically approaching sb. Runtime compilation techniques pay off when the accumulated runtime savings of and the context it is invoked in. It also depends on the used all kernel calls exceed the runtime compilation cost. compiler, compiler options, and processor architecture. We Figure 4 shows the compilation cost using OpenCL as have selected two simple kernels to highlight the principal a reference and KART with different compilers. These are usefulness of runtime compilation for HPC kernels: convolve: One-dimensional linear convolution with an roughly the same timings as measured when executing the • command lines generated by KART by manually typing them offset into the input vector input. This offset could be into a console terminal—the cost of the KART API itself is seen as a way to decompose the input and assign subsets negligible. The linking step took roughly two thirds of the to different threads. Figure 6 shows the KART version. matvec: Matrix vector multiplication with scaling (alpha). time. The timings for the empty kernels show that there is a • large constant cost coming with the command line compilers. To give the compiler’s optimizer some optimization head- This includes starting processes and lots of file operations room, we assume the scaling to be 0.0 or 1.0, and matrix a when handling dependencies like a large set of header and to have just one column. This simulates cases where the library files, all not present in OpenCL. A library interface compiler can remove invariant statements and/or loops. to existing compilers together with a set of small headers Compared to convolve, which is only called once, the and libraries specifically optimized for compilation time could matvec kernel is executed multiple times. Figure 5 shows improve the situation. For commercial compilers, additional the KART version. time is lost for fetching licenses from a file system or network. Both kernels have been compiled via KART (as seen in Caching the compilation results between application runs the listing) and as ordinary C functions with all parameters could mitigate these costs if the actual reuse is high. as arguments (e.g., alpha, rows, and cols for kernel matvec). For all use-cases that only need a single runtime build KART was used to pass the static values via preprocessor per kernel during initialization, a few seconds are tolerable, macros (-D...). given that the kernel runtimes for computational hotspots can The runtimes for the Intel C++ compiler are shown in easily add up to many minutes or even hours during large Figure 7 (the compiler options used were always identical: production runs. For more dynamic use cases, where kernels -restrict -std=c++11 -qopenmp -xHost -O3). are regularly rebuilt as values of compile-time constants need For the matvec kernel the compiler was able to entirely to be adapted, we recommend profiling the performance gain eliminate the loops for alpha=0 and even showed a 3.0×
6 extern "C" Synthetic kernel runtime comparison void matvec_kart(float a[][COLS], w/o KART KART float b[ROWS], matvec, alpha=0, HSW float x[COLS]) { matvec, alpha=1, HSW 2.11 x for (int i = 0; i < ROWS; ++i) for (int j = 0; j < COLS; ++j) convolve, off=0, HSW 14.79 x b[i] += a[i][j] * x[j] * ALPHA; } matvec, alpha=0, KNL
matvec, alpha=1, KNL 3.04 x Fig. 5. The synthetic matvec kernel, a benchmark for matrix vector multipli- convolve, off=0, KNL cations. COLS, ROWS, and ALPHA are the introduced compile-time constants. 16.91 x 0 2 4 6 8 10 runtime [s] extern "C" void convolve_kart(float restrict input, * matvec float* restrict kernel, Fig. 7. Speed-ups for two synthetic kernels (single threaded) and float* restrict output) convolve (OpenMP). The speed-ups do not include the compilation overhead, { as it this would require defining a somehow “realistic” number of kernel calls. #pragma omp parallel for A compile-time known alpha=0 entirely removed the computation loops. The for (int i = 0; i < INPUT_SIZE; ++i) { average compilation times for each kernel ranges from 0.90 to 0.92 s on the float sum = 0; Xeon (HSW) and 3.46 to 3.60 s on the Xeon Phi processor (KNL). for (int j = 0; j < KERNEL_SIZE; ++j) sum += kernel[j] * input[OFF + i + j]; output[i] = sum; HEOM kernel runtime comparison } w/o KART KART } 2x Xeon E5-2630v3 (HSW), AV 1.09 x
2x Xeon E5-2630v3 (HSW), MV Fig. 6. The synthetic convolve kernel, implementing a one dimensional 1.12 x convolution algorithm. , , and are the INPUT_SIZE KERNEL_SIZE OFF Xeon Phi 7210 (KNL), AV introduced compile-time constants. 1.59 x
Xeon Phi 7210 (KNL), MV 1.44 x
0 5 10 15 20 kernel speed-up for alpha=1 where the inner-most loop was runtime per call [ms] removed since there is only one column of matrix a. For the Fig. 8. Runtimes and speed-ups achieved by using KART for the OpenMP convolve kernel, having off, input size, and kernel size known version of the HEOM benchmark. Matrix size and number of matrices read by the compiler could improve kernel runtimes by 16.9×. from the input are used as compile-time constants during runtime compilation. Those cases are just examples to show the potential that can AV is automatic and MV is manual vectorization. Amortization of compile time requires 659 and 7195 calls, 90 % of the theoretical gain (sb − 1) are be unleashed through runtime code generation and optimiza- reached after 7195 and 24431 calls, on Xeon (HSW) and Xeon Phi (KNL), tion. Especially, if the input data has certain properties that respectively. Typical application runs need 4000 to 40000 calls. can be exploited by compiler optimizations of a kernel, the effects can be quite high. absolute runtime was still slower than that of the code emitted B. Real World Applications by the Intel C++ Compiler. 1) HEOM: HEOM is a short for Hierarchical Equations of The Intel Xeon Phi (co)processor benefits much more from Motion, which is a mathematical approach to solving open the compiler optimizations. Its light-weight cores and the miss- quantum systems. It is used, for instance, to simulate the ing L3 cache requires better optimized code than the Haswell energy transfer in photo-active molecular complexes to under- architecture. The effect on the whole HEOM application run- stand recent observations of femto-second laser-experiments time cannot be measured as only the Hexciton kernel has been on photosynthesis molecules found in bacteria and plants. ported to OpenMP. However, it is the computationally most For the GPU HEOM OpenCL implementation [19], a complex part of the solver, and the remaining computations detailed case study [20] including a benchmark is publicly will most likely benefit in a similar way as they work on the available. It compares a variety of OpenCL and OpenMP same data structure using the same constants. The compile- implementations of the Hexciton kernel as well as different time overhead is not relevant for production runs of the HEOM vectorization and optimization strategies. We integrated KART code, as they typically take several hours. into the OpenMP version of the benchmark to build the 2) WSM6 Proxy Code: We integrated KART into the different kernel variants at runtime and enable the use of input- WSM6 proxy code [1] written in Fortran for demonstrating specific constants at kernel compile time. Two constants, the different optimization techniques. WSM6—the WRF SIngle matrix dimension and the number of matrices of the central Moment 6-class Microphysics schema—is part of the Weather HEOM data-structure, are most relevant for the memory access Research and Forecast (WRF) model, widely used for numer- pattern and the loop counts. Figure 8 shows the achieved ical weather prediction. The authors investigated the impact performance gain. The plotted values are from the set of the of known compile-time constants describing loop lengths and best-performing kernel variants using the Intel C++ Compiler. array dimensions on the efficiency of the generated code. The highest speed-up of 2.6× was observed for clang++ on The original benchmark uses the C preprocessor and a set the Xeon CPU with the manually vectorized kernel, but the of Perl scripts to modify the source code to generate a version
7 WSM6 kernel runtime comparison w/o KART KART puting” (Intel PCC) at ZIB. We thank the “The North-German 2x Xeon E5-2630v3 (HSW) Supercomputing Alliance - HLRN” for providing us access to 1.11 x the HLRN-III production system ‘Konrad’ and the Cray TDS Xeon Phi 7210 (KNL) 1.16 x system with Intel KNL nodes. 0 20 40 60 80 100 runtime [ms] REFERENCES
Fig. 9. Runtime comparison and speed-ups for using runtime compilation [1] T. Henderson, J. Michalakes, I. Gokhale, and A. Jha, “Chapter 2 with the WSM6 Kernel on the Xeon (HSW) and Xeon Phi (KNL) architecture. - Numerical Weather Prediction Optimization,” in High Performance Average compilation times are 3.24 s on the Xeon (HSW) and 20.13 s on the Parallelism Pearls, J. Reinders and J. Jeffers, Eds. Boston: Morgan Xeon Phi processor (KNL). Kaufmann, 2015, pp. 7 – 23. [2] S. Siso, “DL MESO Code Modernization.” Intel Xeon Phi Users Group (IXPUG), March 2016, IXPUG Workshop, Ostrava. [3] OpenMP Architecture Review Board, “OpenMP Application Program with compile time constants for every run. This is no longer Interface, Version 4.5,” 2015, http://www.openmp.org/. necessary when using KART. Our modified Kernel uses the [4] Khronos OpenCL Working Group, “The OpenCL Specification, Version preprocessor only to simplify the build process at runtime. For 2.2,” March 2016, https://www.khronos.org/registry/cl/specs/opencl-2.2. pdf. the WSM6 proxy code, KART achieved a speed-up of 1.16× [5] J. Nickolls, I. Buck, M. Garland, and K. Skadron, “Scalable Parallel for the slightly modified kernel (see Figure 9). Programming with CUDA,” Queue, vol. 6, no. 2, pp. 40–53, March 2008. [Online]. Available: http://doi.acm.org/10.1145/1365490.1365500 VI.CONCLUSIONAND OUTLOOK [6] C. Lattner and V. Adve, “LLVM: A Compilation Framework for Lifelong Studies on the optimization of HPC workloads for many- Program Analysis and Transformation,” in CGO, San Jose, CA, USA, March 2004, pp. 75–88, llvm.org. core CPUs demonstrate the impact of available constants on [7] B. Joo,´ “LLVM and QDP-JIT,” https://www.ixpug.org/events/ the optimization capabilities during the compilation step. In ixpug-annual-meeting-2015, September 2015, iXPUG Workshop, particular, the more parameters affecting the data layouts 2015, Berkeley. [8] F. T. Winter, M. A. Clark, R. G. Edwards, and B. Joo,´ “A and loop/branching structures can be provided, the better the Framework for Lattice QCD Calculations on GPUs,” in Proceedings compiler is able to optimize. Compiling hot-spot kernels at of the 2014 IEEE 28th International Parallel and Distributed run-time for the context of their invocation can yield better Processing Symposium, ser. IPDPS ’14. Washington, DC, USA: IEEE Computer Society, 2014, pp. 1073–1082. [Online]. Available: optimizations and thus shorter time-to-solutions. http://dx.doi.org/10.1109/IPDPS.2014.112 We have presented KART, a flexible and easy to use [9] J. Bezanson, S. Karpinski, V. B. Shah, and A. Edelman, “Julia: A framework that allows runtime compilation with a variety of C, fast dynamic language for technical computing,” September 2012, http://julialang.org. C++, and Fortran command line compilers. It supports dynamic [10] J. Bezanson, A. Edelman, S. Karpinski, and V. B. Shah, “Julia: A fresh re-compilation during program execution if kernel parameters approach to numerical computing,” November 2014. such as data sizes are changed, e.g., after load-balancing [11] A. Heinecke, H. Pabst, and G. Henry, “LIBXSMM: A High Performance Library for Small Matrix Multiplications,” steps between nodes or to adjust for different input data http://sc15.supercomputing.org/sites/all/themes/SC15images/tech\ sets. For applications with irregularly distributed data across poster/tech\ poster\ pages/post137.html, November 2015, poster. the compute nodes, our approach can support individually [12] T. Schneider, F. Kjolstad, and T. Hoefler, “MPI Datatype Processing using Runtime Compilation,” in Proceedings of the 20th European MPI optimized kernels for each node. Users’ Group Meeting. ACM, Sep. 2013, pp. 19–24. We have demonstrated the effectiveness of our solution [13] A. Heinecke, M. Klemm, D. Pfluger,¨ A. Bode, and H.-J. Bungartz, “Ex- R for synthetic and real-world kernels. Potential speed-ups can tending a Highly Parallel Data Mining Algorithm to the Intel Many Integrated Core Architecture,” in Euro-Par 2011: Parallel Processing be significant, depending on the specifics of the kernel and Workshops, Bordeaux, France, August 2011, pp. 375–384, lNCS 7156. target architecture. KART considerably accelerated the HEOM [14] NVIDIA, “NVRTC - CUDA Runtime Compilation User Guide,” (1.59×) and WSM6 (1.16×) kernels with an acceptable com- September 2015, http://docs.nvidia.com/cuda/pdf/NVRTC User Guide. pdf. pilation overhead, given the typical runtimes of HPC applica- [15] “OpenMP: Support for the OpenMP language,” http://openmp.llvm.org/, tions. The approach is particularly effective on the Intel Xeon April 2016. Phi (co)processor, which represents a many-core architecture [16] “OpenMP Compilers,” http://openmp.org/wp/openmp-compilers/, September 2016. whose successors are a likely technology for exascale systems. [17] B. Schling, The Boost C++ Libraries. XML Press, 2011. The convenience and efficiency of runtime compilation could [18] M. Noack, F. Wende, T. Steinke, and F. Cordes, “A Unified be further improved if it would be integrated into a pro- Programming Model for Intra- and Inter-Node Offloading on Xeon Phi Clusters,” in International Conference for High Performance gramming model like OpenMP or if compiler vendors would Computing, Networking, Storage and Analysis, SC 2014, New Orleans, provide an—ideally standardized—API to their tools. LA, USA, November 16-21, 2014, 2014, pp. 203–214. [Online]. KART and the benchmarks are available as open source Available: http://dx.doi.org/10.1109/SC.2014.22 [19] C. Kreisbeck, T. Kramer, and A. Aspuru-Guzik, “Scalable High- on GitHub (https://github.com/noma/kart). We expect further Performance Algorithm for the Simulation of Exciton Dynamics. improvements of KART and adoptions by other HPC applica- Application to the Light-Harvesting Complex II in the Presence tions in the near future and are currently working on automatic of Resonant Vibrational Modes,” Journal of Chemical Theory and Computation, vol. 10, no. 9, pp. 4045–4054, 2014, pMID: 26588548. caching mechanisms for compiled kernels. [Online]. Available: http://dx.doi.org/10.1021/ct500629s [20] M. Noack, F. Wende, and K.-D. Oertel, “Chapter 19 - OpenCL: There ACKNOWLEDGMENTS and Back Again,” in High Performance Parallelism Pearls, J. Reinders This work is partially supported by Intel Corporation within and J. Jeffers, Eds. Boston: Morgan Kaufmann, 2015, pp. 355 – 378. the “Research Center for Many-core High-Performance Com-
8