Automatic Generation of Fast BLAS3-GEMM:
A Portable Compiler Approach
Xing Su† Xiangke Liao† Jingling Xue‡ †National Laboratory for Parallel and Distributed Processing, College of Computer, NUDT, Changsha, 410073, China ‡School of Computer Science and Engineering, UNSW, Sydney, NSW 2052, Australia
Abstract the other level-3 BLAS routines can be defined in terms of GEMM is the main computational kernel in BLAS3. Its GEMM and some level 1 and 2 computations [14]. In addi- micro-kernel is either hand-crafted in assembly code or gen- tion, GEMM is also the key library routine for deep learn- erated from C code by general-purpose compilers (guided by ing. Finally, the LINPACK benchmarks rely critically on architecture-specific directives or auto-tuning). Therefore, GEMM for its performance measurements. Therefore, op- either performance or portability suffers. timizing GEMM is the core task in any high-performance We present a POrtable Compiler Approach, POCA, im- BLAS implementation. However, given a C specification of plemented in LLVM, to automatically generate and opti- GEMM, general-purpose compilers are still not able to gen- mize this micro-kernel in an architecture-independent man- erate machine code that achieves near peak performance. ner, without involving domain experts. The key insight is Broadly speaking, there are three approaches for obtain- to leverage a wide range of architecture-specific abstrac- ing optimized loop-based GEMM kernels, (1) assembly pro- tions already available in LLVM, by first generating a vector- gramming, (2) auto-tuning, and (3) directive-based program- ized micro-kernel in the architecture-independent LLVM IR ming, yielding different tradeoffs between performance and and then improving its performance by applying a series of portability. With assembly programming, domain experts domain-specific yet architecture-independent optimizations. write a few innermost loops in GEMM directly in assem- The optimized micro-kernel drops easily in existing GEMM bly code. In the case of auto-tuning, ATLAS [32] generates frameworks such as BLIS and OpenBLAS. Validation fo- different GEMM kernels (with different parameter values) cuses on optimizing GEMM in double precision on two in C and compile them to run on the actual computing sys- architectures. On Intel Sandybridge and AArch64 Cortex- tem to find the best-performing one. Finally, directive-based A57, POCA’s micro-kernels outperform expert-crafted as- programming is embraced by POET [35] and AUGEM [31]. sembly code by 2.35% and 7.54%, respectively, and both Given a GEMM kernel in C, POET inserts annotations into BLIS and OpenBLAS achieve competitive or better perfor- the C code to direct source-to-source compiler transforma- mance once their micro-kernels are replaced by POCA’s. tions and AUGEM uses a template-based method to match predefined patterns in the C code and transforms the matched Categories and Subject Descriptors D.3.4 [Programming C code sequence into an optimized assembly code sequence. Languages]: Processors—Compilers; G.1.3 [Numerical These three approaches make different performance and Analysis]: Numerical Linear Algebra portability tradeoffs. Coding GEMM in assembly by do- Keywords dense linear algebra, GEMM, code optimization main experts can achieve near peak performance but is te- dious and non-portable. Auto-tuning makes the opposite 1. Introduction tradeoff. For example, ATLAS relies on general-purpose Dense linear algebra libraries are fundamental in scien- compilers to generate optimized GEMM kernels for differ- tific computing. Basic Linear Algebra Subprograms (BLAS) ent architectures automatically, thus resulting in portable are routines that provide standard building blocks for per- but sub-optimal performance. In the case of directive-based forming vector-vector operations (level 1), matrix-vector op- programming, POET and AUGEM resort to architecture- erations (level 2), and matrix-matrix operations (level 3). specific annotations and templates, respectively, written still BLAS libraries are widely available. Vendor supplied imple- by domain experts, to guide general-purpose compilers to mentations include Intel MKL, AMD ACML and NVIDIA produce optimized GEMM kernels. In particular, AUGEM cuBLAS. The HPC community have also contributed sev- is shown to generate optimized GEMM kernels for x86 only. eral high-performance BLAS implementations such as AT- How do we obtain near peak yet portable performance LAS [32], GotoBLAS [12], OpenBLAS [37] and BLIS [28]. for GEMM automatically for a wide range of architectures? For the level-3 BLAS (BLAS3), GEMM (GEneral Ma- Despite its algorithmic simplicity, this problem is very chal- trix Multiplication) is the main computational kernel, as
978-1-5090-4931-8/17 c 2017 IEEE 122 CGO 2017, Austin, USA
Accepted for publication by IEEE. c 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. A B C GEMM Blocked Algorithm K N N for jj = 0:N :N 1 c ¡ M K M A =A; B = B[:][jj:jj + N 1]; C = C[:][jj:jj + N 1]; 0 0 c ¡ 0 c ¡ layer 1 Nc Nc K Nc Nc Kc for kk = 0:K :K 1 c ¡ M K M A =A [:][kk:kk + K 1]; B =B [kk:kk + K 1][:]; C =C ; 1 0 c ¡ 1 0 c ¡ 1 0 layer 2 Kc Kc Nc Nc Kc for ii = 1:M :M 1 Mc Mc c ¡ M M A =A [ii:ii + M 1][:]; B =B ; C =C [ii:ii + M 1][:]; 2 1 c ¡ 2 1 2 1 c ¡ layer 3 Kc Nc Nc for j = 1:Nr:Nc 1 Kc ¡ Mc Mc Nr A =A ; B =B [:][j:j + N 1]; C =C [:][j:j + N 1]; 3 2 3 2 r ¡ 3 2 r ¡ layer 4 Nr Kc Nr Nr Mr Mr for i = 0:Mr:Mc 1 Kc ¡ Mc Mc A =A [i:i + M 1][:]; B =B ; C =C [i:i + M 1][:]; 4 3 r ¡ 4 3 4 3 r ¡ layer 5 Kc Nr Nr 1 Mr Mr for k = 0:1:K 1 c ¡ 1 Kc
A5=A4[:][k]; B5=B4[k][:]; C5=C4; l macro-kernel (GEBP) e n
layer 6 r
1 Nr Nr e k
1 ¹
Mr Mr C +=A B ; 5 5 £ 5
layer 7 Figure 1. Structure of blocked DGEMM (the structures of SGEMM, CGEMM and ZGEMM are similar (Section 2)). lenging to solve. Any promising solution is significant due Intel Sandybridge and AArch64 Cortex-A57, POCA’s to the continued proliferation of computer architectures. micro-kernels outperform expert-crafted assembly code In this paper, we present a POrtable Compiler Approach, by 2.35% and 7.54% respectively. In addition, both BLIS POCA, implemented in LLVM, to automatically generate and OpenBLAS achieve competitive or better perfor- and optimize GEMM in an architecture-independent man- mance once their micro-kernels are replaced by POCA’s. ner. Due to the nature of GEMM, it suffices to focus on its • We provide a comprehensive quantitative analysis to un- innermost loop, known as a micro-kernel, µkernel. The key derstand and evaluate the impact of POCA-specific com- insight is to leverage a wide range of architecture-specific piler optimizations on kernel performance. abstractions (e.g., SIMD engines supported and instruction To the best of our knowledge, this is the first compiler ap- latencies) already available in LLVM, by first generating a proach for generating fast GEMM kernels portably, without vectorized µkernel in the architecture-independent LLVM involving domain experts. While the work presented here is IR and then boosting its performance by applying a series of for GEMM in BLAS3, the approach can be applied to the domain-specific yet architecture-independent optimizations. other kernels in BLAS3 such as TRSM and GEMV. The optimized µkernel, obtained without any involvement of domain experts, drops easily in existing GEMM frameworks 2. Structure of GEMM such as GotoBLAS [12], OpenBLAS [37] and BLIS [28]. GEMM comes with four main varaints, SGEMM, DGEMM, We restrict our presentation to GEMM that works on CGEMM and ZGEMM, which operate on four different data double-precision real numbers, known as DGEMM, as in types, single-precision floating-point (S), double-precision prior work [26, 31, 35], for two reasons. First, the basic floating-point (D), complex single-precision floating-point idea behind POCA applies to other variants of GEMM such (C), and complex double-precision floating-point (Z). We as SGEMM, CGEMM and ZGEMM (as discussed in Sec- first describe a blocked algorithm for DGEMM and intro- tion 2). Second, the LINPACK benchmarks, which rely on duce several basic concepts used throughout the paper. We GEMM as the performance-critical routine, must work on then look at SGEMM, CGEMM and ZGEMM briefly. double-precision real numbers in order to build the TOP500 DGEMM performs a matrix-multiply-add operation on list, ranking the world’s most powerful supercomputers. double-precision real numbers, C = βC + αAB, where This paper makes the following contributions: A, B and C are matrices of sizes M ×K, K×N and M ×N, respectively, and α and β are scalars. While this operation is • We introduce a portable compiler approach, POCA for algorithmically simple, so that a 3-deep loop nest suffices to generating highly optimized GEMM fully automatically. accomplish it, a high-performance implementation is quite • We evaluate POCA in optimizing GEMM that works on sophisticated due to the presence of multi-level memory hi- double-precision real numbers on two architectures. On erarchies on modern processors. Figure 1 shows the program
123 structure of a real-world DGEMM kernel implemented via a focus on optimizing GEBP or µkernel that works on single- blocked algorithm. Each loop (in the original 3-deep loop and double-precision real numbers only. nest) is tiled, resulting in a total of six loops (often referred 3. POCA: A Portal Compiler Approach to as layers 1 – 6). Loop tiling, together with data packing and prefetching, serves to improve data locality and overlap To obtain a fast GEMM kernel portably, we focus on × computation and memory access effectively. automatically generating a fast µkernel of size Mr Nr that In this blocked algorithm, the loops over the N, K and works on real numbers portably, where Mr and Nr are input parameters determined by the overall design of GEMM [12]. M matrix dimensions are tiled by sizes Nc, Kc and Mc at layers 1, 2 and 3, respectively. N , K and M are carefully B c c c C = (cpq ) ; selected so that matrix B2 (Kc × Nc) fits into the L3 cache T = (tpq ) ; // initialized to 0 b0 b1 b2 b3 f o r k = 1 : Kc : 1 (if it exists), A3 (Mc × Kc) fits into the L2 cache, and both ′ A = A [:][k] = (ap) ; a0 c00 c01 c02 c03 × × ′ A4 (Mr Kc) and B4 (Kc Nr) fit into the L1 cache. At B = B [k][:] = (bq ) ; a1 c10 c11 c12 c13 f o r p = 0 : Mr −1 : 1 A C layers 4 and 5, the N and M dimensions are further tiled a2 c20 c21 c22 c23 f o r q = 0 : Nr −1 : 1 by sizes Nr and Mr, respectively. As a result, the innermost tpq += ap ∗ bq a3 c30 c31 c32 c33 ¯ ∗ ∗ loop at layer 6 goes over the K dimension for a total of Kc cpq = β cpq + α tpq times, with each iteration performing a rank-1 update on the Figure 2. µkernel with Mr × Nr = 4 × 4 illustrated. Mr × Nr submatrix of C, as shown at layer 7. Note that a new scalar β¯, where β¯ = (kk == 0 ? β : 1), is introduced To ease presentation, we will focus on µkernel given in at layer 7, to ensure that βC is computed only once by loop Figure 2 abstracted from Figure 1. A′, B′ and C are matrices kk at layer 2. N and M are selected so that M elements r r r of sizes M × K , K × N , and M × N , respectively. from A, N elements from B and M × N elements from r c c r r r r r r A (B) is the k-th column (row) of A′ (B′). The rank-1 C can fit into registers, with the M × N submatrix of r r update T + = A × B, where T = (t ), A = (a ) and C residing in registers throughout the innermost loop at pq p B = (b ), is first performed in the two innermost loops layer 6. Finally, A [ii : ii + M − 1][:] and B are packed q 1 c 1 and a multiply-add operation is then performed at the end into continuous buffers at layer 3 in order to ensure their of loop k to update C, as illustrated for the case when consecutive memory access at layers 4 – 7. M × N = 4 × 4. Below we will use a double-precision Goto [12] factors out the innermost three loops at layers r r µkernel of this particular size as a running example. 4 – 6 for computing C += A × B as an architecture- 2 2 2 To generate a fast µkernel kernel portably even though it dependent kernel, known as a macro-kernel and abbreviated looks simple, we must address two major problems: as GEBP (GEneral multiply of a Block of A and a Panel of B). In GotoBLAS [12], and its successor, OpenBLAS [37], µkernel Generation. How do we perform instruction selec- their GEMM kernels are developed based on this factoriza- tion to accomplish the µkernel computation, which is tion, with GEBP coded in assembly. Thus, a hand-crafted expected to be fully vectorized? This problem depends GEBP kernel, together with a set of well-chosen tile sizes on the processor’s ISA. Different ISAs lead to different M , N , K , M and N , will suffice to instantiate a highly vectorized µkernels. Even for the same ISA, different c c c r r × optimized GEMM on a target processor. BLIS [28] factors µkernels can result for different tile sizes Mr Nr given. out an even smaller kernel, the innermost loop at layer 6, µkernel Optimization. How do we perform instruction known as a micro-kernel and denoted as µkernel, as the only scheduling to achieve near peak performance? This architecture-dependent kernel to be coded in assembly. problem is also architecture-dependent. General-purpose While SGEMM shares exactly the same structure as compilers are known to be ineffective, since they cannot DGEMM, CGEMM and ZGEMM are slightly different but exploit domain-specific knowledge well. are easily adapted from SGEMM and DGEMM, respec- POCA represents the first compiler approach for obtaining tively. In general, a complex matrix-multiply-add operation near peak performance portably for µkernel, as illustrated β(Cr + Cii) + α(Ar + Aii)(Br + Bii) is factorized as in Figure 3. POCA is portable, because it operates on top (βCr + αArBr − αAiBi) + (βCi + αArBi + αAiBr)i and of the architecture-specific abstractions already provided by thus implemented in terms of four real matrix-multiply-add general-purpose compilers such as LLVM for a wide range operations. For [CZ]GEMM, A1[ii : ii + MC − 1][:] and of computer architectures. POCA can deliver near peak per- B1 at layer 3 are each packed into two real submatrices of formance, because it is structured into two sequential phases, the same size, one for the real part and one for the imaginary µkernel generation and µkernel optimization. Our µkernel part. As a result, a complex GEBP kernel is decomposed generator produces a vectorized µkernel in the architecture- into four real GEBP kernels with each containing a real independent LLVM IR. The main task performed here is µkernel. All the tile sizes are determined similarly as in the instruction selection. Our µkernel optimizer then applies a case of DGEMM to improve cache locality. Therefore, for series of domain-specific but architecture-independent opti- SGEMM, DGEMM, CGEMM and ZGEMM, it suffices to mizations to produce a highly optimized µkernel. The main task performed here is instruction scheduling.
124 fmla (vector, by element) on AArch64). VectorLoad- Shuffle is adopted on architectures such as x86 that sup- Instruction Info μ port vector shuffling. Otherwise, ScalarLoad-Broadcast is Register Info kernel Genrator SIMD Engine selected, resulting in a scalar load for each element of B and Function Units ... Single-Iteration its subsequent broadcast to all the lanes in a vector register. Scheduling After all the loads for A and B are generated, arithmetic LLVM Abstract Two-Iteration instructions are selected to perform the accumulation opera- Machine Pipelining tion in µkernel. FMA instructions are used, if available. Descriptions Unroll-based Rotating Register Example Figure 4 illustrates how the 4×4 double-precision X86_64 AArch64 ... Allocation µkernel is vectorized on Sandybridge and Cortex-A57 (with Sandybridge Cortex-A57 μkernel Optimizer their architectural features listed in Table 5). On Sandy- bridge, where VL = 4, VectorLoad-Shuffle is adopted. Af- Figure 3. POCA’s framework for generating fast µkernels ter the vector loads for A and B (lines 1 – 2), the three vector portably on top of LLVM’s architecture-specific abstractions. shuffle instructions (lines 3 – 5) are executed to obtain three different permutations of B. Then the outer product A × B 3.1 The µkernel Generator is computed (lines 6 – 13). On Sandybridge, FMA is not We take as input, the tile size Mr × Nr for µkernel, the supported. In lines 14 – 15, the loop induction instructions data type of matrix elements (e.g., float or double), and are executed to enable new vectors A and B to be loaded in the architecture targeted, as shown in Figure 3, and produce the next iteration. The prefetching instructions in lines 16 – a vectorized µkernel in LLVM IR. To perform instruction 18 will be explained shortly in Section 3.1.2. Note that the selection, we carry out vectorization, data prefetching and shuffle instructions used for restoring C0 – C3 to the storage addressing mode optimization, as described below. While order required for C, as shown in Figure 2, are omitted. seemingly standard, these passes are applied portably on top On Cortex-A57, where VL = 2, VectorLoad-Broadcast of LLVM’s architecture-specific abstractions. is adopted. The loads for A and B are vectorized (lines 1 – 3.1.1 Vectorization 4). Then, each scalar element of B is replicated in a distinct vector register (lines 5 – 8). These four scalar broadcasts To achieve near peak performance, vector instructions are have zero cost, as they will be later combined and performed utilized whenever possible. While SIMD accelerators are together with their corresponding fmla instructions (lines 9 ubiquitous in microprocessors, their SIMD features can dif- – 16). For this architecture, FMA instructions are selected. fer greatly. Despite its simplicity, µkernel must be vectorized carefully to achieve portability due to the ISA diversity. 3.1.2 Data Prefetching To vectorize µkernel in Figure 2, we adopt the same Data prefetching is important for µkernel. In Figure 2, ′ ′ design exercised in hand-crafted GEMM variants, such as A and B symbolize A4 (Mr × Kc) and B4 (Kc × Nr) GotoBLAS [12], OpenBLAS [37] and BLIS [28]. The two in Figure 1, respectively. According to Section 2, these tile loops governing the rank-1 update are fully unrolled. C sizes are selected to fit A4 and B4 into the L1 cache. resides in registers as live-ins but the two vectors A and B From Figure 1, we can see that A4 varies but B4 stays are loaded from memory into registers across the iterations unchanged at layer 5. Therefore, two different prefetching of loop k. To ensure consecutive memory access, A′[:][k] and strategies are applied, following GotoBLAS [12], Open- ′ B [k][:] have already been packed into contiguous buffers. BLAS [37] and BLIS [28]. For B4, its rows are prefetched Finally, tile size Mr × Nr has also been selected so that Mr several iterations in advance. For A4, the next Mr × Kc row next − is a multiple of the vector length, denoted VL, expressed in block of A3, A4 := A3[i + Mr : i + 2Mr 1][:], is next terms of the maximum number of data elements residing in a prefetched, as A4 will be used in the next call to µkernel. vector register. Thus, A can always be loaded into Mr/V L Consider the vectorized µkernel for Sandybridge given vector registers with Mr/V L aligned vector loads and C in Figure 4, where the instructions in lines 16 – 18 serve × next occupies exactly Mr Nr/V L vector registers. to prefetch A4 and B4. The base addresses of A4, B4 next Therefore, it suffices to consider how to load an Nr-sized and A4 are represented by AP, BP and AN, respectively. vector B into vector registers. Currently, we make use of Let St be the size of the data type of matrix elements and three strategies, VectorLoad-Broadcast, VectorLoad-Shuffle Sc the cacheline size for the underlying L1 data cache (in and ScalarLoad-Broadcast. To achieve high bandwidth, vec- bytes). For each iteration of loop k in µkernel (Figure 2), a next tor loads (VectorLoad-Broadcast and VectorLoad-Shuffle) column of A4 and a row of B4 are prefetched by issuing are preferred over scalar loads (ScalarLoad-Broadcast). ⌈Mr ∗ St/Sc⌉ and ⌈Nr ∗ St/Sc⌉ prefetching instructions, However, vector loads are considered only when they are respectively. When Mr × Nr = 4 × 4, only ⌈4 ∗ 8/64⌉ = 1 all aligned, which happens when VL divides Nr. Then prefetching instruction is needed each. The prefetch distance there are two cases. VectorLoad-Broadcast is adopted on ar- used for B4 is empirical, ranging typically between 256 – chitectures that support “zero-cost scalar broadcasts” (e.g., 768 bytes. In POCA, 512 bytes are used, by default.
125 1 A0 = vload AP //
126 � function units and their latency and throughput), provided by �� � � � LLVM’s architecture-specific abstraction, are considered. ���� � � �� � � BalancedWKLD We aim to balance the workload for the � �� � �� ����� � � function units at the processor backend. Let FU(Ilast) � �� � � � �� (FU(I)) be the set of function units required for execut- � � �� �� � ���� � � � ing Ilast (I), where Ilast (I) is the instruction most recently �� � � ���� �� � (being) scheduled. We set BalancedW KLD(I) to 1 (true) � � � � � � � if FU(I ) ∩ FU(I) = ∅, since their µops will be dis- �� �� �� last � � patched to different function units, and 0 (false) otherwise �� � �� � �� FPFirst P F F irst(I) returns 1 (true) if I is a floating- point instruction, since I is highly likely to be on a critical Figure 5. Data dependence graph for the 4 × 4 vectorized path in the dependence graph, and 0 (false) otherwise. µkernel given in Figure 4(a) on Sandybridge. MaxHeight, MinDepth, MaxHeight and InstOrder These (EXIT) is linked with the original source (sink) nodes. Each four are standard, already available in LLVM. MaxRelease dependence edge is labeled by the latency of the instruction. favors scheduling instructions such that more instructions I Formally, let be the set of candidate instructions such will be ready to be scheduled afterwards. MinDepth and that the instructions that they depend on have been sched- MaxHeight favor instructions with smaller depths and I → uled. A scheduling heuristic is a function h : Xh that larger heights, respectively. The depth (height) of an in- ∈ ∈ I assigns a score x Xh to each instruction I . Here, struction is measured as the longest distance from ENTRY ≺ Xh is a totally ordered set with a total order Xh . For two (to EXIT) in the dependence graph. Finally, InstOrder is a ∈ I ≺ instructions I1,I2 , h prioritizes I1 if h(I1) Xh h(I2). fallback heuristic that respects the original instruction order. To break ties, list scheduling can be made more effective Example Table 3 gives the schedule for Figure 4(a) on by applying several heuristics to select the best instruction Sandybridge (based on the architectural information given in to be scheduled next. If h(I ) = h(I ), where I ,I ∈ I, 1 Xh 2 1 2 Tables 1 and 5). With MaxReg = 8 assumed, the instruc- for a particular heuristic, then the next heuristic is tried. tions are listed in the order scheduled from left to right. For Table 2. Single-iteration scheduling heuristics. each instruction, the second row gives the cycle at which it Order is dispatched and the third row gives LiveReg immediately Heuristic (h) Range (Xh) ≺ Xh =Xh afterwards. The single-iteration latency is 12 cycles. RegPress B > = This schedule is unconventional. We have obtained it due EarliestCycle N < = to the sophisticated heuristics used for modeling Sandy- BalancedWKLD B > = FPFirst B > = bridge’s out-of-order behavior (provided by LLVM’s ma- MaxRelease N > = chine abstraction). Note that the 4 × 4 µkernel has 4 live- MinDepth N < = MaxHeight N > = in registers, C0, C1, C2 and C3. Thus, LiveReg = 5 af- InstOrder N < n/a ter the first instruction B0 = vload BP has been scheduled. Throughout, LiveReg is made as close to MaxReg as pos- Table 2 lists eight heuristics tried by the µkernel opti- sible, particularly in the middle of a schedule. mizer in that order, with the top four being POCA-specific. Let us select a particular scheduling point to highlight the For each heuristic, the “Range (X )” and “Order (≺ and h Xh capability of our cost model. Why is C0 scheduled before = )” columns give its totally ordered set and total order, Xh B3, M2 and M3 but executed only afterwards? After M1 respectively. N = {0, 1, ···} is the set of natural numbers. is scheduled, LiveReg = MaxReg = 8. Then the set of B = {0, 1}. <, > and = have the traditional meanings. candidate instructions becomes {C0, C1, M2, B3}. We have Below we describe the eight heuristics used: RegP ress(C0) = RegP ress(C1) = 1 and RegP ress(B3) RegPress We keep track of LiveReg and compute the set = RegP ress(M2) = 0. Thus, C0 and C1 are preferred, as of variables, KillSet(I), that are dead after instruction I is scheduling B3 and M2 would cause LiveReg > MaxReg. scheduled. We set RegP ress(I) to 0 (false) if LiveReg > According to the next heuristic, EarliestCycle(C0) = 9 MaxReg immediately afterwards, i.e., if LiveReg + 1− and EarliestCycle(C1) = 10. Thus, C0 is finally selected. | KillSet(I) |> MaxReg and 1 (true) otherwise. Now, the set of candidate instructions is {C1, M2, B3}. EarliestCycle EarliestCycle(I) gives the cycle at which We first compute RegP ress(C1) = RegP ress(M2) = I can be executed, based on EarliestCycle(I′) for all RegP ress(B3) = 1. We then find that EarliestCycle(C1) the instructions I′ already scheduled, thereby favoring the = 10 and EarliestCycle(M2) = EarliestCycle(B3) = earliest-starting instruction. We compute EarliestCycle(I) 6. As M2 and B3 tie with respect to BalacedW KLD, by modeling precisely the underlying architecture, from its F P F irst, MaxRelease and MinDepth, we finally ob- instruction decoding to µop execution. All relevant archi- tain 9 = MaxHeight(B3) > MaxHeight(M2) = 8. tectural features (e.g., out-of-order vs. in-order, issue width, Thus, B3 is selected to be scheduled after C0. However, B3
127 Table 3. Single-iteration scheduling for the double-precision µkernel in Figure 4(a) on Sandybridge (MaxReg = 8). Instruction B0 A0 BP AP PrfA AN PrfB B1 M0 B2 M1 C0 B3 M2 M3 C1 C2 C3 Cycle 0 0 0 0 1 1 1 4 4 5 5 9 6 6 7 10 11 12 LiveReg 566666677887887654
Table 4. The pipelined kernel Pµ obtained by two-iteration pipelining for the schedule in Table 3 on Sandybridge (MaxReg = 8), with the instructions from one iteration in S shown in normal font and the instructions from the next iteration in S′ in bold. Instruction B2 M1 C0 B3 M2 M3 B0’ C1 A0’ BP’ AP’ PrfA’ PrfB’ AN’ C2 B1’ M0’ C3 Cycle 5 5 9 6 6 7 7 10 8 8 9 9 10 9 11 10 11 12 LiveReg 88788787888 8 8 8 7 8 8 7
(at cycle 6) will be dispatched before C0 (at cycle 9) due to cycles each. After pipelining, two iterations take only 20 cy- register renaming and out-of-order dispatching modeled in cles (with 4 cycles in the prologue consisting of the first nine EarliestCycle. (Note that, for Intel Sandybridge, there are instructions in Table 3, 8 cycles in Pµ, and 8 cycles in the 16 256-bit AVX registers but 144 256-bit physical registers.) epilogue consisting of the last nine instructions in Table 3). 3.2.2 Two-Iteration Pipelining By comparing Tables 3 and 4, we note that we have pushed LiveReg to nearly the MaxReg = 8 limit. In this second pass, we apply software pipelining to only two consecutive iterations of µkernel that have been sched- 3.2.3 Unroll-based Rotating Register Allocation uled earlier. The objective is to maximize resource utiliza- In this last pass, we first unroll the pipelined µkernel with tion while pushing LiveReg even closer to MaxReg. its loop body as Pµ and then perform a rotating register allocation. We will choose an unroll factor of UF to ensure Algorithm 1 Two-iteration pipelining. that every variable is allocated to the same register every UF ′ ′ ′ Require: S =[I0,...,In−1],S = [I0,...,In−1] iterations, resulting in a rotating register allocation [10, 13, ¯ ¯ ¯ ∈ ∪ ′ Ensure: Pµ = [I0,..., In−1], where Ii S S 17, 30]. As LiveReg stays near but never exceeds MaxReg in ← 1: Pµ [] the pipelined schedule, as shown in Table 4, the final µkernel 2: i ← 0, j ← 0 3: i < n obtained is highly optimized, free also of register spills. while do ′ 4: if (inserting Ij immediately before Ii ensures that (1) We compute UF by using a directed graph created from LiveReg ⩽ MaxReg, and (2) EarliestCycle(I ) re- k the pipelined kernel Pµ, called Register Transfer Graph mains unchanged as in S, for every Ik, where k ⩾ i) then ′ (RTG). Each node represents a variable in P . Each edge 5: I ← I j ← j + 1 µ j , → 6: else v w represents a possible flow of a physical register, in- ← ← 7: I Ii, i i + 1 dicating that a register assigned to v can be reassigned to w 8: end if 9: Pµ ← Pµ #[I] // append I to Pµ after v is dead, since their live ranges do not interfere. 10: end while Algorithm 2 RTG construction. 11: Pµ ← Pµ[j : j + n − 1] // pipelined kernel 12: return Pµ // j = 0 =⇒ Pµ = S Require: Pµ = [I¯0,..., I¯n−1] Ensure: RTG Let S = [I0,...,In−1] be the schedule for one iteration 1: k ← MaxReg − LiveIn ′ ′ ′ 2: ∆psudo ← {F0,...,Fk−1} and S = [I0,...,In−1] the schedule for the next iteration 3: ∆ ← ∆ obtained in the first pass. We generate a two-iteration (mod- dead psudo 4: for I in [I¯ ,..., I¯ − ] in their sequential order do ulo) schedule by applying Algorithm 1. The basic idea is to 0 n 1 ′ 5: Let LHS(I) be the variable defined by I move j instructions from S into S while keeping the rel- 6: k ← k + |KillSet(I)| ative order of the instructions in each sequence unchanged, 7: ∆dead ← ∆dead ∪ KillSet(I) 8: for d ∈ ∆dead do so that the latency of S, which contains now n + j instruc- → − 9: Add d LHS(I) tions, remains the same. Let Pµ = S[j : n + j 1]. Typi- 10: end for cally, LiveReg is close to MaxReg in the middle of a one- 11: k ← k − 1 12: if k = 0 then iteration schedule (Table 3). Thus, Pµ usually contains all 13: ∆dead ← ϕ the j instructions moved from S′. In theory, this can always 14: end if ′ 15: end for be enforced by overlapping S gradually with a later part of 16: for d ∈ ∆dead and v ∈ ∆psudo do → S. Therefore, Pµ is the pipelined kernel. S[0 : j − 1] is the 17: Add d v ′ 18: end for prologue and S (without the j removed instructions) is the epilogue. In the special case when j = 0, which will never Algorithm 2 constructs the RTG needed. LiveIn gives µ M × N P = S happen for kernel of a well-chosen r r, µ . the set of live-in variables for Pµ (e.g., C0 – C3, A0, B1, M0 Example Continuing from the single-iteration schedule in our example). Thus, ∆psudo represents the set of pseudo given in Table 3 with 18 instructions, Table 4 shows the variables (i.e., registers) available initially. KillSet(I) rep- pipelined kernel Pµ consisting of also 18 instructions, with resents the set of variables that is dead after, i.e., killed by I. ′ half from S (in normal font) and half from S (in bold). We use ∆dead and k to keep track of the set of dead variables Previously, two iterations take 24 cycles to execute, with 12 and the number of free registers, respectively. Due to lines 8
128 – 10, the variable defined by I, i.e., LHS(I) can reuse any RQ1. Can the µkernels generated automatically by POCA register previously allocated to d ∈ ∆dead. Note that ∆dead outperform expert-crafted assembly versions? and k are not updated in sync, because when I is processed, RQ2. Can representative GEMM frameworks, OpenBLAS one free register must be allocated to LHS(I), causing k and BLIS, achieve competitive or better performance to go down by 1 (line 11). However, this free register may once their expert-crafted µkernels are replaced by POCA’s? be any previously assigned to any dead variable in ∆dead. RQ3. How effective are POCA’s optimizations? Thus, ∆dead remains unchanged and is cleared only when We have implemented POCA in LLVM (3.9.0). Its µkernel k = 0 (lines 12 – 14). This ensures that all possible flows generator is a stand-alone module generating vectorized of registers are considered. Even after ∆dead is cleared, ev- µkernels in LLVM IR. Its µkernel optimizer is embed- ery variable is guaranteed to have at least one (dead variable) ded into LLVM’s backend compiler. The “Single-Iteration predecessor, as LiveReg ⩽ MaxReg holds always. Finally, Scheduling” and “Two-Iteration Pipelining” passes replace → ∈ ∈ d v is added, where d ∆dead and v ∆psudo. LLVM’s machine scheduler. The “Unroll-based rotating We can find UF from RTG easily to unroll the pipelined Register Allocation” pass is implemented in terms of LLVM’s µkernel (with Pµ as its loop body) to enable a rotating register allocator by making hints on which physical reg- register allocation. As LiveReg ⩽ MaxReg, every node isters should be allocated to which variables. To optimize in RTG lies in a cycle. For every cycle c, which is not µkernel with POCA, LLVM’s “-O3” is turned on. necessarily an elementary cycle, let Tc be the number of live- POCA is implemented in about 9300 lines of C++, with out variables on c, called its period. Tc is an upper-bound for about 1500 LOC in its µkernel generator and 7800 LOC in the number of live variables in c, since any variable in c can its µkernel optimizer (3800 LOC for scheduling and pipelin- only become live after it predecessor is dead. ing, 2500 for register allocation, and 1500 for others). Let P = {c0, . . . , cm−1} be a partition of RTG, every ci is a cycle. Then UF = LCM(T0,...,Tm−1). A rotating Table 5. Configurations for Sandybridge and Cortext-A57. register allocation can be readily obtained [10, 17, 30]. Processor Sandybridge Cortex-A57 The problem of finding optimal unroll factors for in- Arch X86 64 AArch64 L1 cache 32KB (8-way) 32KB (-) struction scheduling and register allocation remains open. In L2 cache 256KB (8-way) 512KB (-) Hardware Dispatch out-of-order (4-issue) in-order (3-issue) practice, RTG is small, with less than 50 nodes. We find Execution out-of-order out-of-order { − } SIMD AVX (256b) Neon (128b) P = c0, . . . , cm 1 with exhaustive search by preferring a FMA NO YES UF =LCM(T0,...,Tm−1) within [2, 8] if it exists and the BLIS 0.1.8-29 0.1.8-29 smallest otherwise, in several seconds. As P is pipelined GEMM OpenBLAS 0.2.20-dev 0.2.20-dev µ ATLAS 3.10.3 3.10.3 with LiveReg close to MaxReg, the impact of different un- MKL 11.03.03 - roll factors on performance is small (as evaluated later). ICC 16.0.3 - Compilers GCC 4.8.2 6.1.0
�� �� We evaluate POCA on two processors, Intel Sandybridge �� �� �� ��� �� �� �� ��� and AArch64 Cortex-A57, with their hardware and software
�� ��� �� ��� configurations listed in Table 5. Sandybridge represents a series of dominant processors in the server and HPC mar- �� ��� ��� �� �� ��� ��� �� ket from the x86 camp, while Cortex-A57 is the first 64-bit �� �� �� �� AArch64 processor from the ARM camp. Both have their
�� �� own micro-architectures, differing in their cache design, (a) RTG (b) Partition µop dispatchers, and SIMD engines supported. For GEMM frameworks, we have selected two representative hand- Figure 6. RTG for P in Table 4, with one four-cycle parti- µ crafted implementations, OpenBLAS [37] and BLIS [28], tion shown in four different colors. The period of a cycle is one well-known auto-tuning-based implementation, AT- the number of live-out variables drawn in a solid shade. LAS [32], and Intel MKL. All matrices are real double- Example For the pipelined kernel Pµ in Table 4, Figure 6 precision matrices. For compiler-synthesized µkernels, ICC shows its RTG and a partition with four cycles. From the and GCC, are considered, with “-O3” turned on. LLVM is last column in Table 4, we see that Livein = 7. Thus, omitted since it generates poorer code for µkernel. F 0 represents the only pseudo variable initially available. 4.1 RQ1: The µkernel Performance For the partition shown, its four circles are drawn in four different colors. For each cycle, its period is represented We compare POCA with assembly programming, ICC by the number of nodes in a solid shade. Thus, UF = and GCC. As discussed in Section 2, OpenBLAS [37] and BLIS [28] are written by domain experts, with GEBP and LCM(Tblue,Tred,Tgreen,Tyellow)=LCM(4, 1, 2, 1) = 4. µkernel depicted in Figure 1. In OpenBLAS, GEBP is coded 4. Performance Evaluation in assembly. In BLIS, only µkernel is coded in assembly. In our evaluation, we address three research questions: Therefore, its expert-crafted µkernel, identified as BLIS-µ,
129 is selected in our evaluation. For BLIS-µ, Mr × Nr = 8 × 4 18 × × 17 on Sandybridge and Mr Nr = 4 8 on Cortex-A57. 16 For the µkernels generated by POCA for both processors 15 14 (in several seconds in each case), UF = 4. For BLIS-µ, 13 ATLAS OpenBLAS BLIS OpenBLAS-POCA 12 loop unrolling is applied identically. To compare POCA with BLIS-POCA gcc ∈ { } 11 MKL icc each compiler C ICC, GCC , we consider two versions Performance (GFLOPS) 10
256512768 of µkernel, C1 and C4, generated by C, where i in Ci is 10241280153617922048230425602816307233283584384040964352460848645120537656325888614464006656691271687424768079368192 Matrix Size (M=N=K) the unroll factor used. In each case, we wrote a µkernel (a) Sandybridge loop vectorized with low-level SIMD intrinsics in exactly the same way as the vectorized µkernel generated by POCA. 5.5
This loop is then unrolled by a factor of i and compiled 5.0 by C (under “-O3”) to produce Ci. Neither GCC (with 4.5 4.0 -fmodulo-sched) nor ICC emits software-pipelined loops. ATLAS OpenBLAS 3.5 BLIS OpenBLAS-POCA BLIS-POCA gcc 6.0 18.0
97.36 Performance (GFLOPS) 96.53 97.16 96.47 3.0 17.5 95.01 256 512 768 93.54 5.5 102412801536179220482304256028163072332835843840409643524608486451205376563258886144 17.0 88.93 Matrix Size (M=N=K) 16.5 89.10 84.98 16.0 5.0 15.5 (b) Cortex-A57 75.67 15.0 4.5
14.5 Performance (GFLOPS) Performance (GFLOPS) 14.0 4.0 Figure 8. The GEMM performance for large data sets. -¹ -¹ S S I icc-1 I L gcc-1 gcc-4 icc-4 POCA L gcc-1 gcc-4 POCA B B × × × × (a) Sandybridge (Mr Nr = 8 4) (b) Cortex-A57 (Mr Nr = 4 8) synthesized GEMM routines. Figure 8 gives the results. BLIS-POCA is BLIS with its expert-written µkernel being Figure 7. The µkernel performance results (with the replaced by POCA’s. For OpenBLAS, Mr × Nr = 8 × 4 on floating-point efficiency shown on the top of each bar). Sandybridge and Mr × Nr = 4 × 8 on Cortex-A57 are also used [37]. However, the loops at layers 4 – 6 forming GEBP Figure 7 gives our results. For the µkernel loop given (Figure 1) are all coded in assembly. OpenBLAS-POCA is in Figure 2, A′ and B′ are stored in column- and row- OpenBLAS with only its µkernel loop being replaced by major, respectively, to simulate the effect that both are pre- POCA’s and the other two loops still in C. For each GEMM packed into continuous buffers. We set K = 192 on both c routine, the same inputs with 32 matrix sizes are used. Its Sandybridge and Cortex-A57 to ensure that all the matrices execution time is the average of three runs. involved fit into the L1 data cache (with A′ being 12KB, On Sandybridge, MKL is the best performer for large ma- B′ being 6K, and the next A′ to be prefetched into the L1 trices but exhibits unstable behavior for smaller ones. In con- cache (Section 3.1.2)). For each µkernel evaluated, we show trast, ATLAS is the worst performer on both processors due its floating-point performance (measured as the average of to its lacking knowledge about the underlying hardware dur- 5000 runs), together with its floating-point efficiency. ing its auto-tuning process. Compiler-synthesized GEMM POCA is the best performer on both processors. For routines are not competitive as hand-written ones. Sandybridge, POCA’s µkernel generator selects the same in- POCA achieves performance results that are competitive structions as BLIS-µ. For Cortex-A57, POCA’s µkernel gen- as or better than BLIS and OpenBLAS on both processors. erator also behaves identically as BLIS-µ except that POCA Note that µkernel is an important factor affecting the overall selects loads with immediate offsets while BLIS-µ selects GEMM performance, but not the only one. Other important loads with post-indexed immediate offsets (Section 3.1.3). factors include matrix blocking and submatrix scheduling. POCA’s µkernel optimizer is a key contributor to perfor- This explains why BLIS-POCA and OpenBLAS-POCA show mance. As shown in Tables 1 and 5 for Sandybridge, POCA different performance results even on the same architecture. schedules and pipelines instructions to improve both execu- On Sandybridge, POCA achieves competitive perfor- tion time and resource utilization aggressively by keeping mance as BLIS and OpenBLAS. BLIS-POCA is slightly LiveReg as close to MaxReg as possible, since a rotating faster than BLIS, by 1.002x on average. OpenBLAS-POCA register allocation without register spills is guaranteed later. is slightly slower than OpenBLAS, exhibiting an average In contrast, general-purpose compilers and domain experts slowdown of 0.994x, possibly because GEBP in OpenBLAS (to a lesser degree) are more conservative, by scheduling in- runs slightly faster, with two more loops (at layers 4 and 5) structions with smaller LiveReg in order to reduce potential also coded in assembly than OpenBLAS-POCA. register spills later, resulting in poorer resource utilization. On Cortex-A57, POCA is more impressive. OpenBLAS- In fact, no spilling eventually happens for ICC and GCC. POCA outperforms OpenBLAS by 1.065x on average due 4.2 RQ2: The GEMM Performance to a better schedule generated, and BLIS-POCA outperforms The performance advantages of POCA-synthesized µkernels BLIS by 1.067x on average due to also the addressing mode also translate into the performance competitiveness of POCA- optimization (as discussed earlier in Section 4.1).
130 4.3 RQ3: The Optimization Analysis 4.3.2 Loop Unrolling We analyze the impact of POCA’s instruction scheduling Due to our aggressive one-iteration scheduling and two- and loop unrolling on the optimized µkernels generated. iteration pipelining passes, our unroll-based rotating register 4.3.1 Instruction Scheduling allocation pass will select an unroll factor within [2, 8] arbi- trarily, as described in Section 3.2.3. On both Sandybridge POCA schedules instructions in two passes, single-iteration UF = 4 scheduling and two-iteration pipelining. To assess their ef- and Cortex-A57, happens to be selected. 6.0 97.88 18.0 97.51 97.36 97.32 fectiveness individually, we start with a non-scheduled ver- 96.49 96.45 96.61 96.08 96.47 96.16 17.5 95.01 5.5 88.93 sion of µkernel, base, and obtain a scheduled version, sched, 17.0 16.5 5.0 by applying one-iteration scheduling only, and a pipelined 16.0
15.5 4.5 version, piped, by applying two-iteration pipelining only. 15.0 Performance (GFLOPS) ∈ { } Performance (GFLOPS) 14.5 4.0 For each optimization strategy S base, sched, piped , -¹ -¹ IS IS L L B B we consider a total of four unrolled variants, S-1, S-2, S-4 POCA-3 POCA-4 POCA-5 POCA-6 POCA-7 POCA-8 POCA-2 POCA-4 POCA-6 POCA-8 × × × × and S-8, where i in S-i is the unroll factor being applied. (a) Sandybridge (Mr Nr = 8 4) (b) Cortex-A57 (Mr Nr = 4 8)
18.5 18.0 Figure 10. Impact of unroll factors on performance (with 97.36 17.5 95.01 17.0 the floating-point efficiency shown on each bar).
89.90 16.5 89.57 88.29 87.76 87.77 87.56 16.0 87.30 87.38 87.30 86.54 86.48 Figure 10 compares the performance results of BLIS- 15.5 85.37 15.0 µ and the POCA-synthesized µkernels under all possible
Performance (GFLOPS) 14.5 -¹ unroll factors in [2, 8], in the same setting as in Section 4.1. IS L POCA base-1 base-2 base-4 base-8 B sched-1piped-1 sched-2piped-2 sched-4piped-4 sched-8piped-8 × × According to our register allocation pass, these unroll factors (a) Sandybridge (Mr Nr = 8 4) × 6.0 are 4, 5, 6, 7 and 8 for the 8 4 µkernel on Sandybridge and
96.47 5.5 2, 4, 6 and 8 for the 4 × 8 µkernel on Cortex-A57. 88.93
85.87 86.43 5.0 84.91 POCA outperforms BLIS-µ on both processors under all 82.65 82.81 82.61 82.55 82.61 76.82 75.31 74.96 78.86 4.5 the unroll factors. The performance advantages of POCA
4.0
Performance (GFLOPS) over BLIS-µ are already analyzed in Section 4.3.1. Note -¹ IS L POCA base-1 base-2 base-4 base-8 B sched-1piped-1 sched-2piped-2 sched-4piped-4 sched-8piped-8 that the effects of unroll factors on performance are small. × × (b) Cortex-A57 (Mr Nr = 4 8) In general, µkernel is backend-bound. So the instruction Figure 9. Impact of instruction scheduling on performance decoding in the frontend rarely becomes a bottleneck if (with the floating-point efficiency shown on each bar). the µop dispatcher and backend function units work at full capacity, which happens for all the unroll factors evaluated. Figure 9 shows the results, obtained in the same setting On Cortex-A57, POCA-8 does not exhibit a performance as in Section 4.1. On Sandybridge, base, sched and piped drop. On Sandybridge, however, POCA-7 and POCA-8 are exhibit similar performance, with a small variation of less slightly slower than POCA-4 due to more DSB-MITE switch- than 3%. Sandybridge is a high-end server processor with ing (µop cache misses) as UF increases (with 17% and 21% powerful out-of-order µop dispatch and execution, reducing more DSB-MITE switching cycles for POCA-7 and POCA-8 the need for software-level instruction scheduling. While than POCA-4, obtained by Intel VTune Amplifier). applying either sched or piped makes little improvement to These results show that with instructions well scheduled, base, applying both brings a great performance gain, with µkernels can achieve near peak performance under a range POCA outperforming base-1 by 12.5%. of unroll factors. The problem of finding a good unroll factor On Cortex-A57, the situation is a little different: sched in our register allocation pass has thus been made simple. is consistently worse than base. With sched, all the loads are scheduled before all the floating-point instructions for 5. Related Work one iteration of µkernel, making both seriously imbalanced. In addition to the prior work discussed on BLAS in Sec- Despite its out-of-order execution, Cortex-A57 has a in- tion 1, we review some additional work related to this paper. order µop dispatcher with a narrow 3-issue window. Thus, The performance of DLA kernels can be improved with the imbalanced instruction stream makes the µop dispatcher compiler techniques, including SIMD vectorization [9, 19, the bottleneck, resulting in poor utilization of its backend 24, 38, 39], polyhedral optimization [4, 16], and loop function units. However, piped is always better than base, tiling [18, 27, 33]. However, general-purpose compilers lack because the loads and floating-point instructions from two the domain-specific knowledge to achieve near peak per- iterations are interleaved (Tables 1 and 5), creating a better- formance for DLA kernels. Source-to-source transforma- balanced instruction stream. Finally, when sched and piped tions [7, 8, 31], assisted with domain-specific directives, can are combined, POCA outperforms base-1 by 16.7%. be more effective, particularly when combined with an em- Therefore, POCA’s instruction scheduling is effective for pirical tuning engine to ease the auto-tuning process [34, 35]. not only low- and middle-end processors like Cortex-A57 Empirical auto-tuning is widely used for DLA kernels. but also high-end server processors like Sandybridge. The idea starts from [1] and was later adopted by PHiPAC [3]
131 and ATLAS [32]. Their BLAS libraries automatically se- [3] J. Bilmes, K. Asanovic, C.-W. Chin, and J. Demmel. Op- lect the best from a set of optimized DLA kernels by actu- timizing matrix multiply using PHiPAC: A portable, high- ally running them on the actual computing system. A DLA performance, ansi c coding methodology. In SC ’97, pages kernel may be obtained from a general-purpose compiler, 253–260. a domain-specific code generator [2], or a domain expert. [4] U. Bondhugula, A. Hartono, J. Ramanujam, and P. Sadayap- Being compiler-dependent, both auto-tuning and source-to- pan. A practical automatic polyhedral parallelizer and locality source transformations trade performance for portability. optimizer. In PLDI ’08, pages 101–113. Instead of auto-tuning, analytic techniques [15, 22, 36] [5] G. J. Chaitin. Register allocation and spilling via graph color- attempt to determine optimal values for various parameters ing. In CC ’82, pages 98–105. in GEMM, such as block sizes, analytically. Techniques for [6] H. Cui, L. Wang, J. Xue, Y. Yang, and X. Feng. Automatic accelerating BLAS on Intel Xeon Phi processors [26] and library generation for BLAS3 on GPUs. In IPDPS ’11, pages other DLA kernels on GPUs [6, 29] also exist. 255–265. POCA takes a different but complementary approach. [7] H. Cui, J. Xue, L. Wang, Y. Yang, X. Feng, and D. Fan. With domain-specific optimizations performed on top of Extendable pattern-oriented optimization directives. ACM LLVM’s abstract machine descriptions, µkernels with near Trans. Archit. Code Optim., 9(3):14:1–14:37, Oct. 2012. peak performance can be generated automatically for a pro- [8] H. Cui, Q. Yi, J. Xue, and X. Feng. Layout-oblivious compiler cessor and drop easily in existing GEMM frameworks. optimization for matrix computations. ACM Trans. Archit. POCA produces highly optimized µkernels by performing Code Optim., 9(4):35:1–35:20, Jan. 2013. domain-specific scheduling and pipelining while guarantee- [9] A. E. Eichenberger, P. Wu, and K. O’Brien. Vectorization for ing a subsequent rotating register allocation without regis- SIMD architectures with alignment constraints. In PLDI ’04, ter spills. For general-purpose compilers, software pipelin- pages 82–93. ing [11, 17, 20, 21, 23] and register allocation [5, 25] tech- [10] C. Eisenbeis, S. Lelait, and B. Marmol. The meeting graph: niques are mature. Techniques for adopting cyclic interval A new model for loop cyclic register allocation. In PACT ’95, graphs as an alternative representation of interference graph pages 264–267. for register allocation are introduced in [10, 13]. Our RTG [11] L. Gao, Q. H. Nguyen, L. Li, J. Xue, and T. Ngai. Thread- used for producing a rotating register allocation can be re- sensitive modulo scheduling for multicore processors. In garded as a (live-range) interval graph for µkernels. ICPP ’08, pages 132–140. 6. Conclusion [12] K. Goto and R. a. V. D. Geijn. Anatomy of high-performance matrix multiplication. ACM Trans. Math. Softw., 34(3):1–25, In this paper, we present a compiler approach, POCA, 2008. for automatically generating highly optimized GEMM ker- [13] L. J. Hendren, G. R. Gao, E. R. Altman, and C. Mukerji. nels with competitive or better performance compared to A register allocation framework based on hierarchical cyclic expert-crafted GEMM implementations, portably for a wide interval graphs. In CC ’93, pages 176–191. range of computer architectures. The key novelty lies in its [14] B. Kagstr˚ om,¨ P. Ling, and C. van Loan. GEMM-based level domain-specific yet architecture-independent optimizations, 3 BLAS: High-performance model implementations and per- especially its aggressive instruction scheduling techniques formance evaluation benchmark. ACM Trans. Math. Softw., for maintaining the number of live variables to a maximum. 24(3):268–302, Sept. 1998. In future work, we plan to extend POCA to multi-threaded [15] V. Kelefouras, A. Kritikakou, and C. Goutis. A matrix- GEMM and other dense linear algebra kernels. matrix multiplication methodology for single/multi-core ar- Acknowledgments chitectures using SIMD. The Journal of Supercomputing, 68(3):1418–1440, jan 2014. We thank all the reviewers for their constructive com- ments on an earlier draft of this paper. This research is [16] M. Kong, R. Veras, K. Stock, F. Franchetti, L.-N. Pouchet, and partly supported by the National Key Research and Devel- P. Sadayappan. When polyhedral transformations meet SIMD code generation. In PLDI ’13, pages 127–138. opment Program of China (NO.2016YFB0200400), NSFC (NO.61272483, NO.61370018, NO.61402495), and Aus- [17] M. Lam. Software pipelining: An effective scheduling tech- PLDI ’88 tralian Research Council (DP150102109 and DP170103956). nique for VLIW machines. In , pages 318–328. [18] M. D. Lam, E. E. Rothberg, and M. E. Wolf. The cache References performance and optimizations of blocked algorithms. In [1] R. C. Agarwal, F. G. Gustavson, and M. Zubair. Ex- ASPLOS ’91, pages 63–74. ploiting functional parallelism of POWER2 to design high- [19] S. Larsen and S. Amarasinghe. Exploiting superword level performance numerical algorithms. IBM Journal of Research parallelism with multimedia instruction sets. In PLDI ’00, & Development, 38(5):563–576, 1994. pages 145–156. [2] G. Belter, J. G. Siek, I. Karlin, and E. R. Jessup. Automatic [20] D. M. Lavery and W.-M. W. Hwu. Unrolling-based optimiza- generation of tiled and parallel linear algebra routines. In tions for modulo scheduling. In MICRO ’95, pages 327–337. IWAPT’ 10, pages 1811–1820.
132 [21] J. Llosa. Swing modulo scheduling: A lifetime-sensitive ap- [31] Q. Wang, X. Zhang, Y. Zhang, and Q. Yi. AUGEM: Automat- proach. In PACT ’96, pages 80–. ically generate high performance dense linear algebra kernels [22] T. M. Low, F. D. Igual, T. M. Smith, and E. S. Quintana-Ort´ı. on x86 CPUs. In SC ’13, pages 25:1–25:12. Analytical modeling is enough for high-performance BLIS. [32] R. C. Whaley and J. J. Dongarra. Automatically Tuned Linear ACM Trans. Math. Softw., 43(2):12:1–12:18, Aug. 2016. Algebra Software. In SC ’98, pages 1–27. [23] B. R. Rau. Iterative modulo scheduling: An algorithm for [33] J. Xue. Loop Tiling for Parallelism. Kluwer Academic software pipelining loops. In MICRO ’94, pages 63–74. Publishers, Boston, 2000. [24] I. Rosen, D. Nuzman, and A. Zaks. Loop-aware SLP in GCC. [34] Q. Yi. Automated programmable control and parameteriza- In GCC Developers’ Summit, pages 131–142, 2007. tion of compiler optimizations. In CGO ’11, pages 97–106. [25] M. D. Smith, N. Ramsey, and G. Holloway. A generalized [35] Q. Yi, Q. Wang, and H. Cui. Specializing compiler opti- algorithm for graph-coloring register allocation. In PLDI ’04, mizations through programmable composition for dense ma- pages 277–288. trix computations. In MICRO ’14, pages 596–608. [26] T. M. Smith, R. Van De Geijn, M. Smelyanskiy, J. R. Ham- [36] K. Yotov, X. Li, G. Ren, M. J. S. Garzaran, D. Padua, K. Pin- mond, and F. G. Van Zee. Anatomy of High-Performance gali, and P. Stodghill. Is search really necessary to gen- Many-Threaded Matrix Multiplication. In IPDPS ’14, pages erate high-performance BLAS? Proceedings of the IEEE, 1049–1059. 93(2):358–386, Feb 2005. [27] D. G. Spampinato and M. Puschel.¨ A basic linear algebra [37] X. Zhang, Q. Wang, and Y. Zhang. Model-driven level 3 compiler. In CGO ’14, pages 23:23–23:32. BLAS performance optimization on Loongson 3A processor. [28] F. G. Van Zee and R. A. van de Geijn. BLIS: A framework for In IPDPS ’12, pages 684–691. rapidly instantiating BLAS functionality. ACM Trans. Math. [38] H. Zhou and J. Xue. Exploiting mixed SIMD parallelism by Softw., 41(3):14:1–14:33, June 2015. reducing data reorganization overhead. In CGO ’16, pages [29] V. Volkov and J. W. Demmel. Benchmarking GPUs to tune 59–69. dense linear algebra. In SC’08, pages 31:1–31:11. [39] H. Zhou and J. Xue. A compiler approach for exploiting [30] L. Wang, J. Xue, and X. Yang. Reuse-aware modulo schedul- partial simd parallelism. ACM Trans. Archit. Code Optim., ing for stream processors. In DATE ’10, pages 1112–1117. 13(1):11:1–11:26, Mar. 2016.
133