arXiv:1904.05717v1 [cs.MS] 11 Apr 2019 09Cprgthl yteonrato() ulcto r Publication owner/author(s). the by held Copyright 2019 © stertobtentert fcmuainadrt ihwhi with rate and computation of rate the between ratio the As ye .SihadRbr .vnd ej.21.Œ OM aiyof Family MOMMS Œe 2019. Geijn. de van A. Robert and Smith M. Tyler format: Reference ACM ie rrpbih ops nsreso ordsrbt ol to redistribute to or servers on post to republish, or wise, arxmlilcto,dnelna ler,performance algebra, linear dense multiplication, matrix KEYWORDS playi start likely will forward. family going role this prominent Œus, fami superior. the more of come members different particular under and that show different memory, red circumstances, main to to built bandwidth custom of were amount that architectures e curre Œe on the member. which results, one of ical but algorithms is of algorithm” family “Goto’s analytica new state-of-the-art a Œe to “no”. analyti us is guide compelling answer sults provides the that paper evidence Œis empirical and optimal? (near) be con CPUs to future algor on best multiplication current matrix-matrix the computing continue Will memory arises: of question layers a various deteriorate, from retrieved be can data int eblne 1] hl oosagrtmi well-sui is algorithm Goto’s While [17]. balanced be c to this tion for requirements hardware the hardware to into such taking predicted on [2], been trends based operation has memory-bound MMM a and become [22], soon diverging been have ment h uhrs utb ooe.Asrcigwt rdti pe is credit with Abstracting work honored. this be of must components n author(s) for this Copyrights the bear page. copies first that the and on thi tion advantage of commercial ar part copies or that or profit provided all for fee of without granted copies is hard use or classroom digital make to Permission ABSTRACT ueteaon fdt oeetb loigtebokof block the of block allowing the by packing movement larger, be data to of amount the duce ac meietta salvae yrdcn h footpri the reducing of by block the alleviated p is a that be can impediment misses mance (TLB) buffer and translation-lookaside locality, that spatial ing beŠer for buffers contiguous formaŠed of include block algorithm Goto’s a of ing innovations Major cache. L3 the oa oke akdcp faruhysur umti (bloc submatrix square roughly a of computati copy packed orchestrates of a algorithm keep Œe to as 7]. so [6, CPUs current on implemen m practical guided for has algorithm (MMM) Goto’s multiplication matrix so-called the decades, two almost For INTRODUCTION 1 10.1145/nnnnnnn.nnnnnnn DOI: (SC19), 2019 17–27, November CO, Stora Denver, Networking, Computing, Performance High for ference In Algorithms. Multiplication Matrix O:10.1145/nnnnnnn.nnnnnnn DOI: 978-x-xxxx-xxxx-x/YY/MM...$15.00 emsinado e.Rqetprisosfo permissi CO from Denver, permissions SC19, Request fee. a and/or permission o er,tert fpa optto n h eoymove- memory the and computation peak of rate the years, For A nteL ah n akdcp farwpnlof panel row a of copy packed a and cache L2 the in h OM aiyo arxMlilcto Algorithms Multiplication Matrix of Family MOMMS The A . A nteL ah ahrta h 1ccet re- to cache L1 the than rather cache L2 the in rceig fŠ nentoa Con- International Še of Proceedings A n ae of panel and 2pages. 12 ye .SihadRbr .vnd Geijn de van A. Robert and Smith M. Tyler ss eurspirspecific prior requires ists, o aeo distributed or made not e [email protected]. mŠd ocp other- copy To rmiŠed. gt iesdt ACM. to licensed ights okfrproa or personal for work s tc n h ulcita- full the and otice we yohr than others by owned B e n Analysis, and ge, nospecially- into caches , tm for ithms omputa- c the uce tations e to ted show- ybe- ly erfor- mpir- tinue atrix- tof nt stag- ga ng B re- l to s cal on ch nt in k) A 1 ssa loih htmr ffcieyuiie the utilizes effectively more that algorithm an uses eetmmr aest eatfiilyreduced artificially be t to bandwidth layers the memory allows ferent that impleme computer various custom-built a of on compu performance tions of aŠained rate the the reports memory 1 to various Figure relative between deteriorate bandwidths to wil as continue this case layers that the be studies, to hierarchies empirical continue memory and current analysis in through caches show, of speeds relative the memory. main to bandwidth low very with architecture for an implementations MMM various of Comparison 1: Figure nagrtmsmlrt t n h uv aee MOMMS uses labeled curve MKL the believe and it, we to and similar algorithm algorithm an Goto’s uses Goto MOMMS prto ouiietemmr irrh seetvl as effectively as hierarchy possible. memory widely-use the extremely utilize this to operation for algorithms that essential is arch significant. an is such ture on algorithm Goto’s of degradation performance eoytacicre yaMMudrasml oe.I then It contributions: o model. new simple of bound a number lower under a MMM makes tight a by (essentially) incurred an traffic memory establish that [26] al. eal fteeprmn r ie nScin5. Section in given are experiment the of Details i ae rtrvesrcn hoeia eut ySihe Smith by results theoretical recent reviews first paper Œis GFLOPS 100 150 200 250 • • • 50 0 eetagrtm ntefml xii eeca charac- beneficial exhibit family teristics. the in algorithms dif ferent which under scenarios different exposes analytically It the results. state-of-the-art oretical Œese current by simul- cache. explained not of when are levels tradeoffs arise multiple that for hierarchy optimizing between taneously memory transfers the of of number the layers in tradeoffs analyzes It member. a is Goto’s algorithm which of algorithms practical of family (MOMMS) box Sa Multiplication proposes Matrix-matrix it Optimized Multilevel foundation, the a as results theoretical these With 0 0010 0020 0030 4000 3500 3000 2500 2000 1500 1000 500 0 sMMbcmsna eoybud it bound, memory near becomes MMM As m = n = k 1 MKL MOMMS Goto MOMMS ecrelabeled curve Œe . L C 3 3 A 2 ah.Œe cache. C 0 C 3 tation. the n A not l dif- o we , itec- nta- 2 C d 0 nd- t - - SC19, November 17–27, 2019, Denver, CO Tyler M. Smith and Robert A. van de Geijn

It empirically demonstrates the benefits of different algo- where Ci, j is mc nc , Ai is mc k, and Bj is k nc , except at the • × × × rithms on custom-built hardware that allows memory band- margins. Œen we compute the suboperation Ci, j += Ai Bj using widths to be varied. the described algorithm for Z += XY . Now, Ci, j is read from slow Together, these lay the foundation for practical solutions if and memory once at the beginning of the suboperation, and resides in when the balance between computation and memory bandwidth fast memory during the rest of the duration, and Ai and Bj are changes in the future. streamed one row and column at a time from slow memory. Each element of each operand is read once for each i, j, and there are 2 THEORY AND FUNDAMENTAL SHAPES m n such suboperations. Overall, this algorithm incurs mn ⌈ mc ⌉⌈ nc ⌉ We briefly review the state-of-the-art theoretical lower bounds for reads and mn writes for matrix C, mnk reads for matrix A, and ⌈ nc ⌉ the I/O complexity for MMM, and describe algorithms that aŠain mnk √ 4 m reads for matrix B. When mc nc M , the I/O cost is those bounds and thus are optimal for a simple model of a two- ⌈ c ⌉ ≈ ≈ 2mnk √M + 2mn. Œe highest ordered term in the I/O cost of the level memory hierarchy. Œese algorithms become our fundamen- / Resident C algorithm is the same as the I/O lower bound for MMM. tal components from which we compose practical algorithms for Œus the algorithm is essentially optimal. multiple levels of cache, in Section 3. 2.2.2 Resident A and B. Similarly, in the MMM Z += XY , each 2.1 An I/O lower bound for MMM columnof Z can becomputedby the matrix-vectormultiplicationzi += Smith et al. [26] starts with a simple model of memory with two lay- Xyi , where zi and yi are columns of Z and Y , respectively. Œis is ers of memory: a small, fast memory with capacity of M elements illustrated in Figure 2 (middle), Z, X, and Y are the le‰, middle, and and a large, slow memory with unlimited capacity. It shows that right operands, respectively. += any algorithm for ordinary MMM2 must read at least 2mnk √M Consider C AB. Partition: / − 2M elements from slow memory and additionally write at least C0 A0,0 A0,n 1 B0 mn M elements to slow memory. Adding these two lower bounds . . ··· . − . − C . , A . . , B . , gives a lower bound on the number of transfers between slow and → © ª → © ª → © ª ­ ® ­ ® ­ ® fast memory, called the I/O lower bound, of approximately 2mnk √M. ­ Cn 1 ® ­ Am 1,0 Am 1,n 1 ® ­ Bn 1 ® / − − ··· − − − Importantly, this lower bound is tight, modulo lower order terms. ­ ® ­ ® ­ ® where Ci is mc n, Ai,p is mc kc , and Bp is kc n, except at It improves upon previous work [3, 13, 15]. « ¬ × « × ¬ × « ¬ the margins. Œen we compute the suboperation Ci += Ai,pBp using the described MVM-based algorithm for Z += XY . In Resi- 2.2 Resident algorithms for MMM dent A, Ai,p is read from slow memory once at the beginning of the In [26], it is shown that three algorithms, named Resident A, Res- suboperation, and Cj and Bp are streamed from slow memory one ident B, and Resident C, aŠain the lower bound on the number of column at a time. Œe total I/O costs associated with each matrix reads from slow memory3. Additionally, Resident C aŠains the are: mnk reads and mnk writes of elements of C; mk reads of 3 ⌈ kc ⌉ ⌈ kc ⌉ lower bound on the number of writes to slow memory . In each mnk elements of A; and reads of elements of B. If kc nc √M, algorithm, the elements of one of the operand matrices are read ⌈ mc ⌉ ≈ ≈ the input cost is approximately 2mnk √M +nk,and the outputcost from slow memory only once, and each element of the other two / √ is approximately mnk √M. Œe input cost aŠains near the lower operand matrices is reused approximately M times each time it / is brought into fast memory. While the Resident A algorithm was bound on reads from slow memory. described as early as 1991 [18], and all three appear in [10], their Œe Resident B algorithm is the obvious symmetric equivalent to optimality was first noted in [26]. the Resident A algorithm, built upon the suboperation in Figure 2 (right). Its I/O costs mirror that of the Resident A algorithm. 2.2.1 Resident C. += Œe operation MMM Z XY can be com- Œe above descriptions “stream” rows and/or columns of two += T + T + puted by the sequence of rank-1 updates Z x0y0 x1y1 , matrices while keeping a block of the third matrix resident in fast T ··· where xi and yi are a row and column of X and Y , respectively. memory. Notice that one can instead stream row panels instead Œis is illustrated in Figure 2 (le‰), where Z is the square block on of rows and/or column panels instead of columns as long as the the le‰, X is the middle operand, and Y is the operand on the right. “small” dimension of the panel is small relative to the sizes of the T Œe vectors xi and yi are represented by the thin partitions of X block that is resident in fast memory. Œis still achieves the I/O and Y . lower bound modulo a lower order term. Œis insight becomes Suppose we have (larger) matrices C, A, and B. We compute crucial when we discuss blocking for multiple levels of memory. C += AB in the following way. Partition: 2.3 Algorithms for different shapes of MMM C0,0 C0,n 1 A0 · · · − . . . Œe number of reads and writes from slow memory for the Res- C . . , A . , B B B , . . . 0 · · · n 1 ident A, B, and C algorithms depend on the shape of the input →© ª →© ª →  −  ­ ® ­ ® matrices: Œere are cases where one of the algorithms is more ef- ­ Cm 1,0 Cm 1,n 1 ® ­ Am 1 ® ­ − · · · − − ® ­ − ® ficient than the other two, where we define efficiency by flops per 2 We only consider algorithms that compute the i, j element of m n matrix C as memop (I/O operations). Œere are 2mnk flops performed during =« k 1 ¬ « ¬ × γi, j : p=−0 αi,p βp, j where αi,p and βp, j are the i, p and p, j elements of m k × 4 matrix AÍand k n matrix B, respectively. mc and nc must be slightly less than √M to make room for a row of Ai and a 3 × Modulo lower order terms. column of Bj in fast memory. The MOMMS Family of Matrix Multiplication Algorithms SC19, November 17–27, 2019, Denver, CO

Resident C Shape Resident A Shape Resident B Shape

+= += +=

Data in cache.

Data in main memory.

Figure 2: ‡e three shapes of MMM exposed by the algorithms Resident C (le ), Resident A (middle), and Resident B (right). Each algorithm can be implemented as two loops around its corresponding shape.

MMM, and the I/O lower bound is 2mnk √M. Œus our goal for third layer of memory of intermediate size and access cost. In this / efficiency is √M flops per memop. We examine the cases for which case, at the start of a multiplication with submatrices one operand algorithms are efficient, assuming that m, n, and k are at least √M. may reside in slow memory while another resides in some interme- Resident C is efficient if and only if k is large. It reads mnk + diate layer. Furthermore, in many cases reads and writes cannot nc mnk ⌈ ⌉ be overlapped (e.g. main memory is o‰en not dual-ported), and + mn elements from slow memory during MMM. If mc = ⌈ mc ⌉ hence it is more expensive to access elements of C since C must = + nc √M, this is approximately 2mnk √M mn. Œis gives an ef- be both read and wriŠen. One way to address this is to select the 1 / 1 + mn − algorithm where blocks of the operand that is most expensive to ficiency of 2k . When k is large, this is approximately  √M  access are kept in fast memory as much as possible. Another is to √ M. We can analyze Resident A and Resident B similarly. Here adjust the sizes used for the the resident block in fast memory. we ignore the I/O cost for writes. If the sizes of the resident blocks We now walk through an example of the second solution. Sup- are chosen to be equal to √M, Resident B has an efficiency of pose we are employing the Resident A algorithm, with an mc kc 1 × 1 + nk − , which is approximately √M when m is large. Resi- block of A in fast memory. If the cost of accessing an element of B √M 2m   1 costs βB , and accessing an element of C costs βC , when m, n, and k 1 mk dent A has an efficiency of + − , which is approximately 2mc 2kc √ 2n are large, the efficiency in terms of flops per memop is + .  M  βB βC √ M when n is large. βB βC Œis is maximized when mc = M and kc = M. With Œis shows that one must choose the right algorithm depending r βC r βB on the shape of the problem. For each of the Resident A,B, and this, the total cost of I/O (rather than the number of accesses) as- C algorithms, there is a minimal shape that can be implemented sociated with accessing the streamed matrices are equalized and efficiently. For Resident C this occurs when m n √M, and thus the cost minimized (modulo lower order terms). ≈ ≈ k is large. For Resident B this occurs when k n √M, and m ≈ ≈ is large. For Resident A this occurs when m k √M, and n is 2.5 Summary ≈ ≈ large. In each case, the resident matrix fits into fast memory, and Œe ingredients to an efficient algorithm are: (1) Fill fast memory the dimension shared by the other two operands should be large with a submatrix of one of the operands (the resident matrix), (2) so that the cost of moving the resident matrix into fast memory Amortize the I/O cost associated with (1) over enough computa- can be amortized. tion, (3) Choose dimensions for the resident block that equalize Œe fact that one must choose a different algorithm for MMM the I/O costs (rather then the number of accesses) associated with depending on problem shape and size was previously noted for the two streamed matrices. distributed memory MMM [19, 24], and for hierarchical memory MMM [10, 31]. 3 MULTIPLE LEVELS OF CACHE We now extend the ideas from Section 2 to MMM for computers 2.4 A balancing act with multiple layers of fast memory. We carefully build up a par- So far in this section, we have assumed that the costs associated ticular algorithm for a computer with three layers of memory: a with accessing an element is the same, no maŠer if it is an element slow main memory, a medium cache, and a fast cache, with each of A, B, or C. In doing so, we arrived at the following strategy: level of cache faster and smaller in capacity than the one before it. Place a square block of the resident matrix in fast memory, stream- We name the fast and medium caches Lf and Lm, and name their ing the other two from slow memory. Œis amortizes the I/O costs capacities Mf and Mm, respectively. associated with the resident matrix, and equalizes the number of We start by partitioning the matrices so that the computation is accesses of the two streamed matrices. orchestrated as a double loop over a particular subproblem, one of We now re-analyze the Resident A, B, and C algorithms in the the shapes in Figure 2, that that effectively utilizes the Lm cache. case that the costs associated with accessing elements of the dif- Œe question then becomes how to implement the subproblem in ferent operands are unequal. Œis can happen if, e.g., if we add a a way that effectively utilizes Lf . SC19, November 17–27, 2019, Denver, CO Tyler M. Smith and Robert A. van de Geijn

3.1 A motivating example two dimensions. Œe result is one of the other two shapes shown For our motivating example, we choose the Resident A algorithm in Figure 2. We name the outermost of these two loops the Lh 1 outer loop and the innermost the L inner loop. Œis process− is when blocking for Lm. h 1 shown in Figure 3. − Effectively utilizing Lm. To effectively utilize Lm, we select the We note that [10] claimed that it was locally optimal to encounter partitioning that castscomputationin termsofa doubleloop around a subproblem that corresponds to one of the three optimal subprob- the middle shape in Figure 2. We call this subproblem the Lm block- lems at every level of the memory hierarchy. However that paper panel multiply. did not give details on how this could be accomplished, nor did it analyze the claim in terms of any I/O lower bounds. Effectively utilizing Lf . Œe question now becomes how to or- chestrate the Lm block-panel multiply in a way that effectively 3.3 Classifying matrix operands and utilizes Lf . Œis suggests again a double loop around one of the shapes in Figure 2. It is not hard to see that creating a double loop algorithms that again implements a Resident A algorithm is problematic: Par- Œe two loops for Lh 1 have exposed partitions of matrices that − titioning A for Lf would expose panels of B and C that by design differ in terms of access frequency and size. From these properties, are too large to fit into Lm, and these panels are used in multiple we can classify these different matrix partitions. Lf subproblems. Either these panels of B or C would need to be Œe Lh resident block is the block that is designed to remain brought into Lm multiple times or the sizes of the various matrix and reside in the Lh cache during the duration of the Lh subprob- partitions would need to be reduced. Either way the effect would lem. Œe other two operands of an Lh subproblem are called the be that the operation would no longer be near-optimal with re- Lh streamed panels, as small partitions of the streamed panels are spect to the number of transfers between Lm and slower levels of brought into Lh during an iteration of the Lh 1 outer loop, used for − memory. We conclude that the block-panel multiply should be im- computation, and then not used again during the Lh subproblem. plemented in terms of a Resident C or Resident B algorithm. Which Œe Lh 1 inner loop partitions the Lh resident block and one of − of these depends on the choice of the outer and inner loop, which the Lh streamed panels. Œe remaining Lh streamed panel is le‰ we discuss next. unpartitioned. Œe matrix partition not partitioned by the Lh 1 inner loop is used during every iteration of the L inner loop.− Choosing the outer loop for the L cache. In order to aŠain near h 1 f Guided by the principle that each element of the L− subproblem the lower bound, each element in the two long panels of B and C h should only by read into Lh once, it must remain in cache during must be used √M times each time it is brought into Lm. Œis ≈ the entire inner L loop. We name this matrix partition the L leads us to first partition along the n dimension with blocksize n , h 1 h c guest panel. Compare− this to the resident block of the L cache. yielding partitions of B and C that are small enough to fit into the h Œe elements of the Lh guest matrix, like the elements of the Lh Lm cache along with the block of A. resident block, are reused from Lh across iterations of a loop. Œe Še inner loop for the Lf cache. Œe next step is to further parti- difference is that the Lh resident block is reused across every it- tion the matrices to optimize for L . Œe subproblem exposed by eration of the outer Lh 1 loop, and the Lh guest matrix is reused f − each iteration of the L outer loop is a block of A times a skinny across the iterations of the inner Lh 1 loop. f − panel of B updating a skinny panel of C. Œe L inner loop will par- A‰er the two Lh 1 loops, we have exposed one of the three f − tition this subproblem along one of the two dimensions that the Lf shapes associated with our algorithms Resident A, Resident B, and outer loop did not. We can choose either of these. For this example, Resident C. Œe small block that will then reside in Lh 1 will be − we will choose the k dimension (with blocksize kc ). Œis L inner known as the Lh 1 resident block. f − loop exposes a new subproblem that we will call the Lf subprob- Œe algorithms that arise from our methodology can be identi- lem. In this case, the Lf subproblem is a tall and skinny panel of fied by the operand that the resident block is from in each level of A times a kc nc block of B, updating a tall and skinny panel of cache. We introduce a naming convention for the algorithms that × C. If kc nc M , then the L subproblem corresponds to the states the level of cache and the operand that resides in it. For in- ≈ ≈ f f furthest le‰ shapep seen in Figure 2. Œen, the block of B will reside stance if an algorithm has B as the resident block of the L2 cache, in the Lf cache, and the panels of A and C will be streamed in from A as the resident block of the L1 cache, and C as the resident block lower levels of cache for the duration of this subproblem. in registers, it is called B2A1C0.

3.2 Building a Family of Algorithms 3.4 Optimizing for registers In the the motivating example, we started with a problem resem- In our family of algorithms, we think of the register file as L0: the bling the middle shape from Figure 2, and used two loops to par- smallest and fastest level of cache. For practical reasons, it should tition the problem, resulting in a problem resembling one of the be treated as a special case. In many implementations of MMM, other two problem shapes. Œis suggests the following methodol- the innermost kernel implements the Resident C algorithm [7, 11, ogy to optimize for any number of levels of cache: We begin with 29, 30]. Œere are good reasons for this. Œe latency of the com- one of the three shapes in Figure 2, optimizing for the I/O cost for putation instructions dictates that there is a minimum number of the Lh cache. Œen, to optimize for the next smaller and faster level registers that must be used to store elements of C to avoid the in- of cache, the Lh 1 cache, we first partition the problem along the struction latency becoming a boŠleneck. Œe number of elements long dimension,− and then partition along along one of the other of C that are stored in registers must be at least the product of the The MOMMS Family of Matrix Multiplication Algorithms SC19, November 17–27, 2019, Denver, CO

Lh subproblem. Lh 1 outer loop. Lh 1 inner loop. Lh 1 subproblem. − − − += += += += += +=

+= += += += += +=

+= += += += += +=

Resident block of Lh cache (or part of it).

Guest panel of Lh cache.

Resident block of Lh 1 cache. −

Figure 3: Possible scenarios when partitioning for Lh and Lh 1 when Resident A (top), Resident B (middle), or Resident C − (bottom) is encountered in Lh. instruction latency and the number of elementary additions that assumptions that the resident blocks of both Lh and Lh 1 must be − can be performed per cycle [20, 33]. O‰en this means that a signif- square, the I/O cost for Lh when optimizing for both Lh and Lh 1 icant portion of the registers must be dedicated to storing elements can be determined by the ratio M M . − h/ h 1 of C, making it unnatural to use the Resident A or Resident B algo- Sometimes, it is counter-productive− or of limited value to opti- rithms for the registers. Œerefore, it is o‰en the case that there is mize for Lh 1 when optimizing for Lh. In this case: − no choice but to use Resident C for the registers. (1) A simple option is to treat Lh 1 as if it were smaller than − it is, reducing the size of the Lh 1 resident block. − 4 MULTILEVEL CACHE TRADEOFFS (2) If Lh is LRU, another option is to tweak the blocksizes for When simultaneously optimizing for multiple levels of cache, there Lh 1 slightly. Œe portions of the Lh streamed panels that − are tensions between I/O costs at the different levels of cache. must fit into Lh alongside the Lh resident block depends on the tiling of the Lh 1 outerloopbutnot ontheblocksize − of the Lh 1 inner loop. Œerefore, one can tweak the shape 4.1 Optimizing for Lh 1 impacts the Lh I/O cost − − of the Lh 1 resident block accordingly. When simultaneously optimizing for both Lh 1 and Lh, the size of − − (3) A third option is that one could “skip” optimizing for Lh 1 the Lh resident block is reduced relative to its size when optimizing − and instead simultaneously optimize for Lh and Lh 2. only for Lh, since larger portions of the Lh streamed panels must − Blocking for the Lh 1 cache adversely affects the number of fit in Lh. When optimizing for both Lh and Lh 1: − − transfers into and out of the Lh cache but blocking for further At minimum, the L resident matrix and L guest matrix (smaller and faster) levels of cache does not, because the entire • h h must fit into Lh. Lh 2 subproblem fits within the data that must be in the Lh cache. − If Lh is inclusive, meaning that anything in Lh 1 must also • − be in Lh, then there must also be space in Lh for the Lh 1 4.2 Optimizing for Lh impacts the Lh 1 I/O cost resident matrix. − − Simultaneously optimizing for Lh and Lh 1 adversely effects trans- If L is inclusive and has a LRU policy, then in order for − • h fers into and out of Lh. We now argue that it also has an adverse an element to remain in L , fewer than M elements may h h effect on the transfers into and out of Lh. be accessed in between accesses of the element for it to When optimizing for only one cache of size M, the streamed ma- remain in cache and hence every matrix partition exposed trices are each associated with an aggregate I/O cost of mnk √M. by the Lh 1 outer loop must fit in Lh. ≈ / − When optimizing for both Lh and Lh 1, however, the I/O cost asso- − Œese conditions represent a tradeoff between optimizing for Lh ciated with the Lh 1 resident matrix becomes cubic because each − and Lh 1. Œe larger Lh 1 is, the more data must fit into it, and element of the Lh 1 resident matrix is moved into Lh 1 once per − − − − the smaller the Lh resident block can be. With the simplifying Lh subproblem. SC19, November 17–27, 2019, Denver, CO Tyler M. Smith and Robert A. van de Geijn

When optimizing for both Lh and Lh 1, the I/O cost associated 5.1 Experimental setup − with the Lh 1 resident matrix will be mnk √Mh , whereas when We have implemented the described family of algorithms as the − ≈ / only optimizing for the Lh 1 cache, the I/O cost associated with Multilevel Optimized Matrix-Matrix Multiplication Sandbox (MOMMS). − the Lh 1 resident matrix is equal to the number of compulsory MOMMS implements algorithms for MMM by composing compo- − reads and writes. Œe I/O costs for the streamed matrices are not nents like matrix partitioning, packing, and parallelization at com- affected. pile time. MOMMS is wriŠen in Rust [21], a modern system pro- While optimizing for the Lh cache has increased the Lh 1 I/O gramming language focusing on memory safety. Most of this safety − cost, optimizing for Lh+1, Lh+2, etc., does not affect it, because is enforced at compile-time through Rust’s borrow checker. In optimizing for further levels of cache does not reduce the number Rust, memory is freed when it goes out of scope, and there is no of times each element is used every time it is brought into the Lh 1 garbage collector. From Rust, one can call C functions with very − cache. low overhead. For low-level kernels, MOMMS calls the BLIS micro- kernel [29] coded in C and inline assembly language. 4.3 Skipping caches We custom built two computers. One has an Intel i7-7700K CPU with two 8GB DIMMS of DDR4-3200 RAM and a motherboard with We have seen that tradeoffs occur when simultaneously optimiz- an Intel Z270 chipset. Œe other has an Intel i7-5775C CPU with ing for the I/O cost of multiple levels of cache. Sometimes these two 8GB DIMMS of DDR3-2400 RAM and a motherboard with an tradeoffs are too great, so instead of optimizing for both Lh and Intel Z97 chipset. We refer to these computers by their proces- Lh 1, one may forego the Lh 1 cache, and instead simultaneously sor names. We chose the Z270 and Z97 chipset motherboards be- − − optimize for Lh and Lh 2 I/O costs, where the Lh 1 cache is in- cause these are enthusiast motherboards for consumers interested − − termediate between Lh and Lh 2. We call this skipping the Lh 1 in overclocking, and they provide the ability to change the mem- − − cache. ory multiplier. Œe i7-7700K computer has a 4-core Intel Kaby Lake When the Lh 1 cache is skipped, an optimal subproblem is en- CPU with 64KB L , 256KB L , and 6MB L caches. We chose this − 1 2 3 countered at the Lh level and at the Lh 2 level, but not at the Lh 1 because it is a recent readily available Intel processor with an L − − 3 level, However, this does not mean that the Lh 1 is not useful. Re- cache. Œe i7-5775C is a 4-core Intel Broadwell CPU. It also has − call that the Lh guest matrix is reused during each iteration of the 64KB L1, 256KB L2,and 6MB L3 caches. Most notably it has 128MB Lh 2 inner loop. In the right circumstances, this Lh guest matrix of eDRAM, functioning as an L cache. − 4 may be instead reused in the Lh 1 cache, if that cache is skipped. All experiments were performed with hyperthreading disabled. − A userspace CPU governor was used to set the CPUs to the nominal In idealized circumstances, only the L guest matrix should • h CPU frequency: 4.2 GHz for the i7-7700K and 3.3 GHz for the i7- need to be in the Lh 1 cache. 5775C. − If the Lh 1 cache is LRU, then a panel of the Lh resident Œe bandwidth to main memory can be determined by the prod- • − matrix must also fit into the Lh 1 cache. uct of the number of memory channels, the base clock rate, the − If the Lh 1 cache is inclusive, then the Lh 2 resident block number of bytes per transfer, and the memory multiplier. With • − − must also fit into the Lh 1 cache. DDR RAM, this is doubled since it transfers on both the leading − and trailing edges of the clock signal. We increase the ratio of the In this case, the Lh guest matrix is reused from Lh 1, but is not rate of I/O to the rate of computation via the BIOS seŠings. Reduc- − square, and the I/O cost associated with reading the other two ing the memory multiplier and the number of memory channels operands is suboptimal. Furthermore, in many cases this panel decreases the rate of I/O without changing the rate of computa- occupies only a fraction of Lh 1, reducing its size and further in- tion. creasing the I/O cost. − Goto’s algorithm is a member of the MOMMS family. It skips optimizing for the L3 and L1 caches. Since A is the resident ma- 5.2 Optimizing for the L3 cache trix of the L2 cache, and C is the resident matrix of the registers, We here describe an algorithm implemented in MOMMS that op- Goto’s algorithm is named A2C0 according to the convention in timizes for both L3 and L2 labeled B3A2C0. We compare this al- Section 3.3. gorithm to our re-implementation of Goto’s algorithm (also im- plemented in MOMMS), and to vendor and state-of-the-art open source BLAS [4] implementations. Figure 4 compares Goto’s algo- 5 EXPERIMENTS rithm with other MOMMS algorithms optimized for both the I/O We now evaluate algorithms created by our methodology. We do cost of L3 and L2. so by performing experiments on architectures with a varying num- We now describe the B3A2C0 algorithm as implemented for the ber of levels of cache. i7-7700K and illustrated in Figure 4 (second from the le‰). First, we For current CPUs, Goto’s algorithm aŠains excellent performance [28] partition for L3 cache. Œe L3 outer loop partitions the matrices that is difficult to exceed despite the fact that it does not aŠain close in the n dimension with blocksize 768. Œen the L3 inner loop to the I/O lower bound on computers with an L3 cache. In order partitions in the k dimension, also with blocksize 768. Œis reveals to evaluate the I/O cost of different algorithms, we artificially vary a768 768blockof B that becomes the L3 resident matrix. Next, we × the cost of accessing main memory. In all experiments, we perform partition for the L2 cache. Since B is the L3 resident matrix, the L2 double precision MMM. outer loop must be in the m dimension, and it is with blocksize 120. The MOMMS Family of Matrix Multiplication Algorithms SC19, November 17–27, 2019, Denver, CO

Goto’s algorithm B3A2C0 Algorithm A3B2C0 Algorithm C3A2C0 Algorithm Partition n with blocksize 3000 Partition m with blocksize 768 Partition n with blocksize 768 Partition n dimension with blocksize 624

+= += += +=

Partition k with blocksize 192 Partition k with blocksize 768 Partition k with blocksize 768 Partition m dimension with blocksize 624

+= += += +=

Partition m with blocksize 120 Partition n with blocksize 120 Partition m with blocksize 120 Partition k dimension with blocksize 156

+=

+= Partition k with blocksize 192 += +=

Inner kernel += Partition k with blocksize 192 Partition m dimension with blocksize 156 +=

Inner kernel += +=

Block is reused in L3 cache.

Block is reused in L2 cache. += Inner kernel Inner kernel += +=

Figure 4: Four MOMMS algorithms for MMM.

Œe L2 inner loop then partitions the k dimension with blocksize 192, making a block of A resident in L2, and a 120 768 panel of × C the guest matrix of L3. We skip L1, since it is a quarter the size of L2, making it is not beneficial to optimize for both L2 and L1. 100 Œe next two loops make a 4 12 block of C the resident matrix × of registers, and a 192 12 panel of B, the guest panel of L . Œis 2 Roofline Single Channel DDR-800 × GFLOPS guest panel of L is designed to be reused in the (skipped) L cache. 2 1 Roofline Dual Channel DDR-3200 Finally, we call a 4 12 micro-kernel provided by BLIS [29]. × 10 MOMMS Goto We compare this to Goto’s algorithm with similar blocksizes as MOMMS B3A2C0 follows: nc is 3000, kc is 192, mc is 120,mr is 4, and nr is 12. Our im- plementation of Goto’s algorithm uses the same micro-kernel from 1 2 4 8 16 32 64 128256512 BLIS as does B3A2C0. For both algorithms, we parallelize the sec- flops per byte ond loop around the micro-kernel with 4 threads. Œis quadruples the bandwidth requirements of our algorithms without increasing Figure 5: Roofline models for i7-7700K at 4.2GHz with 4 the amount of the L3 cache that must be set-aside for elements of cores under two bandwidth conditions. ‡e y-axis is mea- A [27]. sured and the x-axis is theoretical. ‡e higher points for each algorithm represent high-bandwidth performance and the lower points represent low-bandwidth performance. Rooflines. Œe roofline model is a simple model used to give an upper bound on performance based on the arithmetic intensity of an algorithm for a specific computer [32]. Œe computer is charac- terized by its rateof computationand the rateat which it can trans- model according to their arithmetic intensity and measured per- fer data between main memory and cache. Œe arithmetic intensity formance as a way to explain their performance and to explain of an algorithm is the number of flops per byte transferred between whether or not they could perform beŠer. One can either measure memory and cache during the execution of that algorithm. When the arithmetic intensity of an algorithm or analyze it. We choose the arithmetic intensity is low it is bandwidth bound, and when to analyze the arithmetic intensity of the algorithms ploŠed. the arithmetic intensity is high it is compute bound. Œe roofline When the matrices are large, Goto’s algorithm has an efficiency model is thus a plot where the x-axis is the arithmetic intensity and 1 1 + 1 − the y-axis is maximum rate of computation for that arithmetic in- of k 2n flops per element. With the blocksizes we used,  c c  tensity. Œe roofline that serves as an upper bound on performance this is 23.26 flops per byte. Œe algorithm B3A2C0, witha768 768 × is formed by two linear curves that intersect when the minimum block of B in the L3 cache, has an efficiency of 64 flops per byte. time spent for computation for an algorithm is equal to the min- In Figure 5, we show the roofline model for the i7-7700K for imum time spent for I/O. Algorithms are ploŠed on the roofline the case of one channel of DDR4-800 RAM, and for the case of SC19, November 17–27, 2019, Denver, CO Tyler M. Smith and Robert A. van de Geijn

1 channel of DDR4-800 1 channel of DDR4-1200 1 channel of DDR4-800 2 channels of DDR4-3200

200 200 200 200

GFLOPS 100 100 MOMMS Goto GFLOPS 100 100 MOMMS B3A2C0 MOMMS C A C 0 0 3 2 0 0 1000 2000 3000 4000 0 1000 2000 3000 4000 BLIS m = n = k m = n = k MKL ATLAS 1 channel of DDR4-1600 2 channels of DDR4-3200 0 0 0 1000 2000 3000 4000 0 1000 2000 3000 4000 m = n = k m = n = k 200 200 Figure 7: Comparison with state-of-the-art open source and GFLOPS 100 100 vendor libraries, for both high and low bandwidth scenarios. MOMMS Goto MOMMS B3A2C0 Matrices stored in column-major order. 0 0 0 1000 2000 3000 4000 0 1000 2000 3000 4000 m = n = k m = n = k

Figure 6: Performance on i7-7700K for square matrices, column major order, and our implementations of Goto’s algorithm varying problem size and available bandwidth. Matrices are and B3A2C0 pack matrices the first time they become the resident stored prepacked. or guest matrix at some level of cache. Œis packing (and the fact thatC is not stored hierarchically for Goto’s algorithm) account for the performance difference seen for the Goto and B3A2C0 curves two channels of DDR4-3200 RAM. Œese cases represent the min- between Figures 6 and 7. We see that for high bandwidth scenar- imum and the maximum memory bandwidth that we configure ios, BLIS, the MOMMS implementation of Goto’s algorithm, and the computer for. We plot the modeled efficiency of Goto’s al- B3A2C0 all aŠain roughly 75% of peak, and that MKL outperforms gorithm and the algorithm B3A2C0 against each algorithm’s mea- the other implementations. For low bandwidth, implementations sured performance. Œe roofline plot clearly shows that in the high- that use Goto’s algorithm (or something similar) exhibit poorer bandwidth case, either algorithm is capable of achieving the peak performance as they do not effectively utilize the L3. In this case, performance of the CPU based on its arithmetic intensity, but for C3A2C0 performs best, with B3A2C0 close behind. For large prob- the low-bandwidth case, only B3A2C0 can. Œe improved arith- lem sizes, ATLAS performs almost as well as the algorithms imple- metic intensity is caused by its more effective utilization of the L3 mented in MOMMS that optimize for the L3 I/O cost but it does cache. not perform nearly as well for the high bandwidth case. Varying Bandwidth. Figure 6 reports the achieved performance of Goto’s algorithm and B3A2C0 for square matrices, varying the 5.3 Optimizing for the L4 cache amount of bandwidth to main memory. Packing is o‰en used to In this section, we demonstrate that our methodology can be effi- achieve spatial locality during an algorithm. Otherwise blocks that ciently applied to the Intel i7-5775C, which has four levels of cache, are designed to reside in cache may not be able to do so due to where the L4 cache is 128MB of eDRAM. We implemented an al- cache conflict issues [12]. Packing incurs extra memory move- gorithm called C4A2C0 for this architecture. Figure 8 shows the ments that do not fundamentally need to happen during MMM. loop ordering and the blocksizes used for C4A2C0. In C4A2C0, a Œis paper is concerned with the fundamentals of temporal local- 3600 3600block of C resides in the L4 cache, and a 120 192 block × × ity during MMM, and hence we sidestep the spatial locality issue of A resides in the L2 cache. We decided to skip blocking for the L3 by storing matrices “prepacked” such that every time a matrix is cache, as there is sufficient bandwidth from the L4 cache without partitioned, the blocks are stored contiguously. Œis lets us sepa- optimizing for the number of L3 cache misses. Nevertheless, the 5 rate the issues of temporal and spatial locality in our experiments . guest matrix of the L4 cache, a 192 3600 panel of B, is appropri- × At low bandwidth, B3A2C0 outperforms Goto’s algorithm by ately sized to remain in the L3. C4A2C0 uses the same inner kernel thirty to forty percent. As the and the gap eventually disappears. as B3A2C0. In Figure 9, we compare the performance of Goto’s algorithm Comparing with existing implementations. In Figure 7, we com- and C A C for square matrices across several bandwidths. In pare our implementations of Goto’s algorithm and B A C against 4 2 0 3 2 0 this experiment, matrices are stored hierarchically, and so pack- the dgemm routines in ATLAS [31] (3.10.3), BLIS (0.2.1), and Intel’s ing is not performed. For high bandwidths, Goto’s algorithm and Math Kernel Library (MKL 2017 Release 2) [14]. It would not be C A C exhibit similar performance, but when bandwidth is low, fair to compare against implementations of MMM if we did not 4 2 0 C A C outperforms Goto’s algorithm for large problem sizes. need to pack, so for this experiment, input matrices are stored in 4 2 0 Figure 10 compares the performance on square matrices of our 5Others have avoided packing for practical reasons. BLASFEO [5] operates on so- implementations of Goto’s algorithm and C4A2C0. Here, matrices called panel-major matrices for performance on small matrices. Œe panel-major for- are stored in column-major order and accordingly packing is per- mat is similar to the format used in Goto’s algorithm for the packed panel of B. An- other library,libxsmm[11], also targets small matrices,and operates on column-major formed when partitions of A and B become resident or guest ma- matrices, but does not perform packing. trices of some level of cache. In C4A2C0, C is unpacked when it is The MOMMS Family of Matrix Multiplication Algorithms SC19, November 17–27, 2019, Denver, CO

Partition m dimension with blocksize 3600 1 channel of DDR3-800 1 channel of DDR3-1066 200 200

150 150

+= 100 100 GFLOPS 50 50

0 0 Partition n dimension with blocksize 3600 0 0 5000 10000 15000 5000 10000 15000 m = n = k m = n = k 1 channel of DDR3-1333 2 channels of DDR3-2400 += 200 200 150 150

100 100 GFLOPS Partition k dimension with blocksize 192 50 50 MOMMS Goto MOMMS C4A2C0 0 0 0 0 5000 10000 15000 5000 10000 15000 += m = n = k m = n = k

Figure 9: Performance for square matrices, varying prob- Partition m dimension with blocksize 120 lem size and available bandwidth. Matrices are stored prepacked. += 1 channel of DDR-800 2 channels of DDR3-2400 200 200 Inner kernel 150 150

100 100 += MOMMS Goto GFLOPS MOMMS C4A2C0 50 50 BLIS MKL Block is reused in L4 cache. 0 0 0 0 Block is reused in L3 cache. 5000 10000 15000 5000 10000 15000 m = n = k m = n = k Block is reused in L2 cache. Figure 10: Performance on i7-5775C with an 128MB L4 cache Figure 8: Algorithm C4A2C0 that optimizes for both L4 and for low bandwidth scenario (1 channel of DDR3-800) and a L2. It places a square block of C in L4, and a square block of high bandwidth scenario (2 channels of DDR3-2400). A in L2. L3 is “skipped”, with the and the L4 guest panel, B, reused from L3. Goto’s algorithm adversely affects performance for the low band- width case, causing a noticeable dropoff for larger matrices. We postulate that somehow the way that data is shared by the threads within BLIS, coupled with the larger value of k within BLIS, (or no longer resident in L4. In both Figures 9 and 10, the top of the c graphs is the peak computational rate of the CPU. the smaller mc ) fosters beŠer reuse of data within the L4 cache. With DDR-800, all implementations of MMM on the i7-5775C Because L4 is so large, we ran quite large problems since oth- erwise the matrices would completely fit into cache. Performance outperform those on the i7-7700K, despite the fact that the former for Goto’s algorithm and MKL do not fall off until the problem size processor is two generations older. Œe large L4 cache means that becomes m = n = k 5000. We can see that Goto’s algorithm does blocksizes for the C4A2C0 algorithm can be very large, so the al- ≈ gorithm does not need much bandwidth from main memory, but not optimally use L4 and neither does Intel’s MKL. While BLIS’s performance does not fall off as severely as for the even algorithms that do not take advantage of L4 by using such other implementations when the problem size grows, its overall large blocksizes benefit from having the 128MB cache. Œe large performance is not as high. Œe algorithmic differences between capacity cache can facilitate the hiding of latency to main memory, BLIS and the MOMMS implementation of Goto’s algorithm are par- through techniques such as hardware prefetching. allelism and blocksizes. BLIS uses a larger kc and a smaller mc than MOMMS and parallelizes the 2nd and 3rd loops around the 5.4 Algorithms for different shapes of matrices micro-kernel, whereas MOMMS parallelizes the 2nd loop around Algorithm A3B2C0 partitions the matrices such that a square block the micro-kernel. Modifying either the parallelism or the block- of A is resident in L3 and a block of B is resident in L2. It then sizes so that they match that of the MOMMS implementation of calls an inner kernel updating a panel of C whose elements are in SC19, November 17–27, 2019, Denver, CO Tyler M. Smith and Robert A. van de Geijn

= L3 by multiplying a block of A whose elements are in L2 times a n = k = 600 m 600 panel of B whose elements are in L3. Algorithm C3A2C0 partitions 200 200 the matrices such that a square block of C is resident in the L3 and a block of A is resident in L . It then calls the same inner kernel 2 100

GFLOPS 100 as the algorithm B3A2C0 does. Blocksizes and loop orderings for algorithms A3B2C0 and C3A2C0 are shown in Figure 4. 0 0 Algorithms A3B2C0, B3A2C0, andC3A2C0 represent three choices 0 1000 2000 3000 4000 0 1000 2000 3000 4000 m k = n for blocking for L3 . In Section 2.3, we argued that each of these choices may be optimal for a specific problem shape where two di- m = k = 600 n = 600 mensions are equal to √M3 and the other dimension is large, and 200 200 selecting the wrong algorithm for a problem shape can result in an I/O cost that is 50% greater. 100

GFLOPS 100 On a computer with three levels of cache and low bandwidth, we claim the following: A3B2C0 casts its computation in terms of 0 0 a block-panel multiply, with a block of A in L3, and so it should be 0 1000 2000 3000 4000 0 1000 2000 3000 4000 n m = k the best choice of the three algorithms when m = k √M3, and ≈ m = n = 600 k = 600 n is large. Similarly, B3A2C0 casts its computation in terms of a panel-block multiply, with a block of B in L3, and so it should be 200 200 the best choice of the three algorithms when n = k √M3, and m ≈ is large. Finally, C A C casts its computation in terms of a block 3 2 0 GFLOPS 100 100 dot product multiply, with a block of C in L3, and so it should be the best of the three algorithms when m = n √M3, and k is large. ≈ 0 0 Figure 11 reports results using A B C , B A C , C A C , and 0 1000 2000 3000 4000 0 1000 2000 3000 4000 3 2 0 3 2 0 3 2 0 k m = n Goto’s algorithm, with matrices stored hierarchically and no pack- Square ing is performed. We vary the shape of the matrices. In each case, two of the dimensions are set to 768, and one of the dimensions 200 is varied along the x-axis. Œe experiments were performed on MOMMS Goto MOMMS A3B2C0 the Intel i7-7700K, with the DDR speed set to a single channel of MOMMS B3A2C0

GFLOPS 100 DDR4-800. When the dimension that is allowed to vary is large, the MOMMS C3A2C0 predicted algorithm outperforms the others. We also show perfor- 0 mance when the matrices are square, and the size varies along the 0 1000 2000 3000 4000 m = n = k x-axis. For our algorithms that optimize for the L3 cache, there is very liŠle performance difference between the square case and the case where an algorithm is the “correct” choice. We conclude that Figure 11: Comparison for different shapes on i7-7700K when executing MMM, optimal I/O properties are aŠainable two with low-bandwidth scenario (1 channel of DDR4-800). dimensions are at least the square root of the last level cache size, as long as the third dimension is much larger. Œe algorithms A3B2C0, B3A2C0, andC3A2C0 outperform Goto’s so algorithms that reduce I/O costs are beneficial for the additional algorithm for larger problem sizes in this low bandwidth scenario. reason that they reduce energy usage. Œis is because even when the algorithm is wrong for the problem Many algorithms in libraries that implement higher-level linear shape, the I/O cost is only 50% higher. In comparison, on this com- algebra, such as LAPACK [1] or libflame [9], take advantage of puter, Goto’s algorithm has an I/O cost that is approximately two the fact that MMM implementations in BLAS libraries are efficient times greater than the optimal algorithm. when the k dimension is relatively small, on the order of a cou- ple of hundred, as Goto’s Algorithm reaches its maximal efficiency when k is equal to the blocksize kc . We expect future computers to be bandwidth bound when executing MMM in such situations, and 6 SUMMARY algorithms for MMM that have larger blocksizes will be used. To We have developed a new family of algorithms for MMM that effec- take advantage of algorithms that use larger blocksizes, LAPACK tively utilizes a cache hierarchy with multiple layers of fast mem- and FLAME can use larger blocksizes, however this is currently ory, using two loops at each level of the memory hierarchy. We disadvantageous because the larger their blocksizes are, the more then demonstrated performance improvements over state-of-the- time is spent during inefficient unblocked subproblems. According art implementations of MMM when I/O cost to main memory is to this line of thought, LAPACK and FLAME would benefit from a limiting factor. Algorithms like this are key to delaying the in- algorithms that do not suffer from this weakness. One possibility evitable situation where MMM becomes memory bound. is to use recursive algorithms, as advocated in Peise and Bienti- We chose to focus on potential performance benefits of these nesi [23]. algorithms and demonstrated those benefits in our experiments. Hard drives and other similarly slow storage devices can be However memory movements cost far more energy than flops do [25], thought of as another layer of the memory hierarchy. Because The MOMMS Family of Matrix Multiplication Algorithms SC19, November 17–27, 2019, Denver, CO of this, we believe the methodology in this paper can be used to REFERENCES instantiate out-of-core algorithms for MMM. A major difference [1] Edward Anderson, Zhaojun Bai, Christian Bischof, L Susan Blackford, James between such out-of-core algorithms and the ones in this paper Demmel, Jack Dongarra, Jeremy Du Croz, Anne Greenbaum, Sven Hammarling, Alan McKenney, et al. 1999. LAPACK Users’ guide. SIAM. targeting LRU caches is that out-of-core algorithms may require [2] Kent Czechowski, Casey BaŠaglino, Chris McClanahan, Aparna Chan- explicit transfers to disk and explicit overlapping of I/O and com- dramowlishwaran, and Richard W Vuduc. 2011. Balance Principles for Algorithm-Architecture Co-Design. HotPar 11 (2011), 9–9. putation. [3] Jack Dongarra, Jean-Franc¸ois Pineau, Yves Robert, Zhiao Shi, and Fr´ed´eric Future work is to generalize the MOMMS family of algorithms Vivien. 2008. Revisiting matrix product on master-worker platforms. Interna- to other dense linear algebra operations, much in the way that tional Journal of Foundations of Computer Science 19, 06 (2008), 1317–1336. [4] Jack J. Dongarra, Jeremy Du Croz, Sven Hammarling, and Iain S. Duff. 1990. A Goto’s algorithm [7] was generalized to the rest of the level-3 BLAS [8]. set of level 3 basic linear algebra subprograms. ACM Transactions on Mathemat- Œe key point is that most suboperations during the other level-3 ical So‡ware (TOMS) 16, 1 (1990), 1–17. BLAS operations (that operate on structured matrices) are regular, [5] Gianluca Frison, Dimitris Kouzoupis, Tommaso Sartor, Andrea Zanelli, and Moritz Diehl. 2018. BLASFEO: Basic linear algebra subroutines for embedded unstructured MMM operations [16]. optimization. ACM Transactions on Mathematical So‡ware (TOMS) 44, 4 (2018), 42. [6] Kazushige Goto and Robert van de Geijn. 2002. On reducing TLB misses in matrix multiplication. Technical Report TR-02-55. Department of Computer Sciences, Œe University of Texas at Austin. [7] Kazushige Goto and Robert van de Geijn. 2008. Anatomy of high-performance matrix multiplication. ACM Transactions on Mathematical So‡ware (TOMS) 34, 3 (May 2008), 12. [8] Kazushige Goto and Robert van de Geijn. 2008. High-performance implementa- tion of the level-3 BLAS. ACM Transactions on Mathematical So‡ware (TOMS) 35, 1 (2008), 4. [9] John A. Gunnels, Fred G. Gustavson, Greg M. Henry, and RobertA. vande Geijn. 2001. FLAME: Formal linear algebra methods environment. ACM Transactions on Mathematical So‡ware (TOMS) 27, 4 (2001), 422–455. [10] John A. Gunnels, Greg M. Henry, and Robert A. van de Geijn. 2001. A Family of High-Performance Matrix Multiplication Algorithms. In Proceedings of the International Conference on Computational Sciences. Springer-Verlag, 51–60. [11] Alexander Heinecke, Greg Henry, Maxwell Hutchinson, and Hans Pabst. 2016. LIBXSMM: accelerating small matrix multiplications by runtime code genera- tion. In Proceedings of the International Conference for High Performance Com- puting, Networking, Storage and Analysis. IEEE Press, 84. [12] Greg Henry. 1992. BLAS based on block data structures. Technical Report. Cornell University. [13] Jia-Wei Hong and Hsiang-Tsung Kung. 1981. I/O complexity: Œe red-blue peb- ble game. In Proceedings of the thirteenth annual ACM symposium on Šeory of computing. ACM, 326–333. [14] Intel. [n. d.]. Math Kernel Library. ([n. d.]). hŠps://so‰ware.intel.com/mkl [15] Dror Irony, Sivan Toledo, and Alexander Tiskin. 2004. Communication lower bounds for distributed-memory matrix multiplication. J. Parallel and Distrib. Comput. 64, 9 (2004), 1017–1026. [16] Bo K˚agstr¨om, Per Ling, and Charles Van Loan. 1998. GEMM-basedLevel 3 BLAS: High Performance Model Implementations and Performance Evaluation Bench- mark. ACM Transactions on Mathematical So‡ware (TOMS) 24, 3 (1998), 268– 302. [17] Hsiang-Tsung Kung. 1985. Memory requirements for balanced computer archi- tectures. Journal of Complexity 1, 1 (1985), 147–157. [18] Monica S. Lam, Edward E. Rothberg, and Michael E. Wolf. 1991. Œe cache per- formance and optimizations of blocked algorithms. In ACM SIGARCH Computer Architecture News, Vol. 19. ACM, 63–74. [19] Jin Li, Anthony Skjellum, and Robert D. Falgout. 1996. A poly-algorithm for parallel dense matrix multiplication on two- dimensional process grid topologies. Master’s thesis. Mississippi State University. Department of Computer Science. [20] Tze Meng Low, Francisco D. Igual, Tyler M. Smith, and Enrique S. ‹intana- Ort´ı. 2016. Analytical modeling is enough for high-performance BLIS. ACM Transactions on Mathematical So‡ware (TOMS) 43, 2 (2016), 12. [21] Nicholas D. Matsakis and Felix S. Klock II. 2014. Œe rust language. In ACM SIGAda Ada Leˆers, Vol. 34. ACM, 103–104. [22] JD McCalpin. 1995. Memory bandwidth and machine balance in high perfor- mance computers. IEEE Technical Commiˆee on Computer Architecture (TCCA) Newsleˆer (December 1995). [23] Elmar Peise and Paolo Bientinesi. 2017. Algorithm 979: Recursive Algorithms for Dense Linear Algebrafi”Œe ReLAPACK Collection. ACM Transactions on Mathematical So‡ware (TOMS) 44, 2 (2017), 16. [24] MartinD.Schatz, RobertA. vande Geijn, andJackPoulson. 2016. Parallel matrix multiplication: A systematic journey. SIAM Journal on Scientific Computing 38, 6 (2016), C748–C781. [25] John Shalf, Sudip Dosanjh, and John Morrison. 2010. Exascale computing tech- nology challenges. In International Conference on High Performance Computing for Computational Science. Springer, 1–25. [26] Tyler M. Smith, Bradley Lowery, Julien Langou, and Robert A van de Geijn. 2019. A Tight I/O Lower Bound for Matrix Multiplication. arXiv preprint arXiv:1702.02017 (2019). SC19, November 17–27, 2019, Denver, CO Tyler M. Smith and Robert A. van de Geijn

[27] Tyler M. Smith, Robert van de Geijn, Mikhail Smelyanskiy, Jeff R. Hammond, and Field G. Van Zee. 2014. Anatomy of high-performance many-threaded ma- trix multiplication. In 2014 IEEE 28th International Parallel and Distributed Pro- cessing Symposium. IEEE, 1049–1059. [28] Field G. Van Zee, Tyler M. Smith, Bryan Marker Bryan, Tze Meng Low, Robert van de Geijn, Francisco D. Igual, Mikhail Smelyanskiy, Xianyi Zhang, Michael Kistler, Vernon Austel, John Gunnels, and Lee Killough. 2016. Œe BLIS frame- work: Experiments in portability. ACM Transactions on Mathematical So‡ware (TOMS) 42, 2 (2016), 12. [29] Field G. Van Zee and Robert A. van de Geijn. 2015. BLIS: A framework for rapidly instantiating BLAS functionality. ACM Transactions on Mathematical So‡ware (TOMS) 41, 3 (2015), 14. [30] Qian Wang, Xianyi Zhang, Yunquan Zhang, and Qing Yi. 2013. AUGEM: auto- matically generate high performance dense linear algebra kernels on x86 CPUs. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis. ACM, 25. [31] R. Clint Whaley and Jack J. Dongarra. 1998. Automatically tuned linear algebra so‰ware. In SC’98: Proceedings of the 1998 ACM/IEEE conference on Supercom- puting. IEEE, 38–38. [32] Samuel Williams, Andrew Waterman, and David PaŠerson. 2009. Roofline: An insightful visual performance model for multicore architectures. Commun. ACM 52, 4 (2009), 65–76. [33] Kamen Yotov, Xiaoming Li, Gang Ren, MJS Garzaran, David Padua, Keshav Pingali, and Paul Stodghill. 2005. Is search really necessary to generate high- performance BLAS? Proc. IEEE 93, 2 (2005), 358–386.