Uniform Techniques for Loop Optimizationt

William Pu h Dept. of Computer Science and Institute For Advanced Computer Science Univ. of Maryland, College Park [email protected]

ment into the following parallel code fragment, a task that is Abstract difficult or impossible for other automatic methods. Many different kinds of loop transformations have been described, such as loop interchange, loop skewing and loop fusion. for t :. 2*k+2 to n+k+l do Each transformation requires its own particular set of depen- B[t-–k, H-k] := B[t-2-k, k+l ]*B[k, k] dence analysis tests and paraltelizing a section of code may forall j :. k+l to &k–l do require perforfig a series of transformations. The only way to B[wj, j] := B[Fj, J]+B[&j–1, Jl+B[kj, j–ll+BIM, kl*B[k, j] decide if there is a way of parallelizing a section of code is to try for t :. n+k+2 to 2*n do all possible sequences of transformations, which presents a diffi- forall j := >n to n do cult search problem. B[tcj, j] := B[&J, j]+B[&j–1 , jl+B[*j, j–ll+B[+j, kl*B[k, j] We present a uniform method of performing loop optimization. Rather than optimizing a program by performing a murky Our approach consists of finding a schedule TJx] for each search through a series of transformations, our method consid- statement SUdescribing the moment at which each iteration x of ers a very powerful class of program transformations that in- SF will be executed. For example, the code above is produced cludes any transformation that also can be obtained by any sequence of standard loop transformations. This optimization using the schedules T1[i] = i+k+l/2 and 7’2[i, j] = i+j. A schedule technique uniformly encompasses the effects of parallelization, is feasible if it respects the orderings imposed by the dependen- loop fusion, loop splitting, loop interchange, loop skewing and cies in the program. Different schedules are isomorphic to the statement reordering, as well as transformations not previously effects of different combinations of loop and code transforma- described. Thus, we only need to perform one program trans- tions. We optimize a program by evaluating the code produced formation. We show that standard techniques for representing depen- by different feasible schedules. Finding a feasible schedule re- dencies (dependence directions and dependence distances) are quires a more sophisticated method of analyzing dependencies insufficient for this type of optimization and describe a more than is normally used. The techniques described here are not as powerful technique, dependence relations, that have sufficient fast as simple optimization techniques and are intended to be descriptive power. used in practice-only when simpler techniques are insufficient. The techniques described here, when specialized for “typical 1. Introduction cases,” may prove to be as fast as simpler techniques. Many different kinds of loop transformations have been de- The idea” of equating code-rearrangemen~ with finding scribed [AK87, Poly88, WO189, W0190], such as loop inter- schedules for an iteration space has been examined by several change, loop skewing and loop fusion. However, there is no authors [Lam74, Rib90, W0190, Ban90]. Previous approaches automatic way to decide which sequence of transformations have been severely limited in their ability to deal w;;h imper- need to be applied to parallelize a section of code. We describe fectly nested loops and complicated or transitive dependencies. a new automatic code transformation technique for paralIeliz- In this paper, we propose methods for handling dependencies ing code. If any combination of standard loop transformation and imperfectly nested loops that move this approach substan- can parallelize the code, our techniques will parallelize the tially forward to reaching its full potential as an optimization code. method that can subsume all other loop optimizations. For example, consider the following code fragment: Most previous work on program transformations use data dependence directions and data dependence distances to sum- for 1 := k+l to n do mary the dependencies between two array references. For our B[i, 1] := B[i–1, k+l]*B[k, k] purposes, these techniques are too crude. We use integer pro- for.j := k+l to n do gramming techniques [o represent and evaluate dependencies B[i, j] := B[i, j]+B[l–1, j]+B[i, j–ll+B[i, kl*B[k, j] exactly. This approach provides more information about the Our methods can automatically transform this code frag- types of transformations allowable and also allows us to prop- erly handle transitive dependencies. Integer programming is a NP-Complete problem and solv- t This research is supported by NSF grant CCR-8908900. ing such problems may require exponential time. However, we Permission to copy without fee all or part of this material is have found that problems arising in practice can be efficiently granted provided that the copies are not made or distributed for manipulated, simplified and solved. We make use of the direct commercial advantage, the ACM copyright notice and the Omega test [Pug91], an integer programming simplification title of tha publication and its date appear, and notice is given and decision alg&ithm desig;ed ~or use in dependence analy- that copying is by permission of the Association for Computing sis. Most of the problems encountered can be solved in 1-3 Machinery. To copy otherwise, or to republish, requires a fee milliseconds (on a Dccstation 3100, a 12 MIPS workstation). and/or specific permission. ~ 1991 ACM 0-89791-434-1/91/0006/0341 ...$l .50 3 41 operation Description Definition FoG The composition of F with G (z, z)~FOG=~ys.t. (x, y)c GA(y, z)e F F-1 The inverse of F (x, y)= F-l=(y, x)=F FuG The union of F and G (x, y)= FuGe(x, y)= Fv(x, y)EG FnG The intersection of F and G (x, y)e FnG@(x, y) EFA(x, y)6G I Identity function (X,X)=1 F* The transitive closure of F least fixed point of F’ = I u F o P F\S The relation F restricted to domain S (x, y) EF\S=(X, y)e FAXe S F/S The relation F restricted to range S (x, y)c F/ S=(X, y)= FAy=S F;G The pointwise sum of F and G (x, y+z)e F; G~(x, y)e FA(x, z)c G F~G The pointwise difference of F and G (x, y-z)~F~G ~(x, y)c FA(x, z)e G SUT The union of S and T XESUTWXESVXET SnT The intersection of S and T xeSnT@XeSAxcT domain(F) The domain of F xedomuin(F) =3ys.t. (x, y)e F range(F) The rarwe of F yenmge(F)=3 xs. t.(x, y)e F 3XES Is S em~ty? Returns ‘~alse” if S = 0 otherwise some tuple x in S Tab [ – Operations on tuple sets and relations, where F and G are tuple relations and S and T are sets of tuples

To achieve the best results from the methods described in (e.g., we do not consider relations take as input both l-tuples this paper, we must apply techniques such as constant propa- and 2-tuples). gation, forward substitution, induction variable substitution, We use integer programming techniques to represent and scalar expansion and variable copying [ZC91] to programs manipulate relations and sets. A relation is defined by a collec- before we attempt to optimize them. tion of linear equalities and inequalities involving the elements This paper is broken into the following sections. Section 2 of the input and output tuples, symbolic constants and wild- describes a class of operations on sets and relations of integers cards. Such a relation contains exact those mappings of integer tuples (the implementations of these operations are described tuples to integer tuples that satisfy the constraints. A set is de- in Appendix 1). Section 3 describes how to obtain a set of fined by a collection of linear equalities and inequalities involv- scheduling dependencies for a code fragment. Section 4 dis- ing the elements of the input and output tuples, symbolic con- cus ses how to find feasible schedules for a set of statements. stants and wildcards, and such a set contains exact those inte- Section 5 show how a feasible schedule is used to rearrange ger tuples that satisfy the constraints. An example of a relation code. Section 6 discusses the relationship between different from 2-tuples [sl, SJ to 2-tuples [tl, f2] is { fl = sl+l; fz= sl+2s2 ] schedules and optimizations. Section 7 shows that our tech- and an example of a set of 2-tuples [sI, S2] is { 1s SI < S2<10 }. niques can find an optimization equivalent to any combination In the representation of a set, a system of linear equalities of a series of standard loop optimization. Section 8 discusses and inequalities can refer to the components of tuples in the set, related work and Section 9 summarizes the paper. Sections 4,5 (denoted as Sl, S2,.... sk) symbolic constants (denoted by the use and 6 show the parallelization of two examples that could not of typewriter font) and wildcards (any other variables). For ex- be automatically parallelized by standard techniques. ample, the set of 2-tuples represented by { SI + S2 = 2ci ) is the Appendix 1 describes the implementation of the operations set of all tuples [x, y] such that x+y is even, and the set of described in Section 3. Appendix 2 gives an overview of the 2-tuples represented by { SI + S2< n ] is the set of all tuples [x, y] Omega test, which is used for manipulating the integer pro- such that x+y is at most n. gramming problems that arise in the methods described here. The representation of a relation is similar, except that the lin- The methods presented in the main body of the paper only dis- ear equalities and inequalities for a relation can refer to the cuss highly parallel schedules (schedules that take at most components of the input (i.e., argument) tuple (denoted as Sl, O(vsn) time, where m is the number of statements and n is the 52, ....sk) and also the components of the output (i.e., result) maximum number of iterations of any loop). Appendix 4 tuple (denoted as tl, t2,....tk.),as well as referring to wildcards briefly discusses nested time, a method that allows our tech- and symbolic constants. niques to be used even in circumstances where no highly paral- Note that we can accommodate integer division and lel schedule exists. Appendix 4 shows the parallelization of an remainder operations. For example, if m is a known constant, iterative smoothing program, and Appendix 5 shows the paral- x=ydivrn can berepresented by{ O .... ek] = [fl, fz .... fk,] (p] to denote the relation that would be more fully represented by

342 {SI =el; s~=e~ .... s~=e~ fl =fi; t~ =~~ .... f~=~p;P). upward. Convention dependence analysis would determine For example, the relation G denoted by that there is a data dependence from s3[i, j] to s3[i, j+l ] and require that s3[i, j] happen before s3[i, j+l]. If we assume machine {sl+l=s~; t1=s1;1compiler to make the assumption that addition and multiplica- 52 B[l, j] := B[i, j] + B[i–1, j] + B[i, j–l] + B[i, kl*B[k, j] tion in machine arithmetic is associative. Although this is a false assumption, it often does not produce significant differ- Using the techniques describe in this section, we can easily ences in the results of the program and can produce significant determine the dependence relations shown in Table 2, which speed-ups. Assume ss[i, j] adds b U] to a[i] and the j loop counts are shown graphically in Figure 1. The from and to columns

1 Con~tionaI assimment is an assignment which is executed only if a b~lean 2 mere any ~any ofier fact~~concerning properties an algorithm must have guard is true. Standard assignment statements are just condltmnal m order to be able to be executed efficiently on a SIMD distributed- assignments with a boolean guard oft rue. memory machine, some of which are discussed in [Wolfe901.

343 Direct dependencies: dependence Initially, we consider only highly parallel # dependence relation range from to direction integer affine schedules. A highly paraHel 1 sz[i, k+l] 5 sl[i+l] k

n- [ n 3 we may ~i~h to ~ched& larger sections of the graph at a time by, allOwmg

us to consider additional transformation that might reduce overhead in Figure 1- Graphical illustration dependencies for Example 1 of the final code.

344 “ F31

rl~r,-...=-ULe z - uuIIIpuIuIg/- —.—...,-- awcr. .._.. waI-uepeIKKIIut5.-.l. J-— _-J_-.:.- of all the edges from p to q. There may be edges from a vertex Nominating, testing and transferring candidate schedules back to itself. We enumerate possible schedules, starting with schedules with We can find the complete set of self-dependencies for a the smallest l–norm (the l-norm is the Manhattan metric; the statement by repeatedly trimming this graph until only the ver- sum of the absolute values of the coefficients). Note that con- tex for the statement of interest in left. We first remove any stant offsets in timing functions cancel out and can be ignored vertices not in the same strongly connected component as the when examining self-dependencies. This enumeration process vertex of interest. We then perform the following process until takes into account the limitations imposed by the specific con- only the vertex of interest remains. (Nofe: the process we use is stant self-dependence distances of SPseen so far. For example, very similar to the process of deriving a regular expression for assume the schedule for sJi, j] is of the form Tp[i, j] = ~ i + $ j a NFA). + Cp.We might test the schedule Tp[i, j] = j and be told that it is If there are two edges going between the same two vertices illegal because of a self-dependence sP[i, j] S sJi+ 1, j–l ]. Given (in the same direction), we replace them with a single edge la- this information, we know to nominate only schedules such beled with the union of the dependencies of the two original that ~– ~P >0, edges. If a vertex x has a single self-edge labeled with a depen- Once we have a schedule that appears to satisfy all transitive dence F, we replace x with two new vertices, x’ and x“, such dependencies for SP, we check the schedule against reduction that all edges that went to x now go to x’, all edges that left x and broadcast dependencies. If it passes those tests, we transfer now leave from x“ and there is an edge from x’ to x“ labeled the schedule to the other statements in the SCC. Assume we with the dependence F*. When we eliminate a vertex with no have decided on a candidate schedule for SPand we wish to self-edges, we generate a new edge for each combination of in- determine the constraints this produces on the schedule for Sq coming and outgoing edges, labeled with the compositions of Let F be a relation that describes the dependence SP5s~ and let the edges. Figure 2 shows an example of computing the exact G be the relation that describes the dependences~ 5 Sp We then self-dependencies for S1.Note that S1is not a feedback vertex, so have we are required to compute transitive closures. (TP o F-’)[xl < Tq [xl < (Tp o G)[xl. Note that since roe are only nominating candidate schedules, we From this information, we can often derive the only allowable can use approximations that report false negatives form for the schedule of s?. If this does not provide us with (approximations that might report no dependence exists when enough information, we can combine it with enumeration in fact one does exist). At this stage, approximations that pro- methods. duce a small number of false negatives are preferable to ap- Testing and Aligning Schedules proximations that produce a larger number of false positives. Now that we have developed a set of candidate schedules for This is particularly important since we do not currently have ef- all statements, we can test and align them. To do this, we sim- fective and general methods for computing F’ (we can approx- ply check that the schedules imply that if SPIXI S sJyl, then TPIxI imate F%for any F as 1 u F u F o F u F o F o F; for certain spe- < T,Jy] and if SP[X] SR s.$y] or SP[X] SB s~[y], then Tp[x] # Tq[y]. cial forms, exact methods for computing F* are described in Note that we allow each schedule to include a constant offset. Appendix 1). When only considering self-dependencies, the constant offsets Computing self-dependence distances drop out. When comparing dependencies between two differ- If F describes all the self-dependences of a statement SPon itself, ent statements, the constant offsets become very important, As then dd = rarzge(F ~ 1) is the set of all self-dependence distances will be seen, allowing fractional offsets is sometimes essential for SP.If TP is a candidate timing function for SF,then to aligning schedules. In Section 4, we will see that fractional (TP \dd /([f] (t s 0))) is empty iff TPis a legal timing function for Sp If it is not empty, 4 *Ctualq,, we don>tn~~d fractional offsets. Given a schedule with fractional we can determine a specific constant self-dependence distance offsets, we can obtain an equivalent fbut longer) schedule by multiplying of SPthat is violated by TV by an appropriate constant. For example, rather than using the schedules TI [i] = 2+k+0.5 and T2[i, j] = i+j in Example 1, we could use the schedules

345 Iteration Iteration Iteration Iteration k+l space of 52 n k+l space of 52 n space of sl space of S1 1 j j 1 k+l @@

. ,:..:.:... i ,:......

@#@@ 43 1 G

Figure 3 – Illustration of the schedules TI [i] = i+k+7z and T2[i, J = Figure 4 – Hlustration of the schedules Tl[i] = 2i–Y2 and T21i, j] = i+j i+j f or Example 1, with some of the more constraining dependencies for Example 1, along with some of the more constraining dependen- for this example. Each wavefront shows the set of tasks performed cies for this exam le. The wavefronts shown are for f equal to 1’ at a particular moment of time. The figure shows the wavefronts at t 2k+3%, 2k+4, 2k+4 /2 2k+5 and 2k+5~2. In this schedule, S1 gets equal to 2k+4, 2k+41/2 and 2k+5. The wavefronts for the two itera- executed as late as possible. tion spaces are synchronized. In this schedule, SI gets executed as early as possible.

offsets in schedules produce the effects of statement reordering, The only solutions to these constraints with integer coefficients and the integer parts of the constant offset produce the effec~s are Tl[fi = i+k+cl, O < c1 <1 or Tl[i] = 2i+cl, –1’< c1 <0. These of loop fusion and splitting. schedules are shown graphically in Figures 3 and 4. We con- We test and align schedules bygenerating all constraints sider using the first (motivated by the fact that all other things imposed bydirect dependencies and seeing if there is some set being equal, it is better to have unary coefficients). Having a of constant offsets that can be given to the schedules so as to candidate set of schedules, we check the schedules against all satisfy the constraints. Since different constant offsets produce the direct dependencies. For our example, we have already different code, we produce a simplified set of constraints that checked the schedules against most of the dependencies while allows us to examine the different possibilities for constant transferring S2’Sschedule to S1.We find that the schedules TI [i] off sets. = i+k+~/2 and T2[i, j] = i+j are legal with respect to all the Note that this is the one point where we are forbidden to use dependencies. any approximation to the true dependencies that gives false Example 2, part 2- scheduling negatives (that fails to report a dependence where there actu- The transitive self-dependencies for Example 2 are ally is one). If we used approximate methods previously, we sl[i, j] 5 s2[i+2, j+2] S s1[i+2, j+3] will catch any potential problems here. sl[i, j] ~s2[i+Z j+zl 3 Sl[i+s, j+ll Example 1, part 2- scheduling s.Ji, j] S sl[i, j+l] S s2[i+2, j+31 Before considering transitive dependencies, we first examine s2[i, j] 6 sl[i+l, j–l] 5 s2[i+3, j+ll direct self-dependencies and see how much they restrict the These dependencies give us a fair degree of freedom in candidate schedules. Let Tl[i] = CXli + Y1k + c1 and TJi, j] = ~z i scheduling thk code. We consider the case where we know m is + 1%j + I’Z k + C2. Without loss of generality, we assume C2= O. small compared to n and therefore would prefer to parallelize We enumerate possible schedules for S2, with the following the i loop. This corresponds to schedules of Tl[i, j] = j+cl and results: T2[i, j] = j+c2 (that both meet our nomination criteria). Without Try T2[i, j] = i: fails, due to dependence distance of (0,1) loss of generality, we assume C2= O,We now have to check that (restricts schedules to ones in which ~z > O) these schedules are feasible. Since we already have a schedule Try Tz[i, j] = j: fails, due to dependence distance of (1,0) for both dependencies, we simply have to check that we can (restricts schedules to ones in which az > O) align the two schedules, based on the cross–statement depen- Try T2[i, j] = i+j succeeds dencies: While we could examine the transitive self-dependencies for .@i, j–l] 5 sl[i, j] a T2[i, j–l] < T1[i, j] + j–1< j+cl SI to restrict candidates for a schedule for Sl, we illustrate our sJi-1, j+l] S sl[i, j] a T2[i–1, j+l] < Tl[i, j] * j+l< j+cl method of transferring schedules. We use the dependencies sl[i–2, j–21 5 sz[i, j] * Tl[i–2, j-2] < T2[i, j] ~ j–2+cl < j between SI and S2to transfer the schedule: This set of constraints simplifies to 1< c1 <2 that can be satis- Dependence Inferred schedule constraint range fied with c1 = 1%, giving the schedules TI [i, j] = j+ll/2 and sl[i] 3 sz[i, i] Tl[i] < T2[i, i] = 2i k+l SiSn T2[i, j] = j. sl[i] S s2[i+l, i] Tl[i] < T2[i+l, i] = 2i+l k+l

346 segments such that during a single segment, one set of state- for all f and ct s.t. 2k+2 s fs 2n and k+l, f-n s a 5 f-(k+l), n. ments is scheduled. During the whole unit time interval around t,, we decide to ex- We use loops to advance through time by whole units. ecute all statements that execute at time f. – 1< fs f., rather Within the body of a time loop with loop variable tUnit, we than f, s f < fU+l. This realignment allows S1[(t% )–k–’/z ] to might execute all statements that execute at times f for fUnit < f execute at t– 1/2 during the period 2k+2 s f < n+k+l (which c fUnif+l, or we could choose to execute all statements that are aligns the start times for S1and S2). This generates the following supposed to execute at times f for fUnif –1 < f < fUnit. Picking code: different rounding methods for fractional times effects the code for t := 2“k+2 to n+k+l do that is generated. Executing statements scheduled for a unit in- sl[t – 1– k] terval around a whole time unit simply produces several dis- forall a := max(k+l, t-n) to min(n, &k–l) do tinct groups of statements to be executed in order. Statements s2[t – a, U] scheduled to execute at the same moment can be executed in for t := n+k+2 to 2*n do parallel (or in any order). forall a := max(k+l, t-n) to min(n, t-k-1) do Once we have selected rounding methods for fractional sz[t – a, a] times, we must generate the appropriate loops to execute all of the iterations of a statement that must be performed at any We find we can resolve the min and max computations stati- moment. Consider a schedule TP[i, j] = ai + ~j + yk + c. Let 7’Ji, cally, producing j] = cti + ~j be the variable portion of the schedule and let g = for t := 2*k+2 to n+k+l do gcd(ct,~). If g >1, then SP will only be executed every g time sl[t – 1– k] steps, and appropriate code must be generated. Regardless of forall a := k+l to t-k-1 do sz[t -a, a] whether g >1, consider the following matrix for t := n+k+2 to 2*n do forall a := tin to n do sz[t –a, a] Q=[a:p:l Substituting back the bodies of the statements and renaming a toj produces: [“:g‘ig][;l=[f!l for t := 2“k+2 to n+k+l do We will use s as the loop variable for our non-temporal loop. B[t-–k, t-l-k] := B[t-2-k, k+ll * B[k, k] We pick u and b to be integers such that Det[Q] =+1 (so that Q forall j := k+l to t-k-1 do is unimodular [Ban90]). Therefor, Q is invertible: B[&j, j] := B[*j, j] + B[tij-1, j] + B[kj, j-1] + B[tcj, k]*B[k, j] for t := n+k+2 to 2“n do [a:g’(gr[f!l=[;l forallj := ~n to n do Performing this calculation and substituting the results into the B[tnj, j] := B[Gj, j] + B[Gj-1, j] + B[~j, j-l] loop bounds for i and j gives appropriate loop bounds for t/g + B[kj, k]*B[k, j] ands. Example 2, part 3- code rearrangement Similar techniques are used for schedules involving more For example 2, the schedules are Tl[i, j] = j+l~z and T2[i, j] = j. than two loop variables, although extending a schedule involv- This tells us that we must execute SI [a, (f+v2 )–1 721 at time ing more than two loop variables into a unimodular matrix can f+ 1/2 for 4< fs m and execute s2[a, t] at time t for 3 s fs m–1. be more difficult. Consider the schedule T@[i,j,k] = 6i+10j+15k. This produces the following code. This can be extended to a unimodular matrix as: [610151 forall a := 3 to n do s2[a, 3]; for t := 4 to m-l do forall a := 3 to n do sz[a, t]; forall a := 3 to n do S1[CC,*1]; To extend a row [al, LJ2,.... am] with a gcd of 1 into a unimod- forall i := 3 to n do sl[a, m–l]; ular matrix, we can use a row with a 1 in position L and O else- Substituting back the bodies of the statements produces: where to transform the problem into finding a unimodular extension of [al, az, .... a~.l~a~+l, .... uJ. If there e~ist two compo- forall i := 3 to n do b[i, 3] := g(c[i–2, 1]); nents of the row that have a gcd of 1, we use this technique to for t := 4 to m–1 do reduce the problem to finding a unimodular extension of those forall i := 3 ton do b[i, t] := g(c[i–2, t-2]); two components. If no such pair exists, we reduce the problem forall i := 3 to n do c[l, Fll := f(b[i, t-21, b[i–1, t]); down into finding a unimodular extension of a row of three forall i := 3 to n do c[i, m–1] := f(b[i, m–21, b[i–1, ml); elements, and use more elaborate techniques that can find a Note that the tightness of the schedule did not allow us any unimodular extension of a row of three elements. flexibility in aligning the schedule. Consider what would hap- The Omega test’s ability to perform simplification is used ex- pen if S2in the original code was diffenmt: tensively in calculating the lower and upper bounds of the resulting loops. fori:=3 tondo Example 1, part 3- code rearrangement forj := 4 to m–1 do Remember that TI [i] = i+k+~2, so we need to execute .$1: C[i, J] := f(b[l, j–l], b[i–1, j+ll); S1[(f+% )–k– ‘/2] at time f+ 72 for all f such that k+l < (f+’/2 )– S2: b[i, j] := g(c[i-2, j-3]); k– $’2

347 equal, the computations for S1and S2can be performed in parallel. This would allow us to produce the code (for clarity, the THEOREM 1. Let P be any program consisting solely of looping original statements have not been substituted back): constructs and conditional assignments. Let P’ be any opti- mized version of P, subject to the following constraints: forall a := 3 to n do sz[ct, 4] ● The program P’ consists of sequential loops, parallel forall a := 3 to n do sz[a, 5] loops, ~obegin/coend constru~ts and conditional as- for t := 6 to m-1 do signments such that the body of each parallel loop forall a := 3 to n do contains no sequential code. Cobegin ● Each conditional assignment statement sl[il, i= . . .. im] in P 52[% t] appears once in P’ as sJfl, ~2,. ... /J, where fl, fz, .... fm sl [a, t-2] are affine functions of the loop indices surrounding the coend occurrence ofs, in P’. forall a := 3 to n do Sl[a, m–2] ● The execution of P’ involves executing the same iterations forall a := 3 to n do sl[ci, m–1] of the same statements as P, in some order that respects the ordering implied by the dependencies between the 6. The relation between schedules and statements and ordering between the iterations of the optimization statements in P. Schedules of the form TP[i, j] = i+% or TP[i, j] = j+cP correspond Then the methods described in this paper can derive P from P. to parallelizing St along j and i respectively. Schedules of the PROOF: The restrictions imposed on P’ allow us to direct form Tp[i, j] = i+ji-cp, Tp[i, j] = i+2j+cp or Tp[i, j] = i-j+cp corre- derive an affine schedule for the time at which each statement spond to parallelization using different wavefronts or loop sz[il, iz, . . .. im] is executed (involving nested time iff SXappears skewing. Fractional offsets in schedules produce the effects of in nested sequential loops). Therefore P’ specifies a complete statement reordering, and the integer parts of the constant offset of schedules for the statements of P. Since P’ must respect set produce the effects of loop fusion and splitting. the ordering imposed by the dependencies of P, the schedule 7. Completeness Results for P must respect those dependencies. Therefore, our methods could derive the schedule that produces P’. ❑ We claim that a transformation equivalent to my transformation that can be obtained via some combination of THEOREM 2. If P’ is derived from P by some combination of ● statement reordering, . loop interchange, statement reordering, loop interchange, loop fusion, loop skewing, loop splitting, loop reversal, loop distribution and loop ● loop fusion, parallelization, and P’ doesn’t contain any sequential code in- ● loop skewing, side of parallel loops, then P’ meets the restrictions described in . loop splittin~ Theorem 1. ● loop reversal, PROOF: None of these optimization duplicate statements, ● loop alignment, and each of them replaces the indices with an affine function of ● loop distribution and the original indices (possibly the identity function). Since the ● loop parallelization composition of a series of affine functions is an affine function, will be found by our optimization methods. Our methods can find optimizations that no combination of the above techniques the statements in P’ must be obtained by replacing the original loop indices with affine functions of the indices. Since any op- will find (as was seen in Example 1 and 2), but it is difficult to timization that did not respect the ordering require by P would describe the complete set of optimization our techniques be a faulty optimization, P’ must respect the ordering of the rnkzht find. “We should note one type of code transformation our current iterations of P imposed by the dependencies. methods cannot produce a parallel loop with a loop body con- COROLLARY 1. If P’ is derived from P by some combination of taining sequential code. For example, given the program loop optimizations discussed in Theorem 2, our methods can fori:=l tondo derive a program P“ from P that has the same inherent paral- for J:.2 tom do lelism as P’. a[i, j] := a[i, j] + a[i, j–l] PROOF.’ Given P’, we can derive a program P“ that contains no sequential code inside of parallel loops by performing loop our methods could not produce the optimization below. distribution and loop interchange to move parallel loops in- forall i := 1 to n do ward. We are assured that we can do these transformations forj := 2 tom do because no dependencies can exist between different iterations a[i, j] := a[i, j] + a[i, j–l] of a parallel loop. The program P“ has the same inherent paral- However, our methods could produce the code lelism as P’, and by Theorems 2 and 1, our methods can derive P“. •1 forj :=2 tom do fora,ll i := 1 to n do 8. Related work a[i, j] := a[i, j] + a[i, j–l] Methods for performing individual optimization have been described by a long list of authors, including [Ban79, AK87, that, except for synchronization overhead, has the same run- Nico87, W0189, LYZ89]. ning time. We are looking at ways to overcome this limitation. Whitfield and Soffa [WS90] looked at the interactions be- For the moment, we assume that a latter step of the optimiza- tween optirnizations and the order in which they should be tion process may look at moving sequential code inside of par- performed. The only loop optimization they considered were allel loops. loop interchanging and loop fusion. (They also consider strip- mining and loop unrolling, which are optimizations that would

348 be applied after the optimizations in this paper were performed), They found that loop interchange should be per- 9. Conclusion formed before loop fusion, although it is not clear this would In this paper, we have described uniform methods for performing loop optimization. These techniques automatically con- still hold if they allowed loop splitting. sider all possible combinations of standard optimization (such Hudson Ribas [Rib90] discusses techniques for manipulating dependencies that are similar to ours. His techniques can be as parallelfzaticm, loop fusion, loop splitting, loop interchange used to calculate the dependencies between two statements, but and loop skewing) and additional optimization. This allows our techniques to find ways of parallelizing code that could not he does not provide methods that allow the wide range of operations on dependencies supported by our works. be parallelized by previous techniques. Besides being of practical use, our techniques hold the promise of providing a simpler Wavefront methods language for discussing loop optimization. The idea of equating code-rearrangement with finding sched- We have shown that our techniques allows us to easily par- ules for an iteration space has been examined by several allel two programs that are difficult or impossible to parallelize authors [Lam74, Rib90, W0190, Ban90]. However, previous using standard techniques. approaches have been limited in their ability to deal with im- These techniques can easily be adapted for single assign- perfectly nested loops and complicated dependencies. ment programming languages with lazy arrays. In this setting, The intent of Banerjee’s paper [Ban90] is very close to our only a single write can be performed to any array location and own. However, Banerjee only discusses perfectly nested double dependencies always go from a write to reads of the location. loops with constant distance vectors, and only considered There are some kinds of loop optimization, such as node transformations that can be obtained by combinations of loop splitting [Poly88] and having sequential code inside of parallel reversal, loop interchange and loop skewing. loops, that cannot be described by our current methods. We are Scheduling of recurrence equations currently exploring methods of extending our techniques. The problem of finding a set of schedules for statements that respect the dependencies between the statements is closely re- References lated to the topic of parallel scheduling of recurrence equations. [AK871 R. Allen and K. Kennedy, “Automatic Translation of Doing this for a set of uniform recurrence equations was closely FORTRAN Programs to Vector Form”. ACM TOPLAS, Vol. studied in [KMW67]. An example of a set of k-dimensional 9, No. 4, Oct. 1987,491-542. uniform recurrence equations is [Ban79] U. Banerjee, “Speedup of ordinary programs: Ph. D Thesis, Report No. UIUCDCS-R-79-989, Dept. of Computer Q,(X) = fl(az(X – ‘WI), fZ3(X-ZU2)) Sciences, University of LBinois at Urbana-Champaign, 1979. a2(x) = f2(aI(X-~3), a2(~-@, a3(x–@, @3(x–@) [Ban88] U. Banerjee, “Dependence Analysis for Supercomputing~ a3(x) = f3(a2(x–r4r7)), Kluwer Academic Publishers, 1988. where x is an integer k-vector and WI, . . .. Wz are constant integer [Ban90] U. Banerjee, “Unimodular Transformations of Double k-vectors. These equations are uniform because all equations Loops,” Proc. of 3rd Workshop on Programming are defined over the same number of dimensions and because Languages and Compilers for Parallel Computing, Irvine, each the value of each equation at a point x only depends on CA, August 1990. equation values located at constant offsets from x. Recurrence [Chen86] Marina Chen, “A Design Methodology for Synthesizing equations are only defined at integer coordinates. Generally, we ParaUef Algorithms and Architectures: Journal of ParaHel wish to determine the values of the recurrence equations at the and Distributed Computing, page 461-491, 1986. integer coordinates within a convex polyhedron of k–space and [DE73] G.B. Dantzin and B.C. Eaves, Fourier-Motzkin Elimination any needed values outside that area are supplied as input. and Its Dual, ]ournal of Combmatorial Theory (A) 14 (1973), Determining a schedule for a single uniform recurrence 288-297. equation was solved in [KMW67]. Determining schedules for a [Grun90] D. Grunwald, “Data Dependence The L Test Revisted; set of two or more uniform recurrence equations is more diffi- 1990 International Conference on Parallel Processing, Penn cult [KMW67, Chen86, RK88]. The recent work is in the context State Univ. Press, 1990, pages 11-220-11223, of attempting to compile uniform recurrence equations into a [KKP90] X. Kong, D. Klappholz and K. Psarris, “The I Test: A new form suitable for execution on processor arrays. test for subscript data depend ence,” 1990 International We solve an even more difficult problem, because our de- Con~ererrce on Parallel Processing, Penn State Univ. Press, pendencies are not uniform. In particular, different recurrence 1990, pages 11-204-11211. equations (i.e., statements) are defined on different number of [KMW671 R.M. Karp, R.E. Miller and S. Winograd, “The organization of compu tations for uniform recurrence equations,” ]. ACM dimensions (i.e., are at different nesting depths) and the loca- Vol 14, pp563-590, 1967. tions from which ai(~) might need values are not restricted to be constant offsets from x. [Lam74] L. Lamport, “The Parallel Execution of DO Loops: Cornrn. There is a direct relation between a set of dependencies and oftheACM, Vol. 17, No. 2, 1974, [LYZ89] Z. Li, P.C.Yew and C.Q. Xhu, “Data dependence analysis a set of recurrence equations that require the same schedule. on multi-dimensional array references,” Proc. of ACM 1989 For example, a schedule that satisfies the set of dependencies Int. Conf. on Supercomputing, Crete, Greece, July 1989. s2[i, j–l] 5 sl[i, j], s21i–Lj+ll S sl[i, j], sdi-Zj-21 ~ ~2[ij j] [Nico87] A. Aiken and A. Nicolau, “Loop Quantization, or unwind- can be satisfied if and only if the same schedule can be used to ing done right: Proceedings of the 1987 ACM International compute the recurrence equations Conference on Supercomputing, Springer-Verlag LNCS 298, %(x) = fl( Uz(x - ( 0, 1 )), @2(x – ( 1,-1 )) ) May 1987. az(x) = /2( fh(x – <2,2 )) ) [P01y58] C. Polychronopoulos, Parallel Programming and Compilers, Kluwer Academic Publishers, 1988.

[Pug911 William Pugh. The Omega Test: a fact and practical integer programming algorithm for dependence analysis, Tech. Report 5 Such ,~ Ae ~bili~ ~ompose dependence, which is essential tO sched~ing #91-50, Institute for Advanced Computer Studies, Univ. of

imperfected nssted loops. Maryland, Coltege Park, Aprit 1991.

349 [Ribas90] Hudson Ribas, Obtaining Dependence Vectors for Nested- Transitive Closure Loop Computations, 1990 International Conference on It is often difficult or impossible to derive a simple, closed form Parallel Procesing. for the transitive closure of a dependency. Fortunately, in the [RK88] S.K. Rao and T. Kailath, “Regular Iterative Algorithms and situations in which we need to compute the closure of a depen- their Implementation on Processor Arrays,” Proc. of the dency, we can use approximations that produce false positives IEEE, Vol 76, No. 3, pp 259-269, March 1988. and /or false negatives. It is often more efficient in practice to &Vo189] Michael Wolfe, Optimizing Supercompilers for approximate P as some finite sequence such as .supercomp~tms, The MIT Pr~s, 1989. F*= IuFu(Fo F)u(FoF oHu(Fo FoFo F) i3Vo190] Michael Wolfe, Massive Parallelism through program Exact computation of a closed form for the closure of a Restructuring, Symposium on Frontiers on Massively dependency Parallel Computation, College Park, MD, October, 1990 We calcuIate a closed form for the transitive closure of a rw7s90] D. Whitfield and M.L. Soffa, “An Approach to Ordering dependency only when the dependence involves a constant Optimizing Transfonnations~ Pro.. of the ACM SIGPLAN dependence distance. More specifically, assume the constraints Symposium on the Principles and Practice of Parallel Programming, Seattle, March 1990, pages 137-146. for a dependence F can be represented as shown below (where P contains only inequality constraints and ~ is a vector of lWT90] Michael Wolfe and Chau-Wen Tseng. The Power Test for Data Dependence, Tech. Report CS/E90-015, Oregon constant offsets): Graduate Institute, August 1993. C(F)=~=~+~&P [ZC91] Hans Zims and Barbara Chapman. Supercompilers for In this case, the constraints for the transitive closure of F are Parallel and Vector Computers, ACM Press, 1991. F’=Iu F+ [ZY90] Z. Li and P.C. Yew, “Some Results on Exact Data C(F’)=iii=?+;~&~>O Dependency Analysis: lnnguagffi and Compilers for Parallel Computing, Edited by Gelernter, Nicolau and Padua, MIT &7=a+; &P[l+; +zl&P[7+&, ?+&+:] Press, 1990, pages 374401. Appendix 2: The Omega test Appendix 1: Implementation of Tuple Relations The Omega test [Pug91] can be used to decide if a set of linear In this section, we describe our implementation of operations equalities and inequalities has an integer solution. Also, it can on tuple relations and sets. A relation is represented by the be used to simplify an integer programming problem by elimin- union of a list of simple relations. A simple relation is repre- ating wildcard variables. The full details of the Omega test are sented by a set of linear equalities and inequalities involving described in [Pug91]. We give a few details of the important the input variables, the output variables, symbolic constants aspects here. Although the Omega test can take exponential time, it appears to take reasonable time in practice (no more and wildcards. than several milliseconds to decide or simplify the problems We do this because in general we cannot produce a closed that arise in practice). form for the union of two relations. All operations distribute across union, so this does not pose any fundamental problems. Simplifying equality tests In the following, we use C(F) to denote set of constraints for An equality testis a test of the forma + bT x = O. We calculate g, the GCD of b. If a is not evenly divisible by g, there is no way to a relation or set F. We also use; to denote the set of input variables for a relation or a set, ~ to denote the set of output vari- satisfy the constraint. Otherwise, we divide a and b by g. ables for a relation and ii or O(JI,+) to denote a set of additional Simplifying and Tightening inequality tests wildcard variables. Given a set of constraints C, we use C[= + An inequality test is a test of the forma + bT x 20. We calculate &] to denote the set of constraints produced by renaming; to & g, the GCD of b and replace a and b with l-a/gJ and b/g. In the following, F and G denote relations, S and T denote sets. Eliminating wild cards constrained by linear equalities From a set of linear equality involving a wild card, we maybe C(F-1) = C(F) [~+ 7’,7’ + 3] able to find a valid substitution for the wild card that will allow C(FO G)= C(F)[3’ + @ & C(G)[? + ~] us to eliminate it. Methods similar to the generalized GCD test C(Fn G) = C(F) & C(G) can be used [Ban88, WT90], as can other methods we have C(F \ S)= C(F)& C(S) devised that appear simpler in practice [Pug91]. C(F / S)= C(F)& C(S)[~ -+ II Checking for redundant or impossible inequalities C(F+G)=C(F)[~ +ii]&C(G)[~ +~]&~=d+~ We can simplify the inequality tests by checking when a C(FL G) = C(F)[~ + ~] & C(G)[~ + ~] & ~ = ?t-~ combination of some inequality constraints imply or contradict C(Sn T) = C(S) & C(T) another inequality constraint. For example, x >5 implies x >3 C(domain(F)) = C(F)U’ -+ ~1 and is contradicted by -x >4. In such situations, we can elimi- C(range(F)) = C(F)G’ + ~, Z + S] nate the implied constraint or decide that the entire system is For operations that introduce additional wildcard variables, infeasible if we find an impossible constraint. We could determine all redundant or impossible inequalities the Omega test is used to simplify the sets of constraints using integer programming techniques, but this probably (possibly eliminating some of the additional wildcard vari- would not be cost effective We can efficiently check to see if an ables). inequality is implied or contradicted by any single or pair of The above methods involve slightly more work than is other dependencies. In practice, we have found it sufficient to needed. By keeping the constraints is specially ordered form, check each pair of dependencies to see if they contradict each we can do part (but not all) of the work involved in simplifying other or if one of the pair is redundant. the results of operations such as composition by using matrix Eliminating unused wild cards constrained by linear multiplication. However, the above is a concise description of inequalities what gets calculated, if not necessarily how it gets calculated. If a wildcard is involved in inequalities tests but not equality Using more complicated representations can give speed tests and all the coefficients of the wildcard are unary, we elim- improvements of a factor of 2 or so.

350 ina te the variable by performing exact integer Fourier-Motzkin This extensions requires us to usc a forall loop on k and i. The variable elimination [DE73, WT90, F’ug91 ] on it. loop bounds wc derive arc: Testing if a relation or set k vacuous 6< f < 2p+n+m-2 We can use the Omega test [R@] to determine if an integer I

Original program for c := 1 ton–1 do for r := n–l downto c do fork: =1 topdo q [c, s’] cosTheta[r,c], sinTheta[r,c] := for i := 2 to n–1 do computeRotation(a[r+ 1, c], a[r, c]) forj := 2 to m–1 do fork: =c tondo S1[Ki, ~] a[i, j] := (a[i, fill + a[l–1, j] + a[i+l, j] + a[i, j+l] )/4; Sz[c,r, k] a[r, k], a[r+l, k] := Dependencies: cosTheta[r,c]*a[r, k] + sinTheta[r,c]*a[r+ 1, k], sl [k,i, j] S 51[k’,i, j+l ] l

Try ‘TI[k, i, j] = k fails due to dd distance of (0,0,1) try T2[c, r, k] = c: fails, due to dd distance of (0,-1,0) Try Tl[k, i, j] = j fails due to dd distance of (0,1,0) iry T2[c, r, k] = –r: fails, duc to dd distance of (1,0,0) Try T1[k, i, )1 = i+; fails due to dd distance of (1,0,-1) try T2[c, r, k] = c-n fails, due to dd distance of (1,1,0) Try Tl[k, i, j] = 2k-ti+j: succeeds fry T2[c, r, k] = 2C–K fails, due to broadcast dd distance of (0,0,#0) Finding an appropriate code transformation try T2[c, r, k] = 2c–r+k succeeds We first find a unimodular extension of schedule: 211 100 [1010

351 Direct dependencies range from to S2[C,r, k] S S2[C’,r–1, k] l

Broadcast “dependencies” S2[C,r, k] 5B S2[C,r, k’] l

Next, we transfer this schedule to S1. In doing so, we use the simplification abilities of the Omega test tommove wild card variables (kandc’) fromthe constraints:

dependence constraint SI[c, r] 5 S2[C,r, k] TI[c, r] < 2c–r+k, c < k Tl[c, r] < 3c–r sz[c’, r–1, cl 6 Sl[c, r] 2c’+c–r+l < Tl[c, r], c’ < c 3c–r–1 < Tl[c, r] S2[C’,r, c] S sl[c, r] 2c’+c-r < Tl[c, r], c’ < c 3c–r-2 < Tl[c, r] s2[c’, r+l, c] 5 sl[c, r] 2c’+c-r–1

This gives TIIc, r] = 3c–r–%, and we round fractions so as to execute tUnit-l < f s tunifinone iteration of the time loop. Finding an appropriate code transformation We find that we can perform appropriate unimodular extensions of the schedule by looping on c for SI and on c and k for S2. For S2,we obtain the following loop bounds: 4-n< f<2n-l 1, t-n< f S n–1, (n-l+f)/3 c, t--cs k 5 n–l +t–2c, n–l For Sl, we obtain the following loop bounds:

4-n < (t+l/2)< 2n-2

1, (t+% )/2 S c S (n–l+t+Y2 )/3, n-1 Generated Code

fort: =4–nto2n–ldo forall c := max(l, rt/2~ to mti~(n – 1 + t)/3], n–1) do Sl[c, 3C – t] forall c := max(l, t – n) to mtnt(n – 1 + t)/3J, n–1) do feral.1 k := max(c, t -c) to min(n– 1 + t – 2c, n) do S2[C, 20+k–t, k] Scheduling, transformation and resulting code, Take 2 Without worrying about broadcast dependencies, we would settle on schedules of T2[c, r, k] = 2c–r and Tl[c, r] = 2c–r~z. This would have produced the following code

fort: =3–nton–ldo forall c := max(l, t) to mtn~(n – 1 + t)/2j, n–1) do Sl[c, 2C - t] forall c := max(l, t) to minfl(n -1 + t)/2J, n–1) do forallk := c ton do S2[C, 20-t, k]

352