University of London Imperial College of Science Technology and Medicine Department of Computing Exploiting Data-Parallelism in Functional Languages Guido Karel Jouret Submitted in part fulfillment of the requirements for the degree of Doctor of Philosophy in Engineering of the University of London and the Diploma of Imperial College of Science Technology and Medicine, Friday, September 27,1991 A b s t r a c t Existing sequential languages are inherently unsuitable for data-parallel programming because of the von-Neumann execution model built-in to the semantics of these languages. Consideration of the requirements for data-parallel programming leads to the development of a declarative language based on the functional style. The powerful abstraction mechanism provided by functional languages allow the capabilities of data-parallel architectures to be presented via a set of higher-order primitive operations defined on an underlying array data-type. Functional programs consisting of composition of these data-parallel operations therefore naturally exhibit data-parallelism. A set of appropriate primitive data-parallel operations is added to a simple functional language. Derived operations are defined in terms of this initial set and sample application programs demonstrate the elegance and advantages of data-parallel programming in a functional language. A compilation scheme for compiling this extended functional language to an abstract data-parallel machine architecture is developed. An abstract data-parallel architecture, the Planar Abstract Machine (PAM) is presented. PAM is a dual-instruction architecture as two forms of basic instructions exist: those that operate on scalar (word- size) values, and those that take planar (multiple word) values as operands. A compilation scheme generates planar-code from conventional, user-defined functions, translating all basic operations and control structures into planar forms suitable for data-parallel execution. This permits the use of fully-general user-defined functions (e.g. with recursion, conditional statements, algebraic data-types) in parallel. Efficiency concerns and possible optimizations are discussed. The development of a methodology for optimizing and parallelizing programs by the transformation of computation and communication operations is developed. Further possible avenues of research, including the development of primitive operations on alternative aggregate data- structures, implementation of the data-parallel model of computation on MIMD systems, and the role of transformation in the possible development of systolic algorithms are discussed. He who sees the Infinite in all things sees God. He who sees the Ratio only sees himself only. Therefore God becomes as we are, that we may be as he - William Blake acknow ledgements A thesis is an individual and often solitary journey on the path to understanding and knowledge. The satisfaction gained at the conclusion is not from arriving, but from what has been learned along the way, both about the subject matter at hand and of one’s potentials and limitations. A number of individuals have been instrumental in my education and deserve particular acknowledgement My supervisor John Darlington provided unfailing support, freedom, and encouragement, consistent since the beginning, when I hadn’t the foggiest idea of what I was talking about. Advice (both solicited and unsolicited) was generously provided by David Lillie, Andrew Bennett, Ross Paterson, David Sharp, and many of the other members of the Functional Programming Section at Imperial. Tony Field and Peter Harrison elicit my gratitude for their forthright (constructive) criticisms which improved the presentation of the thesis considerably. I never would have considered trying for a Ph.D. if it weren’t for the moral-support provided by my world-wide circle of friends, specifically: Hilary Wilkinson, Chris Marriott, Alan Spidle, Steve Fernandez, and most of all, Kenn Frankel, whose confidence in my abilities vastly exceeded my own. My brother Dirk and my sister Erika have supported me since the beginning. For putting up with my anxieties and worries over these past few years, and for never losing faith in me, Nadege Ferrero gets my deepest thanks. I dedicate this thesis to my parents, who never let me think that I wasn’t up to the task and who selflessly invested a great deal of time, love, and encouragement to make this thesis happen. Over a period of 25 years we moved across 3 continents, 6 schools, and 4 languages but throughout all these changes, their commitment and devotion remained undiminished. This thesis is for them. T a b l e o f C o n t e n t s 1. Introduction.............................................................................................................................. 13 1.1. Motivation...................................................................................................................... 14 1.1.1. Exploiting Parallelism in the Presence of Dependencies................................14 1.1.2. von-Neumann Constraints on Imperative Languages.....................................15 1.1.3. Data-Parallel Programming................................................................................16 1.2. Rationale..........................................................................................................................18 1.2.1. The Case for Functional Languages.................................................................. 18 1.2.2. Process-Parallelism............................................................................................. 19 1.2.3. Data-Parallelism & Monolithic Operations..................................................... 20 1.2.3.1. Algebraic Data-Types (ADTs)................................................................ 21 1.2.3.2. Arrays.......................................................................................................... 23 1.2.4. Parallel Architectures...........................................................................................24 1.3. Difficulties with the Data-Parallel Model.................................................................. 26 1.3.1 Limitations of SIMD...........................................................................................26 1.3.2. Conceptual Difficulties with Data-Parallelism.................................................27 1.3.3. Shortcomings of Adapted von-Neumann Languages.....................................28 1.4. Philosophy of Approach...............................................................................................29 1.4.1. Design Principles.................................................................................................29 1.4.2. Bias Towards SIMD...........................................................................................30 1.4.3. A Hybrid Approach to Transformation............................................................30 1.4.4. Objective...............................................................................................................31 1.5. Statement of Originality............................................................................................... 31 1.6. Thesis Overview............................................................................................................32 2. A Functional Language with Data-Parallel Operations...................................................... 35 2.1. Introduction....................................................................................................................36 2.2. A Lazy, Higher-Order, Functional Language............................................................36 2.3. The Array Data-Type....................................................................................................37 2.3.1. Multi-Dimensional Arrays.................................................................................38 2.3.2. Nested Arrays.......................................................................................................38 2.4. Primitive Data-Parallel Operators................................................................................39 2.5. Derived Operators.........................................................................................................43 2.5.1. map, map2............................................................................................................ 43 2.5.2. update....................................................................................................................45 2.5.3. permute................................................................................................................. 45 2.5.4. shift........................................................................................................................ 45 2.5.5. scan........................................................................................................................ 45 2.5.6. reduce....................................................................................................................46 2.5.7. scatter.................................................................................................................... 46 2.5.8. gather.....................................................................................................................47 2.6. Alternative Aggregates................................................................................................. 47 2.6.1. Classification of Aggregate