Getting to the Point. Index Sets and Parallelism-Preserving Autodiff for Pointful Array Programming
Getting to the Point. Index Sets and Parallelism-Preserving Autodiff for Pointful Array Programming ADAM PASZKE, Google Research, Poland DANIEL JOHNSON, Google Research, Canada DAVID DUVENAUD, University of Toronto, Canada DIMITRIOS VYTINIOTIS, DeepMind, United Kingdom ALEXEY RADUL, Google Research, USA MATTHEW JOHNSON, Google Research, USA JONATHAN RAGAN-KELLEY, Massachusetts Institute of Technology, USA DOUGAL MACLAURIN, Google Research, USA We present a novel programming language design that attempts to combine the clarity and safety of high-level functional languages with the efficiency and parallelism of low-level numerical languages. We treat arraysas eagerly-memoized functions on typed index sets, allowing abstract function manipulations, such as currying, to work on arrays. In contrast to composing primitive bulk-array operations, we argue for an explicit nested indexing style that mirrors application of functions to arguments. We also introduce a fine-grained typed effects system which affords concise and automatically-parallelized in-place updates. Specifically, an associative accumulation effect allows reverse-mode automatic differentiation of in-place updates in a way that preserves parallelism. Empirically, we benchmark against the Futhark array programming language, and demonstrate that aggressive inlining and type-driven compilation allows array programs to be written in an expressive, “pointful” style with little performance penalty. CCS Concepts: • Software and its engineering ! Functional languages; Parallel programming lan- guages; • Mathematics of computing ! Automatic differentiation. Additional Key Words and Phrases: array programming, automatic differentiation, parallel acceleration 1 INTRODUCTION Recent years have seen a dramatic rise in the popularity of the array programming model. The model was introduced in APL [Iverson 1962], widely popularized by MATLAB, and eventually made its way into Python.
[Show full text]