The Functional Design of Scientific Computing Library
Total Page：16
File Type：pdf, Size：1020Kb
The Functional Design of Scientific Computing Library Liang Wang University of Cambridge, UK ABSTRACT in Python (such as Caffe [10] and TensorFlow [5]), they Owl is an emerging library developed in the OCaml lan- often provide Python bindings to take advantage of the guage for scientific and engineering computing. It focuses existing numerical infrastructure in Numpy and Scipy. on providing a comprehensive set of high-level numerical On the other hand, the supporting of basic scien- functions so that developers can quickly build up any data tific computing in OCaml is rather fragmented. There analytical applications. After over one-year intensive devel- have been some initial efforts (e.g., Lacaml [2], Oml [4], opment and continuous optimisation, Owl has evolved into a Pareto, and etc.), but their APIs are either too low-level powerful software system with competitive performance to to offer satisfying productivity, or the designs overly mainstream numerical libraries. Meanwhile, Owl’s overall focus on a specific problem domain. Moreover, incon- architecture remains simple and elegant, its small code base sistent data representation and careless use of abstract can be easily managed by a small group of developers. types make it difficult to pass data across different li- In this paper, we first present Owl’s design, core compo- braries. Consequently, developers often have to write nents, and its key functionality. We show that Owl benefits a significant amount of boilerplate code just to finish greatly from OCaml’s module system which not only allows rather trivial numerical tasks. As we can see, there is us to write concise generic code with superior performance, a severe lack of a general purpose numerical library in but also leads to a very unique design to enable parallel and OCaml ecosystem. We believe OCaml per se is a good distributed computing. OCaml’s static type checking signif- candidate for developing such a general purpose numer- icantly reduces potential bugs and accelerates the develop- ical library for two important reasons: 1) we can write ment cycle. We also share the knowledge and lessons learnt functional code as concise as that in Python with type- from building up a full-fledge system for scientific comput- safety; 2) OCaml code often has much superior perfor- ing with the functional programming community. mance comparing to dynamic languages such as Python and Julia. However, designing and developing a full-fledged nu- 1. INTRODUCTION merical library is a non-trivial task, despite that OCaml Thanks to the recent advances in machine learning has been widely used in system programming such as and deep neural networks, there is a huge demand on MirageOS [15]. The key difference between the two various numerical tools and libraries in order to facili- is obvious and interesting: system libraries provide a tate both academic researchers and industrial develop- lean set of APIs to abstract complex and heterogeneous ers to fast prototype and test their new ideas, then de- physical hardware, whilst numerical library offer a fat arXiv:1707.09616v3 [cs.MS] 28 Aug 2018 velop and deploy analytical applications at large scale. set of functions over a small set of abstract number Take deep neural network as an example, Google invests types. heavily in TensorFlow [5] while Facebook promotes their When Owl project started in 2016, we were imme- PyTorch [16]. Beyond these libraries focusing on one diately confronted by a series of fundamental questions specific numerical task, the interest on general purpose like: "what should be the basic data types", "what should tools like Python and Julia [1] also grows fast. be the core data structures", "what modules should be Python has been one popular choice among devel- designed", and etc. In the following development and opers for fast prototyping analytical applications, one performance optimisation, we also tackled many research important reason is because Scipy [12] and Numpy [3] and engineering challenges on a wide range of different two libraries, tightly integrated with other advanced topics such as software engineering, language design, functionality such as plotting, offer a powerful environ- system and network programming, and etc. ment which lets developers write very concise code to In this paper, we will introduce Owl1, a new general finish complicated numerical tasks. As a result, even for the frameworks which were not originally developed 1https://github.com/ryanrhymes/owl 1 purpose numerical library for scientific computing de- • Base is the basis of all other libraries in Owl. veloped in OCaml.2 We show that Owl benefits greatly Base defines core data structures, exceptions, and from OCaml's module system which not only allows us part of numerical functions. Because it contains to write concise generic code with superior performance, pure OCaml code so the applications built atop but also leads to a very unique design to enable par- of Base can be safely compiled into native code, allel and distributed computing. OCaml's static type bytecode, Javascript, even into unikernels. Fortu- checking significantly reduces potential bugs and accel- nately, majority of Owl's advanced functions are erate the development cycle. We would like to share implemented in pure OCaml. the knowledge and lessons learnt from building up a full-fledge system for scientific computing with the func- • Owl is the backbone of the numerical subsystem. tional programming community. It depends on Base but replaces some pure OCaml functions with C implementations (e.g. vectorised 2. ARCHITECTURE math functions inNdarray module). Mixing C code into the library limits the choice of backends (e.g. Owl is a complex library consisting of numerous func- browsers and MirageOS) but gives us significant tions (over 6500 by the end of 2017), we have strived for performance improvement when running applica- a modular design to make sure that the system is flex- tions on CPU. ible enough to be extended in future. In the following, we will present its architecture briefly. • Zoo [20] is designed for packaging and sharing 2.1 Hierarchy code snippets among users. This module targets small scripts and light numerical functions which may not be suitable for publishing on the heavier Owl Actor OPAM system. The code is distributed via gists on Composable Services Github, and Zoo is able to resolve the dependency GPGPU Dataset & NLP Neural Network Model Zoo and automatically pull in and cache the code. Map-Reduce • Top is the Toplevel system of Owl. It automati- Classic Analytics Engine cally loads both Owl and Zoo, and installs multiple Algorithmic Linear Algebra Plot Differentiation Parameter pretty printers for various data types. Server Engine Maths & Stats Regression Optimisation The subsystem on the right is called Actor Sub- Peer-to-Peer Engine system [19] which extends Owl's capability to parallel Core and distributed computing. The addition of Actor sub- Ndarray & CBLAS, LAPACKE Parallel & Synchronous system makes Owl fundamentally different from main- Matrix Interface Distributed Engine Parallel Machine stream numerical libraries such as Scipy and Julia. The core idea is to transform a user application from se- Figure 1: Two subsystems: Owl on the left handles quential execution mode into parallel mode (using var- numerical calculations while Actor on the right takes ious computation engines) with minimal efforts. The care of parallel and distributed computing. method we used is to compose two subsystems together with functors to generate the parallel version of the Figure 1 gives a bird view of Owl's system architec- module defined in the numerical subsystem. We will ture, i.e. the two subsystems and their modules. The elaborate this idea and its design in detail in Section subsystem on the left part is Owl's Numerical Sub- 5.1. system. The modules contained in this subsystem fall into three categories: (1) core modules contains basic 3. CORE DATA STRUCTURE data structures and foreign function interfaces to other N-dimensional array and matrix are the building blocks libraries (e.g., CBLAS and LAPACKE); (2) classic ana- of Owl library, their functionality are implemented in lytics contains basic mathematical and statistical func- Ndarray and Matrix modules respectively. Matrix is tions, linear algebra, regression, optimisation, plotting, a special case of n-dimensional array, and in fact many and etc.; (3) composable service includes more advanced functions in Matrix module call the corresponding func- numerical techniques such as deep neural network, nat- tions in Ndarray directly. ural language processing, data processing and service For n-dimensional array and matrix, Owl supports deployment tools. both dense and sparse data structures. The dense data The numerical subsystem is further organised in a structure is built atop of OCaml's native Bigarray mod- stack of smaller libraries, as follows. ule hence it can be easily interfaced with other libraries 2Owl can be installed with OPAM "opam install owl". like BLAS and LAPACK. Owl also supports both single 2 and double precisions for both real and complex num- Algorithm 1 Get and Set a slice in an ndarray ber. Therefore, Owl essentially has covered all the nec- 1: let x = sequential [j10; 10; 10j];; essary number types in most common scientific com- 2: let a = x.$f [[0;4]; [6;-1]; [-1;0]] g;; putations. The functions in Ndarray modules can be 3: x.$f [[0;4]; [6;-1]; [-1;0]] g <- zeros (shape a);; divided into following three groups. • The first group contains the vectorised mathemat- to the ndarrays with dimensionality greater than four. ical functions such as sin, cos, relu, and etc. Owl exploits the extended indexing operators recently • The second group contains the high-level function- introduced in OCaml 4.06 to provide a unified indexing ality to manipulate arrays and matrices, e.g., in- and slicing interface. dex, slice, tile, repeat, pad, and etc.