DEFIne: A Fluent Interface DSL for Deep Learning Applications Nina Dethlefs Ken Hawick The Digital Centre The Digital Centre School of Engineering and School of Engineering and Computer Science Computer Science University of Hull, UK University of Hull, UK [email protected] [email protected] ABSTRACT or medical imaging [23], natural language processing [42], Recent years have seen a surge of interest in deep learning computer vision [18], and artificial intelligence in general models that outperform other machine learning algorithms [5], in some tasks even exceeding human performance [53]. on benchmarks across many disciplines. Most existing deep Despite the great success of deep learning and an increas- learning libraries facilitate the development of neural nets ing number of libraries and frameworks that are available, by providing a mathematical framework that helps users im- a substantial amount of parameter tuning is required to de- plement their models more efficiently. This still represents velop an effective model for a new domain, see e.g. the ran- a substantial investment of time and effort, however, when dom search solution in [8]. There are few guidelines on how the intention is to compare a range of competing models many layers a deep learning model should use, or what acti- quickly for a specific task. We present DEFIne, a fluent vation, loss function and optimiser will lead to an adequate interface DSL for the specification, optimisation and evalu- representation of the input data, as these details depend cru- ation of deep learning models. The fluent interface is im- cially on the target data [6]. Developers will typically start plemented through method chaining. DEFIne is embedded from a basic set of rules of thumb and then experiment to in Python and is build on top of its most popular deep find the best setup. learning libraries, Keras and Theano. It extends these with In this paper, we present a fluent interface DSL that aims common operations for data pre-processing and represen- to facilitate the process of developing deep learning models tation as well as visualisation of datasets and results. We for new domains. A fluent interface [19] is one where syntac- test our framework on three benchmark tasks from different tic features of the hosting language are used to good effect to domains: heart disease diagnosis, hand-written digit recog- construct an DSL that is embedded in a host language and nition and weather forecast generation. Results in terms that captures the jargon, the commands and other notions of accuracy, runtime and lines of code show that our DSL of the requisite application domain [20, 31]. Our fluent inter- achieves equivalent accuracy and runtime to state-of-the-art face is implemented through method chaining as explained models, while requiring only about 10 lines of code per ap- in Section 4. The general idea is to abstract away from im- plication. plementational details and integrate standard operations in data representation or model definition into a DSL that is embedded in Python and built directly on top of the most CCS Concepts popular deep learning libraries, Keras [10] and Theano [58]. •Software engineering ! Reusable software; •Artificial We present experiments in three different domains that are Intelligence ! Learning; relevant to real-world applications: heart disease diagnosis, hand-written digit recognition and weather forecast genera- Keywords tion. Our results in terms of accuracy, runtime and lines of code suggest that our DSL is able to provide sufficient flex- Domain-specific languages, deep learning ibility to programmers to express the representational and mathematical peculiarities of individual domains, while at 1. INTRODUCTION the same time enhancing the readability and maintainabil- Deep learning has received a lot of interest in recent years ity of code. The research contributions of this paper are: in both academic and industrial contexts. It is increasingly used in applications such as stock market prediction [17] 1. A DSL framework that performs automatic data anal- ysis, pre-processing and choice of hyper-parameters for Permission to make digital or hard copies of all or part of this work for personal or deep learning applications; see Section 4. classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation 2. An evaluation in three different application domains to on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or demonstrate the flexibility of the DSL across datasets; republish, to post on servers or to redistribute to lists, requires prior specific permission see Section 5 for the datasets and Section 6 for results. and/or a fee. Request permissions from [email protected]. RWDSL ’17, February 04 2017, Austin, TX, USA 3. A reduction in code size by up to a factor of 5 at equiv- © 2017 Copyright held by the owner/author(s). Publication rights licensed to ACM. alent performance to recent state-of-the-art results; see ISBN 978-1-4503-4845-4/17/02. $15.00 Section 6. DOI: http://dx.doi.org/10.1145/3039895.3039898 2. BACKGROUND together into a back-end framework or library and allow Computer programming language designers have a major them to be invoked by the application user through appro- goal of helping programmers express ideas and algorithms priate compact programming language features in a span- concisely and clearly. Even the most elegant programming ning DSL. DSLs are particularly effective when a whole fam- languages however sometimes falter in this goal, when pro- ily of problems [4] can be identified. Once a subset of special grams become bigger and more complex. The standard com- cases have been solved, it is often feasible and efficient to in- puter science \divide and conquer" approach is to abstrac- ductively design a DSL framework [32] that then addresses tify ideas and component parts of a large program into a the whole family of problems rather than continuing to solve framework or software library that can be separately de- each individually. This is a very productive approach. veloped, tested and hidden away, and invoked only when Spinellis [54] described some well known usage patterns of needed. This can considerably lower the amount of source DSLs but although ideas such as language-oriented program- code and hence concepts that the programmer needs to hold ming [66] have been reported in the programming literature on their screen or in their mind all at once and is a key to since 1994, the development and deployment of DSLs is still managing large-scale complex software development [62]. a relatively new area with most activity reported only over Many different techniques and tools have been introduced the last 15 years [63, 19, 43, 39]. Many application domains in modern programming languages to help this abstraction including: simulations [25]; business applications [51]; im- and lowering of code complexity including: subroutines and age processing [55]; database systems [41]; materials physics functions; modules and packages; and classes and objects problems [26], or other complex systems [27] could be sup- where data structures and operational code are combined ported in this manner [63, 54]. In this present paper we together to form abstract data types. While these help, ap- develop a DSL for deep-learning applications. plication developers are still often easily overwhelmed by DSLs are used in generating programming languages and the size and sheer complexity of programs [36]. These dis- tools themselves [16, 59], but other areas of reported success- tractions take away effort from addressing the application ful DSL use to date include: communications and telephony problems - particularly when carrying out work in computa- [50, 15]; real-time- embedded systems [24] and field pro- tional experiments when each experiment must be carefully grammable gate array device deployment applications [13]; and repeatably programmed. distributed and computational grid applications [33]; and Different programming languages [52] have varied relative mathematical [9] and equation-based problem formulation strengths, advantages and specialities appropriate to various [40, 28, 57]. Parallel computing [38] is also a promising area sorts of applications. General-purpose programming lan- for use of DSLs, whereby multiple versions of a program guages often have a lot of legacy operational and idiomatic suited to different parallel architectures could be generated features that can obscure the essence of an application and by a single DSL specification. sometimes make it unnecessarily complex for developers. DSLs approaches are thus now being employed in many A relatively new technique is to create a domain-specific application areas [12]. At the time of writing, their use is programming language (DSL) [19, 63, 21, 14] that provides still not completely widespread although this appears to be the programmer with a very high-level vehicle to formulate accelerating as better development tools become available application ideas. The goal is for the language to focus con- and as more positive user and programmer experiences are cisely on only those concepts and aspects that are directly reported. relevant to the particular application problem domain and to hide away any \boiler plate" source code that is just an 3. DEEP LEARNING FRAMEWORKS artefact of the underpinning programming language. The Deep Learning can be approached in a number of ways. application domain is often a business problem using busi- In this section we present some background on principle ap- ness jargon and language or for the discussion below it could proaches, followed by a explanation of how existing libraries use the scientific terminology particular to complex deep can be used. learning systems. A DSL can be implemented as either a full-blown language 3.1 Overview of deep learning models using all the necessary compiler [2] and language builder Two aspects that are particular relevant to our present environment apparatus to aid the programmer [35].