Vate: Runtime Adaptable Probabilistic Programming for Java Daniel Goodman Adam Pocock Oracle Labs Oracle Labs [email protected] Jason Peck Guy Steele Oracle Labs Oracle Labs Abstract encode domain specific knowledge into them, or explain how Inspired by earlier work on Augur, Vate is a probabilistic they reach their results. For these reasons, domain experts programming language for the construction of JVM based may choose to construct explicit probabilistic models. Typi- probabilistic models with an Object-Oriented interface. As cally these models can be easily described, but implementing a compiled language it is able to examine the dependency them requires the user to construct custom inference code. graph of the model to produce optimised code that can be This is labour intensive and potentially error prone. The con- dynamically targeted to different platforms. Using Gibbs siderable effort acts as a deterrent to modifying the models Sampling, Metropolis–Hastings and variable marginalisation for either general development or domain specific target- it can handle a range of model types and is able to efficiently ing. Probabilistic programming addresses this problem by infer values, estimate probabilities, and execute models. providing a high-level means of describing the model, typ- ically through a language [2, 5, 11, 15, 18] or through an Keywords: Probabilistic Programming, Java, Augur, MCMC, API embedded in another language [1, 13, 17, 19]. Models Parallel Programming described in probabilistic programs have the operations by ACM Reference Format: which the values of the model are inferred provided by the Daniel Goodman, Adam Pocock, Jason Peck, and Guy Steele. 2021. compiler or API, saving the user from this error prone work, Vate: Runtime Adaptable Probabilistic Programming for Java. In and reducing the burden of adapting models. The 1st Workshop on Machine Learning and Systems (EuroMLSys’21), Vate compiles models to JVM classes that encapsulate the April 26, 2021, Online, United Kingdom. ACM, New York, NY, USA, complexity of the different model operations, providing the 8 pages. https://doi.org/10.1145/3437984.3458835 user with a clean Object-Oriented interface. The language is single assignment, but designed to be familiar to Java 1 Introduction programmers, making models more accessible to the devel- In this paper we introduce the design and implementation of oper community. Models constructed in Vate perform three Vate, a probabilistic programming language [4] for creating classes of operation: Value inference, where the value of any models for JVM-based applications. Inspired by earlier work variables that are not fixed can be inferred; Probability infer- on Augur [18], models are constructed to allow dynamic ence, where the probabilities of individual parameters and switching of execution environment. Currently this is re- the whole model can be estimated; Conventional execution, stricted to single threaded and Fork-Join [10] JVM execution which can be used to generate outputs from a trained model environments, but we plan to extend to GPUs, clusters, and in cases such as linear regression or to produce synthetic runtime environments such as Callisto [8]. data sets. The values of variables within the model can be While deep learning models have advanced many tasks, fixed at compile time or at run time. they typically require large amounts of data, and it is hard to Value inference is performed primarily through Gibbs sam- pling [3] using conjugate priors when possible and reverting Permission to make digital or hard copies of all or part of this work for to marginalising out sampled variables for finite discrete dis- personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that tributions, and localised Metropolis–Hastings [9] inference copies bear this notice and the full citation on the first page. Copyrights for unbounded or continuous distributions. The results of for components of this work owned by others than the author(s) must this inference can be: all the sampled values modulo any re- be honored. Abstracting with credit is permitted. To copy otherwise, or quested burning in and thinning; the value when the model republish, to post on servers or to redistribute to lists, requires prior specific was in its most probable state (Maximum a Posteriori, MAP); permission and/or a fee. Request permissions from [email protected]. EuroMLSys’21, April 26, 2021, Online, United Kingdom or no values saved. These policies are set for the whole model © 2021 Copyright held by the owner/author(s). Publication rights licensed and can be modified on a per variable basis. to ACM. ACM ISBN 978-1-4503-8298-4/21/04...$15.00 https://doi.org/10.1145/3437984.3458835 EuroMLSys’21, April 26, 2021, Online, United Kingdom Daniel Goodman, Adam Pocock, Jason Peck, and Guy Steele 1.1 Probabilistic Models code in Figure 3 takes a sequence of observed coin flips At a high level probabilistic models consist of parameterised and determine the most likely coin biases and transition random variables, constant parameters supplied by the user, probabilities. and relationships between the variables, parameters and data. 2.1 Model Code Random variables encapsulate simpler probability distribu- tions which can be easily sampled from producing values. Models are declared via the keyword model followed by the The values of the random variables and the values sampled model name, any arguments, and a block containing the from them can be inferred in a variety of ways. body. The arguments and the body are valid java syntax, For example a random variable representing a Uniform with the exception of the syntactic sugar for initialising ar- distribution parameterised with the values 0 and 1 will when rays of constants and setting loop bounds. In Java the body sampled provide values in the range »0 ... 1¼ with uniform describes a sequence of operations to perform, in a proba- probability, and will not provide any values outside of this bilistic model it describes the relationship between variables range. This sampled value can then be used as an input so the value on the left of an assignment can effect values on to another random variable. If we use it to parameterise a the right. Variables named in the model will become fields Bernoulli distribution, it could represent the bias of a coin. in the compiled model allowing properties of these variables Repeated sampling from the Bernoulli distribution provides to be queried or set. values that represent repeated flipping of the coin. If we then Variables within the model consist of base types such as provide a set of observed results of flipping a real coin we int and double, arrays, and random variables representing can use value inference to determine the most like bias of the instances of the distributions supported by Vate. Random coin and the variance of this value. For this simple model a variables are constructed either via a call to a constructor, for closed form solution exists rendering the probabilistic model example new Beta(1.0, 1.0), or via statically imported unnecessary, but for more complex models such as the one factory methods as in the example. The method observe used as an example in Section 2 this is not the case. is used to tie the values generated by the model to data provided by the user for inference. One or more values can be drawn from a random variable by the sample method, if 1.2 Contributions sampled values are not observed, they can be changed during The contributions of this paper are as follows: the inference steps. To simplify the compilation and understandability of the • A language for writing probabilistic programming model, all values are single assignment. This means that to models that is familiar to Java developers. sum the values in an array some additional functionality • A compiler and runtime system for compiling models is required. To achieve this the method reduce is included. to Java class files, allowing more efficient execution This takes as its arguments an array of values of type X, a that models built out of runtime API calls. unit value of type X, and an associative function taking two • An Object-Oriented design for compiled models that values of type X and returning one value of type X. These enforces the separation of the model implementation are used to build a binary tree with a lambda at each node and the system that uses the model. and either an array element or a unit value at each leaf. This • An Object-Oriented design allowing the execution en- will be executed to provide the return value. vironment to be dynamically changed so models writ- Models can be split into reusable methods, which are not ten once can take best advantage of the system they required to exist in the same files, but can be formed into are used on. libraries, allowing families of models with a similar structure The rest of this paper first describes the language and to be created without having to recreate the entire model. resulting models from a user’s perspective before examin- The syntax of methods is a subset of that used in Java. An ing some of the implementation details. It briefly looks at example can be seen in Figure 2. Because the model will performance before considering how Vate differs from other ultimately be represented by a Directed Acyclic Graph (DAG) probabilistic programming frameworks. before the inference code is constructed, recursive models are not currently supported. 2 Model Presentation In this section we describe the construction and use of a 2.2 Compiled Model model. As an example we use a simple Hidden Markov Model The compilation of the model generates a collection of classes (HMM).