Google Vizier: A Service for Black-Box Optimization Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, D. Sculley fdgg, bsolnik, smoitra, gpk, karro, [email protected] Google Research Pittsburgh, PA, USA ABSTRACT In this paper we discuss a state-of-the-art system for black{ Any sufficiently complex system acts as a black box when box optimization developed within Google, called Google it becomes easier to experiment with than to understand. Vizier, named after a high official who offers advice to rulers. Hence, black-box optimization has become increasingly im- It is a service for black-box optimization that supports several portant as systems have become more complex. In this paper advanced algorithms. The system has a convenient Remote we describe Google Vizier, a Google-internal service for per- Procedure Call (RPC) interface, along with a dashboard and forming black-box optimization that has become the de facto analysis tools. Google Vizier is a research project, parts of parameter tuning engine at Google. Google Vizier is used which supply core capabilities to our Cloud Machine Learning 1 to optimize many of our machine learning models and other HyperTune subsystem. We discuss the architecture of the systems, and also provides core capabilities to Google's Cloud system, design choices, and some of the algorithms used. Machine Learning HyperTune subsystem. We discuss our re- quirements, infrastructure design, underlying algorithms, and 1.1 Related Work advanced features such as transfer learning and automated Black{box optimization makes minimal assumptions about early stopping that the service provides. the problem under consideration, and thus is broadly appli- cable across many domains and has been studied in multiple KEYWORDS scholarly fields under names including Bayesian Optimiza- Black-Box Optimization, Bayesian Optimization, Gaussian tion [2, 25, 26], Derivative{free optimization [7, 24], Sequen- Processes, Hyperparameters, Transfer Learning, Automated tial Experimental Design [5], and assorted variants of the Stopping multiarmed bandit problem [13, 20, 29]. Several classes of algorithms have been proposed for the 1 INTRODUCTION problem. The simplest of these are non-adaptive procedures such as Random Search, which selects xt uniformly at ran- Black{box optimization is the task of optimizing an objective dom from X at each time step t independent of the previous function f : X ! with a limited budget for evaluations. R points selected, fx휏 : 1 ≤ 휏 < tg, and Grid Search, which The adjective \black{box" means that while we can eval- selects along a grid (i.e., the Cartesian product of finite sets uate f(x) for any x 2 X, we have no access to any other of feasible values for each parameter). Classic algorithms information about f, such as gradients or the Hessian. When such as SimulatedAnnealing and assorted genetic algo- function evaluations are expensive, it makes sense to carefully rithms have also been investigated, e.g., Covariance Matrix and adaptively select values to evaluate; the overall goal is Adaptation [16]. for the system to generate a sequence of xt that approaches Another class of algorithms performs a local search by the global optimum as rapidly as possible. selecting points that maintain a search pattern, such as a sim- Black box optimization algorithms can be used to find the plex in the case of the classic Nelder{Mead algorithm [22]. best operating parameters for any system whose performance More modern variants of these algorithms maintain simple can be measured as a function of adjustable parameters. It models of the objective f within a subset of the feasible has many important applications, such as automated tuning regions (called the trust region), and select a point xt to of the hyperparameters of machine learning systems (e.g., improve the model within the trust region [7]. learning rates, or the number of hidden layers in a deep neural More recently, some researchers have combined powerful network), optimization of the user interfaces of web services techniques for modeling the objective f over the entire feasible (e.g. optimizing colors and fonts to maximize reading speed), region, using ideas developed for multiarmed bandit problems and optimization of physical systems (e.g., optimizing airfoils for managing explore / exploit trade-offs. These approaches in simulation). are fundamentally Bayesian in nature, hence this literature Permission to make digital or hard copies of part or all of this work goes under the name Bayesian Optimization. Typically, the for personal or classroom use is granted without fee provided that model for f is a Gaussian process (as in [26, 29]), a deep copies are not made or distributed for profit or commercial advantage neural network (as in [27, 31]), or a regression forest (as and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. in [2, 19]). For all other uses, contact the owner/author(s). Many of these algorithms have open-source implemen- KDD '17, August 13-17, 2017, Halifax, NS, Canada tations available. Within the machine learning community, © 2017 Copyright held by the owner/author(s). ACM ISBN 978-1-4503-4887-4/17/08. https://doi.org/10.1145/3097983.3098043 1https://cloud.google.com/ml/ examples include, e.g., HyperOpt2, MOE3, Spearmint4, and scale as O(n3) with the number of training points. Thus, once AutoWeka5, among many others. In contrast to such software we've collected a large number of completed Trials, we may packages, which require practitioners to set them up and run want to switch to using a more scalable algorithm. them locally, we opted to develop a managed service for At the same time, we want to allow ourselves (and advanced black{box optimization, which is more convenient for users users) the freedom to experiment with new algorithms or but involves additional design considerations. special-case modifications of the supported algorithms ina manner that is safe, easy, and fast. Hence, we've built Google 1.2 Definitions Vizier as a modular system consisting of four cooperating Throughout the paper, we use to the following terms to processes (see Figure 1) that update the state of Studies in the describe the semantics of the system: central database. The processes themselves are modular with A Trial is a list of parameter values, x, that will lead to a several clean abstraction layers that allow us to experiment single evaluation of f(x). A trial can be \Completed", which with and apply different algorithms easily. means that it has been evaluated and the objective value Finally we want to allow multiple trials to be evaluated f(x) has been assigned to it, otherwise it is \Pending". in parallel, and allow for the possibility that evaluating the A Study represents a single optimization run over a feasible objective function for each trial could itself be a distributed space. Each Study contains a configuration describing the process. To this end we define Workers, responsible for evalu- feasible space, as well as a set of Trials. It is assumed that ating suggestions, and identify each worked by a persistent f(x) does not change in the course of a Study. name (a worker handle) that persists across process preemp- A Worker refers to a process responsible for evaluating a tions or crashes. Pending Trial and calculating its objective value. 2.2 Basic User Workflow 2 SYSTEM OVERVIEW To use Vizier, a developer may use one of our client libraries This section explores the design considerations involved in (currently implemented in C++, Python, Golang), which will implementing black-box optimization as a service. generate service requests encoded as protocol buffers [15]. The basic workflow is extremely simple. Users specify a study 2.1 Design Goals and Constraints configuration which includes: Vizier's design satisfies the following desiderata: ∙ Identifying characteristics of the study (e.g. name, ∙ Ease of use. Minimal user configuration and setup. owner, permissions). ∙ Hosts state-of-the-art black-box optimization algorithms. ∙ The set of parameters along with feasible sets for each ∙ High availability (c.f., Section 2.3.1 for details); Vizier does constrained ∙ Scalable to millions of trials per study, thousands of optimization over the feasible set. parallel trial evaluations per study, and billions of stud- Given this configuration, basic use of the service (with each ies. trial being evaluated by a single process) can be implemented ∙ Easy to experiment with new algorithms. as follows: ∙ Easy to change out algorithms deployed in production. # Register this client with the Study, creating it if For ease of use, we implemented Vizier as a managed ser- # necessary. vice that stores the state of each optimization. This approach client.LoadStudy(study config, worker handle) drastically reduces the effort a new user needs to get upand while (not client.StudyIsDone()): running; and a managed service with a well-documented and # Obtain a trial to evaluate. stable RPC API allows us to upgrade the service without user trial = client.GetSuggestion() effort. We provide a default configuration for our managed # Evaluate the objective function at the trial parameters. service that is good enough to ensure that most users need metrics = RunTrial(trial) never concern themselves with the underlying optimization # Report back the results. algorithms. client.CompleteTrial(trial, metrics) The default option allows the service to dynamically select a recommended black{box algorithm along with low{level Here RunTrial is the problem{specific evaluation of the settings based on the study configuration. We choose to objective function f. Multiple named metrics may be reported make our algorithms stateless, so that we can seamlessly back to Vizier, however one must be distinguished as the switch algorithms during a study, dynamically choosing the objective value f(x) for trial x.