EUROGRAPHICS 2015 / O. Sorkine-Hornung and M. Wimmer Volume 34 (2015), Number 2 (Guest Editors) Generating Design Suggestions under Tight Constraints with Gradient-based Probabilistic Programming Daniel Ritchie Sharon Lin Noah D. Goodman Pat Hanrahan Stanford University Figure 1: Physical realizations of stable structures generated by our system. To create these structures, we write programs that generate random structures (e.g. a random tower or a randomly-perturbed arch), constrain the output of the program to be near static equilibrium, and then sample from the constrained output space using Hamiltonian Monte Carlo. Abstract We present a system for generating suggestions from highly-constrained, continuous design spaces. We formulate suggestion as sampling from a probability distribution; constraints are represented as factors that concentrate probability mass around sub-manifolds of the design space. These sampling problems are intractable using typical random walk MCMC techniques, so we adopt Hamiltonian Monte Carlo (HMC), a gradient-based MCMC method. We implement HMC in a high-performance probabilistic programming language, and we evaluate its ability to efficiently generate suggestions for two different, highly-constrained example applications: vector art coloring and designing stable stacking structures. 1. Introduction suggestions: given a model of the design space, computers can synthesize examples that their users might never have Considering multiple possibilities is critical in design. Ex- thought of independently. posure to different examples facilitates creativity—for in- stance, prototyping multiple alternatives can lead to better- In computer graphics, probabilistic inference has become quality final designs [KDK14, DGK∗10]. Exploring the popular for computer-aided suggestion in domains as diverse whole space of creative options seems to help people avoid as color selection and furniture layout [LRFH13,YYW∗12]. fixation and overcome their unconscious biases [JS91]. In this framework, the user provides a model of the de- Computation can assist with this exploration by generating sign space by expressing her preferences as soft constraints, © 2015 The Author(s) Computer Graphics Forum © 2015 The Eurographics Association and John Wiley & Sons Ltd. Published by John Wiley & Sons Ltd. D. Ritchie et al. / Generating Design Suggestions under Tight Constraints or factors. These factors are combined into a probabil- generally-useful constraints, such as physical stability and ity distribution, and sampling from this distribution using symmetry. We compare the performance of HMC to clas- Markov Chain Monte Carlo (MCMC) provides an efficient sical random walk MCMC on these two examples, demon- suggestion-generation mechanism. Working with inference strating that HMC provides both qualitatively and quantita- can be made easier through probabilistic programming.A tively better design space exploration in the presence of tight probabilistic program defines a random process (e.g. con- constraints. structing a random scene); inference then amounts to rea- Our contributions are: soning about the space of possible executions of that pro- cess under constraints [GMR∗08]. Inference algorithms only 1. The introduction of Hamiltonian Monte Carlo to handle need access to the elementary random choices made by the tight constraints in probabilistic design suggestion. program, allowing them to be used in virtually any design 2. An efficient implementation of HMC in a general-purpose application domain. probabilistic programming language. Real design applications feature a range of constraints, 3. An evaluation of this implementation through two repre- from vague preferences that loosely shape the design space sentative example applications: vector art coloring and de- (“Make this object a reddish color”) to strict requirements signing stacking structures. that eliminate entire regions of the space as undesirable We view HMC as one new tool in a toolbox that needs (“This container must hold one liter of liquid, up to man- to grow in order to make probabilistic computational design ufacturing tolerance”). But the tighter these constraints, the efficient and easy to use. more ill-conditioned the underlying probability distribution becomes. As we will show, the random walk MCMC meth- ods typically used for design suggestion break down when 2. Background and Related Work faced with tight constraints, especially in high-dimensional 2.1. Design Space Exploration design spaces. Design space exploration in computer graphics can be traced To work around this problem, developers can implement back at least as far as the seminal work on Design Galleries complex, application-specific MCMC algorithms that ex- by Marks and colleagues [MAB∗97]. Exploration can be di- ploit knowledge of constraint structure [JM12, SW14]. This vided into two phases: generating suggestions and navigat- strategy does not scale, however, as it requires new al- ing between those suggestions. Our work focuses on gener- gorithms be developed for each new application. General- ating suggestions; other researchers have examined the nav- purpose solutions would be preferrable, especially for use igation problem [BYMW13, UIM12]. with probabilistic programming. Researchers have experimented with different algorith- In this paper, we take a step toward solving this problem mic frameworks for generating design suggestions, includ- by adopting a different sampling algorithm: Hamiltonian ing genetic algorithms [XZCOC12], nonlinear manifold ex- Monte Carlo (HMC). HMC is used in Bayesian statistics ploration [YYPM11], and probabilistic inference [JTRS12b, to train predictive models with many parameters [Nea10]. TLL∗11,MSL∗11]. Our system uses probabilistic inference, It excels when the posterior distribution of the parame- and the particular inference algorithm it relies on, Hamil- ters given training data causes some parameters to become tonian Monte Carlo, shares some mathematical similarities highly correlated—the same statistical problem as design with manifold exploration methods. variables being strongly coupled by tight constraints. Its per- formance comes from using the gradient of the probabil- Design domains can contain discrete variables, contin- ity distribution to take less-random walks through the state uous variables, or some combination of both. Several ex- space. This gradient can be computed automatically, making isting design suggestion methods operate on purely dis- HMC a general-purpose, application-agonistic tool. HMC crete design spaces, including shape generation by part operates on continuous design domains (i.e subsets of Rn). combination [KCKK12, JTRS12a] and tiled pattern synthe- This property makes it a tool well-suited to graphics appli- sis [YBY∗13]. In contrast, our work focuses on continu- cations, since they often feature many continuous quantities ous design spaces, which are often used to model quanti- (positions, directions, dimensions, colors, etc.) ties such as positions, directions, sizes, and colors. In the mixed discrete/continuous regime, an important subclass of To evaluate the usefulness of HMC for design sugges- design spaces are those where discrete choices dictate the tion tasks, we implemented the algorithm in Quicksand, structure of a continuous parameter set [YYW∗12,FRS∗12]. an open-source probabilistic programming language em- The techniques presented in this paper can be also applied to bedded in the Terra language for high-performance com- the continuous subsets of these domains, for a fixed setting puting [Rit14, DHA∗13]. We then use our implementation of the discrete choices. to generate suggestions for two different example appli- cations: vector art coloring and designing stacking struc- Probability distributions over design spaces are typi- tures. These applications employ several challenging and cally complex, and researchers have explored techniques © 2015 The Author(s) Computer Graphics Forum © 2015 The Eurographics Association and John Wiley & Sons Ltd. D. Ritchie et al. / Generating Design Suggestions under Tight Constraints to make sampling from them more tractable. Parallel tem- 1 pering, which assists samplers when probability mass is s = 0.5 concentrated around multiple modes, is one notable exam- 0 ∗ ∗ 0:1 ple [TLL 11,MSL 11,LRFH13]. In contrast, the techniques −0.5 we present help when probability mass is concentrated along −1 thin manifolds. The two methods can be used in concert if −1 −0.5 0 0.5 1 a design space exhibits both multi-modality and manifold 1 1 structure. 0.5 0.5 s = 0:005 0 0 2.2. HMC Applications −0.5 −0.5 −1 −1 HMC has been applied in other areas that require searching −1 −0.5 0 0.5 1 −1 −0.5 0 0.5 1 through complex design spaces. It has found use in trajectory Density MH HMC ∗ optimization for robot motion planning [ZRD 13]. It has Figure 2: Tight constraints in action on a simple 2D ex- also been applied in 3D printing for estimating and correct- ample. Top left: The probability density of Equation 1 with ing material shrinkage during the printing process [HZSD]. s = 0:1. Top middle: Samples drawn from this density using Both of these efforts are concerned with optimization prob- MH. Bottom left: The probability density of Equation 1 with lems: they attempt to find the best possible solution in a large s = 0:005. Bottom middle: Samples drawn from this density design space. In contrast, we seek to explore large sets of using MH. Bottom right: Samples drawn from this density possibilites in design spaces.