End-to-End Learning for Structured Prediction Energy Networks
David Belanger 1 Bishan Yang 2 Andrew McCallum 1
Abstract minimization (LeCun et al., 2006):
Structured Prediction Energy Networks (SPENs) ˆy = arg miny Ex(y), (1) are a simple, yet expressive family of struc- where E ( ) depends on x and learned parameters. tured prediction models (Belanger & McCal- x · lum, 2016). An energy function over candidate This approach includes factor graphs (Kschischang et al., structured outputs is given by a deep network, 2001), e.g., conditional random fields (CRFs) (Laf- and predictions are formed by gradient-based ferty et al., 2001), and many recurrent neural networks optimization. This paper presents end-to-end (Sec. 2.1). Output constraints can be enforced using learning for SPENs, where the energy function constrained optimization. Compared to feed-forward ap- is discriminatively trained by back-propagating proaches, energy-based approaches often provide better op- through gradient-based prediction. In our ex- portunities to inject prior knowledge about likely outputs perience, the approach is substantially more ac- and often have more parsimonious models. On the other curate than the structured SVM method of Be- hand, energy-based prediction requires non-trivial search in langer & McCallum (2016), as it allows us to the exponentially-large space of outputs, and search tech- use more sophisticated non-convex energies. We niques often need to be designed on a case-by-case basis. provide a collection of techniques for improving Structured prediction energy networks (SPENs) (Belanger the speed, accuracy, and memory requirements & McCallum, 2016) help reduce these concerns. They of end-to-end SPENs, and demonstrate the power can capture high-arity interactions among components of of our method on 7-Scenes image denoising and y that would lead to intractable factor graphs and provide CoNLL-2005 semantic role labeling tasks. In a mechanism for automatic structure learning. This is ac- both, inexact minimization of non-convex SPEN complished by expressing the energy function in Eq. (1) energies is superior to baseline methods that use as a deep architecture and forming predictions by approxi- simplistic energy functions that can be mini- mately optimizing y using gradient descent. mized exactly. While providing the expressivity and generality of deep networks, SPENs also maintain the useful semantics of en- ergy functions: domain experts can design architectures to 1. Introduction capture known properties of the data, energy functions can In a variety of application domains, given an input x we be combined additively, and we can perform constrained y seek to predict a structured output y. For example, given optimization over . Most importantly, SPENs provide a noisy image, we predict a clean version of it, or given for black-box interaction with the energy, via forward and a sentence we predict its semantic structure. Often, it is back-propagation. This allows practitioners to explore a insufficient to employ a feed-forward predictor y = F (x), wide variety of models without the need to hand-design since this may have prohibitive sample complexity, fail to corresponding prediction methods. model global interactions among outputs, or fail to enforce Belanger & McCallum (2016) train SPENs using a struc- hard output constraints. Instead, it can be advantageous to tured SVM (SSVM) loss (Taskar et al., 2004; Tsochan- define the prediction function implicitly in terms of energy taridis et al., 2004) and achieve competitive performance on simple multi-label classification tasks. Unfortunately, 1University of Massachusetts, Amherst 2Carnegie Mel- lon University. Correspondence to: David Belanger
Monte Carlo sampling from the energy surface (Mnih & Therefore, these methods may be undesirable even for Hinton, 2005; Hinton et al., 2006; Ngiam et al., 2011). Re- problems where exact energy minimization is tractable. cently, Zhai et al. (2016) trained energy-based density mod- For non-convex E (y), gradient-based prediction will only els for anomaly detection by exploiting the connections be- x find a local optimum. Amos et al. (2017) present input- tween denosing autoencoders, energy-based models, and convex neural networks (ICNNs), which employ an easy- score matching (Vincent, 2011). to-implement method for constraining the parameters of a SPEN such that the energy is convex with respect to 2.3. Learning with Exact Energy Minimization y, but perhaps non-convex with respect to the parameters.
Let (yˆ, y⇤) be a non-negative task-specific cost function One simply uses convex, non-decreasing non-linearities for comparing yˆ and the ground truth y⇤. Belanger & and only non-negative parameters in any part of the com- McCallum (2016) employ a structured SVM (SSVM) loss putation graph downstream from y. Here, prediction will (Taskar et al., 2004; Tsochantaridis et al., 2004): return the global optimum, but convexity, especially when achieved this way, may impose a strong restriction on the max [ (y, yi) Ex (y)+Ex (yi)] , (3) expressivity of the energy. Their construction is a suffi- y i i + xi,yi cient condition for achieving convexity, but there are con- { X } vex energies that disobey this property. Our experiments where [ ] = max(0, ). Each step of minimizing Eq. (3) · + · present results for instances of ICNNs. In general, non- by subgradient descent requires loss-augmented inference: convex SPENS perform better.
min ( (y, yi)+Ex (y)) . (4) y i 3. Learning with Unrolled Optimization For differentiable (y, yi), a local optimum of Eq. (4) can obtained using first-order methods. The methods of Sec. 2.3 are unreliable with non-convex energies because we cannot simply use the output of in- Solving Eq. (4) probes the model for margin violations. exact energy minimization as a drop-in replacement for If none exist, the gradient of the loss with respect to the the exact minimizer. Instead, a collection of prior work parameters is zero. Therefore, SSVM performance does has performed end-to-end learning of gradient-based pre- not degrade gracefully with optimization errors in the in- dictors (Gregor & LeCun, 2010; Domke, 2012; Maclau- ner prediction problem, since inexact energy minimization rin et al., 2015; Andrychowicz et al., 2016; Wang et al., may fail to discover margin violations that exist. Perfor- 2016; Metz et al., 2017; Greff et al., 2017). Rather than mance can be recovered if Eq. (4) returns a lower bound, reasoning about the energy minimum as an abstract quan- eg. by solving an LP relaxation (Finley & Joachims, 2008). tity, the authors pose a specific gradient-based algorithm However, this is not possible in general. In Sec. 6.1.3 we for approximate energy minimization and optimize its em- compare the image denoising performance of SSVM learn- pirical performance using back-propagation. This is a form ing vs. this paper’s end-to-end method. Overall, we have of direct risk minimization (Tappen et al., 2007; Stoyanov found SSVM learning to be unstable and difficult to tune et al., 2011; Domke, 2013). for non-convex energies in applications more complex than the multi-label classification experiments of Belanger & Consider simple gradient descent: McCallum (2016). T d y = y ⌘ E (y ). (5) The implicit function theorem offers an alternative frame- T 0 t dy x t t=1 work for training energy-based predictors (Foo et al., 2008; X Samuel & Tappen, 2009). See Domke (2012) for an To learn the energy function end-to-end, we can back- overview. While a naive implementation requires inverting propagate through the unrolled optimization Eq. (5) for Hessians, one can solve the product of an inverse Hessian fixed T . With this, it can be rendered API-equivalent to and a vector using conjugate gradients, which can leverage a feed-forward network that takes x as input and returns the techniques discussed in Sec. 3 as a subroutine. To per- a prediction for y, and can thus be trained using standard form reliably, the method unfortunately requires exact en- methods. Furthermore, certain hyperparameters, such as ergy minimization and many conjugate gradient iterations. the learning rates ⌘t, are trainable (Domke, 2012). Overall, both of these learning algorithms only update the This backpropagation requires non-standard interaction energy function in the neighborhoods of the ground truth with a neural-network library because Eq. (5) computes and the predictions of the current model. On the other hand, gradients in the forward pass, and thus it must compute it may be advantageous to shape the entire energy surface second order terms in the backwards pass. We can save such that is exhibits certain properties, e.g., gradient de- space and computation by avoiding instantiating Hessian scent converges quickly when initialized well (Sec. 4.2). terms and instead directly computing Hessian-vector prod- End-to-End SPENs ucts. These can be achieved three ways. First, the method probability simplex on D elements. of Pearlmutter (1994) is exact, but requires non-trivial code While this rounding introduces poorly-understood sources modifications. Second, some libraries construct computa- of error, it has worked well for non-convex energy-based tion graphs for gradients that are themselves differentiable. prediction in multi-label classification (Belanger & Mc- Third, we can employ finite-differences (Domke, 2012). Callum, 2016), sequence tagging (Vilnis et al., 2015), and It is clear that Eq. (5) can be naturally extended to cer- translation (Hoang et al., 2017). tain alternative optimization methods, such as gradient de- w h w h Both [0, 1] and ⇥ are Cartesian products of prob- scent with momentum, or L-BFGS (Liu & Nocedal, 1989; ⇥ D ability simplices, and it is easy to adopt existing methods Domke, 2012). These require an additional state vector ht for projected gradient optimization over the simplex. that is evolved along with yt across iterations. Andrychow- icz et al. (2016) unroll gradient-descent, but employ a First, it is natural to apply Euclidean projected gradient de- w h learned non-linear RNN to perform per-coordinate updates scent. Over [0, 1] ⇥ , we have: to y. End-to-end learning is also applicable to special-case y = Clip [y ⌘ E (y )] , (6) energy minimization algorithms for graphical models, such t+1 0,1 t tr x t as mean-field inference and belief propagation (Domke, This is unusable for end-to-end learning, however, since 2013; Chen et al., 2015; Tompson et al., 2014; Li & Zemel, back-propagation through the projection will yield 0 gradi- 2014; Hershey et al., 2014; Zheng et al., 2015). ents whenever yt ⌘t Ex(yt) / [0, 1]. This is similarly r 2w h problematic for projection onto D⇥ (Duchi et al., 2008). 4. End-to-End Learning for SPENs Alternatively, we can apply entropic mirror descent, ie. We now present details for applying the methods of the pre- projected gradient with distance measured by KL diver- w h vious section to SPENs. We first describe considerations gence (Beck & Teboulle, 2003). For y D⇥ , we have: for learning SPENs defined for the convex relaxation of dis- 2 y = SoftMax (log(y ) ⌘ E (y )) (7) crete labeling problems. Then, we describe how to encour- t+1 t tr x t age our models to optimize quickly in practice. Finally, we present methods for improving the speed and memory This is suitable for end-to-end learning, but the updates are overhead of SPEN implementations. similar to an RNN with sigmoid non-linearities, which is vulnerable to vanishing gradients (Bengio et al., 1994). Our experiments unroll either Eq. (5) or an analogous version implementing gradient descent with momentum. Instead, we have found it useful to avoid constrained op- We compute Hessian-vector products using the finite- timization entirely, by optimizing un-normalized logits lt, difference method of (Domke, 2012), which allows black- with yt = SoftMax(lt): box interaction with the energy. l = l ⌘ E (SoftMax(l )) . (8) t+1 t tr x t We avoid the RNN-based approach of Andrychowicz et al. (2016) because it diminishes the semantics of the energy, as Here, the updates to lt are additive, and thus will be less the interaction between the optimizer and gradients of the susceptible to vanishing gradients (Hochreiter & Schmid- energy is complicated. In recent work, Gygli et al. (2017) huber, 1997; Srivastava et al., 2015; He et al., 2016). propose an alternative learning method that fits the energy Finally, Amos et al. (2017) present the bundle entropy function such that E ( ) ( , y⇤), where is defined x · ⇡ · method for convex optimization with simplex constraints, as in Sec. 2.3. This is an interesting direction for future along with a method for differentiating the output of the research, as it allows for non-differentiable . The advan- optimizer. End-to-end learning for Eq. (10) can be per- tage of end-to-end learning, however, is that it provides a formed using generic learning software, since the unrolled energy function that is precisely tuned for a particular test- optimization obeys the API of a feed-forward predictor, but time energy minimization procedure. unfortunately this is not true for their method. Future work should consider their method, however, as it performs very 4.1. End-to-End Learning for Discrete Problems rapid energy minimization. To apply SPENs to a discrete structured prediction prob- lem, we relax to a constrained continuous problem, apply 4.2. Learning to Optimize Quickly SPEN prediction, and then round to a discrete output. For We next enumerate methods for learning a model such example, for tagging each pixel of a h w image with a that gradient-based energy minimization converges to high- ⇥w h w h binary label, we would relax from 0, 1 ⇥ to [0, 1] ⇥ , { } quality y quickly. When using such methods, we have and if the pixels can take on one of D values, we would found it important to maintain the same optimization con- w h w h relax from y 0,...,D ⇥ to ⇥ , where is the 2{ } D D figuration, such as T , at both train and test time. End-to-End SPENs
First, we can encourage rapid optimization by defining our 5. Recommended SPEN Architectures for loss function as a sum of losses on every iterate yt, rather End-to-End Learning than only on the final one. Let `(yt, y⇤) be a differentiable loss between an iterate and the ground truth. We employ To train SPENs end-to-end, we write Eq. (5) as:
T d yT = Init(F (x)) ⌘t E(yt ; F (x)). (10) T dy 1 t=1 L = w `(y , y⇤), (9) X T t t t=1 Here, Init( ) is a differentiable procedure for predicting an X · initial iterate y0. Following Belanger & McCallum (2016), we also employ Ex(y)=E(y ; F (x)), where the de- where wt is a non-negative weight. This encourages the pendence of Ex(y) on x comes by way of a parametrized model to achieve high-quality predictions early. It has the feature function F (x). This is useful because test-time pre- additional benefit that it reduces vanishing gradients, since diction can avoid back-propagation in F (x). a learning signal is introduced at every timestep. Our ex- 1 We have found it useful in practice to employ an energy periments use wt = T t+1 . that splits into global and local terms: Second, for the simplex-constrained problems of Sec. 4.1, g l we smooth the energy with an entropy term i H(yi). E(y ; F (x)) = E (y ; F (x)) + Ei(yi ; F (x)). (11) This introduces extra strong convexity, which helps im- i P X prove convergence. It also strengthens the parallel between g SPEN prediction and marginal inference in a Markov ran- Here, i indexes the components of y and E (y ; F (x)) is dom field, where the inference objective is expected energy an arbitrary global energy function. The modeling benefits plus entropy (Koller & Friedman, 2009, p. 385). of the local terms are similar to the benefits of using local factors in popular factor graph models. We also can use the Third, we can set T to a small value. Of course, this guaran- local terms to provide an implementation of Init( ). tees that optimization converges quickly on the train data. · Here, we lose the contract that Eq. (10) is even perform- We pretrain F (x) by training the feed-forward predictor Init(F (x)). We also stabilize learning by first clamping the ing energy minimization, since it hasn’t converged, but this g may be acceptable if predictions are accurate. For example, local terms for a few epochs while updating E (y ; F (x)). some experiments achieve good performance with T =3. To back-propagate through Eq. (10), the energy function In future work, it may be fruitful to directly penalize con- must be at least twice differentiable with respect to y. d Therefore, we can’t use non-linearities with discontinuous vergence criteria, such as yt yt 1 and Ex(yt) . k k k dyt k gradients. Instead of ReLUs, we use a SoftPlus with a rea- sonably high temperature. Note that F (x) and Init( ) can 4.3. Efficient Implementation · be arbitrary networks that are sub-differentiable with re- Since we can explicitly encourage our model to converge spect to their parameters. quickly, it is important to exploit fast convergence at train time. Eq. (10) is unrolled for a fixed T . However, if op- 6. Experiments timization converges at T0
SPENs are useful because we can select architectures based Here, DNN(y) is a general deep convolutional network that on their suitability for the data, not whether they support takes an image and returns a number. The architecture in model-specific algorithms. our experiments consists of a 7 7 32 convolution, a ⇥ ⇥ SoftPlus, another 7 7 32 convolution, a SoftPlus, a ⇥ ⇥ 1 1 1 convolution, and finally spatial average pooling. 6.1. Image Denoising ⇥ ⇥ w h The method of Wang et al. (2016) cannot handle this prior. Let x [0, 1] ⇥ be an observed grayscale image. We 2 assume that it is a noisy realization of a latent clean image XPERIMENTAL ETUP w h 6.1.2. E S y [0, 1] ⇥ , which we estimate using MAP inference. 2 Consider a Gaussian noise model with variance 2 and a We evaluate on the 7-Scenes dataset (Newcombe et al., prior P(y). The associated energy function is: 2011), where we seek to denoise depth measurements from 2 2 a Kinect sensor. Our data processing and hyperparam- y x 2 log P(y). (12) k k2 eters are designed to replicate the setup of Wang et al. Here, the feature network is the identity. The first term is (2016), who demonstrate state-of-the art results for energy- the local energy network and the second, which does not minimization-based denoising on the dataset. We train us- depend on x, is the global energy network. ing random 96 128 crops from 200 images of the same ⇥ scene and report PSNR (higher is better) for 5500 images There are three general families for the prior. First, it can be from different scenes. We treat 2 as a trainable parameter hard-coded. Second, it can be learned by approximate den- and minimize the mean-squared-error of y. sity estimation. Third, given a collection of x, y pairs, { } we can perform supervised learning, where the prior’s pa- 6.1.3. RESULTS AND DISCUSSION rameters are discriminatively trained such that the output of a particular algorithm for minimizing Eq. (12) is high- Example outputs are given in Figure 1 and Table 1 com- quality. End-to-end learning has proven to be highly suc- pares PSNR. BM3D is a widely-used non-parametric cessful for the third approach (Tappen et al., 2007; Barbu, method (Dabov et al., 2007). FilterForest (FF) adaptively 2009; Schmidt et al., 2010; Sun & Tappen, 2011; Domke, selects denoising filters for each location (Fanello et al., 2012; Wang et al., 2016), and thus it is important to evalu- 2014). ProximalNet (PN) is the system of Wang et al. ate the methods of this paper on the task. (2016). FOE-20 is an attempt to replicate PN using end-to- end SPEN learning. We unroll 20 steps of gradient descent 6.1.1. IMAGE PRIORS with momentum 0.75 and use the modification in Eq. (14). Note it performs similarly to PN, which unrolls 5 iterations Much of the existing work on end-to-end training for de- of sophisticated optimization. Note that we can obtain 37.0 noising considers some form of a field-of-experts (FOE) PSNR using a feed-forward convnet with a similar archi- ` prior (Roth & Black, 2005). We consider an 1 version, tecture to our DeepPrior, but without spatial pooling. which assigns high probability to images with sparse acti- vations from K learned filters: The next set of results consider improved instances of the FOE model. First, FOE-20+ is identical to FOE-20, ex- P(y) exp (fk y) 1 . (13) cept that it employs the average loss Eq. (9), uses a mo- / k ⇤ k ! mentum constant of 0.25, and treats the learning rates ⌘ Xk t Wang et al. (2016) perform end-to-end learning for as trainable parameters. We find that this results in both Eq. (13), by unrolling proximal gradient methods that ana- better performance and faster convergence. Of course, we could achieve fast convergence by simply setting T to be lytically handle the non-differentiable `1 term. small. In response, we consider FOE-3. This only unrolls This paper assumes we only have black-box interaction for T =3iterations and obtains superior performance. with the energy. In response, we alter Eq. (13) such that it is twice differentiable, so that we can unroll generic first- The final three results are with the DNN prior Eq. (15). DP- order optimization methods. We approximate Eq. (13) by 20 unrolls 20 steps of gradient descent with a momentum leveraging a SoftPlus with temperature 25, replacing by: constant of 0.25. The gain in performance is substantial, |·| especially considering that a PSNR of 30 can be obtained SoftAbs(y)=0.5 SoftPlus(y)+0.5 SoftPlus( y). (14) with elementary signal processing. Similar to FOE-3 vs. FOE-20+, we experience a modest performance gain us- The principal advantage of learning algorithms that are not ing DP-3, which only unrolls for 3 gradient steps but is hand-crafted to the problem structure is that they provide otherwise identical. the opportunity to employ more expressive energies. In re- sponse, we also consider a deeper prior, given by: Finally, the FOE-SSVM and DP-SSVM configurations use SSVM training. We find that FOE-SSVM performs P(y) exp ( DNN(y)) . (15) / End-to-End SPENs
6.2. Semantic Role Labeling Semantic role labeling (SRL) predicts the semantic struc- ture of predicates and arguments in sentences (Gildea & Jurafsky, 2002). For example, in the sentence “I want to buy a car,” the verbs “want” and “buy” are two predicates, and “I” is an argument that refers to the wanter and buyer, Ground Truth Noisy Input “to buy a car” is the thing wanted, and “a car” is the thing bought. Given predicates, we seek to identify arguments and their semantic roles in relation to each predicate. For- mally, given a set of predicates p in a sentence x and a set of candidate argument spans a, we assign a discrete seman- tic role r to each pair of predicate and argument, where r can be either a pre-defined role label or an empty label. We FOE-20 FOE-20+ evaluate SRL instead of, for example, noun-phrase chunk- ing (Lacoste-Julien et al., 2012), since it is a more chal- lenging task, where the outputs are subject to substantially more complex non-local constraints. Existing work imposes hard constraints on r, such as ex- cluding overlapping arguments and repeated core roles dur- DeepPrior-20 DeepPrior-3 ing prediction. The objective is to minimize the energy: min E(r ; x, p, a) s.t. r (x, p, a), (16) Figure 1. Example Denoising Outputs r 2Q where (x, p, a) is set of feasible joint role assignments. Q BM3D FF PN FOE-20 FOE-SSVM This constrained optimization problem can be solved us- 35.46 35.63 36.31 36.41 37.7 ing integer linear programming (ILP) (Punyakanok et al., FOE-20+ FOE-3 DP-20 DP-3 DP-SSVM 2008) or its relaxations (Das et al., 2012). These meth- 37.34 37.62 40.3 40.4 38.7 ods rely on the output of local classifiers that are un- aware of structural constraints during training. More Table 1. Denoising Results (PSNR) recently, Tackstr¨ om¨ et al. (2015) account for the con- straint structure using dynamic programming at train time. FitzGerald et al. (2015) extend this using neural net- work features and show improved results. competitively with the other FOE configurations. This is not surprising, since the FOE prior is convex. However, fit- 6.2.1. DATA AND PREPROCESSING AND BASELINES ting the DeepPrior with an SSVM is inferior to using end- to-end learning. The performance is very sensitive to the We consider the CoNLL 2005 shared task data (Carreras energy minimization hyperparameters. &Marquez` , 2005), with standard data splits and offi- cial evaluation scripts. We apply similar preprocessing In these experiments, it is superior to only unroll for a few as Tackstr¨ om¨ et al. (2015). This includes part-of-speech iterations for end-to-end learning. One possible reason is tagging, dependency parsing, and using the parse to gener- that a shallow unrolled architecture is easier to train. Trun- ate candidate arguments. cated optimization with respect to y may also provide an interesting prior over outputs (Duvenaud et al., 2016). It is Our baseline is an arc-factored model for the conditional also observed in Wang et al. (2014) that better energy min- probability of the predicate-argument arc labels: imization for FOE models may not improve PSNR. Often P(r x, p, a)=⇧iP(ri x, p, a). (17) unrolling for 20 iterations results in over-smoothed outputs. | | where P(ri x, p, a) exp g(ri, x, p, a) . Here, each We are unable achieve reasonable performance with an | / conditional distribution is given by a multiclass logistic re- ICNN (Amos et al., 2017), which restricts all of the param- gression model. See Appendix A.2.1 for details of the ar- eters of the convolutions to be positive. Unfortunately, this chitecture and training procedure for our baseline. hinders the ability of the filters in the prior to act as edge detectors or encourage local smoothness. Both of these are When using the negative log of Eq. (18) as an energy in important for high-quality denoising. Note that the `1 FOE Eq. (16), there are variety of methods for finding a near- is convex, even without the restrictive ICNN constraint. optimal r (x, p, a). First, we can employ simple 2Q End-to-End SPENs heuristics for locally resolving constraint violation. The Dev Test Test Local + H system uses Eq. (18) and these. We can instead Model (WSJ) (WSJ) (Brown) 3 Local + H 78.0 79.7 69.7 use the AD message passing algorithm (Martins et al., 3 Local + AD 78.2 80.0 69.9 2011) to solve the LP relaxation of this constrained prob- SPEN + H 79.0 80.7 69.3 3 lem. We use Local + AD to refer to this system. Since the SPEN + AD3 79.0 80.7 69.4 3 LP solution may not be integral, we post-process the AD Tackstr¨ om¨ (Local) 77.9 79.3 70.2 output using the same heuristics as Local + H. Tackstr¨ om¨ (Structured) 78.6 79.9 71.3 FitzGerald (Local) 78.4 79.4 70.9 6.2.2. SPEN MODEL FitzGerald (Structured) 78.3 79.4 71.2 The SPEN performs continuous optimization over the re- Table 2. SRL Results (F1) laxed set y for each discrete label r , where A i 2 A i is the number of possible roles. The preprocessing gener- ates sparse predicate-argument candidates, but we optimize them during training. Note that Zhou & Xu (2015) obtain over the complete bipartite graph between predicates and n m slightly better performance with alternative RNN methods. arguments to support vectorization. We have y ⇥ , 2 A We were unable to outputerform the Local systems using a where n and m are the max number of predicates and argu- SPEN system trained with an SSVM loss. ments. Invalid arcs are constrained to the empty label. We select our SPEN configuration by maximizing perfor- We employ a pretrained version of Eq. (18) to provide the mance of SPEN + AD3 on the dev data. Our best system local energy term of a SPEN. This is augmented with global unrolls for 10 iterations, trains per-iteration learning rates, terms that couple the outputs together. See Appendix A.2.2 uses no momentum, and unrolls Eq. (8). Overall, SPEN for details of the architecture we use. It has terms, for ex- + AD3 performs the best of all systems on the WSJ test ample, that apply a deep network to the feature representa- data. We expect our diminished performance on the Brown tions of all of the arcs selected for a given predicate. test set is due to overfitting. The Brown set is not from the As with Tackstr¨ om¨ et al. (2015), we seek to account for same source as the train, dev, and test WSJ data. SPENs constraints (x, p, a) during both inference and learn- are more susceptible to overfitting because the expressive Q ing, rather than only imposing them via post-processing. global term introduces many parameters. Therefore, we include additional energy terms that encode Note that SPEN + AD3 and SPEN + H performs identi- membership in (x, p, a) as twice-differentiable soft con- 3 Q cally, whereas LOCAL + AD and LOCAL + H do not. straints that can be applied to y. All of the constraints in This is because our learned global energy encourages con- (x, p, a) express that certain arcs cannot co-occur. For Q straint satisfaction during gradient-based optimization of y. example, two arguments cannot attach to the same pred- Using the method of Amos et al. (2017) for restricting the icate if the arguments correspond to spans of tokens that energy to be convex wrt y, we obtain 80.3 on the test set. overlap. Consider general binary variables a and b with corresponding relaxations a,¯ ¯b [0, 1]. We convert the con- 2 straint (a b) into an energy function ↵SoftPlus(¯a+¯b 1), 7. Conclusion and Future Work ¬ ^ where ↵ is a learned parameter. SPENs are a flexible, expressive framework for structured We consider the SPEN + H and SPEN + AD3 configura- prediction, but training them can be challenging. This pa- tions, which employ heuristics or AD3 to enforce the out- per provides a new end-to-end training method that enables put constraints. Rather than applying these methods to the high performance on considerably more complex tasks than probabilities from Eq. (18), we use the soft prediction out- those of Belanger & McCallum (2016). We unroll an ap- put by energy minimization. proximate energy minimization algorithm into a differen- tiable computation graph that is trainable by gradient de- 6.2.3. RESULTS AND DISCUSSION scent. The approach is user-friendly in practice because it returns not just an energy function but also a test-time pre- Table 2 contains results on the CoNLL 2005 WSJ dev and diction procedure that has been tailored for it. test sets and the Brown test set. We compare the SPEN and Local systems with the best non-ensemble systems In the future, it may be useful to employ more sophisticated of Tackstr¨ om¨ et al. (2015) and FitzGerald et al. (2015), unrolled optimizers, perhaps where the optimizer’s hyper- which have similar overall setups as us for feature extrac- parameters are a learned function of x, and to perform it- tion and for the parametrization of the local energy terms. erative optimization in a learned feature space, rather than For these, ‘Local’ fits Eq. (18) without regard for the out- output space. Finally, we could model gradient-based pre- put constraints, whereas ‘Structured’ explicitly considers diction as a sequential decision making problem and train the energy using value-based reinforcement learning. End-to-End SPENs
Acknowledgments Das, Dipanjan, Martins, Andre´ FT, and Smith, Noah A. An exact dual decomposition algorithm for shallow seman- Many thanks to Justin Domke, Tim Vieiria, Luke Vilnis, tic parsing with constraints. In Conference on Lexical and Shenlong Wang for helpful discussions. The first and and Computational Semantics, 2012. third authors were supported in part by the Center for Intel- ligent Information Retrieval and in part by DARPA under Domke, Justin. Generic methods for optimization-based agreement number FA8750-13-2-0020. The second author modeling. In AISTATS, 2012. was supported in part by DARPA under contract number FA8750-13-2-0005. The U.S. Government is authorized Domke, Justin. Learning graphical model parameters with to reproduce and distribute reprints for Governmental pur- approximate marginal inference. Pattern Analysis and poses notwithstanding any copyright notation thereon. Any Machine Intelligence, 2013. opinions, findings and conclusions or recommendations ex- Duchi, John, Shalev-Shwartz, Shai, Singer, Yoram, and pressed in this material are those of the authors and do not Chandra, Tushar. Efficient projections onto the l 1-ball necessarily reflect those of the sponsor. for learning in high dimensions. In ICML, 2008.
References Duvenaud, David, Maclaurin, Dougal, and Adams, Ryan P. Early stopping as nonparametric variational inference. In Amos, Brandon, Xu, Lei, and Kolter, J Zico. Input-convex AISTATS, 2016. deep networks. ICML, 2017. Fanello, Sean Ryan, Keskin, Cem, Kohli, Pushmeet, Izadi, Andrychowicz, Marcin, Denil, Misha, Gomez, Sergio, Shahram, Shotton, Jamie, Criminisi, Antonio, Pattacini, Hoffman, Matthew W, Pfau, David, Schaul, Tom, and Ugo, and Paek, Tim. Filter forests for learning data- de Freitas, Nando. Learning to learn by gradient descent dependent convolutional kernels. In CVPR, 2014. by gradient descent. NIPS, 2016. Finley, Thomas and Joachims, Thorsten. Training struc- Barbu, Adrian. Training an active random field for real- tural svms when exact inference is intractable. In ICML, time image denoising. IEEE Transactions on Image Pro- 2008. cessing, 18(11):2451–2462, 2009. FitzGerald, Nicholas, Tackstr¨ om,¨ Oscar, Ganchev, Kuz- Beck, Amir and Teboulle, Marc. Mirror descent and non- man, and Das, Dipanjan. Semantic role labeling with linear projected subgradient methods for convex opti- neural network factors. In EMNLP, pp. 960–970, 2015. mization. Operations Research Letters, 31(3), 2003. Foo, Chuan-sheng, Do, Chuong B, and Ng, Andrew Y. Ef- Belanger, David and McCallum, Andrew. Structured pre- ficient multiple hyperparameter learning for log-linear diction energy networks. In ICML, 2016. models. In NIPS, 2008.
Bengio, Yoshua, Simard, Patrice, and Frasconi, Paolo. Geoffrey, Zweig and Padmanabhan, Mukund. Exact alpha- Learning long-term dependencies with gradient descent beta computation in logarithmic space with application is difficult. IEEE transactions on neural networks, 5(2): to map word graph construction. 2000. 157–166, 1994. Gildea, Daniel and Jurafsky, Daniel. Automatic labeling of Brakel, Philemon,´ Stroobandt, Dirk, and Schrauwen, Ben- semantic roles. Computational linguistics, 28(3):245– jamin. Training energy-based models for time-series im- 288, 2002. putation. JMLR, 14, 2013. Greff, Klaus, Srivastava, Rupesh K, and Schmidhuber, Carreras, Xavier and Marquez,` Llu´ıs. Introduction to Jurgen.¨ Highway and residual networks learn unrolled the conll-2005 shared task: Semantic role labeling. In iterative estimation. ICLR, 2017. CoNLL, 2005. Gregor, Karol and LeCun, Yann. Learning fast approxima- Chen, Liang-Chieh, Papandreou, George, Kokkinos, Ia- tions of sparse coding. In ICML, 2010. sonas, Murphy, Kevin, and Yuille, Alan L. Semantic im- age segmentation with deep convolutional nets and fully Gygli, M., Norouzi, M., and Angelova, A. Deep Value connected crfs. ICLR, 2015. Networks Learn to Evaluate and Iteratively Refine Struc- tured Outputs. In ICML, 2017. Dabov, Kostadin, Foi, Alessandro, Katkovnik, Vladimir, and Egiazarian, Karen. Image denoising by sparse 3-d He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, transform-domain collaborative filtering. IEEE Transac- Jian. Deep residual learning for image recognition. In tions on image processing, 16(8):2080–2095, 2007. CVPR, 2016. End-to-End SPENs
Hershey, John R, Roux, Jonathan Le, and Weninger, Felix. Metz, Luke, Poole, Ben, Pfau, David, and Sohl-Dickstein, Deep unfolding: Model-based inspiration of novel deep Jascha. Unrolled generative adversarial networks. ICLR, architectures. arXiv preprint arXiv:1409.2574, 2014. 2017.
Hinton, Geoffrey, Osindero, Simon, Welling, Max, and Mikolov, Tomas, Sutskever, Ilya, Chen, Kai, Corrado, Teh, Yee-Whye. Unsupervised discovery of nonlinear Greg S, and Dean, Jeff. Distributed representations of structure using contrastive backpropagation. Cognitive words and phrases and their compositionality. In NIPS, science, 30(4):725–731, 2006. 2013.
Hoang, Cong Duy Vu, Haffari, Gholamreza, and Cohn, Mnih, Andriy and Hinton, Geoffrey. Learning nonlinear Trevor. Decoding as continuous optimization in neural constraints with contrastive backpropagation. In IJCNN, machine translation. arXiv preprint:1701.02854, 2017. 2005.
Hochreiter, Sepp and Schmidhuber, Jurgen.¨ Long short- Newcombe, Richard A, Izadi, Shahram, Hilliges, Otmar, term memory. Neural computation, 1997. Molyneaux, David, Kim, David, Davison, Andrew J, Kohi, Pushmeet, Shotton, Jamie, Hodges, Steve, and Kingma, Diederik and Ba, Jimmy. Adam: A method for Fitzgibbon, Andrew. Kinectfusion: Real-time dense sur- stochastic optimization. ICLR, 2015. face mapping and tracking. In IEEE international sym- posium on Mixed and augmented reality, 2011. Koller, Daphne and Friedman, Nir. Probabilistic graphical models: principles and techniques. MIT press, 2009. Ngiam, Jiquan, Chen, Zhenghao, Koh, Pang W, and Ng, Andrew Y. Learning deep energy models. In ICML, Kschischang, Frank R, Frey, Brendan J, and Loeliger, 2011. H-A. Factor graphs and the sum-product algorithm. IEEE Transactions on information theory, 47(2):498– Pearlmutter, Barak A. Fast exact multiplication by the hes- 519, 2001. sian. Neural computation, 6(1):147–160, 1994.
Lacoste-Julien, Simon, Jaggi, Martin, Schmidt, Mark, Punyakanok, Vasin, Roth, Dan, and Yih, Wen-tau. The im- and Pletscher, Patrick. Block-coordinate frank-wolfe portance of syntactic parsing and inference in semantic optimization for structural svms. arXiv preprint role labeling. Computational Linguistics, 34, 2008. arXiv:1207.4747, 2012. Roth, Stefan and Black, Michael J. Fields of experts: A Lafferty, John, McCallum, Andrew, and Pereira, Fernando. framework for learning image priors. In CVPR, 2005. Conditional random fields: Probabilistic models for seg- Samuel, Kegan GG and Tappen, Marshall F. Learning op- menting and labeling sequence data. In ICML, 2001. timized map estimates in continuously-valued mrf mod- LeCun, Yann, Chopra, Sumit, Hadsell, Raia, Ranzato, M, els. In CVPR, 2009. and Huang, F. A tutorial on energy-based learning. Pre- Schmidt, Uwe, Gao, Qi, and Roth, Stefan. A generative dicting Structured Data, 1, 2006. perspective on mrfs in low-level vision. In CVPR, 2010. Lewis, Bil. Debugging backwards in time. arXiv preprint Srivastava, Rupesh K, Greff, Klaus, and Schmidhuber, cs/0310016, 2003. Jurgen.¨ Training very deep networks. In NIPS, 2015. Li, Yujia and Zemel, Richard S. Mean-field networks. Stoyanov, Veselin, Ropson, Alexander, and Eisner, Jason. ICML Workshop on Learning Tractable Probabilistic Empirical risk minimization of graphical model param- Models, 2014. eters given approximate inference, decoding, and model structure. In AISTATS, 2011. Liu, Dong C and Nocedal, Jorge. On the limited memory bfgs method for large scale optimization. Mathematical Sun, Jian and Tappen, Marshall F. Learning non-local programming, 45(1):503–528, 1989. range markov random field for image restoration. In CVPR, 2011. Maclaurin, Dougal, Duvenaud, David, and Adams, Ryan P. Gradient-based hyperparameter optimization through re- Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc V. Sequence versible learning. In ICML, 2015. to sequence learning with neural networks. In NIPS, 2014. Martins, Andre´ FT, Figeuiredo, Mario AT, Aguiar, Pe- dro MQ, Smith, Noah A, and Xing, Eric P. An aug- Tackstr¨ om,¨ Oscar, Ganchev, Kuzman, and Das, Dipanjan. mented lagrangian approach to constrained map infer- Efficient inference and structured learning for semantic ence. In ICML, 2011. role labeling. TACL, 2015. End-to-End SPENs
Tappen, Marshall F, Liu, Ce, Adelson, Edward H, and Free- man, William T. Learning gaussian conditional random fields for low-level vision. In CVPR, 2007. Taskar, B., Guestrin, C., and Koller, D. Max-margin Markov networks. NIPS, 2004. Tompson, Jonathan J, Jain, Arjun, LeCun, Yann, and Bre- gler, Christoph. Joint training of a convolutional network and a graphical model for human pose estimation. In Advances in neural information processing systems, pp. 1799–1807, 2014. Tsochantaridis, Ioannis, Hofmann, Thomas, Joachims, Thorsten, and Altun, Yasemin. Support vector machine learning for interdependent and structured output spaces. In ICML, 2004. Vilnis, Luke, Belanger, David, Sheldon, Daniel, and Mc- Callum, Andrew. Bethe projections for non-local infer- ence. UAI, 2015. Vincent, Pascal. A connection between score matching and denoising autoencoders. Neural Computation, 2011. Wang, Shenlong, Schwing, Alex, and Urtasun, Raquel. Efficient inference of continuous markov random fields with polynomial potentials. In NIPS, 2014. Wang, Shenlong, Fidler, Sanja, and Urtasun, Raquel. Prox- imal deep structured models. In NIPS, 2016. Zhai, Shuangfei, Cheng, Yu, Lu, Weining, and Zhang, Zhongfei. Deep structured energy based models for anomaly detection. In ICML, 2016. Zheng, Shuai, Jayasumana, Sadeep, Romera-Paredes, Bernardino, Vineet, Vibhav, Su, Zhizhong, Du, Dalong, Huang, Chang, and Torr, Philip HS. Conditional random fields as recurrent neural networks. In ICCV, 2015. Zhou, Jie and Xu, Wei. End-to-end learning of semantic role labeling using recurrent neural networks. In ACL, 2015.