On Autoencoders and Score Matching for Energy Based Models Kevin Swersky*
[email protected] Marc'Aurelio Ranzatoy
[email protected] David Buchman*
[email protected] Benjamin M. Marlin*
[email protected] Nando de Freitas*
[email protected] *Department of Computer Science, University of British Columbia, Vancouver, BC V6T 1Z4, Canada yDepartment of Computer Science, University of Toronto, Toronto, ON M5S 2G4, Canada Abstract a long history in particular application areas including modeling natural images. We consider estimation methods for the class of continuous-data energy based mod- Recently, more sophisticated latent variable EBMs for els (EBMs). Our main result shows that es- continuous data including the PoT (Welling et al., timating the parameters of an EBM using 2003), mPoT (Ranzato et al., 2010b), mcRBM (Ran- score matching when the conditional distri- zato & Hinton, 2010), FoE (Schmidt et al., 2010) and bution over the visible units is Gaussian cor- others have become popular models for learning rep- responds to training a particular form of reg- resentations of natural images as well as other sources ularized autoencoder. We show how different of real-valued data. Such models, also called gated Gaussian EBMs lead to different autoencoder MRFs, leverage latent variables to represent higher architectures, providing deep links between order interactions between the input variables. In these two families of models. We compare the very active research area of deep learning (Hinton the score matching estimator for the mPoT et al., 2006), these models been employed as elemen- model, a particular Gaussian EBM, to several tary building blocks to construct hierarchical models other training methods on a variety of tasks that achieve very promising performance on several including image denoising and unsupervised perceptual tasks (Ranzato & Hinton, 2010; Bengio, feature extraction.