DOCSLIB.ORG

Restricted Boltzmann Machine and Deep Belief Network: Tutorial and Survey

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Restricted Boltzmann Machine and Deep Belief Network: Tutorial and Survey To appear as a part of an upcoming textbook on dimensionality reduction and manifold learning. Restricted Boltzmann Machine and Deep Belief Network: Tutorial and Survey Benyamin Ghojogh [email protected] Department of Electrical and Computer Engineering, Machine Learning Laboratory, University of Waterloo, Waterloo, ON, Canada Ali Ghodsi [email protected] Department of Statistics and Actuarial Science & David R. Cheriton School of Computer Science, Data Analytics Laboratory, University of Waterloo, Waterloo, ON, Canada Fakhri Karray [email protected] Department of Electrical and Computer Engineering, Centre for Pattern Analysis and Machine Intelligence, University of Waterloo, Waterloo, ON, Canada Mark Crowley [email protected] Department of Electrical and Computer Engineering, Machine Learning Laboratory, University of Waterloo, Waterloo, ON, Canada Abstract to be useful for modeling the physical systems statisti- This is a tutorial and survey paper on Boltz- cally (Huang, 1987). One of these systems was the Ising mann Machine (BM), Restricted Boltzmann Ma- model which modeled interacting particles with binary chine (RBM), and Deep Belief Network (DBN). spins (Lenz, 1920; Ising, 1925). Later, the Ising model We start with the required background on prob- was found to be able to be a neural network (Little, 1974). abilistic graphical models, Markov random field, Hence, Hopfield network was proposed which modeled an Gibbs sampling, statistical physics, Ising model, Ising model in a network for modeling memory (Hopfield, and the Hopfield network. Then, we intro- 1982). Inspired by the Hopfield network (Little, 1974; duce the structures of BM and RBM. The con- Hopfield, 1982), which was itself inspired by the physi- ditional distributions of visible and hidden vari- cal Ising model (Lenz, 1920; Ising, 1925), Hinton et. al. ables, Gibbs sampling in RBM for generating proposed Boltzmann Machine (BM) and Restricted Boltz- variables, training BM and RBM by maximum mann Machine (RBM) (Hinton & Sejnowski, 1983; Ack- likelihood estimation, and contrastive divergence ley et al., 1985). These models are energy-based models are explained. Then, we discuss different possi- (LeCun et al., 2006) and the names come from the Boltz- ble discrete and continuous distributions for the mann distribution (Boltzmann, 1868; Gibbs, 1902) used in variables. We introduce conditional RBM and these models. A BM has weighted links between two lay- how it is trained. Finally, we explain deep be- ers of neurons as well as links between neurons of every layer. RBM restricts these links to not have links between arXiv:2107.12521v1 [cs.LG] 26 Jul 2021 lief network as a stack of RBM models. This paper on Boltzmann machines can be useful in neurons of a layer. BM and RBM take one of the layers various fields including data science, statistics, as the layer of data and the other layer as a representation neural computation, and statistical physics. or embedding of data. BM and RBM are special cases of the Ising model whose weights (coupling parameters) are learned. BM and RBM are also special cases of the Hop- 1. Introduction field network whose weights are learned by maximum like- lihood estimation rather than the Hebbian learning method Centuries ago, the Boltzmann distribution (Boltzmann, (Hebb, 1949) which is used in the Hopfield network. 1868), also called the Gibbs distribution (Gibbs, 1902), was proposed. This energy-based distribution was found The Hebbian learning method, used in the Hopfield net- work, was very weak and could not generalize well to un- seen data. Therefore, backpropagation (Rumelhart et al., 1986) was proposed for training neural networks. Back- propagation was gradient descent plus the chain rule tech- Restricted Boltzmann Machine and Deep Belief Network: Tutorial and Survey 2 nique. However, researchers found out soon that neural The remainder of this paper is as follows. We briefly review networks cannot get deep in their number of layers. This is the required background on probabilistic graphical models, because, in deep networks, gradients become very small in Markov random field, Gibbs sampling, statistical physics, the initial layers after many chain rules from the last layers Ising model, and the Hopfield network in Section2. The of network. This problem was called vanishing gradients. structure of BM and RBM, Gibbs sampling in RBM, train- This problem of networks plus the glory of theory in ker- ing RBM by maximum likelihood estimation, contrastive nel support vector machines (Boser et al., 1992) resulted in divergence, and training BM are explained in Section3. the winter of neural networks in the last years of previous Then, we introduce different cases of states for units in century until around 2006. RBM in Section4. Conditional RBM and DBN are ex- During the winter of neural networks, Hinton tried to save plained in Sections5 and6, respectively. Finally, Section7 neural networks from being forgotten in the history of ma- concludes the paper. chine learning. So, he returned to his previously pro- posed RBM and proposed a learning method for RBM with Required Background for the Reader the help of some other researchers including Max Welling This paper assumes that the reader has general knowledge (Hinton, 2002; Welling et al., 2004). They proposed train- of calculus, probability, linear algebra, and basics of opti- ing the weights of BM and RBM using maximum likeli- mization. The required background on statistical physics is hood estimation. BM and RBM can be seen as genera- explained in the paper. tive models where new values for neurons can be gener- ated using Gibbs sampling (Geman & Geman, 1984). Hin- 2. Background ton noticed RBM because he knew that the set of weights 2.1. Probabilistic Graphical Model and Markov between every two layers of a neural network is an RBM. Random Field It was in the year 2006 (Hinton & Salakhutdinov, 2006; Hinton et al., 2006) that he thought it is possible to train a A Probabilistic Graphical Model (PGM) is a grpah-based network in a greedy way (Bengio et al., 2007) where the representation of a complex distribution in the possibly weights of every layer of network is trained using RBM high dimensional space (Koller & Friedman, 2009). In training. This stack of RBM models with a greedy algo- other words, PGM is a combination of graph theory and rithm for training was named Deep Belief Network (DBN) probability theory. In a PGM, the random variables are rep- (Hinton et al., 2006; Hinton, 2009). DBN allowed the net- resented by nodes or vertices. There exist edges between works to become deep by preparing a good initialization of two variables which have interaction with one another in weights (using RBM training) for backpropagation. This terms of probability. Different conditional probabilities can good starting point for backpropagation optimization did be represented by a PGM. There exist two types of PGM not face the problem of vanishing gradients anymore. Since which are Markov network (also called Markov random the breakthrough in 2006 (Hinton & Salakhutdinov, 2006), field) and Bayesian network (Koller & Friedman, 2009). the winter of neural networks started to end gradually be- In the Markov network and Bayesian network, the edges cause the networks could get deep to become more nonlin- of graph are undirected and directed, respectively. BM and ear and handle more nonlinear data. RBM are Markov networks (Markov random field) because their links are undirected (Hinton, 2007). DBN was used in different applications including speech recognition (Mohamed et al., 2009; Mohamed & Hinton, 2.2. Gibbs Sampling 2010; Mohamed et al., 2011) and action recognition (Tay- Gibbs sampling, firstly proposed by (Geman & Geman, lor et al., 2007). Hinton was very excited about the suc- 1984), draws samples from a d-dimensional multivari- cess of RBM and was thinking that the future of neural ate distribution (X) using d conditional distributions networks belongs to DBN. However, his research group P (Bishop, 2006). This sampling algorithm assumes that the proposed two important techniques which were the ReLu conditional distributions of every dimension of data condi- activation function (Glorot et al., 2011) and the dropout tioned on the rest of coordinates are simple to draw samples technique (Srivastava et al., 2014). These two regulariza- from. tion methods prevented overfitting (Ghojogh & Crowley, 2019) and resolved vanishing gradients even without RBM In Gibbs sampling, we desire to sample from a multivariate d pre-training. Hence, backpropagation could be used alone distribution P(X) where X 2 R . Consider the notation d > if the new regularization methods were utilized. The suc- R 3 x := [x1; x2; : : : ; xd] . We start from a random d- cess of neural networks was found out more (LeCun et al., dimensional vector in the range of data. Then, we sample 2015) by its various applications, for example in in image the first dimension of the first sample from the distribution recognition (Karpathy & Fei-Fei, 2015). of the first dimension conditioned on the other dimensions. We do it for all dimensions, where the j-th dimension is This is a tutorial and survey paper on BM, RBM, and DBN. Restricted Boltzmann Machine and Deep Belief Network: Tutorial and Survey 3 sampled as (Ghojogh et al., 2020): where ln(:) is the natural logarithm. The internal energy is defined as: x ∼ (x j x ; : : : ; x ; x ; : : : ; x ): (1) j P j 1 j−1 j+1 d @ U(β) := β F (β): (6) We do this for all dimensions until all dimensions of the @β first sample are drawn. Then, starting from the first sam- Therefore, we have: ple, we repeat this procedure for the dimensions of the sec- @ −1 @Z ond sample. We iteratively perform this for all samples; U(β) = (− ln(Z)) = however, some initial samples are not yet valid because the @β Z @β −βE(x) algorithm has started from a not-necessarily valid vector. (3) X e (2) X = E(x) = (x)E(x): (7) We accept all samples after some burn-in iterations.
Load more

Recommended publications
  • Deep Belief Networks for Phone Recognition
    Deep Belief Networks for phone recognition Abdel-rahman Mohamed, George Dahl, and Geoffrey Hinton Department of Computer Science University of Toronto {asamir,gdahl,hinton}@cs.toronto.edu Abstract Hidden Markov Models (HMMs) have been the state-of-the-art techniques for acoustic modeling despite their unrealistic independence assumptions and the very limited representational capacity of their hidden states. There are many proposals in the research community for deeper models that are capable of modeling the many types of variability present in the speech generation process. Deep Belief Networks (DBNs) have recently proved to be very effective for a variety of ma- chine learning problems and this paper applies DBNs to acoustic modeling. On the standard TIMIT corpus, DBNs consistently outperform other techniques and the best DBN achieves a phone error rate (PER) of 23.0% on the TIMIT core test set. 1 Introduction A state-of-the-art Automatic Speech Recognition (ASR) system typically uses Hidden Markov Mod- els (HMMs) to model the sequential structure of speech signals, with local spectral variability mod- eled using mixtures of Gaussian densities. HMMs make two main assumptions. The first assumption is that the hidden state sequence can be well-approximated using a first order Markov chain where each state St at time t depends only on St−1. Second, observations at different time steps are as- sumed to be conditionally independent given a state sequence. Although these assumptions are not realistic, they enable tractable decoding and learning even with large amounts of speech data. Many methods have been proposed for relaxing the very strong conditional independence assumptions of standard HMMs (e.g.
    [Show full text]
  • Natural Reweighted Wake-Sleep
    Natural Reweighted Wake-Sleep Csongor Várady Riccardo Volpi Max Planck Institute for Romanian Institute of Mathematics in the Sciences Science and Technology Leipzig, Germany Cluj-Napoca, Romania [email protected] [email protected] Luigi Malagò Nihat Ay Romanian Institute of Max Planck Institute for Science and Technology Mathematics in the Sciences Cluj-Napoca, Romania Leipzig, Germany [email protected] [email protected] Abstract Natural gradient has been successfully employed in a wide range of optimization problems. However, for the training of neural networks the resulting increase in computational complexity sets a limitation to its practical application. Helmholtz Machines are a particular type of generative models, composed of two Sigmoid Belief Networks, commonly trained using the Wake-Sleep algorithm. The locality of the connections in this type of networks induces sparsity and a particular structure for the Fisher information matrix that can be exploited for the evaluation of its inverse, allowing the efficient computation of the natural gradient also for large networks. We introduce a novel algorithm called Natural Reweighted Wake-Sleep, a geometric adaptation of Reweighted Wake-Sleep, based on the computation of the natural gradient. We present an experimental analysis of the algorithm in terms of speed of convergence and the value of the log-likelihood, both with respect to number of iterations and training time, demonstrating improvements over non-geometric baselines. 1 Introduction Deep generative models have been successfully employed in unsupervised learning to model complex and high dimensional distributions thanks to their ability to extract higher-order representations of the data and thus generalize better [16, 8].
    [Show full text]
  • A Survey Paper on Deep Belief Network for Big Data
    International Journal of Computer Engineering & Technology (IJCET) Volume 9, Issue 5, September-October 2018, pp. 161–166, Article ID: IJCET_09_05_019 Available online at http://iaeme.com/Home/issue/IJCET?Volume=9&Issue=5 Journal Impact Factor (2016): 9.3590(Calculated by GISI) www.jifactor.com ISSN Print: 0976-6367 and ISSN Online: 0976–6375 © IAEME Publication A SURVEY PAPER ON DEEP BELIEF NETWORK FOR BIG DATA Azhagammal Alagarsamy Research Scholar, Bharathiyar University, Coimbatore, Tamil Nadu, India Assistant Professor, Department of Computer Science, Ayya Nadar Janaki Ammal College, Sivakasi, Tamilnadu Dr. K. Ruba Soundar Head, Department of CSE, PSR Engineering College, Sivakasi, Tamilnadu, India ABSTRACT Deep Belief Network (DBN) , as one of the deep learning architecture also noticeable machine learning technique, used in many applications like image processing, speech recognition and text mining. It uses supervised and unsupervised method to understand features in hierarchical architecture for the classification and pattern recognition. Recent year development in technologies has enabled the collection of large data. It also poses many challenges in data mining and processing due its nature of large volume, variety, velocity and veracity. In this paper we review the research of DBN for Big data. Key words: DBN, Big Data, Deep Belief Network, Classification. Cite this Article: Azhagammal Alagarsamy and Dr. K. Ruba Soundar, A Survey Paper on Deep Belief Network for Big Data. International Journal of Computer Engineering and Technology, 9(5), 2018, pp. 161-166. http://iaeme.com/Home/issue/IJCET?Volume=9&Issue=5 1. INTRODUCTION Deep learning is also named as deep structured learning or hierarchal learning.
    [Show full text]
  • Auto-Encoding a Knowledge Graph Using a Deep Belief Network
    ABSTRACT We started with a knowledge graph of connected entities and descriptive properties of those entities, from which, a hierarchical representation of the knowledge graph is derived. Using a graphical, energy-based neural network, we are able to show that the structure of the hierarchy can be internally captured by the neural network, which allows for efficient output of the underlying equilibrium distribution from which the data are drawn. AUTO-ENCODING A Robert A. Murphy [email protected] KNOWLEDGE GRAPH USING A DEEP BELIEF NETWORK A Random Fields Perspective Table of Contents Introduction .................................................................................................................................................. 2 GloVe for Knowledge Expansion ................................................................................................................... 2 The Approach ................................................................................................................................................ 3 Deep Belief Network ................................................................................................................................. 4 Random Field Setup .............................................................................................................................. 4 Random Field Illustration ...................................................................................................................... 5 Restricted Boltzmann Machine ................................................................................................................
    [Show full text]
  • Unsupervised Pre-Training of a Deep LSTM-Based Stacked Autoencoder for Multivariate Time Series Forecasting Problems Alaa Sagheer 1,2,3* & Mostafa Kotb2,3
    www.nature.com/scientificreports OPEN Unsupervised Pre-training of a Deep LSTM-based Stacked Autoencoder for Multivariate Time Series Forecasting Problems Alaa Sagheer 1,2,3* & Mostafa Kotb2,3 Currently, most real-world time series datasets are multivariate and are rich in dynamical information of the underlying system. Such datasets are attracting much attention; therefore, the need for accurate modelling of such high-dimensional datasets is increasing. Recently, the deep architecture of the recurrent neural network (RNN) and its variant long short-term memory (LSTM) have been proven to be more accurate than traditional statistical methods in modelling time series data. Despite the reported advantages of the deep LSTM model, its performance in modelling multivariate time series (MTS) data has not been satisfactory, particularly when attempting to process highly non-linear and long-interval MTS datasets. The reason is that the supervised learning approach initializes the neurons randomly in such recurrent networks, disabling the neurons that ultimately must properly learn the latent features of the correlated variables included in the MTS dataset. In this paper, we propose a pre-trained LSTM- based stacked autoencoder (LSTM-SAE) approach in an unsupervised learning fashion to replace the random weight initialization strategy adopted in deep LSTM recurrent networks. For evaluation purposes, two diferent case studies that include real-world datasets are investigated, where the performance of the proposed approach compares favourably with the deep LSTM approach. In addition, the proposed approach outperforms several reference models investigating the same case studies. Overall, the experimental results clearly show that the unsupervised pre-training approach improves the performance of deep LSTM and leads to better and faster convergence than other models.
    [Show full text]
  • Introduction Boltzmann Machine: Learning
    Introduction to Neural Networks Probability and neural networks Initialize standard library files: Off[General::spell1]; SetOptions[ContourPlot, ImageSize → Small]; SetOptions[Plot, ImageSize → Small]; SetOptions[ListPlot, ImageSize → Small]; Introduction Last time From energy to probability. From Hopfield net to Boltzmann machine. We showed how the Hopfield network, which minimized an energy function, could be viewed as finding states that increase probabil- ity. Neural network computations from the point of view of statistical inference By treating neural network learning and dynamics in terms of probability computations, we can begin to see how a common set of tools and concepts can be applied to: 1. Inference: as a process which makes the best guesses given data. E.g. find H such that p(H | data) is biggest. 2. Learning: a process that discovers the parameters of a probability distribution. E.g. find weights such that p(weights | data) is biggest 3. Generative modeling: a process that generates data from a probability distribution. E.g. draw samples from p(data | H) Today Boltzmann machine: learning the weights Probability and statistics overview Introduction to drawing samples Boltzmann Machine: Learning We've seen how a stochastic update rule improves the chances of a network evolving to a global minimum. Now let's see how learning weights can be formulated as a statistical problem. 2 Lect_13_Probability.nb The Gibbs distribution again Suppose T is fixed at some value, say T=1. Then we could update the network and let it settle to ther- mal equilibrium, a state characterized by some statistical stability, but with occasional jiggles. Let Vα represent the vector of neural activities.
    [Show full text]
  • A Video Recognition Method by Using Adaptive Structural Learning Of
    A Video Recognition Method by using Adaptive Structural Learning of Long Short Term Memory based Deep Belief Network Shin Kamada Takumi Ichimura Advanced Artificial Intelligence Project Research Center, Advanced Artificial Intelligence Project Research Center, Research Organization of Regional Oriented Studies, Research Organization of Regional Oriented Studies, Prefectural University of Hiroshima and Faculty of Management and Information System, 1-1-71, Ujina-Higashi, Minami-ku, Prefectural University of Hiroshima Hiroshima 734-8558, Japan 1-1-71, Ujina-Higashi, Minami-ku, E-mail: [email protected] Hiroshima 734-8558, Japan E-mail: [email protected] Abstract—Deep learning builds deep architectures such as function that classifies an given image or detects an object multi-layered artificial neural networks to effectively represent while predicting the future, is required. Understanding of time multiple features of input patterns. The adaptive structural series video is expected in various kinds industrial fields, learning method of Deep Belief Network (DBN) can realize a high classification capability while searching the optimal network such as human detection, pose or facial estimation from video structure during the training. The method can find the optimal camera, autonomous driving system, and so on [10]. number of hidden neurons of a Restricted Boltzmann Machine LSTM (Long Short Term Memory) is a well-known method (RBM) by neuron generation-annihilation algorithm to train the for time-series prediction and is applied to deep learning given input data, and then it can make a new layer in DBN methods[11]. The method enabled the traditional recurrent by the layer generation algorithm to actualize a deep data representation.
    [Show full text]
  • Deep Reinforcement Learning with Experience Replay Based on SARSA
    Deep Reinforcement Learning with Experience Replay Based on SARSA Dongbin Zhao, Haitao Wang, Kun Shao and Yuanheng Zhu Key Laboratory of Management and Control for Complex Systems Institute of Automation Chinese Academy of Sciences, Beijing 100190, China [email protected], [email protected], [email protected], [email protected] Abstract—SARSA, as one kind of on-policy reinforcement designed to deal with decision-making problems, it has run into learning methods, is integrated with deep learning to solve the great difficulties when handling high dimension data. With the video games control problems in this paper. We use deep development of feature detection method like DL, such convolutional neural network to estimate the state-action value, problems are to be well solved. and SARSA learning to update it. Besides, experience replay is introduced to make the training process suitable to scalable A new method, called deep reinforcement learning (DRL), machine learning problems. In this way, a new deep emerges to lead the direction of advanced AI research. DRL reinforcement learning method, called deep SARSA is proposed combines excellent perceiving ability of DL with decision- to solve complicated control problems such as imitating human making ability of RL. In 2010, Lange [10] proposed a typical to play video games. From the experiments results, we can algorithm which applied a deep auto-encoder neural network conclude that the deep SARSA learning shows better (DANN) into a visual control task. Later, Abtahi and Fasel [11] performances in some aspects than deep Q learning. employed a deep belief network (DBN) as the function approximation to improve the learning efficiency of traditional Keywords—SARSA learning; Q learning; experience replay; neural fitted-Q method.
    [Show full text]
  • Factor Analysis Using Delta-Rule Wake-Sleep Learning
    Communicated by Robert Jacobs Factor Analysis Using Delta-Rule Wake-Sleep Learning Radford M. Neal Department of Statistics and Department of Computer Science, University of Toronto, Downloaded from http://direct.mit.edu/neco/article-pdf/9/8/1781/813800/neco.1997.9.8.1781.pdf by guest on 24 September 2021 Toronto M5S 1A1, Canada Peter Dayan Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA, 02139 U.S.A. We describe a linear network that models correlations between real-valued visible variables using one or more real-valued hidden variables—a fac- tor analysis model. This model can be seen as a linear version of the Helmholtz machine, and its parameters can be learned using the wake- sleep method, in which learning of the primary generative model is as- sisted by a recognition model, whose role is to fill in the values of hidden variables based on the values of visible variables. The generative and recognition models are jointly learned in wake and sleep phases, using just the delta rule. This learning procedure is comparable in simplicity to Hebbian learning, which produces a somewhat different representa- tion of correlations in terms of principal components. We argue that the simplicity of wake-sleep learning makes factor analysis a plausible al- ternative to Hebbian learning as a model of activity-dependent cortical plasticity. 1 Introduction Statistical structure in a collection of inputs can be found using purely local Hebbian learning rules (Hebb, 1949), which capture second-order aspects of the data in terms of principal components (Linsker, 1988; Oja, 1989).
    [Show full text]
  • The Helmholtz Machine Revisited, EPFL2012
    Introduction Variational Approximation Relevant special cases Learning with non-factorized qs Extending the model over time Final remarks The Helmholtz machine revisited Danilo Jimenez Rezende BMI-EPFL November 8, 2012 Danilo Jimenez Rezende BMI-EPFL The Helmholtz machine revisited Introduction Variational Approximation Relevant special cases Learning with non-factorized qs Extending the model over time Final remarks 1 Introduction 2 Variational Approximation 3 Relevant special cases 4 Learning with non-factorized qs 5 Extending the model over time 6 Final remarks Danilo Jimenez Rezende BMI-EPFL The Helmholtz machine revisited Only X0 is observed. Introduction Variational Approximation Relevant special cases Learning with non-factorized qs Extending the model over time Final remarks Helmholtz machines Helmholtz machines [Dayan et al., 1995, Dayan and Hinton, 1996, Dayan, 2000] are directed graphical models with a layered structure: XL Complete data likelihood: L−1 g Y g p(X jθ ) = p(Xl jXl+1; θ )p(XL); X1 l=0 X0 Danilo Jimenez Rezende BMI-EPFL The Helmholtz machine revisited Introduction Variational Approximation Relevant special cases Learning with non-factorized qs Extending the model over time Final remarks Helmholtz machines Helmholtz machines [Dayan et al., 1995, Dayan and Hinton, 1996, Dayan, 2000] are directed graphical models with a layered structure: XL Complete data likelihood: L−1 g Y g p(X jθ ) = p(Xl jXl+1; θ )p(XL); X1 l=0 Only X0 is observed. X0 Danilo Jimenez Rezende BMI-EPFL The Helmholtz machine revisited Smooth units: g g g p(Xl jXl+1) = N(Xl ; tanh(Wl Xl+1 + Bl ); Σl ): Parameters θg = fW g ; Bg ; Σg g.
    [Show full text]
  • Deep Belief Networks Based Feature Generation and Regression For
    Deep Belief Networks Based Feature Generation and Regression for Predicting Wind Power Asifullah Khan, Aneela Zameer*, Tauseef Jamal, Ahmad Raza Email Addresses: ([asif, aneelaz, jamal]@pieas.edu.pk, [email protected]) Department of Computer Science, Pakistan Institute of Engineering and Applied Sciences (PIEAS), Nilore, Islamabad, Pakistan. Aneela Zameer*: Email: [email protected]; [email protected] Phone: +92 3219799379 Fax: 092 519248600 Page 1 of 31 ABSTRACT Wind energy forecasting helps to manage power production, and hence, reduces energy cost. Deep Neural Networks (DNN) mimics hierarchical learning in the human brain and thus possesses hierarchical, distributed, and multi-task learning capabilities. Based on aforementioned characteristics, we report Deep Belief Network (DBN) based forecast engine for wind power prediction because of its good generalization and unsupervised pre-training attributes. The proposed DBN-WP forecast engine, which exhibits stochastic feature generation capabilities and is composed of multiple Restricted Boltzmann Machines, generates suitable features for wind power prediction using atmospheric properties as input. DBN-WP, due to its unsupervised pre-training of RBM layers and generalization capabilities, is able to learn the fluctuations in the meteorological properties and thus is able to perform effective mapping of the wind power. In the deep network, a regression layer is appended at the end to predict sort-term wind power. It is experimentally shown that the deep learning and unsupervised pre-training capabilities of DBN based model has comparable and in some cases better results than hybrid and complex learning techniques proposed for wind power prediction. The proposed prediction system based on DBN, achieves mean values of RMSE, MAE and SDE as 0.124, 0.083 and 0.122, respectively.
    [Show full text]
  • Deep Belief Networks
    Probabilistic Graphical Models Statistical and Algorithmic Foundations of Deep Learning Eric Xing Lecture 11, February 19, 2020 Reading: see class homepage © Eric Xing @ CMU, 2005-2020 1 ML vs DL © Eric Xing @ CMU, 2005-2020 2 Outline q An overview of DL components q Historical remarks: early days of neural networks q Modern building blocks: units, layers, activations functions, loss functions, etc. q Reverse-mode automatic differentiation (aka backpropagation) q Similarities and differences between GMs and NNs q Graphical models vs. computational graphs q Sigmoid Belief Networks as graphical models q Deep Belief Networks and Boltzmann Machines q Combining DL methods and GMs q Using outputs of NNs as inputs to GMs q GMs with potential functions represented by NNs q NNs with structured outputs q Bayesian Learning of NNs q Bayesian learning of NN parameters q Deep kernel learning © Eric Xing @ CMU, 2005-2020 3 Outline q An overview of DL components q Historical remarks: early days of neural networks q Modern building blocks: units, layers, activations functions, loss functions, etc. q Reverse-mode automatic differentiation (aka backpropagation) q Similarities and differences between GMs and NNs q Graphical models vs. computational graphs q Sigmoid Belief Networks as graphical models q Deep Belief Networks and Boltzmann Machines q Combining DL methods and GMs q Using outputs of NNs as inputs to GMs q GMs with potential functions represented by NNs q NNs with structured outputs q Bayesian Learning of NNs q Bayesian learning of NN parameters
    [Show full text]