DOCSLIB.ORG

Random Backpropagation and the Deep Learning Channel

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Random Backpropagation and the Deep Learning Channel Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19) Learning in the Machine: Random Backpropagation and the Deep Learning Channel (Extended Abstract)∗ Pierre Baldi1y , Peter Sadowski1 and Zhiqin Lu2 1Department of Computer Science, University of California, Irvine 2Department of Mathematics, University of California, Irvine [email protected], [email protected], [email protected] Abstract matrices, a requirement known as the weight symmetry prob- lem that has long been an objection to the hypothesis that Random backpropagation (RBP) is a variant of the biological neurons learn via gradient descent (e.g. [Crick, backpropagation algorithm for training neural net- 1989]). New learning algorithms that do not require full gra- works, where the transpose of the forward matrices dient calculations could lead to more efficient neuromorphic are replaced by fixed random matrices in the calcu- hardware and could help explain learning in the brain. lation of the weight updates. It is remarkable both Are gradients really needed for learning in deep neural net- because of its effectiveness, in spite of using ran- works (NNs)? Recent work suggests they are not (e.g. [Jader- dom matrices to communicate error information, berg et al., 2017]). In the random backpropagation algorithm and because it completely removes the requirement (RBP) [Lillicrap et al., 2016], deep layers of a NN learn of maintaining symmetric weights in a physical useful representations even when the forward weight matri- neural system. To better understand RBP, we com- ces are replaced with fixed, random matrices in the back- pare different algorithms in terms of the informa- propagation equations. This algorithm differs from greedy tion available locally to each neuron. In the pro- unsupervised layer-wise approaches [Hinton et al., 2006; cess, we derive several alternatives to RBP, includ- Bengio et al., 2007] because the deep weights depend on in- ing skipped RBP (SRBP), adaptive RBP (ARBP), formation about the targets, and it differs from greedy super- sparse RBP, and study their behavior through simu- vised layer-wise approaches [Gilmer et al., 2017; Mostafa et lations. These simulations show that many variants al., 2017] because the deep weights depend on the NN output are also robust deep learning algorithms, but that layer, and hence all the other weights. the derivative of the transfer function is important In this work we connect the RBP algorithm to the notion of in the learning rule. Finally, we prove several math- the deep learning channel that communicates error informa- ematical results including the convergence to fixed tion from the output layer to the deep hidden layers [Baldi and points of linear chains of arbitrary length, the con- Sadowski, 2016]. This channel is necessary to converge to vergence to fixed points of linear autoencoders with critical points of the objective, and can be studied using tools decorrelated data, the long-term existence of solu- from information and complexity theory. We classify learn- tions for linear systems with a single hidden layer ing algorithms by the information that is transmitted along and convergence in special cases, and the conver- this channel, and our analysis leads to several new learning gence to fixed points of non-linear chains, when the algorithms, which we analyze through experiments on the derivative of the activation functions is included. MNIST [LeCun et al., 1998], CIFAR-10 [Krizhevsky and Hinton, 2009], and HIGGS [Baldi et al., 2014] benchmark 1 Introduction data sets. Furthermore, we prove that these algorithms con- verge to a global optimum of the objective function for im- Modern artificial neural networks are optimized using portant special cases. gradient-based algorithms. Gradients can be computed rel- atively efficiently via the backpropagation algorithm, but the gradient at each weight generally depends on both the data 2 Random Backpropagation Algorithms and all other weights in the network. This high degree of in- In this work we consider layered, feed-forward, neural net- terdependence costs energy, both in biological neural systems works in which neurons in layer h are fully-connected to the and in the artificial neural networks simulated using digital neurons in the previous layer h − 1. Layer activity Oh is computers. Furthermore, the calculation of the gradients in computed as a function of the preceding layer as the backpropagation algorithm includes the forward weight ∗This paper is an extended abstract of the article Baldi, et al., Oh , f h(Sh) Artificial intelligence,260:1–35, 2018. h , h h−1 yContact Author S W O for 1 < h ≤ L (1) 6348 Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19) where O0 = I is the input data and f h is a non-linear acti- These learning rules can be compared in terms of the in- vation function. We focus here on supervised learning with formation required to update each weight. RBP solves the typical output activation functions and loss functions, includ- weight-symmetry problem by removing the dependency of ing linear, sigmoid, and softmax output layers, for which the the update on the forward weights in the backpropagation derivative of the loss E with respect to SL for a single input- step; the updates still depend on every other weight in the target pair (I;T ) is given by network, but all that information is subsumed by the error signal at the output, T − OL. In SRBP, we also remove the h @E dependency of ∆W on the derivative of the transfer func- L f l 0; l > h L = O − T: (2) tion in the downstream layers, ( ) . Despite these @S differences, we show that the these learning algorithms (as The backpropagation algorithm works by first computing well as adaptive variants) still converge to critical points of gradients at each neuron recursively, then computing the gra- the objective function in network architectures conducive to dients at the weights. These gradients are then used to update mathematical analysis, unlike other alternative deep learning the weights: algorithms such as greedy, layer-wise “pre-training.” In addition, we introduce the idea of adaptive random backpropagation (ARBP), where the backpropagation matri- @E (T − OL); for h = L Bh , − = ces in the learning channel are initialized randomly, then pro- @Sh (f h)0 (W h+1)T Bh+1; for h < L gressively adapted during learning using the product of the @E corresponding forward and backward signals, so that ∆W h = −η = ηOh−1Bh (BP) @W h ∆Ch = ηOhRh+1: where the derivative (f h)0 is evaluated at Oh, and η is the learning rate. In random backpropagation, the gradients Bh In this case, the forward channel becomes the learning chan- are replaced with a randomized error signal Rh defined recur- nel for the backward weights. This adaptive behavior can also sively be used with the skip version (ASRBP). In all these algorithms, the weight updates in the last layer are equivalent to those of BP, so (T − OL); for h = L BP=RBP=SRBP=ARBP=ASRBP in the top layer. They only Rh , (f h)0 ChRh+1; for h < L differ in the way they train the hidden layers. In experiments, h h−1 h we also compare to the case where only the top layer is ∆W = ηO R (RBP) trained, and the hidden layers remain fixed after a random h initialization. with constant matrices fC g1≤h≤L replacing the transpose of the weight matrices in each layer. In skip random back- propagation [Baldi et al., 2018], the randomized error signals 3 Results are sent directly to the deep layers rather than propagating 3.1 Mathematical Results through each intermediate layer. Through mathematical analysis, we prove that RBP and SRBP converge to a fixed point corresponding to the global (T − OL); for h = L optimum of the training set loss for the following neural net- Rh , (f h)0 ChRL; for h < L works architectures, starting from almost any set of initial h h−1 h weights (except for a set of measure 0). Proofs for the case of ∆W = ηO R (SRBP) ARBP are provided in [Baldi et al., 2017]. where random matrix Ch now connects layer h to the output • A chain of single linear neurons of arbitrary length layer. ([1;:::; 1]). • An expansive architecture of linear neurons [1; N; 1]. BP RBP SRBP • A compressive architecture of linear neurons [N; 1;N]. Output • A simple [1; 1; 1] architecture, with a power function non-linearity in the hidden neuron of the form f(x) = xµ. Setting µ = 1=3 for instance gives an S-shaped acti- vation. Furthermore, we show that this system generally does not converge for µ =6 1 when the derivative of the transfer function is omitted from the learning rule. Input For the linear architectures, under a set of standard assump- tions, we can derive a set of polynomial, autonomous, ordi- Figure 1: The path of the error signal (red) from an output neuron to nary differential equations (ODEs) for the average time evo- a deep, hidden neuron in backpropagation (BP), random backpropa- lution of the weights under the different learning algorithms. gation (RBP), and skip random backpropagation (SRBP). As soon as there is more than one variable and the system is 6349 Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19) non-linear, there is no general theory to understand the cor- 1.0 responding behavior. In fact, even in two dimensions, the problem of understanding the upper bound on the number and relative position of the limit cycles of a system of the form dx=dt = P (x; y) and dy=dt = Q(x; y), where P and Q are 0.9 polynomials of degree n is open–in fact this is Hilbert’s 16- BP th problem in the field of dynamical systems [Smale, 1998; SBP Ilyashenko, 2002].
Load more

Recommended publications
  • Backpropagation and Deep Learning in the Brain
    Backpropagation and Deep Learning in the Brain Simons Institute -- Computational Theories of the Brain 2018 Timothy Lillicrap DeepMind, UCL With: Sergey Bartunov, Adam Santoro, Jordan Guerguiev, Blake Richards, Luke Marris, Daniel Cownden, Colin Akerman, Douglas Tweed, Geoffrey Hinton The “credit assignment” problem The solution in artificial networks: backprop Credit assignment by backprop works well in practice and shows up in virtually all of the state-of-the-art supervised, unsupervised, and reinforcement learning algorithms. Why Isn’t Backprop “Biologically Plausible”? Why Isn’t Backprop “Biologically Plausible”? Neuroscience Evidence for Backprop in the Brain? A spectrum of credit assignment algorithms: A spectrum of credit assignment algorithms: A spectrum of credit assignment algorithms: How to convince a neuroscientist that the cortex is learning via [something like] backprop - To convince a machine learning researcher, an appeal to variance in gradient estimates might be enough. - But this is rarely enough to convince a neuroscientist. - So what lines of argument help? How to convince a neuroscientist that the cortex is learning via [something like] backprop - What do I mean by “something like backprop”?: - That learning is achieved across multiple layers by sending information from neurons closer to the output back to “earlier” layers to help compute their synaptic updates. How to convince a neuroscientist that the cortex is learning via [something like] backprop 1. Feedback connections in cortex are ubiquitous and modify the
    [Show full text]
  • Implementing Machine Learning & Neural Network Chip
    Implementing Machine Learning & Neural Network Chip Architectures Using Network-on-Chip Interconnect IP Implementing Machine Learning & Neural Network Chip Architectures USING NETWORK-ON-CHIP INTERCONNECT IP TY GARIBAY Chief Technology Officer [email protected] Title: Implementing Machine Learning & Neural Network Chip Architectures Using Network-on-Chip Interconnect IP Primary author: Ty Garibay, Chief Technology Officer, ArterisIP, [email protected] Secondary author: Kurt Shuler, Vice President, Marketing, ArterisIP, [email protected] Abstract: A short tutorial and overview of machine learning and neural network chip architectures, with emphasis on how network-on-chip interconnect IP implements these architectures. Time to read: 10 minutes Time to present: 30 minutes Copyright © 2017 Arteris www.arteris.com 1 Implementing Machine Learning & Neural Network Chip Architectures Using Network-on-Chip Interconnect IP What is Machine Learning? “ Field of study that gives computers the ability to learn without being explicitly programmed.” Arthur Samuel, IBM, 1959 • Machine learning is a subset of Artificial Intelligence Copyright © 2017 Arteris 2 • Machine learning is a subset of artificial intelligence, and explicitly relies upon experiential learning rather than programming to make decisions. • The advantage to machine learning for tasks like automated driving or speech translation is that complex tasks like these are nearly impossible to explicitly program using rule-based if…then…else statements because of the large solution space. • However, the “answers” given by a machine learning algorithm have a probability associated with them and can be non-deterministic (meaning you can get different “answers” given the same inputs during different runs) • Neural networks have become the most common way to implement machine learning.
    [Show full text]
  • Training Autoencoders by Alternating Minimization
    Under review as a conference paper at ICLR 2018 TRAINING AUTOENCODERS BY ALTERNATING MINI- MIZATION Anonymous authors Paper under double-blind review ABSTRACT We present DANTE, a novel method for training neural networks, in particular autoencoders, using the alternating minimization principle. DANTE provides a distinct perspective in lieu of traditional gradient-based backpropagation techniques commonly used to train deep networks. It utilizes an adaptation of quasi-convex optimization techniques to cast autoencoder training as a bi-quasi-convex optimiza- tion problem. We show that for autoencoder configurations with both differentiable (e.g. sigmoid) and non-differentiable (e.g. ReLU) activation functions, we can perform the alternations very effectively. DANTE effortlessly extends to networks with multiple hidden layers and varying network configurations. In experiments on standard datasets, autoencoders trained using the proposed method were found to be very promising and competitive to traditional backpropagation techniques, both in terms of quality of solution, as well as training speed. 1 INTRODUCTION For much of the recent march of deep learning, gradient-based backpropagation methods, e.g. Stochastic Gradient Descent (SGD) and its variants, have been the mainstay of practitioners. The use of these methods, especially on vast amounts of data, has led to unprecedented progress in several areas of artificial intelligence. On one hand, the intense focus on these techniques has led to an intimate understanding of hardware requirements and code optimizations needed to execute these routines on large datasets in a scalable manner. Today, myriad off-the-shelf and highly optimized packages exist that can churn reasonably large datasets on GPU architectures with relatively mild human involvement and little bootstrap effort.
    [Show full text]
  • Predrnn: Recurrent Neural Networks for Predictive Learning Using Spatiotemporal Lstms
    PredRNN: Recurrent Neural Networks for Predictive Learning using Spatiotemporal LSTMs Yunbo Wang Mingsheng Long∗ School of Software School of Software Tsinghua University Tsinghua University [email protected] [email protected] Jianmin Wang Zhifeng Gao Philip S. Yu School of Software School of Software School of Software Tsinghua University Tsinghua University Tsinghua University [email protected] [email protected] [email protected] Abstract The predictive learning of spatiotemporal sequences aims to generate future images by learning from the historical frames, where spatial appearances and temporal vari- ations are two crucial structures. This paper models these structures by presenting a predictive recurrent neural network (PredRNN). This architecture is enlightened by the idea that spatiotemporal predictive learning should memorize both spatial ap- pearances and temporal variations in a unified memory pool. Concretely, memory states are no longer constrained inside each LSTM unit. Instead, they are allowed to zigzag in two directions: across stacked RNN layers vertically and through all RNN states horizontally. The core of this network is a new Spatiotemporal LSTM (ST-LSTM) unit that extracts and memorizes spatial and temporal representations simultaneously. PredRNN achieves the state-of-the-art prediction performance on three video prediction datasets and is a more general framework, that can be easily extended to other predictive learning tasks by integrating with other architectures. 1 Introduction
    [Show full text]
  • Q-Learning in Continuous State and Action Spaces
    -Learning in Continuous Q State and Action Spaces Chris Gaskett, David Wettergreen, and Alexander Zelinsky Robotic Systems Laboratory Department of Systems Engineering Research School of Information Sciences and Engineering The Australian National University Canberra, ACT 0200 Australia [cg dsw alex]@syseng.anu.edu.au j j Abstract. -learning can be used to learn a control policy that max- imises a scalarQ reward through interaction with the environment. - learning is commonly applied to problems with discrete states and ac-Q tions. We describe a method suitable for control tasks which require con- tinuous actions, in response to continuous states. The system consists of a neural network coupled with a novel interpolator. Simulation results are presented for a non-holonomic control task. Advantage Learning, a variation of -learning, is shown enhance learning speed and reliability for this task.Q 1 Introduction Reinforcement learning systems learn by trial-and-error which actions are most valuable in which situations (states) [1]. Feedback is provided in the form of a scalar reward signal which may be delayed. The reward signal is defined in relation to the task to be achieved; reward is given when the system is successfully achieving the task. The value is updated incrementally with experience and is defined as a discounted sum of expected future reward. The learning systems choice of actions in response to states is called its policy. Reinforcement learning lies between the extremes of supervised learning, where the policy is taught by an expert, and unsupervised learning, where no feedback is given and the task is to find structure in data.
    [Show full text]
  • Double Backpropagation for Training Autoencoders Against Adversarial Attack
    1 Double Backpropagation for Training Autoencoders against Adversarial Attack Chengjin Sun, Sizhe Chen, and Xiaolin Huang, Senior Member, IEEE Abstract—Deep learning, as widely known, is vulnerable to adversarial samples. This paper focuses on the adversarial attack on autoencoders. Safety of the autoencoders (AEs) is important because they are widely used as a compression scheme for data storage and transmission, however, the current autoencoders are easily attacked, i.e., one can slightly modify an input but has totally different codes. The vulnerability is rooted the sensitivity of the autoencoders and to enhance the robustness, we propose to adopt double backpropagation (DBP) to secure autoencoder such as VAE and DRAW. We restrict the gradient from the reconstruction image to the original one so that the autoencoder is not sensitive to trivial perturbation produced by the adversarial attack. After smoothing the gradient by DBP, we further smooth the label by Gaussian Mixture Model (GMM), aiming for accurate and robust classification. We demonstrate in MNIST, CelebA, SVHN that our method leads to a robust autoencoder resistant to attack and a robust classifier able for image transition and immune to adversarial attack if combined with GMM. Index Terms—double backpropagation, autoencoder, network robustness, GMM. F 1 INTRODUCTION N the past few years, deep neural networks have been feature [9], [10], [11], [12], [13], or network structure [3], [14], I greatly developed and successfully used in a vast of fields, [15]. such as pattern recognition, intelligent robots, automatic Adversarial attack and its defense are revolving around a control, medicine [1]. Despite the great success, researchers small ∆x and a big resulting difference between f(x + ∆x) have found the vulnerability of deep neural networks to and f(x).
    [Show full text]
  • Introduction to Machine Learning
    Introduction to Machine Learning Perceptron Barnabás Póczos Contents History of Artificial Neural Networks Definitions: Perceptron, Multi-Layer Perceptron Perceptron algorithm 2 Short History of Artificial Neural Networks 3 Short History Progression (1943-1960) • First mathematical model of neurons ▪ Pitts & McCulloch (1943) • Beginning of artificial neural networks • Perceptron, Rosenblatt (1958) ▪ A single neuron for classification ▪ Perceptron learning rule ▪ Perceptron convergence theorem Degression (1960-1980) • Perceptron can’t even learn the XOR function • We don’t know how to train MLP • 1963 Backpropagation… but not much attention… Bryson, A.E.; W.F. Denham; S.E. Dreyfus. Optimal programming problems with inequality constraints. I: Necessary conditions for extremal solutions. AIAA J. 1, 11 (1963) 2544-2550 4 Short History Progression (1980-) • 1986 Backpropagation reinvented: ▪ Rumelhart, Hinton, Williams: Learning representations by back-propagating errors. Nature, 323, 533—536, 1986 • Successful applications: ▪ Character recognition, autonomous cars,… • Open questions: Overfitting? Network structure? Neuron number? Layer number? Bad local minimum points? When to stop training? • Hopfield nets (1982), Boltzmann machines,… 5 Short History Degression (1993-) • SVM: Vapnik and his co-workers developed the Support Vector Machine (1993). It is a shallow architecture. • SVM and Graphical models almost kill the ANN research. • Training deeper networks consistently yields poor results. • Exception: deep convolutional neural networks, Yann LeCun 1998. (discriminative model) 6 Short History Progression (2006-) Deep Belief Networks (DBN) • Hinton, G. E, Osindero, S., and Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18:1527-1554. • Generative graphical model • Based on restrictive Boltzmann machines • Can be trained efficiently Deep Autoencoder based networks Bengio, Y., Lamblin, P., Popovici, P., Larochelle, H.
    [Show full text]
  • Deep Learning Architectures for Sequence Processing
    Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright © 2021. All rights reserved. Draft of September 21, 2021. CHAPTER Deep Learning Architectures 9 for Sequence Processing Time will explain. Jane Austen, Persuasion Language is an inherently temporal phenomenon. Spoken language is a sequence of acoustic events over time, and we comprehend and produce both spoken and written language as a continuous input stream. The temporal nature of language is reflected in the metaphors we use; we talk of the flow of conversations, news feeds, and twitter streams, all of which emphasize that language is a sequence that unfolds in time. This temporal nature is reflected in some of the algorithms we use to process lan- guage. For example, the Viterbi algorithm applied to HMM part-of-speech tagging, proceeds through the input a word at a time, carrying forward information gleaned along the way. Yet other machine learning approaches, like those we’ve studied for sentiment analysis or other text classification tasks don’t have this temporal nature – they assume simultaneous access to all aspects of their input. The feedforward networks of Chapter 7 also assumed simultaneous access, al- though they also had a simple model for time. Recall that we applied feedforward networks to language modeling by having them look only at a fixed-size window of words, and then sliding this window over the input, making independent predictions along the way. Fig. 9.1, reproduced from Chapter 7, shows a neural language model with window size 3 predicting what word follows the input for all the. Subsequent words are predicted by sliding the window forward a word at a time.
    [Show full text]
  • Unsupervised Speech Representation Learning Using Wavenet Autoencoders Jan Chorowski, Ron J
    1 Unsupervised speech representation learning using WaveNet autoencoders Jan Chorowski, Ron J. Weiss, Samy Bengio, Aaron¨ van den Oord Abstract—We consider the task of unsupervised extraction speaker gender and identity, from phonetic content, properties of meaningful latent representations of speech by applying which are consistent with internal representations learned autoencoding neural networks to speech waveforms. The goal by speech recognizers [13], [14]. Such representations are is to learn a representation able to capture high level semantic content from the signal, e.g. phoneme identities, while being desired in several tasks, such as low resource automatic speech invariant to confounding low level details in the signal such as recognition (ASR), where only a small amount of labeled the underlying pitch contour or background noise. Since the training data is available. In such scenario, limited amounts learned representation is tuned to contain only phonetic content, of data may be sufficient to learn an acoustic model on the we resort to using a high capacity WaveNet decoder to infer representation discovered without supervision, but insufficient information discarded by the encoder from previous samples. Moreover, the behavior of autoencoder models depends on the to learn the acoustic model and a data representation in a fully kind of constraint that is applied to the latent representation. supervised manner [15], [16]. We compare three variants: a simple dimensionality reduction We focus on representations learned with autoencoders bottleneck, a Gaussian Variational Autoencoder (VAE), and a applied to raw waveforms and spectrogram features and discrete Vector Quantized VAE (VQ-VAE). We analyze the quality investigate the quality of learned representations on LibriSpeech of learned representations in terms of speaker independence, the ability to predict phonetic content, and the ability to accurately re- [17].
    [Show full text]
  • Approaching Hanabi with Q-Learning and Evolutionary Algorithm
    St. Cloud State University theRepository at St. Cloud State Culminating Projects in Computer Science and Department of Computer Science and Information Technology Information Technology 12-2020 Approaching Hanabi with Q-Learning and Evolutionary Algorithm Joseph Palmersten [email protected] Follow this and additional works at: https://repository.stcloudstate.edu/csit_etds Part of the Computer Sciences Commons Recommended Citation Palmersten, Joseph, "Approaching Hanabi with Q-Learning and Evolutionary Algorithm" (2020). Culminating Projects in Computer Science and Information Technology. 34. https://repository.stcloudstate.edu/csit_etds/34 This Starred Paper is brought to you for free and open access by the Department of Computer Science and Information Technology at theRepository at St. Cloud State. It has been accepted for inclusion in Culminating Projects in Computer Science and Information Technology by an authorized administrator of theRepository at St. Cloud State. For more information, please contact [email protected]. Approaching Hanabi with Q-Learning and Evolutionary Algorithm by Joseph A Palmersten A Starred Paper Submitted to the Graduate Faculty of St. Cloud State University In Partial Fulfillment of the Requirements for the Degree of Master of Science in Computer Science December, 2020 Starred Paper Committee: Bryant Julstrom, Chairperson Donald Hamnes Jie Meichsner 2 Abstract Hanabi is a cooperative card game with hidden information that requires cooperation and communication between the players. For a machine learning agent to be successful at the Hanabi, it will have to learn how to communicate and infer information from the communication of other players. To approach the problem of Hanabi the machine learning methods of Q- learning and Evolutionary algorithm are proposed as potential solutions.
    [Show full text]
  • Machine Learning V/S Deep Learning
    International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056 Volume: 06 Issue: 02 | Feb 2019 www.irjet.net p-ISSN: 2395-0072 Machine Learning v/s Deep Learning Sachin Krishan Khanna Department of Computer Science Engineering, Students of Computer Science Engineering, Chandigarh Group of Colleges, PinCode: 140307, Mohali, India ---------------------------------------------------------------------------***--------------------------------------------------------------------------- ABSTRACT - This is research paper on a brief comparison and summary about the machine learning and deep learning. This comparison on these two learning techniques was done as there was lot of confusion about these two learning techniques. Nowadays these techniques are used widely in IT industry to make some projects or to solve problems or to maintain large amount of data. This paper includes the comparison of two techniques and will also tell about the future aspects of the learning techniques. Keywords: ML, DL, AI, Neural Networks, Supervised & Unsupervised learning, Algorithms. INTRODUCTION As the technology is getting advanced day by day, now we are trying to make a machine to work like a human so that we don’t have to make any effort to solve any problem or to do any heavy stuff. To make a machine work like a human, the machine need to learn how to do work, for this machine learning technique is used and deep learning is used to help a machine to solve a real-time problem. They both have algorithms which work on these issues. With the rapid growth of this IT sector, this industry needs speed, accuracy to meet their targets. With these learning algorithms industry can meet their requirements and these new techniques will provide industry a different way to solve problems.
    [Show full text]
  • Linear Prediction-Based Wavenet Speech Synthesis
    LP-WaveNet: Linear Prediction-based WaveNet Speech Synthesis Min-Jae Hwang Frank Soong Eunwoo Song Search Solution Microsoft Naver Corporation Seongnam, South Korea Beijing, China Seongnam, South Korea [email protected] [email protected] [email protected] Xi Wang Hyeonjoo Kang Hong-Goo Kang Microsoft Yonsei University Yonsei University Beijing, China Seoul, South Korea Seoul, South Korea [email protected] [email protected] [email protected] Abstract—We propose a linear prediction (LP)-based wave- than the speech signal, the training and generation processes form generation method via WaveNet vocoding framework. A become more efficient. WaveNet-based neural vocoder has significantly improved the However, the synthesized speech is likely to be unnatural quality of parametric text-to-speech (TTS) systems. However, it is challenging to effectively train the neural vocoder when the target when the prediction errors in estimating the excitation are database contains massive amount of acoustical information propagated through the LP synthesis process. As the effect such as prosody, style or expressiveness. As a solution, the of LP synthesis is not considered in the training process, the approaches that only generate the vocal source component by synthesis output is vulnerable to the variation of LP synthesis a neural vocoder have been proposed. However, they tend to filter. generate synthetic noise because the vocal source component is independently handled without considering the entire speech To alleviate this problem, we propose an LP-WaveNet, production process; where it is inevitable to come up with a which enables to jointly train the complicated interactions mismatch between vocal source and vocal tract filter.
    [Show full text]