DOCSLIB.ORG

Semi-Supervised Learning with Sparse Autoencoders in Automatic Speech Recognition

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Semi-Supervised Learning with Sparse Autoencoders in Automatic Speech Recognition DEGREE PROJECT IN COMPUTER SCIENCE AND ENGINEERING, SECOND CYCLE, 30 CREDITS STOCKHOLM, SWEDEN 2016 Semi-Supervised Learning with Sparse Autoencoders in Automatic Speech Recognition AKASH KUMAR DHAKA KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF COMPUTER SCIENCE AND COMMUNICATION Semi-Supervised Learning with Sparse Autoencoders in Automatic Speech Recognition AKASH KUMAR DHAKA Master in Machine Learning Date: November 2016 Supervisor: Giampiero Salvi Examiner: Danica Kragic Swedish title: Semi-övervakad inlärning med glesa autoencoders i automatisk taligenkänning School of Computer Science and Communication Abstract This work is aimed at exploring semi-supervised learning techniques to improve the performance of Automatic Speech Recognition systems. Semi-supervised learning takes advantage of unlabeled data in order to improve the quality of the representations ex- tracted from the data. The proposed model is a neural network where the weights are updated by minimizing the weighted sum of a supervised and an unsupervised cost function, simultaneously. These costs are evaluated on the labeled and unlabeled por- tions of the data set, respectively. The combined cost is optimized through mini-batch stochastic gradient descent via standard backpropagation. The model was tested on a phone classification task on the TIMIT American English data set and on a written digit classification task on the MNIST data set. Our results show that the model outperforms a network trained with standard backpropagation on the labelled material alone. The results are also in line with state-of-the-art graph-based semi-supervised training methods. Sammanfattning Detta arbete syftar till att utforska halvövervakade inlärningstekniker (eng: semi-supervised learning techniques) för att förbättra prestandan hos automatiska taligenkänningssystem. Halvövervakad maskininlärning använder sig av data ej märkt med klasstillhörighets- information för att förbättra kvaliteten hos den från datan extraherade representationen. Modellen som beskrivs i arbetet är ett neuralt nätverk där vikterna uppdateras genom att samtidigt minimera den viktade summan av en övervakad och en oövervakad kostnads- funktion. Dessa kostnadsfunktioner evalueras på den märkta respektive den omärkta da- tamängden. De kombinerade kostnadsfunktionerna optimeras genom gradient descent med hjälp av traditionell backpropagation. Modellen har evaluerats genom en fonklassificeringsuppgift på datamängden TIMIT American English, samt en sifferklassificeringsuppgift på datamängden MNIST. Resul- taten visar att modellen presterar bättre än ett nätverk tränat med backpropagation på endast märkt data. Resultaten är även konkurrenskraftiga med rådande state of the art, grafbaserade halvövervakade inlärningsmetoder. Contents 1 Introduction 1 1.1 The Speech Recognition Problem . .1 1.2 Motivation For the Thesis . .2 1.3 Research Questions . .3 1.4 Assumptions . .3 1.5 Report Structure . .3 2 Relevant Theory 4 2.1 Automatic Speech Recognition and Phone Classification . .4 2.2 Feature Extraction . .6 2.3 Acoustic Modelling . .7 2.4 MLPs and Deep Neural Networks . .9 2.5 Autoencoders . 11 2.5.1 Manifold Learning with Autoencoders . 11 2.5.2 Sparse Autoencoders . 12 2.5.3 Performance of Sparse Autoencoders . 12 2.5.4 Applications of Autoencoders . 13 2.6 Semi-Supervised Learning . 13 2.7 Assumptions . 14 3 Related Work 15 3.1 Deep Neural Networks in ASR . 15 3.1.1 Deep Belief Networks . 15 3.1.2 Recurrent Neural Networks . 15 3.2 Examples of Semi-Supervised Learning Methods . 16 3.2.1 Heuristic based SSL/Self-Training . 16 3.2.2 Transductive SVMs . 16 3.2.3 Entropy Based Semi Supervised Learning . 17 3.2.4 Graph based SSL . 17 3.2.5 Semi-Supervised Learning with generative models . 18 3.3 Autoencoder Based Semi-Supervised Learning . 19 4 Method 20 4.1 The Model . 20 4.2 Evaluation . 22 4.3 Monitoring and Debugging . 23 4.3.1 Design Choices/Tuning Hyperparameters . 23 4.3.2 Learning Rate . 24 4.3.3 Batch Size . 25 4.3.4 Weight Initialization . 25 4.3.5 Number of Hidden Units . 25 4.3.6 Momentum . 26 4.3.7 Activation Function . 26 4.3.8 Training Epochs . 26 4.3.9 Additive Noise . 26 4.3.10 Alpha . 26 4.3.11 Gradient Checking . 27 5 Experiment Setup and Results 28 5.1 Data . 28 5.1.1 MNIST . 28 5.1.2 TIMIT . 28 5.2 Experimental Setup . 30 5.3 Practical Setup . 31 5.4 Results . 31 6 Discussion, Conclusion and Future Work 36 6.1 Hypotheses discussed . 36 6.1.1 H.1 Do Semi-Supervised Sparse Autoencoders perform better than neu- ral networks on phone classification? . 36 6.1.2 H.2 Does the above result generalize to other domains? . 36 6.1.3 H.3: Do Semi-Supervised Sparse Autoencoders perform better than GBL SSL methods on phoneme classification? . 36 6.2 Evaluation Method . 37 6.3 Effect of α in the model . 37 6.4 Future Work . 37 6.5 Society and Ethics . 38 7 Appendix 39 Bibliography 40 Chapter 1 Introduction With the invention of computers, the question of whether machines could be made to understand human speech emerged. In more recent years, speech technology has started to change the way we live by becoming an important tool for communication and inter- action with devices. The recent improvements in Spoken Language systems have greatly improved Human-Machine Communication. Personal Digital Assistance (PDA) systems are an example of an intelligent dialogue management system. They have been very pop- ular with the recent launch of products like Apple Siri, Amazon Alexa and Google Allo. Besides human-machine interaction, speech technology has also been applied in assisted human-human communication. There could be several barriers even when humans com- municate with each other. One of the most prominent of those barriers occurs if the two speakers do not speak a common language. In the past, and to great extent in present days, this was solved by means of a human interpreter. Speech-to-speech Translation sys- tems are, however, reaching sufficient quality to be of help for example for travellers. These systems accept spoken input in one language and output a spoken translation of the input in a target language. In all the above examples, a key component is Automatic Speech Recognition (ASR). This system has the task of translating spoken utterances into their textual transcription that can be more easily handled in the rest of the system. In dialogue systems, this tex- tual representation is fed to a language understanding module that extracts the semantic information to be handled by a dialogue manager. The dialogue manager, in turns, can decide to formulate a spoken response by means of a language generation system and a speech synthesis system. In speech-to-speech translation, instead, the output of the ASR is fed to an automatic translation system, and the translation is then converted to speech by means of speech synthesis. 1.1 The Speech Recognition Problem Although humans recognize speech in their mother tongue effortlessly, the problem of automatic speech recognition presents a number of challenges. A source of complexity is due to the large variations in speech based on region, accent, age, gender, emotions, physical and mental well-being of the speaker. Another complication, if compared to many classification tasks, is that speech is a continuous stream of units hierarchically combined into speech sounds, syllables, words, phrases and utterances. A speech rec- ognizer must therefore be able to handle sequences of patterns. 1 2 CHAPTER 1. INTRODUCTION The way the speech recognition problem is approached is by means of statistical meth- ods that can incorporate the variation and model sequences of acoustic events in a ro- bust way. The design of these models makes extensive use of domain knowledge com- ing from the fields of linguistics and phonetics, and incorporates this knowledge into a machine learning framework that can learn the variability of the spoken units from large collections of recordings of spoken utterances. The building blocks of the statistical models are short segments of speech that can be considered to be stationary. These seg- ments (or the corresponding models) are then combined to form phonemic segments. A phoneme is the smallest linguistic unit that can distinguish between two words. Phone- mic models are then combined into words and phrases by using lexical and grammatical rules. Although each language uses a specific set of phonemes, there is a large overlap between languages, because the number of sounds that we can produce is constraint by the physics of our speech organs. An example of phoneme classification for the Ameri- can English is reported in Appendix 7.1. 1.2 Motivation For the Thesis In order to learn the associations between the constituent speech units and the corre- sponding sounds, most speech recognition methods require carefully annotated speech recordings. The increasing interest in speech based applications has produced large amount of such data for many languages with a sufficiently broad consumer basis. However, these linguistic resources are extremely expensive to produce because they require in- tense expert labour (phonetic transcriptions are usually created by phoneticians). A con- sequence of this is that most speech databases are not publicly available, and even re- searchers must pay royalties in order to use them. Another consequence is that speech technology, and in particular speech recognition, does not easily reach speakers of lan- guages spoken by minorities. This work will specifically target improvements in ASR in a semi-supervised set- ting, therefore reducing the need for annotated material. The existing methods in Semi- supervised learning in ASR are based on Graph based Learning or self-training using Neural Networks. Graph based learning is computationally very intensive, while self- training is based on heuristics and prone to error due to wrong predictions. Recently, learning through neural networks has been found to be scalable on industrial levels and has given state-of-the-art results in automatic speech recognition.
Load more

Recommended publications
  • Predrnn: Recurrent Neural Networks for Predictive Learning Using Spatiotemporal Lstms
    PredRNN: Recurrent Neural Networks for Predictive Learning using Spatiotemporal LSTMs Yunbo Wang Mingsheng Long∗ School of Software School of Software Tsinghua University Tsinghua University [email protected] [email protected] Jianmin Wang Zhifeng Gao Philip S. Yu School of Software School of Software School of Software Tsinghua University Tsinghua University Tsinghua University [email protected] [email protected] [email protected] Abstract The predictive learning of spatiotemporal sequences aims to generate future images by learning from the historical frames, where spatial appearances and temporal vari- ations are two crucial structures. This paper models these structures by presenting a predictive recurrent neural network (PredRNN). This architecture is enlightened by the idea that spatiotemporal predictive learning should memorize both spatial ap- pearances and temporal variations in a unified memory pool. Concretely, memory states are no longer constrained inside each LSTM unit. Instead, they are allowed to zigzag in two directions: across stacked RNN layers vertically and through all RNN states horizontally. The core of this network is a new Spatiotemporal LSTM (ST-LSTM) unit that extracts and memorizes spatial and temporal representations simultaneously. PredRNN achieves the state-of-the-art prediction performance on three video prediction datasets and is a more general framework, that can be easily extended to other predictive learning tasks by integrating with other architectures. 1 Introduction
    [Show full text]
  • A Survey on Data Collection for Machine Learning a Big Data - AI Integration Perspective
    1 A Survey on Data Collection for Machine Learning A Big Data - AI Integration Perspective Yuji Roh, Geon Heo, Steven Euijong Whang, Senior Member, IEEE Abstract—Data collection is a major bottleneck in machine learning and an active research topic in multiple communities. There are largely two reasons data collection has recently become a critical issue. First, as machine learning is becoming more widely-used, we are seeing new applications that do not necessarily have enough labeled data. Second, unlike traditional machine learning, deep learning techniques automatically generate features, which saves feature engineering costs, but in return may require larger amounts of labeled data. Interestingly, recent research in data collection comes not only from the machine learning, natural language, and computer vision communities, but also from the data management community due to the importance of handling large amounts of data. In this survey, we perform a comprehensive study of data collection from a data management point of view. Data collection largely consists of data acquisition, data labeling, and improvement of existing data or models. We provide a research landscape of these operations, provide guidelines on which technique to use when, and identify interesting research challenges. The integration of machine learning and data management for data collection is part of a larger trend of Big data and Artificial Intelligence (AI) integration and opens many opportunities for new research. Index Terms—data collection, data acquisition, data labeling, machine learning F 1 INTRODUCTION E are living in exciting times where machine learning expertise. This problem applies to any novel application that W is having a profound influence on a wide range of benefits from machine learning.
    [Show full text]
  • Almost Unsupervised Text to Speech and Automatic Speech Recognition
    Almost Unsupervised Text to Speech and Automatic Speech Recognition Yi Ren * 1 Xu Tan * 2 Tao Qin 2 Sheng Zhao 3 Zhou Zhao 1 Tie-Yan Liu 2 Abstract 1. Introduction Text to speech (TTS) and automatic speech recognition (ASR) are two popular tasks in speech processing and have Text to speech (TTS) and automatic speech recog- attracted a lot of attention in recent years due to advances in nition (ASR) are two dual tasks in speech pro- deep learning. Nowadays, the state-of-the-art TTS and ASR cessing and both achieve impressive performance systems are mostly based on deep neural models and are all thanks to the recent advance in deep learning data-hungry, which brings challenges on many languages and large amount of aligned speech and text data. that are scarce of paired speech and text data. Therefore, However, the lack of aligned data poses a ma- a variety of techniques for low-resource and zero-resource jor practical problem for TTS and ASR on low- ASR and TTS have been proposed recently, including un- resource languages. In this paper, by leveraging supervised ASR (Yeh et al., 2019; Chen et al., 2018a; Liu the dual nature of the two tasks, we propose an et al., 2018; Chen et al., 2018b), low-resource ASR (Chuang- almost unsupervised learning method that only suwanich, 2016; Dalmia et al., 2018; Zhou et al., 2018), TTS leverages few hundreds of paired data and extra with minimal speaker data (Chen et al., 2019; Jia et al., 2018; unpaired data for TTS and ASR.
    [Show full text]
  • Self-Supervised Learning
    Self-Supervised Learning Andrew Zisserman Slides from: Carl Doersch, Ishan Misra, Andrew Owens, Carl Vondrick, Richard Zhang The ImageNet Challenge Story … 1000 categories • Training: 1000 images for each category • Testing: 100k images The ImageNet Challenge Story … strong supervision The ImageNet Challenge Story … outcomes Strong supervision: • Features from networks trained on ImageNet can be used for other visual tasks, e.g. detection, segmentation, action recognition, fine grained visual classification • To some extent, any visual task can be solved now by: 1. Construct a large-scale dataset labelled for that task 2. Specify a training loss and neural network architecture 3. Train the network and deploy • Are there alternatives to strong supervision for training? Self-Supervised learning …. Why Self-Supervision? 1. Expense of producing a new dataset for each new task 2. Some areas are supervision-starved, e.g. medical data, where it is hard to obtain annotation 3. Untapped/availability of vast numbers of unlabelled images/videos – Facebook: one billion images uploaded per day – 300 hours of video are uploaded to YouTube every minute 4. How infants may learn … Self-Supervised Learning The Scientist in the Crib: What Early Learning Tells Us About the Mind by Alison Gopnik, Andrew N. Meltzoff and Patricia K. Kuhl The Development of Embodied Cognition: Six Lessons from Babies by Linda Smith and Michael Gasser What is Self-Supervision? • A form of unsupervised learning where the data provides the supervision • In general, withhold some part of the data, and task the network with predicting it • The task defines a proxy loss, and the network is forced to learn what we really care about, e.g.
    [Show full text]
  • Reinforcement Learning in Supervised Problem Domains
    Technische Universität München Fakultät für Informatik Lehrstuhl VI – Echtzeitsysteme und Robotik reinforcement learning in supervised problem domains Thomas F. Rückstieß Vollständiger Abdruck der von der Fakultät für Informatik der Technischen Universität München zur Erlangung des akademischen Grades eines Doktors der Naturwissenschaften (Dr. rer. nat.) genehmigten Dissertation. Vorsitzender: Univ.-Prof. Dr. Daniel Cremers Prüfer der Dissertation 1. Univ.-Prof. Dr. Patrick van der Smagt 2. Univ.-Prof. Dr. Hans Jürgen Schmidhuber Die Dissertation wurde am 30. 06. 2015 bei der Technischen Universität München eingereicht und durch die Fakultät für Informatik am 18. 09. 2015 angenommen. Thomas Rückstieß: Reinforcement Learning in Supervised Problem Domains © 2015 email: [email protected] ABSTRACT Despite continuous advances in computing technology, today’s brute for- ce data processing approaches may not provide the necessary advantage to win the race against the ever-growing amount of data that can be wit- nessed over the last decades. In this thesis, we discuss novel methods and algorithms that are capable of directing attention to relevant details and analysing it in sequence to overcome the processing bottleneck and to keep up with this data explosion. In the first of three parts, a novel exploration technique for Policy Gradi- ent Reinforcement Learning is presented which replaces traditional ad- ditive random exploration with state-dependent exploration, exploring on a higher, more strategic level. We will show how this new exploration method converges faster and finds better global solutions than random exploration can. The second part of this thesis will introduce the concept of “data con- sumption” and discuss means to minimise it in supervised learning tasks by deriving classification as a sequential decision process and ma- king it accessible to Reinforcement Learning methods.
    [Show full text]
  • Infinite Variational Autoencoder for Semi-Supervised Learning
    Infinite Variational Autoencoder for Semi-Supervised Learning M. Ehsan Abbasnejad Anthony Dick Anton van den Hengel The University of Adelaide {ehsan.abbasnejad, anthony.dick, anton.vandenhengel}@adelaide.edu.au Abstract with whatever labelled data is available to train a discrimi- native model for classification. This paper presents an infinite variational autoencoder We demonstrate that our infinite VAE outperforms both (VAE) whose capacity adapts to suit the input data. This the classical VAE and standard classification methods, par- is achieved using a mixture model where the mixing coef- ticularly when the number of available labelled samples is ficients are modeled by a Dirichlet process, allowing us to small. This is because the infinite VAE is able to more ac- integrate over the coefficients when performing inference. curately capture the distribution of the unlabelled data. It Critically, this then allows us to automatically vary the therefore provides a generative model that allows the dis- number of autoencoders in the mixture based on the data. criminative model, which is trained based on its output, to Experiments show the flexibility of our method, particularly be more effectively learnt using a small number of samples. for semi-supervised learning, where only a small number of The main contribution of this paper is twofold: (1) we training samples are available. provide a Bayesian non-parametric model for combining autoencoders, in particular variational autoencoders. This bridges the gap between non-parametric Bayesian meth- 1. Introduction ods and the deep neural networks; (2) we provide a semi- supervised learning approach that utilizes the infinite mix- The Variational Autoencoder (VAE) [18] is a newly in- ture of autoencoders learned by our model for prediction troduced tool for unsupervised learning of a distribution x x with from a small number of labeled examples.
    [Show full text]
  • Unsupervised Speech Representation Learning Using Wavenet Autoencoders Jan Chorowski, Ron J
    1 Unsupervised speech representation learning using WaveNet autoencoders Jan Chorowski, Ron J. Weiss, Samy Bengio, Aaron¨ van den Oord Abstract—We consider the task of unsupervised extraction speaker gender and identity, from phonetic content, properties of meaningful latent representations of speech by applying which are consistent with internal representations learned autoencoding neural networks to speech waveforms. The goal by speech recognizers [13], [14]. Such representations are is to learn a representation able to capture high level semantic content from the signal, e.g. phoneme identities, while being desired in several tasks, such as low resource automatic speech invariant to confounding low level details in the signal such as recognition (ASR), where only a small amount of labeled the underlying pitch contour or background noise. Since the training data is available. In such scenario, limited amounts learned representation is tuned to contain only phonetic content, of data may be sufficient to learn an acoustic model on the we resort to using a high capacity WaveNet decoder to infer representation discovered without supervision, but insufficient information discarded by the encoder from previous samples. Moreover, the behavior of autoencoder models depends on the to learn the acoustic model and a data representation in a fully kind of constraint that is applied to the latent representation. supervised manner [15], [16]. We compare three variants: a simple dimensionality reduction We focus on representations learned with autoencoders bottleneck, a Gaussian Variational Autoencoder (VAE), and a applied to raw waveforms and spectrogram features and discrete Vector Quantized VAE (VQ-VAE). We analyze the quality investigate the quality of learned representations on LibriSpeech of learned representations in terms of speaker independence, the ability to predict phonetic content, and the ability to accurately re- [17].
    [Show full text]
  • A Primer on Machine Learning
    A Primer on Machine Learning By instructor Amit Manghani Question: What is Machine Learning? Simply put, Machine Learning is a form of data analysis. Using algorithms that “ continuously learn from data, Machine Learning allows computers to recognize The core of hidden patterns without actually being programmed to do so. The key aspect of Machine Learning Machine Learning is that as models are exposed to new data sets, they adapt to produce reliable and consistent output. revolves around a computer system Question: consuming data What is driving the resurgence of Machine Learning? and learning from There are four interrelated phenomena that are behind the growing prominence the data. of Machine Learning: 1) the ever-increasing volume, variety and velocity of data, 2) the decrease in bandwidth and storage costs and 3) the exponential improve- ments in computational processing. In a nutshell, the ability to perform complex ” mathematical computations on big data is driving the resurgence in Machine Learning. 1 Question: What are some of the commonly used methods of Machine Learning? Reinforce- ment Machine Learning Supervised Machine Learning Semi- supervised Machine Unsupervised Learning Machine Learning Supervised Machine Learning In Supervised Learning, algorithms are trained using labeled examples i.e. the desired output for an input is known. For example, a piece of mail could be labeled either as relevant or junk. The algorithm receives a set of inputs along with the corresponding correct outputs to foster learning. Once the algorithm is trained on a set of labeled data; the algorithm is run against the same labeled data and its actual output is compared against the correct output to detect errors.
    [Show full text]
  • Learning from Noisy Labels with Deep Neural Networks: a Survey Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, Jae-Gil Lee
    UNDER REVIEW 1 Learning from Noisy Labels with Deep Neural Networks: A Survey Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, Jae-Gil Lee Abstract—Deep learning has achieved remarkable success in 100% 100% numerous domains with help from large amounts of big data. Noisy w/o. Reg. Noisy w. Reg. 75% However, the quality of data labels is a concern because of the 75% Clean w. Reg Gap lack of high-quality labels in many real-world scenarios. As noisy 50% 50% labels severely degrade the generalization performance of deep Noisy w/o. Reg. neural networks, learning from noisy labels (robust training) is 25% Accuracy Test 25% Training Accuracy Training Noisy w. Reg. becoming an important task in modern deep learning applica- Clean w. Reg tions. In this survey, we first describe the problem of learning with 0% 0% 1 30 60 90 120 1 30 60 90 120 label noise from a supervised learning perspective. Next, we pro- Epochs Epochs vide a comprehensive review of 57 state-of-the-art robust training Fig. 1. Convergence curves of training and test accuracy when training methods, all of which are categorized into five groups according WideResNet-16-8 using a standard training method on the CIFAR-100 dataset to their methodological difference, followed by a systematic com- with the symmetric noise of 40%: “Noisy w/o. Reg.” and “Noisy w. Reg.” are parison of six properties used to evaluate their superiority. Sub- the models trained on noisy data without and with regularization, respectively, sequently, we perform an in-depth analysis of noise rate estima- and “Clean w.
    [Show full text]
  • Introduction to Deep Learning in Signal Processing & Communications with MATLAB
    Introduction to Deep Learning in Signal Processing & Communications with MATLAB Dr. Amod Anandkumar Pallavi Kar Application Engineering Group, Mathworks India © 2019 The MathWorks, Inc.1 Different Types of Machine Learning Type of Machine Learning Categories of Algorithms • Output is a choice between classes Classification (True, False) (Red, Blue, Green) Supervised Learning • Output is a real number Regression Develop predictive (temperature, stock prices) model based on both Machine input and output data Learning Unsupervised • No output - find natural groups and Clustering Learning patterns from input data only Discover an internal representation from input data only 2 What is Deep Learning? 3 Deep learning is a type of supervised machine learning in which a model learns to perform classification tasks directly from images, text, or sound. Deep learning is usually implemented using a neural network. The term “deep” refers to the number of layers in the network—the more layers, the deeper the network. 4 Why is Deep Learning So Popular Now? Human Accuracy Source: ILSVRC Top-5 Error on ImageNet 5 Vision applications have been driving the progress in deep learning producing surprisingly accurate systems 6 Deep Learning success enabled by: • Labeled public datasets • Progress in GPU for acceleration AlexNet VGG-16 ResNet-50 ONNX Converter • World-class models and PRETRAINED PRETRAINED PRETRAINED MODEL MODEL CONVERTER MODEL MODEL TensorFlow- connected community Caffe GoogLeNet IMPORTER PRETRAINED Keras Inception-v3 MODEL IMPORTER MODELS 7
    [Show full text]
  • 4 Perceptron Learning
    4 Perceptron Learning 4.1 Learning algorithms for neural networks In the two preceding chapters we discussed two closely related models, McCulloch–Pitts units and perceptrons, but the question of how to find the parameters adequate for a given task was left open. If two sets of points have to be separated linearly with a perceptron, adequate weights for the comput- ing unit must be found. The operators that we used in the preceding chapter, for example for edge detection, used hand customized weights. Now we would like to find those parameters automatically. The perceptron learning algorithm deals with this problem. A learning algorithm is an adaptive method by which a network of com- puting units self-organizes to implement the desired behavior. This is done in some learning algorithms by presenting some examples of the desired input- output mapping to the network. A correction step is executed iteratively until the network learns to produce the desired response. The learning algorithm is a closed loop of presentation of examples and of corrections to the network parameters, as shown in Figure 4.1. network test input-output compute the examples error fix network parameters Fig. 4.1. Learning process in a parametric system R. Rojas: Neural Networks, Springer-Verlag, Berlin, 1996 78 4 Perceptron Learning In some simple cases the weights for the computing units can be found through a sequential test of stochastically generated numerical combinations. However, such algorithms which look blindly for a solution do not qualify as “learning”. A learning algorithm must adapt the network parameters accord- ing to previous experience until a solution is found, if it exists.
    [Show full text]
  • Learning from Minimally Labeled Data with Accelerated Convolutional Neural Networks Aysegul Dundar Purdue University
    Purdue University Purdue e-Pubs Open Access Dissertations Theses and Dissertations 4-2016 Learning from minimally labeled data with accelerated convolutional neural networks Aysegul Dundar Purdue University Follow this and additional works at: https://docs.lib.purdue.edu/open_access_dissertations Part of the Computer Sciences Commons, and the Electrical and Computer Engineering Commons Recommended Citation Dundar, Aysegul, "Learning from minimally labeled data with accelerated convolutional neural networks" (2016). Open Access Dissertations. 641. https://docs.lib.purdue.edu/open_access_dissertations/641 This document has been made available through Purdue e-Pubs, a service of the Purdue University Libraries. Please contact [email protected] for additional information. Graduate School Form 30 Updated ¡¢ ¡£¢ ¡¤ ¥ PURDUE UNIVERSITY GRADUATE SCHOOL Thesis/Dissertation Acceptance This is to certify that the thesis/dissertation prepared By Aysegul Dundar Entitled Learning from Minimally Labeled Data with Accelerated Convolutional Neural Networks For the degree of Doctor of Philosophy Is approved by the final examining committee: Eugenio Culurciello Chair Anand Raghunathan Bradley S. Duerstock Edward L. Bartlett To the best of my knowledge and as understood by the student in the Thesis/Dissertation Agreement, Publication Delay, and Certification Disclaimer (Graduate School Form 32), this thesis/dissertation adheres to the provisions of Purdue University’s “Policy of Integrity in Research” and the use of copyright material. Approved by Major Professor(s): Eugenio Culurciello Approved by: George R. Wodicka 04/22/2016 Head of the Departmental Graduate Program Date LEARNING FROM MINIMALLY LABELED DATA WITH ACCELERATED CONVOLUTIONAL NEURAL NETWORKS A Dissertation Submitted to the Faculty of Purdue University by Aysegul Dundar In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy May 2016 Purdue University West Lafayette, Indiana ii To my family: Nezaket, Cengiz and Deniz.
    [Show full text]