Introduction to Machine Learning CMU-10701 Deep Learning
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
Face Recognition: a Convolutional Neural-Network Approach
98 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 8, NO. 1, JANUARY 1997 Face Recognition: A Convolutional Neural-Network Approach Steve Lawrence, Member, IEEE, C. Lee Giles, Senior Member, IEEE, Ah Chung Tsoi, Senior Member, IEEE, and Andrew D. Back, Member, IEEE Abstract— Faces represent complex multidimensional mean- include fingerprints [4], speech [7], signature dynamics [36], ingful visual stimuli and developing a computational model for and face recognition [8]. Sales of identity verification products face recognition is difficult. We present a hybrid neural-network exceed $100 million [29]. Face recognition has the benefit of solution which compares favorably with other methods. The system combines local image sampling, a self-organizing map being a passive, nonintrusive system for verifying personal (SOM) neural network, and a convolutional neural network. identity. The techniques used in the best face recognition The SOM provides a quantization of the image samples into a systems may depend on the application of the system. We topological space where inputs that are nearby in the original can identify at least two broad categories of face recognition space are also nearby in the output space, thereby providing systems. dimensionality reduction and invariance to minor changes in the image sample, and the convolutional neural network provides for 1) We want to find a person within a large database of partial invariance to translation, rotation, scale, and deformation. faces (e.g., in a police database). These systems typically The convolutional network extracts successively larger features return a list of the most likely people in the database in a hierarchical set of layers. We present results using the [34]. -
CNN Architectures
Lecture 9: CNN Architectures Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 9 - 1 May 2, 2017 Administrative A2 due Thu May 4 Midterm: In-class Tue May 9. Covers material through Thu May 4 lecture. Poster session: Tue June 6, 12-3pm Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 9 - 2 May 2, 2017 Last time: Deep learning frameworks Paddle (Baidu) Caffe Caffe2 (UC Berkeley) (Facebook) CNTK (Microsoft) Torch PyTorch (NYU / Facebook) (Facebook) MXNet (Amazon) Developed by U Washington, CMU, MIT, Hong Kong U, etc but main framework of Theano TensorFlow choice at AWS (U Montreal) (Google) And others... Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 9 - 3 May 2, 2017 Last time: Deep learning frameworks (1) Easily build big computational graphs (2) Easily compute gradients in computational graphs (3) Run it all efficiently on GPU (wrap cuDNN, cuBLAS, etc) Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 9 - 4 May 2, 2017 Last time: Deep learning frameworks Modularized layers that define forward and backward pass Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 9 - 5 May 2, 2017 Last time: Deep learning frameworks Define model architecture as a sequence of layers Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 9 - 6 May 2, 2017 Today: CNN Architectures Case Studies - AlexNet - VGG - GoogLeNet - ResNet Also.... - NiN (Network in Network) - DenseNet - Wide ResNet - FractalNet - ResNeXT - SqueezeNet - Stochastic Depth Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 9 - 7 May 2, 2017 Review: LeNet-5 [LeCun et al., 1998] Conv filters were 5x5, applied at stride 1 Subsampling (Pooling) layers were 2x2 applied at stride 2 i.e. -
Deep Learning Architectures for Sequence Processing
Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright © 2021. All rights reserved. Draft of September 21, 2021. CHAPTER Deep Learning Architectures 9 for Sequence Processing Time will explain. Jane Austen, Persuasion Language is an inherently temporal phenomenon. Spoken language is a sequence of acoustic events over time, and we comprehend and produce both spoken and written language as a continuous input stream. The temporal nature of language is reflected in the metaphors we use; we talk of the flow of conversations, news feeds, and twitter streams, all of which emphasize that language is a sequence that unfolds in time. This temporal nature is reflected in some of the algorithms we use to process lan- guage. For example, the Viterbi algorithm applied to HMM part-of-speech tagging, proceeds through the input a word at a time, carrying forward information gleaned along the way. Yet other machine learning approaches, like those we’ve studied for sentiment analysis or other text classification tasks don’t have this temporal nature – they assume simultaneous access to all aspects of their input. The feedforward networks of Chapter 7 also assumed simultaneous access, al- though they also had a simple model for time. Recall that we applied feedforward networks to language modeling by having them look only at a fixed-size window of words, and then sliding this window over the input, making independent predictions along the way. Fig. 9.1, reproduced from Chapter 7, shows a neural language model with window size 3 predicting what word follows the input for all the. Subsequent words are predicted by sliding the window forward a word at a time. -
Persian Handwritten Digit Recognition Using Combination of Convolutional Neural Network and Support Vector Machine Methods
572 The International Arab Journal of Information Technology, Vol. 17, No. 4, July 2020 Persian Handwritten Digit Recognition Using Combination of Convolutional Neural Network and Support Vector Machine Methods Mohammad Parseh, Mohammad Rahmanimanesh, and Parviz Keshavarzi Faculty of Electrical and Computer Engineering, Semnan University, Iran Abstract: Persian handwritten digit recognition is one of the important topics of image processing which significantly considered by researchers due to its many applications. The most important challenges in Persian handwritten digit recognition is the existence of various patterns in Persian digit writing that makes the feature extraction step to be more complicated.Since the handcraft feature extraction methods are complicated processes and their performance level are not stable, most of the recent studies have concentrated on proposing a suitable method for automatic feature extraction. In this paper, an automatic method based on machine learning is proposed for high-level feature extraction from Persian digit images by using Convolutional Neural Network (CNN). After that, a non-linear multi-class Support Vector Machine (SVM) classifier is used for data classification instead of fully connected layer in final layer of CNN. The proposed method has been applied to HODA dataset and obtained 99.56% of recognition rate. Experimental results are comparable with previous state-of-the-art methods. Keywords: Handwritten Digit Recognition, Convolutional Neural Network, Support Vector Machine. Received January 1, 2019; accepted November 11, 2019 https://doi.org/10.34028/iajit/17/4/16 1. Introduction years, they are good alternatives to handcraft feature extraction method. Optical Character Recognition (OCR) is one of the attractive topics of Artificial Intelligence [3, 6, 15, 23, 24]. -
Introduction-To-Deep-Learning.Pdf
Introduction to Deep Learning Demystifying Neural Networks Agenda Introduction to deep learning: • What is deep learning? • Speaking deep learning: network types, development frameworks and network models • Deep learning development flow • Application spaces Deep learning introduction ARTIFICIAL INTELLIGENCE Broad area which enables computers to mimic human behavior MACHINE LEARNING Usage of statistical tools enables machines to learn from experience (data) – need to be told DEEP LEARNING Learn from its own method of computing - its own brain Why is deep learning useful? Good at classification, clustering and predictive analysis What is deep learning? Deep learning is way of classifying, clustering, and predicting things by using a neural network that has been trained on vast amounts of data. Picture of deep learning demo done by TI’s vehicles road signs person background automotive driver assistance systems (ADAS) team. What is deep learning? Deep learning is way of classifying, clustering, and predicting things by using a neural network that has been trained on vast amounts of data. Machine Music Time of Flight Data Pictures/Video …any type of data Speech you want to classify, Radar cluster or predict What is deep learning? • Deep learning has its roots in neural networks. • Neural networks are sets of algorithms, modeled loosely after the human brain, that are designed to recognize patterns. Biologocal Artificial Biological neuron neuron neuron dendrites inputs synapses weight axon output axon summation and cell body threshold dendrites synapses cell body Node = neuron Inputs Weights Summation W1 x1 & threshold W x 2 2 Output Inputs W3 Output x3 y layer = stack of ∑ ƒ(x) neurons Wn xn ∑=sum(w*x) Artificial neuron 6 What is deep learning? Deep learning creates many layers of neurons, attempting to learn structured representation of big data, layer by layer. -
Face Recognition Using Popular Deep Net Architectures: a Brief Comparative Study
future internet Article Face Recognition Using Popular Deep Net Architectures: A Brief Comparative Study Tony Gwyn 1,* , Kaushik Roy 1 and Mustafa Atay 2 1 Department of Computer Science, North Carolina A&T State University, Greensboro, NC 27411, USA; [email protected] 2 Department of Computer Science, Winston-Salem State University, Winston-Salem, NC 27110, USA; [email protected] * Correspondence: [email protected] Abstract: In the realm of computer security, the username/password standard is becoming increas- ingly antiquated. Usage of the same username and password across various accounts can leave a user open to potential vulnerabilities. Authentication methods of the future need to maintain the ability to provide secure access without a reduction in speed. Facial recognition technologies are quickly becoming integral parts of user security, allowing for a secondary level of user authentication. Augmenting traditional username and password security with facial biometrics has already seen impressive results; however, studying these techniques is necessary to determine how effective these methods are within various parameters. A Convolutional Neural Network (CNN) is a powerful classification approach which is often used for image identification and verification. Quite recently, CNNs have shown great promise in the area of facial image recognition. The comparative study proposed in this paper offers an in-depth analysis of several state-of-the-art deep learning based- facial recognition technologies, to determine via accuracy and other metrics which of those are most effective. In our study, VGG-16 and VGG-19 showed the highest levels of image recognition accuracy, Citation: Gwyn, T.; Roy, K.; Atay, M. as well as F1-Score. -
Advancements in Image Classification Using Convolutional Neural Network
This paper has been accepted and presented on 2018 Fourth International Conference on Research in Computational Intelligence and Communication Networks. This is preprint version and original proceeding will be published in IEEE Xplore. Advancements in Image Classification using Convolutional Neural Network Farhana Sultana Abu Sufian Paramartha Dutta Department of Computer Science Department of Computer Science Department of CSS University of Gour Banga University of Gour Banga Visva-Bharati University West Bengal, India West Bengal, India West Bengal, India Email: [email protected] Email: sufi[email protected] Email: [email protected] Abstract—Convolutional Neural Network (CNN) is the state- LeCun et al. introduced the practical model of CNN [6] [7] of-the-art for image classification task. Here we have briefly and developed LeNet-5 [8]. Training by backpropagation [9] discussed different components of CNN. In this paper, We have algorithm helped LeNet-5 recognizing visual patterns from explained different CNN architectures for image classification. Through this paper, we have shown advancements in CNN from raw pixels directly without using any separate feature engi- LeNet-5 to latest SENet model. We have discussed the model neering mechanism. Also fewer connections and parameters description and training details of each model. We have also of CNN than conventional feedforward neural networks with drawn a comparison among those models. similar network size, made model training easier. But at that Keywords—AlexNet, Capsnet, Convolutional Neural Network, time in spite of several advantages, the performance of CNN Deep learning, DenseNet, Image classification, ResNet, SENet. in intricate problems such as classification of high-resolution image, was limited by the lack of large training data, lack of I. -
Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
Global Sparse Momentum SGD for Pruning Very Deep Neural Networks Xiaohan Ding 1 Guiguang Ding 1 Xiangxin Zhou 2 Yuchen Guo 1, 3 Jungong Han 4 Ji Liu 5 1 Beijing National Research Center for Information Science and Technology (BNRist); School of Software, Tsinghua University, Beijing, China 2 Department of Electronic Engineering, Tsinghua University, Beijing, China 3 Department of Automation, Tsinghua University; Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, China 4 WMG Data Science, University of Warwick, Coventry, United Kingdom 5 Kwai Seattle AI Lab, Kwai FeDA Lab, Kwai AI platform [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] Abstract Deep Neural Network (DNN) is powerful but computationally expensive and memory intensive, thus impeding its practical usage on resource-constrained front- end devices. DNN pruning is an approach for deep model compression, which aims at eliminating some parameters with tolerable performance degradation. In this paper, we propose a novel momentum-SGD-based optimization method to reduce the network complexity by on-the-fly pruning. Concretely, given a global compression ratio, we categorize all the parameters into two parts at each training iteration which are updated using different rules. In this way, we gradually zero out the redundant parameters, as we update them using only the ordinary weight decay but no gradients derived from the objective function. As a departure from prior methods that require heavy human works to tune the layer-wise sparsity ratios, prune by solving complicated non-differentiable problems or finetune the model after pruning, our method is characterized by 1) global compression that automatically finds the appropriate per-layer sparsity ratios; 2) end-to-end training; 3) no need for a time-consuming re-training process after pruning; and 4) superior capability to find better winning tickets which have won the initialization lottery. -
Recurrent Neural Network
Recurrent Neural Network TINGWU WANG, MACHINE LEARNING GROUP, UNIVERSITY OF TORONTO FOR CSC 2541, SPORT ANALYTICS Contents 1. Why do we need Recurrent Neural Network? 1. What Problems are Normal CNNs good at? 2. What are Sequence Tasks? 3. Ways to Deal with Sequence Labeling. 2. Math in a Vanilla Recurrent Neural Network 1. Vanilla Forward Pass 2. Vanilla Backward Pass 3. Vanilla Bidirectional Pass 4. Training of Vanilla RNN 5. Vanishing and exploding gradient problems 3. From Vanilla to LSTM 1. Definition 2. Forward Pass 3. Backward Pass 4. Miscellaneous 1. More than Language Model 2. GRU 5. Implementing RNN in Tensorflow Part One Why do we need Recurrent Neural Network? 1. What Problems are Normal CNNs good at? 2. What is Sequence Learning? 3. Ways to Deal with Sequence Labeling. 1. What Problems are CNNs normally good at? 1. Image classification as a naive example 1. Input: one image. 2. Output: the probability distribution of classes. 3. You need to provide one guess (output), and to do that you only need to look at one image (input). P(Cat|image) = 0.1 P(Panda|image) = 0.9 2. What is Sequence Learning? 1. Sequence learning is the study of machine learning algorithms designed for sequential data [1]. 2. Language model is one of the most interesting topics that use sequence labeling. 1. Language Translation 1. Understand the meaning of each word, and the relationship between words 2. Input: one sentence in German input = "Ich will stark Steuern senken" 3. Output: one sentence in English output = "I want to cut taxes bigly" (big league?) 2. -
Neural Networks and Backpropagation
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Neural Networks and Backpropagation Neural Net Readings: Murphy -- Matt Gormley Bishop 5 Lecture 20 HTF 11 Mitchell 4 April 3, 2017 1 Reminders • Homework 6: Unsupervised Learning – Release: Wed, Mar. 22 – Due: Mon, Apr. 03 at 11:59pm • Homework 5 (Part II): Peer Review – Expectation: You Release: Wed, Mar. 29 should spend at most 1 – Due: Wed, Apr. 05 at 11:59pm hour on your reviews • Peer Tutoring 2 Neural Networks Outline • Logistic Regression (Recap) – Data, Model, Learning, Prediction • Neural Networks – A Recipe for Machine Learning Last Lecture – Visual Notation for Neural Networks – Example: Logistic Regression Output Surface – 2-Layer Neural Network – 3-Layer Neural Network • Neural Net Architectures – Objective Functions – Activation Functions • Backpropagation – Basic Chain Rule (of calculus) This Lecture – Chain Rule for Arbitrary Computation Graph – Backpropagation Algorithm – Module-based Automatic Differentiation (Autodiff) 3 DECISION BOUNDARY EXAMPLES 4 Example #1: Diagonal Band 5 Example #2: One Pocket 6 Example #3: Four Gaussians 7 Example #4: Two Pockets 8 Example #1: Diagonal Band 9 Example #1: Diagonal Band 10 Example #1: Diagonal Band Error in slides: “layers” should read “number of hidden units” All the neural networks in this section used 1 hidden layer. 11 Example #1: Diagonal Band 12 Example #1: Diagonal Band 13 Example #1: Diagonal Band 14 Example #1: Diagonal Band 15 Example #2: One Pocket -
Arxiv:1907.08798V1 [Astro-Ph.SR] 20 Jul 2019 Keywords: Sun: Coronal Mass Ejections (Cmes) - Techniques: Image Processing
Draft version July 23, 2019 Typeset using LATEX preprint style in AASTeX62 A New Automatic Tool for CME Detection and Tracking with Machine Learning Techniques Pengyu Wang,1 Yan Zhang,1 Li Feng,2 Hanqing Yuan,1 Yuan Gan,1 Shuting Li,2, 3 Lei Lu,2 Beili Ying,2, 3 Weiqun Gan,2 and Hui Li2 1Department of Computer Science and Technology, Nanjing University, 210023 Nanjing, China 2Key Laboratory of Dark Matter and Space Astronomy, Purple Mountain Observatory, Chinese Academy of Sciences, 210034 Nanjing, China 3School of Astronomy and Space Science, University of Science and Technology of China, Hefei, Anhui 230026, China Submitted to ApJS ABSTRACT With the accumulation of big data of CME observations by coronagraphs, automatic detection and tracking of CMEs has proven to be crucial. The excellent performance of convolutional neural network in image classification, object detection and other com- puter vision tasks motivates us to apply it to CME detection and tracking as well. We have developed a new tool for CME Automatic detection and tracking with MachinE Learning (CAMEL) techniques. The system is a three-module pipeline. It is first a su- pervised image classification problem. We solve it by training a neural network LeNet with training labels obtained from an existing CME catalog. Those images containing CME structures are flagged as CME images. Next, to identify the CME region in each CME-flagged image, we use deep descriptor transforming to localize the common object in an image set. A following step is to apply the graph cut technique to finely tune the detected CME region. -
Deep Layer Aggregation
Deep Layer Aggregation Fisher Yu Dequan Wang Evan Shelhamer Trevor Darrell UC Berkeley Abstract Visual recognition requires rich representations that span levels from low to high, scales from small to large, and resolutions from fine to coarse. Even with the depth of fea- Dense Connections Feature Pyramids tures in a convolutional network, a layer in isolation is not enough: compounding and aggregating these representa- tions improves inference of what and where. Architectural + efforts are exploring many dimensions for network back- bones, designing deeper or wider architectures, but how to best aggregate layers and blocks across a network deserves Deep Layer Aggregation further attention. Although skip connections have been in- Figure 1: Deep layer aggregation unifies semantic and spa- corporated to combine layers, these connections have been tial fusion to better capture what and where. Our aggregation “shallow” themselves, and only fuse by simple, one-step op- architectures encompass and extend densely connected net- erations. We augment standard architectures with deeper works and feature pyramid networks with hierarchical and aggregation to better fuse information across layers. Our iterative skip connections that deepen the representation and deep layer aggregation structures iteratively and hierarchi- refine resolution. cally merge the feature hierarchy to make networks with better accuracy and fewer parameters. Experiments across architectures and tasks show that deep layer aggregation improves recognition and resolution compared to existing riers, different blocks or modules have been incorporated branching and merging schemes. to balance and temper these quantities, such as bottlenecks for dimensionality reduction [29, 39, 17] or residual, gated, and concatenative connections for feature and gradient prop- agation [17, 38, 19].