Public Datasets & Deep Learning Model Repositories
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
Backpropagation and Deep Learning in the Brain
Backpropagation and Deep Learning in the Brain Simons Institute -- Computational Theories of the Brain 2018 Timothy Lillicrap DeepMind, UCL With: Sergey Bartunov, Adam Santoro, Jordan Guerguiev, Blake Richards, Luke Marris, Daniel Cownden, Colin Akerman, Douglas Tweed, Geoffrey Hinton The “credit assignment” problem The solution in artificial networks: backprop Credit assignment by backprop works well in practice and shows up in virtually all of the state-of-the-art supervised, unsupervised, and reinforcement learning algorithms. Why Isn’t Backprop “Biologically Plausible”? Why Isn’t Backprop “Biologically Plausible”? Neuroscience Evidence for Backprop in the Brain? A spectrum of credit assignment algorithms: A spectrum of credit assignment algorithms: A spectrum of credit assignment algorithms: How to convince a neuroscientist that the cortex is learning via [something like] backprop - To convince a machine learning researcher, an appeal to variance in gradient estimates might be enough. - But this is rarely enough to convince a neuroscientist. - So what lines of argument help? How to convince a neuroscientist that the cortex is learning via [something like] backprop - What do I mean by “something like backprop”?: - That learning is achieved across multiple layers by sending information from neurons closer to the output back to “earlier” layers to help compute their synaptic updates. How to convince a neuroscientist that the cortex is learning via [something like] backprop 1. Feedback connections in cortex are ubiquitous and modify the -
Automated Elastic Pipelining for Distributed Training of Transformers
PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers Chaoyang He 1 Shen Li 2 Mahdi Soltanolkotabi 1 Salman Avestimehr 1 Abstract the-art convolutional networks ResNet-152 (He et al., 2016) and EfficientNet (Tan & Le, 2019). To tackle the growth in The size of Transformer models is growing at an model sizes, researchers have proposed various distributed unprecedented rate. It has taken less than one training techniques, including parameter servers (Li et al., year to reach trillion-level parameters since the 2014; Jiang et al., 2020; Kim et al., 2019), pipeline paral- release of GPT-3 (175B). Training such models lel (Huang et al., 2019; Park et al., 2020; Narayanan et al., requires both substantial engineering efforts and 2019), intra-layer parallel (Lepikhin et al., 2020; Shazeer enormous computing resources, which are luxu- et al., 2018; Shoeybi et al., 2019), and zero redundancy data ries most research teams cannot afford. In this parallel (Rajbhandari et al., 2019). paper, we propose PipeTransformer, which leverages automated elastic pipelining for effi- T0 (0% trained) T1 (35% trained) T2 (75% trained) T3 (100% trained) cient distributed training of Transformer models. In PipeTransformer, we design an adaptive on the fly freeze algorithm that can identify and freeze some layers gradually during training, and an elastic pipelining system that can dynamically Layer (end of training) Layer (end of training) Layer (end of training) Layer (end of training) Similarity score allocate resources to train the remaining active layers. More specifically, PipeTransformer automatically excludes frozen layers from the Figure 1. Interpretable Freeze Training: DNNs converge bottom pipeline, packs active layers into fewer GPUs, up (Results on CIFAR10 using ResNet). -
Convolutional Neural Network Transfer Learning for Robust Face
Convolutional Neural Network Transfer Learning for Robust Face Recognition in NAO Humanoid Robot Daniel Bussey1, Alex Glandon2, Lasitha Vidyaratne2, Mahbubul Alam2, Khan Iftekharuddin2 1Embry-Riddle Aeronautical University, 2Old Dominion University Background Research Approach Results • Artificial Neural Networks • Retrain AlexNet on the CASIA-WebFace dataset to configure the neural • VGG-Face Shows better results in every performance benchmark measured • Computing systems whose model architecture is inspired by biological networf for face recognition tasks. compared to AlexNet, although AlexNet is able to extract features from an neural networks [1]. image 800% faster than VGG-Face • Capable of improving task performance without task-specific • Acquire an input image using NAO’s camera or a high resolution camera to programming. run through the convolutional neural network. • Resolution of the input image does not have a statistically significant impact • Show excellent performance at classification based tasks such as face on the performance of VGG-Face and AlexNet. recognition [2], text recognition [3], and natural language processing • Extract the features of the input image using the neural networks AlexNet [4]. and VGG-Face • AlexNet’s performance decreases with respect to distance from the camera where VGG-Face shows no performance loss. • Deep Learning • Compare the features of the input image to the features of each image in the • A subfield of machine learning that focuses on algorithms inspired by people database. • Both frameworks show excellent performance when eliminating false the function and structure of the brain called artificial neural networks. positives. The method used to allow computers to learn through a process called • Determine whether a match is detected or if the input image is a photo of a training. -
Machine Learning Is Driving an Innovation Wave in Saas Software
Machine Learning is Driving an Innovation Wave in SaaS Software By Jeff Houston, CFA [email protected] | 512.364.2258 SaaS software vendors are enhancing their soluti ons with machine learning (ML) algorithms. ML is the ability of a computer to either automate or recommend appropriate actions by applying probability to data with a feedback loop that enables learning. We view ML as a subset of artificial intelligence (AI), which represents a broad collection of tools that include/leverage ML, such as natural language processing (NLP), self-driving cars, and robotics as well as some that are tangent to ML, such as logical rule-based algorithms. Although there have been multiple “AI Winters” since the 1950s where hype cycles were followed by a dearth of funding, ML is now an enduring innovation driver, in our opinion, due in part to the increasing ubiquity of affordable cloud-based processing power, data storage—we expect the pace of breakthroughs to accelerate. automation, customer support, marketing, and human resources) through both organic and M&A These innovations will make it easier for companies initiatives. As for vertical-specific solutions, we think to benefit beyond what is available with traditional that a new crop of vendors will embrace ML and business intelligence (BI) solutions, like IBM/ achieve billion-dollar valuations. VCs are making big Cognos and Tableau, that simply describe what bets that these hypotheses will come to fruition as happened in the past. New ML solutions are demonstrated by the $5B they poured into 550 amplifying intelligence while enabling consistent and startups using AI as a core part of their solution in better-informed reasoning, behaving as thought 20162. -
Tiny Imagenet Challenge
Tiny ImageNet Challenge Jiayu Wu Qixiang Zhang Guoxi Xu Stanford University Stanford University Stanford University [email protected] [email protected] [email protected] Abstract Our best model, a fine-tuned Inception-ResNet, achieves a top-1 error rate of 43.10% on test dataset. Moreover, we We present image classification systems using Residual implemented an object localization network based on a Network(ResNet), Inception-Resnet and Very Deep Convo- RNN with LSTM [7] cells, which achieves precise results. lutional Networks(VGGNet) architectures. We apply data In the Experiments and Evaluations section, We will augmentation, dropout and other regularization techniques present thorough analysis on the results, including per-class to prevent over-fitting of our models. What’s more, we error analysis, intermediate output distribution, the impact present error analysis based on per-class accuracy. We of initialization, etc. also explore impact of initialization methods, weight decay and network depth on system performance. Moreover, visu- 2. Related Work alization of intermediate outputs and convolutional filters are shown. Besides, we complete an extra object localiza- Deep convolutional neural networks have enabled the tion system base upon a combination of Recurrent Neural field of image recognition to advance in an unprecedented Network(RNN) and Long Short Term Memroy(LSTM) units. pace over the past decade. Our best classification model achieves a top-1 test error [10] introduces AlexNet, which has 60 million param- rate of 43.10% on the Tiny ImageNet dataset, and our best eters and 650,000 neurons. The model consists of five localization model can localize with high accuracy more convolutional layers, and some of them are followed by than 1 objects, given training images with 1 object labeled. -
Deep Learning Is Robust to Massive Label Noise
Deep Learning is Robust to Massive Label Noise David Rolnick * 1 Andreas Veit * 2 Serge Belongie 2 Nir Shavit 3 Abstract Thus, annotation can be expensive and, for tasks requiring expert knowledge, may simply be unattainable at scale. Deep neural networks trained on large supervised datasets have led to impressive results in image To address this limitation, other training paradigms have classification and other tasks. However, well- been investigated to alleviate the need for expensive an- annotated datasets can be time-consuming and notations, such as unsupervised learning (Le, 2013), self- expensive to collect, lending increased interest to supervised learning (Pinto et al., 2016; Wang & Gupta, larger but noisy datasets that are more easily ob- 2015) and learning from noisy annotations (Joulin et al., tained. In this paper, we show that deep neural net- 2016; Natarajan et al., 2013; Veit et al., 2017). Very large works are capable of generalizing from training datasets (e.g., Krasin et al.(2016); Thomee et al.(2016)) data for which true labels are massively outnum- can often be obtained, for example from web sources, with bered by incorrect labels. We demonstrate remark- partial or unreliable annotation. This can allow neural net- ably high test performance after training on cor- works to be trained on a much wider variety of tasks or rupted data from MNIST, CIFAR, and ImageNet. classes and with less manual effort. The good performance For example, on MNIST we obtain test accuracy obtained from these large, noisy datasets indicates that deep above 90 percent even after each clean training learning approaches can tolerate modest amounts of noise example has been diluted with 100 randomly- in the training set. -
Market Update January 2020 a Machine Learning & Artificial Intelligence Market Update
PRIVATE & CONFIDENTIAL Market Update January 2020 A Machine Learning & Artificial Intelligence Market Update B Canaccord Genuity Overview / Update Artificial Intelligence Machine Learning f (x) Deep Learning Page 1 Driven by your success. Machine Learning (“ML”) and Artificial Intelligence (“AI”) continue to generate strong levels of attention and excitement in the marketplace, based on the promise of self-correcting algorithms driving increased intelligence and automation across a number of mission critical applications and use cases When ML & AI were first introduced as concepts that would impact the IT landscape, most companies in the sector were limited to collections of data scientists or technologies in search of use cases – today there are defined categories emerging and companies with real traction in ML/AI, as well as a growing set of tangible use cases While there is real innovation and traction occurring in ML/AI, in some cases it is still difficult to understand where certain companies truly play in the ML ecosystem and the unique value that each brings to the table – this presentation aims to provide a framework to understand the ML landscape First we look to define and better understand ML/AI technology, both the underlying algorithms as well as data science platforms, operational frameworks and advanced analytics solutions which leverage and / or optimize core ML/AI technologies – these are also described as “AI Infrastructure” From a category perspective, we focus primarily on horizontal platforms which can provide data -
The History Began from Alexnet: a Comprehensive Survey on Deep Learning Approaches
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 1 The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches Md Zahangir Alom1, Tarek M. Taha1, Chris Yakopcic1, Stefan Westberg1, Paheding Sidike2, Mst Shamima Nasrin1, Brian C Van Essen3, Abdul A S. Awwal3, and Vijayan K. Asari1 Abstract—In recent years, deep learning has garnered I. INTRODUCTION tremendous success in a variety of application domains. This new ince the 1950s, a small subset of Artificial Intelligence (AI), field of machine learning has been growing rapidly, and has been applied to most traditional application domains, as well as some S often called Machine Learning (ML), has revolutionized new areas that present more opportunities. Different methods several fields in the last few decades. Neural Networks have been proposed based on different categories of learning, (NN) are a subfield of ML, and it was this subfield that spawned including supervised, semi-supervised, and un-supervised Deep Learning (DL). Since its inception DL has been creating learning. Experimental results show state-of-the-art performance ever larger disruptions, showing outstanding success in almost using deep learning when compared to traditional machine every application domain. Fig. 1 shows, the taxonomy of AI. learning approaches in the fields of image processing, computer DL (using either deep architecture of learning or hierarchical vision, speech recognition, machine translation, art, medical learning approaches) is a class of ML developed largely from imaging, medical information processing, robotics and control, 2006 onward. Learning is a procedure consisting of estimating bio-informatics, natural language processing (NLP), cybersecurity, and many others. -
Kernel Descriptors for Visual Recognition
Kernel Descriptors for Visual Recognition Liefeng Bo University of Washington Seattle WA 98195, USA Xiaofeng Ren Dieter Fox Intel Labs Seattle University of Washington & Intel Labs Seattle Seattle WA 98105, USA Seattle WA 98195 & 98105, USA Abstract The design of low-level image features is critical for computer vision algorithms. Orientation histograms, such as those in SIFT [16] and HOG [3], are the most successful and popular features for visual object and scene recognition. We high- light the kernel view of orientation histograms, and show that they are equivalent to a certain type of match kernels over image patches. This novel view allows us to design a family of kernel descriptors which provide a unified and princi- pled framework to turn pixel attributes (gradient, color, local binary pattern, etc.) into compact patch-level features. In particular, we introduce three types of match kernels to measure similarities between image patches, and construct compact low-dimensional kernel descriptors from these match kernels using kernel princi- pal component analysis (KPCA) [23]. Kernel descriptors are easy to design and can turn any type of pixel attribute into patch-level features. They outperform carefully tuned and sophisticated features including SIFT and deep belief net- works. We report superior performance on standard image classification bench- marks: Scene-15, Caltech-101, CIFAR10 and CIFAR10-ImageNet. 1 Introduction Image representation (features) is arguably the most fundamental task in computer vision. The problem is highly challenging because images exhibit high variations, are highly structured, and lie in high dimensional spaces. In the past ten years, a large number of low-level features over images have been proposed. -
Shikhar-Cv.Pdf
SHIKHAR SHARMA [email protected] Senior Research SDE, Microsoft Research http://www.shikharsharma.com/ RESEARCH INTERESTS • Deep Learning • Machine Learning • Recurrent Neural Networks • Adversarial Learning • Computer Vision • Task-oriented Dialogue RESEARCH STATEMENT SUMMARY My long-term research interest lies in advancing the capabilities of current artificial intelligence systems. I am highly interested in neural network architectures augmented with memory and attention mechanisms. My recent research deals with image synthesis and visually-grounded multi-modal dialogue systems, which stand to gain significant advantage from using memory and attention similar to how we as humans do. Prior to this, I have worked on visual attention models for action recognition and video description. I have broad interests in deep learning research in natural language processing and computer vision, which I believe are important research areas on the path to true artificial intelligence. EDUCATION University of Toronto Aug'14 - Feb'16 M.Sc. in Computer Science with the Deep Learning and Machine Learning group GPA: 3.83/4 • Thesis title: Action Recognition and Video Description using Visual Attention • Thesis supervisor: Prof. Ruslan Salakhutdinov Indian Institute of Technology (IIT), Kanpur Jul'10 - Jun'14 B.Tech. in Computer Science GPA: 9.3/10 • Thesis title: Speech Recognition using Deep Belief Networks and Hidden Markov Models • Thesis supervisor: Prof. Harish Karnick PUBLICATIONS yequal contribution Theses - Shikhar Sharma. 2016. \Action Recognition and Video Description using Visual Attention." Masters Thesis, University of Toronto Journal papers - Ashesh Jain, Shikhar Sharma, Thorsten Joachims, Ashutosh Saxena. 2015. \Learning preferences for ma- nipulation tasks from online coactive feedback." The International Journal of Robotics Research (IJRR), 34, 1296-1313 Conference and Workshop papers - Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz, Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahou, Yoshua Bengio, Graham W. -
Profiles in Innovation: Artificial Intelligence
EQUITY RESEARCH | November 14, 2016 Artificial intelligence is the apex technology of the information era. In the latest in our Profiles in Innovation Heath P. Terry, CFA series, we examine how (212) 357-1849 advances in machine [email protected] learning and deep learning Goldman, Sachs & Co. have combined with more Jesse Hulsing powerful computing and an (415) 249-7464 ever-expanding pool of data [email protected] to bring AI within reach for Goldman, Sachs & Co. companies across Mark Grant industries. The development (212) 357-4475 [email protected] of AI-as-a-service has the Goldman, Sachs & Co. potential to open new markets and disrupt the Daniel Powell (917) 343-4120 playing field in cloud [email protected] computing. We believe the Goldman, Sachs & Co. ability to leverage AI will Piyush Mubayi become a defining attribute (852) 2978-1677 of competitive advantage [email protected] for companies in coming Goldman Sachs (Asia) L.L.C. years and will usher in a Waqar Syed resurgence in productivity. (212) 357-1804 [email protected] Goldman, Sachs & Co. PROFILESIN INNOVATION Artificial Intelligence AI, Machine Learning and Data Fuel the Future of Productivity Goldman Sachs does and seeks to do business with companies covered in its research reports. As a result, investors should be aware that the firm may have a conflict of interest that could affect the objectivity of this report. Investors should consider this report as only a single factor in making their investment decision. For Reg AC certification and other important disclosures, see the Disclosure Appendix, or go to www.gs.com/research/hedge.html. -
Using Neural Networks for Image Classification Tim Kang San Jose State University
View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by SJSU ScholarWorks San Jose State University SJSU ScholarWorks Master's Projects Master's Theses and Graduate Research Spring 5-18-2015 Using Neural Networks for Image Classification Tim Kang San Jose State University Follow this and additional works at: https://scholarworks.sjsu.edu/etd_projects Part of the Artificial Intelligence and Robotics Commons Recommended Citation Kang, Tim, "Using Neural Networks for Image Classification" (2015). Master's Projects. 395. DOI: https://doi.org/10.31979/etd.ssss-za3p https://scholarworks.sjsu.edu/etd_projects/395 This Master's Project is brought to you for free and open access by the Master's Theses and Graduate Research at SJSU ScholarWorks. It has been accepted for inclusion in Master's Projects by an authorized administrator of SJSU ScholarWorks. For more information, please contact [email protected]. Using Neural Networks for Image Classification San Jose State University, CS298 Spring 2015 Author: Tim Kang ([email protected]), San Jose State University Advisor: Robert Chun ([email protected]), San Jose State University Committee: Thomas Austin ([email protected]), San Jose State University Committee: Thomas Howell ([email protected]), San Jose State University Abstract This paper will focus on applying neural network machine learning methods to images for the purpose of automatic detection and classification. The main advantage of using neural network methods in this project is its adeptness at fitting nonlinear data and its ability to work as an unsupervised algorithm. The algorithms will be run on common, publically available datasets, namely the MNIST and CIFAR10, so that our results will be easily reproducible.