Introduction to Deep Learning in Signal Processing & Communications with MATLAB Dr. Amod Anandkumar Pallavi Kar Application Engineering Group, Mathworks India © 2019 The MathWorks, Inc.1 Different Types of Machine Learning Type of Machine Learning Categories of Algorithms • Output is a choice between classes Classification (True, False) (Red, Blue, Green) Supervised Learning • Output is a real number Regression Develop predictive (temperature, stock prices) model based on both Machine input and output data Learning Unsupervised • No output - find natural groups and Clustering Learning patterns from input data only Discover an internal representation from input data only 2 What is Deep Learning? 3 Deep learning is a type of supervised machine learning in which a model learns to perform classification tasks directly from images, text, or sound. Deep learning is usually implemented using a neural network. The term “deep” refers to the number of layers in the network—the more layers, the deeper the network. 4 Why is Deep Learning So Popular Now? Human Accuracy Source: ILSVRC Top-5 Error on ImageNet 5 Vision applications have been driving the progress in deep learning producing surprisingly accurate systems 6 Deep Learning success enabled by: • Labeled public datasets • Progress in GPU for acceleration AlexNet VGG-16 ResNet-50 ONNX Converter • World-class models and PRETRAINED PRETRAINED PRETRAINED MODEL MODEL CONVERTER MODEL MODEL TensorFlow- connected community Caffe GoogLeNet IMPORTER PRETRAINED Keras Inception-v3 MODEL IMPORTER MODELS 7 Deep Learning is Versatile MATLAB Examples Available Here 8 Many Network Architectures for Deep Learning Directed Acyclic Series Network Recurrent Network Graph Network and more (GAN, DQN,…) AlexNet ResNet LSTM YOLO R-CNN 9 Convolutional Neural Networks Edges Shapes Objects A lot of data is required Convolution layers Fully-connected layers 10 Deep Learning Inference in 4 Lines of Code • >> net = alexnet; • >> I = imread('peacock.jpg') • >> I1 = imresize(I,[227 227]); • >> classify(net,I1) • ans = • categorical • peacock 11 Understanding network behavior using visualizations Filters … Deep Dream Activations ▪ Custom visualizations – Example: Class Activation Maps (See blog post) 12 Visualization Technique – Deep Dream deepDreamImage(... net, ‘fc8', channel, 'NumIterations', 50, ... 'PyramidLevels', 4,... 'PyramidScale', 1.25); Synthesizes images that strongly activate a channel in a particular layer Example Available Here 13 What is Training? Images Network Training Labels Trained Deep Neural Network Large Data Set During training, neural network architectures learn features directly from the data without the need for manual feature extraction 14 What Happens During Training? AlexNet Example Training Data Labels Layer weights are learned during training 15 Visualize Network Weights During Training AlexNet Example Training Data Labels First Convolution Layer 16 Visualize Features Learned During Training AlexNet Example Sample Training Data Features Learned by Network 17 Visualize Features Learned During Training AlexNet Example Sample Training Data Features Learned by Network 18 Deep Learning Workflow ACCESS AND EXPLORE LABEL AND PREPROCESS DEVELOP PREDICTIVE INTEGRATE MODELS WITH DATA DATA MODELS SYSTEMS Files Data Augmentation/ Hardware-Accelerated Desktop Apps Transformation Training Databases Labeling Automation Hyperparameter Tuning Enterprise Scale Systems Sensors Import Reference Network Visualization Embedded Devices and Models Hardware 19 Deep Learning Challenges Data ▪ Handling large amounts of data ▪ Labeling thousands of signals, images & videos ▪ Transforming, generating, and augmenting data (for different domains) Training and Testing Deep Neural Networks ▪ Accessing reference models from research ▪ Understanding network behaviour ▪ Optimizing hyperparameters ▪ Training takes hours-days Rapid and Optimized Deployment ▪ Desktop, web, cloud, and embedded hardware 20 Working with Signal Data Signals Sequence Fully Connected Labels Layers Layers Trained Model Signals Transform Train Convolutional Trained Model Labels Network (CNN) 21 Long Short Term Memory Networks from RNNs ▪ Recurrent Neural Network that carries a memory cell throughout the process ▪ Sequence Problems – Long term dependency does not work well with RNNs c0 C1 Ct 22 RNN to LSTM Cell state Input gate Forget gate Update Output gate 23 Example: Speaker Gender Recognition Using Deep Learning LSTM Network for Audio based Speaker Classification audioDatastore 1. Train − Male − Female Training Data Training Time-varying Mozilla Common features: - MFCC Voice Dataset - Pitch - ... 2. Predict − Male − Female Test Data Test 24 Some audio and speech applications following CNN workflows 27 https://www.isca-speech.org/archive/interspeech_2015/papers/i15_1478.pdf 28 Solution2: Speech Command Recognition with Deep Learning(MATLAB) Integrated Analytic Preprocessed Models and Algorithms Systems Raw Data Data . 29 43 44 45 46 Data augmentation allows training more advanced networks and generating more robust models Original Simulate: Dataset - lower voice - varying level of environmental interference - recording hardware effects - distortions due to use of different microphones - different speaker vs microphone positions or room sizes in enclosed environments - ... 50 % Cycle continuously and automatically through files in datastore mfccFile = zeros(numel(ads.Files),numMfcc) while hasdata(ads) [data,info] = read(ads); Serial % do something with data end % Partition and explicit parfor parallel processing pattern N = numpartitions(ads); parfor index=1:N subads = partition(ads,N,index); while hasdata(subads) data = read(subads); % do something with data end end Parallel % Tall array and "implicit" parallel processing pattern T = tall(ads); dcOffsets = cellfun(@(x) mean(x), T,'UniformOutput',false); gather(dcOffsets); 51 Package Speech Recognition App 52 53 BREAK 54 Deep Learning Challenges Data Not a deep learning expert ✓ Handling large amounts of data ▪ Labeling thousands of signals, images & videos ▪ Transforming, generating, and augmenting data (for different domains) Training and Testing Deep Neural Networks ✓ Understanding network behavior ▪ Accessing reference models from research ▪ Optimizing hyperparameters ▪ Training takes hours-days Rapid and Optimized Deployment ▪ Desktop, web, cloud, and embedded hardware 55 BREAK 56 57 Speech 2 text intro 58 59 60 61 62 Audio Labeler ▪ Work on collections of recordings or record new audio directly within the app ▪ Navigate dataset and playback interactively ▪ Define and apply labels to – Entire files – Regions within files ▪ Import and export audio folders, label definitions and datastores 63 Apps for Data Labeling 64 Label Images Using Image Labeler App 65 Accelerate Labeling With Automation Algorithms Learn More 66 Perform Bootstrapping to Label Large Datasets Images Videos Train Network Labeling Apps Ground Truth Improve More Network Ground Truth Propose Labels Automation Algorithm 67 Example – Semantic Segmentation Available Here ▪ Classify pixels into 11 classes – Sky, Building, Pole, Road, Pavement, Tree, SignSymbol, Fence, Car, Pedestrian, Bicyclist ▪ CamVid dataset Brostow, Gabriel J., Julien Fauqueur, and Roberto Cipolla. "Semantic object classes in video: A high-definition ground truth database." Pattern Recognition Letters Vol 30, Issue 2, 2009, pp 88-97. 68 Example: Singing Voice Separation ▪ Source separation ▪ Based on U-Net architecture 69 Synthetically generating labeled data 70 Modulation Classification with Deep Learning ▪ Generate synthetic modulated signals ▪ Apply channel impairments ▪ Train a CNN to classify modulation types 71 Deep Learning Challenges Data ✓ Handling large amounts of data ✓ Labeling thousands of signals, images & videos ✓ Transforming, generating, and augmenting data (for different domains) Training and Testing Deep Neural Networks ✓ Understanding network behavior ▪ Accessing reference models from research ▪ Optimizing hyperparameters ▪ Training takes hours-days Rapid and Optimized Deployment ▪ Desktop, web, cloud, and embedded hardware 72 Transfer Learning 8 lines of MATLAB Code %% Create a datastore Load Training Data 1 imds = imageDatastore(‘Data’,... 'IncludeSubfolders',true,'LabelSource','foldernames'); 2 num_classes = numel(unique(imds.Labels)); %% Load Referece Network Load Pre-Trained Network 3 net = alexnet; 4 layers = net.Layers %% Replace Final Layers 5 layers(end-2) = Replace Final Layers fullyConnectedLayer(num_classes,'Name',[‘fc5']); 6 layers(end) = classificationLayer('Name','classOut'); %% Set Training Options & Train Network 7 trainOpts = trainingOptions('sgdm’,... Train Network on 'MiniBatchSize', 64); New Data 8 net = trainNetwork(imds, layers, trainOpts); 73 Tune Hyperparameters to Improve Training Many hyperparameters ▪ depth, layers, solver options, learning rates, regularization, … Techniques ▪ Parameter sweep ▪ Bayesian optimization 74 Keras-Tensorflow Importer 75 Model Exchange and Co-execution MATLAB Python Model Exchange Co-execution 76 Model Exchange With Deep Learning Frameworks TensorFlow- PyTorch Caffe2 TensorFlow Keras MXNet ONNX MATLAB Cognitive Core ML Caffe Chainer Toolkit ONNX = Open Neural Network Exchange Format 77 Interoperate With Deep Learning Frameworks – Use Cases Deployment PyTorch Caffe2 TensorFlow Export Augmentation MXNet ONNX MATLAB and Optimization Import Visualization Core ML Cognitive Chainer Toolkit and Debugging ONNX = Open Neural Network Exchange Format 78 Model Exchange With Deep Learning Frameworks