Introduction to in Signal Processing & Communications with MATLAB

Dr. Amod Anandkumar Pallavi Kar Application Engineering Group, Mathworks India

© 2019 The MathWorks, Inc.1 Different Types of

Type of Machine Learning Categories of Algorithms

• Output is a choice between classes Classification (True, False) (Red, Blue, Green) • Output is a real number Regression Develop predictive (temperature, stock prices) model based on both Machine input and output data Learning

Unsupervised • No output - find natural groups and Clustering Learning patterns from input data only

Discover an internal representation from input data only

2 What is Deep Learning?

3 Deep learning is a type of supervised machine learning in which a model learns to perform classification tasks directly from images, text, or sound.

Deep learning is usually implemented using a neural network.

The term “deep” refers to the number of layers in the network—the more layers, the deeper the network.

4 Why is Deep Learning So Popular Now?

Human Accuracy

Source: ILSVRC Top-5 Error on ImageNet 5 Vision applications have been driving the progress in deep learning producing surprisingly accurate systems

6 Deep Learning success enabled by:

• Labeled public datasets

• Progress in GPU for acceleration

AlexNet VGG-16 ResNet-50 ONNX Converter • World-class models and PRETRAINED PRETRAINED PRETRAINED MODEL MODEL CONVERTER MODEL MODEL TensorFlow- connected community Caffe GoogLeNet IMPORTER PRETRAINED Keras Inception-v3 MODEL IMPORTER MODELS 7 Deep Learning is Versatile MATLAB Examples Available Here

8 Many Network Architectures for Deep Learning

Directed Acyclic Series Network Recurrent Network Graph Network

and more (GAN, DQN,…)

AlexNet ResNet LSTM YOLO R-CNN

9 Convolutional Neural Networks Edges Shapes Objects

A lot of data is required

Convolution layers Fully-connected layers 10 Deep Learning Inference in 4 Lines of Code

• >> net = alexnet; • >> I = imread('peacock.jpg') • >> I1 = imresize(I,[227 227]); • >> classify(net,I1) • ans = • categorical • peacock

11 Understanding network behavior using visualizations

Filters …

Deep Dream Activations

▪ Custom visualizations – Example: Class Activation Maps (See blog post)

12 Visualization Technique – Deep Dream

deepDreamImage(... net, ‘fc8', channel, 'NumIterations', 50, ... 'PyramidLevels', 4,... 'PyramidScale', 1.25);

Synthesizes images that strongly activate a channel in a particular layer

Example Available Here 13 What is Training?

Images Network Training Labels Trained Deep Neural Network Large Data Set

During training, neural network architectures learn features directly from the data without the need for manual feature extraction

14 What Happens During Training? AlexNet Example

Training Data Labels

Layer weights are learned during training

15 Visualize Network Weights During Training AlexNet Example

Training Data Labels First Convolution Layer

16 Visualize Features Learned During Training AlexNet Example

Sample Training Data Features Learned by Network 17 Visualize Features Learned During Training AlexNet Example

Sample Training Data Features Learned by Network 18 Deep Learning Workflow

ACCESS AND EXPLORE LABEL AND PREPROCESS DEVELOP PREDICTIVE INTEGRATE MODELS WITH DATA DATA MODELS SYSTEMS

Files Data Augmentation/ Hardware-Accelerated Desktop Apps Transformation Training

Databases Labeling Automation Hyperparameter Tuning Enterprise Scale Systems

Sensors Import Reference Network Visualization Embedded Devices and Models Hardware

19 Deep Learning Challenges

Data ▪ Handling large amounts of data ▪ Labeling thousands of signals, images & videos ▪ Transforming, generating, and augmenting data (for different domains) Training and Testing Deep Neural Networks ▪ Accessing reference models from research ▪ Understanding network behaviour ▪ Optimizing hyperparameters ▪ Training takes hours-days Rapid and Optimized Deployment ▪ Desktop, web, cloud, and embedded hardware

20 Working with Signal Data

Signals Sequence Fully Connected Labels Layers Layers Trained Model

Signals Transform Train Convolutional Trained Model Labels Network (CNN) 21 Long Short Term Memory Networks from RNNs

▪ that carries a memory cell throughout the process ▪ Sequence Problems – Long term dependency does not work well with RNNs

c0 C1 Ct

22 RNN to LSTM

Cell state

Input gate Forget gate

Update

Output gate

23 Example: Speaker Gender Recognition Using Deep Learning LSTM Network for Audio based Speaker Classification

audioDatastore 1. Train

− Male

− Female Training Data Training Time-varying Mozilla Common features: - MFCC Voice Dataset - Pitch - ...

2. Predict

− Male

− Female Test Data Test

24 Some audio and speech applications following CNN workflows

27 https://www.isca-speech.org/archive/interspeech_2015/papers/i15_1478.pdf

28 Solution2: Speech Command Recognition with Deep Learning(MATLAB)

Integrated Analytic Preprocessed Models and Algorithms Systems Raw Data Data

. . .

29 43 44 45 46 Data augmentation allows training more advanced networks and generating more robust models

Original Simulate: Dataset - lower voice - varying level of environmental interference - recording hardware effects - distortions due to use of different microphones - different speaker vs microphone positions or room sizes in enclosed environments - ... 50 % Cycle continuously and automatically through files in datastore mfccFile = zeros(numel(ads.Files),numMfcc) while hasdata(ads) [data,info] = read(ads);

Serial % do something with data end

% Partition and explicit parfor parallel processing pattern N = numpartitions(ads);

parfor index=1:N subads = partition(ads,N,index); while hasdata(subads) data = read(subads); % do something with data end

end Parallel

% Tall array and "implicit" parallel processing pattern T = tall(ads);

dcOffsets = cellfun(@(x) mean(x), T,'UniformOutput',false); gather(dcOffsets); 51 Package Speech Recognition App

52 53 BREAK

54 Deep Learning Challenges

Data Not a deep learning expert ✓ Handling large amounts of data ▪ Labeling thousands of signals, images & videos ▪ Transforming, generating, and augmenting data (for different domains) Training and Testing Deep Neural Networks ✓ Understanding network behavior ▪ Accessing reference models from research ▪ Optimizing hyperparameters ▪ Training takes hours-days Rapid and Optimized Deployment ▪ Desktop, web, cloud, and embedded hardware

55 BREAK

56 57 Speech 2 text intro

58 59 60 61 62 Audio Labeler

▪ Work on collections of recordings or record new audio directly within the app

▪ Navigate dataset and playback interactively

▪ Define and apply labels to – Entire files – Regions within files

▪ Import and export audio folders, label definitions and datastores

63 Apps for Data Labeling

64 Label Images Using Image Labeler App

65 Accelerate Labeling With Automation Algorithms

Learn More

66 Perform Bootstrapping to Label Large Datasets

Images Videos Train Network Labeling Apps Ground Truth

Improve More Network Ground Truth Propose Labels

Automation Algorithm

67 Example – Semantic Segmentation Available Here

▪ Classify pixels into 11 classes – Sky, Building, Pole, Road, Pavement, Tree, SignSymbol, Fence, Car, Pedestrian, Bicyclist

▪ CamVid dataset Brostow, Gabriel J., Julien Fauqueur, and Roberto Cipolla. "Semantic object classes in video: A high-definition ground truth database." Letters Vol 30, Issue 2, 2009, pp 88-97. 68 Example: Singing Voice Separation

▪ Source separation ▪ Based on U-Net architecture

69 Synthetically generating labeled data

70 Modulation Classification with Deep Learning

▪ Generate synthetic modulated signals ▪ Apply channel impairments ▪ Train a CNN to classify modulation types

71 Deep Learning Challenges

Data ✓ Handling large amounts of data ✓ Labeling thousands of signals, images & videos ✓ Transforming, generating, and augmenting data (for different domains) Training and Testing Deep Neural Networks ✓ Understanding network behavior ▪ Accessing reference models from research ▪ Optimizing hyperparameters ▪ Training takes hours-days Rapid and Optimized Deployment ▪ Desktop, web, cloud, and embedded hardware

72 Transfer Learning 8 lines of MATLAB Code %% Create a datastore Load Training Data 1 imds = imageDatastore(‘Data’,... 'IncludeSubfolders',true,'LabelSource','foldernames'); 2 num_classes = numel(unique(imds.Labels)); %% Load Referece Network Load Pre-Trained Network 3 net = alexnet; 4 layers = net.Layers %% Replace Final Layers 5 layers(end-2) = Replace Final Layers fullyConnectedLayer(num_classes,'Name',[‘fc5']); 6 layers(end) = classificationLayer('Name','classOut'); %% Set Training Options & Train Network 7 trainOpts = trainingOptions('sgdm’,... Train Network on 'MiniBatchSize', 64); New Data 8 net = trainNetwork(imds, layers, trainOpts);

73 Tune Hyperparameters to Improve Training

Many hyperparameters ▪ depth, layers, solver options, learning rates, regularization, …

Techniques ▪ Parameter sweep ▪ Bayesian optimization

74 Keras-Tensorflow Importer

75 Model Exchange and Co-execution

MATLAB Python

Model Exchange Co-execution

76 Model Exchange With Deep Learning Frameworks

TensorFlow- PyTorch Caffe2 TensorFlow Keras

MXNet ONNX MATLAB

Cognitive Core ML Caffe Chainer Toolkit

ONNX = Open Neural Network Exchange Format

77 Interoperate With Deep Learning Frameworks – Use Cases

Deployment PyTorch Caffe2 TensorFlow

Export

Augmentation MXNet ONNX MATLAB and Optimization Import

Visualization Core ML Cognitive Chainer Toolkit and Debugging

ONNX = Open Neural Network Exchange Format 78 Model Exchange With Deep Learning Frameworks

Caffe Model Importer ▪ importCaffeLayers

▪ importCaffeNetwork TensorFlow-Keras Model Importer ▪ importKerasLayers

▪ importKerasNetwork ONNX Converter ▪ importONNXNetwork ONNX Converter ▪ exportONNXNetwork

79 Model Exchange and Co-execution

MATLAB Python

Model Exchange Co-execution

80 MATLAB-Python Co-Execution – The ‘How’ Call Python file from Call TensorFlow commands from MATLAB MATLAB

TF_code.py

81 MATLAB-Python Co-Execution – Method A

1. Copy the code into a .PY file 2. Wrap entry point in a function import tensorflow as tf from tf import keras

def myTFCode(): for x in y: a.foo()

def foo(x): TF_code.py return x + 1

3. Add module to Python path and then call from MATLAB via: py.myModule.myTFCode();

82 Deep Learning on CPU, GPU, Multi-GPU and Clusters H OW TO TARGET ?

Single Single CPU CPU Single GPU

Single CPU, Multiple GPUs

On-prem server with Cloud GPUs GPUs (AWS) 83 MATLAB Containers for NVIDIA GPU Cloud & DGX

84 Deep Learning Challenges

Data ✓ Handling large amounts of data ✓ Labeling thousands of signals, images & videos ✓ Transforming, generating, and augmenting data (for different domains) Training and Testing Deep Neural Networks ✓ Accessing reference models from research ✓ Understanding network behaviour ✓ Optimizing hyperparameters ✓ Training takes hours-days Rapid and Optimized Deployment ▪ Desktop, web, cloud, and embedded hardware

85 Deploying Deep Learning Application

Embedded Hardware Desktop, Web, Cloud Application logic

Code Application Generation Deployment

86 MATLAB Production Server is an application server that publishes MATLAB code as APIs that can be called by other applications

Analytics Development

MATLAB MATLAB Compiler SDK Deploy

Package Code / test Access

Enterprise Application Integrate MATLAB Production Server Data sources / Worker processes applications < > Request Broker Mobile / Web Application

rd Scale and secure 3 party dashboard

87 MATLAB support for Cloud

88 89 IAAS to PAAS

90 Automatic Cluster Creation

91 Get Public IP for access

92 Server Machine VM access • MPS console endpoint: https://xxx.xx.xx.xx

93 Deploying Deep Learning Application

Embedded Hardware Desktop, Web, Cloud Application logic

Code Application Generation Deployment

94 Solution- GPU/MATLAB Coder for Deep Learning Deployment Target Libraries

TensorRT & GPU Coder cuDNN Application Libraries logic

Intel MKL-DNN Library MATLAB Coder ARM Compute Library 95 Deep Learning Deployment Workflows

INFERENCE ENGINE DEPLOYMENT INTEGRATED APPLICATION DEPLOYMENT

Trained Pre- Post- DNN processing processing

cnncodegen codegen

Portable target code Portable target code

96 Single Image Inference on Titan Xp using cuDNN

TensorFlow (1.8.0)

MXNet (1.2.1)

GPU Coder (R2018b)

PyTorch (0.3.1)

Intel® Xeon® CPU 3.6 GHz - NVIDIA libraries: CUDA9 - cuDNN 7 97 NVIDIA Hardware Support Package (HSP) Simple out-of-box targeting for NVIDIA boards:

INFERENCE ENGINE DEPLOYMENT

Jetson Drive platform Trained DNN

cnncodegen

Portable target code

98 Summary- MATLAB Deep Learning Framework DEPLOYMENT Intel MKL-DNN Library

Access Data Design + Train NVIDIA TensorRT & cuDNN Libraries

▪ Manage large image sets ▪ Acceleration with GPU’s ▪ Automate image labeling ▪ Scale to clusters ARM ▪ Easy access to models Compute Library

99 Summary

✓ Create and validate Deep learning models ✓ Automate ground truth labeling ✓ Access large amount of data from cluster/cloud ✓ Interoperability with Deep learning frameworks ✓ Visualization and hyperparameter tuning ✓ Seamlessly scale training to GPUs, clusters and cloud ✓ Deployment on embedded targets and web services

100 Available Here 101 102 103