AI Techniques in MATLAB for Signal, Time-Series, and Text Data Matthias Sommer © 2015 The MathWorks, Inc.1 AI and Deep Learning for Signals in the News 2 The Use of Deep Learning is Growing Across Industries Aerospace, Defense and Communications Consumer Electronics and Digital Health Communications devices, Multi-standard communications security receivers, drone recognition Voice assistants Digital health Automotive Industrial Automation Sensor processing, Voice control enabled automated driving Condition monitoring Predictive maintenance Infotainment 3 Application Examples Using MATLAB – Audio and Speech Speech Command Recognition Music Genre Classification (a.k.a. "Keyword Spotting") https://www.mathworks.com/help/audio/examples/music- https://www.mathworks.com/help/deeplearning/e genre-classification-using-wavelet-time-scattering.html xamples/deep-learning-speech-recognition.html 4 Application Examples Using MATLAB – Industrial and Radar sensors ECG Signal Classification Modulation Classification https://www.mathworks.com/help/signal/examples/classify- ecg-signals-using-long-short-term-memory-networks.html http://www.mathworks.com/help/comm/examples/ modulation-classification-with-deep-learning.html 5 Agenda ▪ Deep Learning – Basic ideas ▪ Deep Learning Model Development for Signals, Time Series, and Text ▪ Conclusions 6 What is Deep Learning? Deep learning is a type of machine learning in which a model learns from examples. 7 Common Network Architectures - Signal Processing Convolutional Neural Networks (CNN) Time-Frequency Transformation Long Short Term Memory (LSTM) Networks Feature Engineering 8 Deep Learning Workflow CREATE AND ACCESS PREPROCESS AND DEVELOP PREDICTIVE ACCELERATE AND DATASETS TRANSFORM DATA MODELS DEPLOY Data sources Pre-Processing Import Reference Models/ Desktop Apps Design from scratch Simulation and Transformation Hardware-Accelerated Enterprise Scale Systems augmentation Training Analyze and tune Data Labeling Feature extraction Embedded Devices and hyperparameters Hardware 9 Deep Learning Workflow Challenges – Signals and Time Series Limited Data Data Labeling Availabiliy Deep learning models only as good as training data Deployment and Domain-specific data Limited reference Scaling to various processing desirable research platforms 10 Agenda ▪ Deep Learning – Basic ideas ▪ Deep Learning Model Development for Signals, Time Series, and Text – Data – Processing and transformation – Model design and optimization – Acceleration, prototyping, and deployment ▪ Conclusions 11 Agenda ▪ Deep Learning – Basic ideas ▪ Deep Learning Model Development for Signals, Time Series, and Text – Data – Processing and transformation – Model design and optimization – Acceleration, prototyping, and deployment ▪ Conclusions 12 Current Investments – Models vs. Data Research Industry Models and Algorithms Datasets From "Troubleshooting deep neural networks" (Josh Tobin et al., Jan 2019) 13 What does a large dataset look like?audioDatastore How to navigate, index, read (aka "ingest") a largeimageDatastore dataset? fileDatastore Custom Datastores How to... ▪ Build a list of all data and labels? ▪ Review basic statistics about available data? ▪ Select data subsets without nested for loops, dir, ls, what, ... aplenty? ▪ Jointly read data and labels? ▪ Automatically distribute computations? 14 Use appropriate tools to help you label signals ▪ Programmatically… ▪ … or via Apps 15 What if available data isn't enough? PitchOriginal shift Augmented Dataset N times as Original much data Dataset Data augmentation allows building more complex and more robust models 16 Simulation is key if recording and labelling real-world data is impractical or unreasonable – Communications Signals 17 Simulation is key if recording and labelling real-world data is impractical or unreasonable – Radar Signals Radar Target Simulation Micro-Doppler Analysis 18 Agenda ▪ Deep Learning – Basic ideas ▪ Deep Learning Model Development for Signals, Time Series, and Text – Data – Processing and transformation – Model design and optimization – Acceleration, prototyping, and deployment ▪ Conclusions 19 Common types of network architectures used in signal processing and text analytics applications Convolutional Neural Networks (CNN) Time-Frequency Transformation Long Short Term Memory (LSTM) Networks Feature Engineering 20 Time-Frequency Transformations Reframe To frequency (e.g. Buffer) ... (e.g. FFT) frequency (bin frequency #) Time (samples) Time Time (samples) Time (frame #) Time (frame #) Easiest to More compact for Best resolution, for understand and speech & audio non-periodic Better resolution at implement applications signals low frequencies Basic spectrogram Perceptually-spaced (e.g. Constant Q transform Mel, Bark) Spectrogram Wavelet scalogram 21 Extracting Features from Signals: Application-Agnostic Examples BW measurements Spectral statistics Harmonic analysis Octave spectrum Frequency domain Time domain 22 Automated Feature Extraction: Wavelet Scattering Deep Feature Extraction Averaging Averaging Averaging Nonlinearity Nonlinearity Classifier Nonlinearity Wavelet Convolution Wavelet Wavelet Convolution Wavelet Wavelet Convolution Wavelet Fixed Fixed Fixed ▪ Can relieve requirements on amount of data and model complexity – Featured in leader-boards a number of research competitions ▪ Framework for extracting features [1] [1] Joan Bruna, and Stephane Mallat, P. 2013. Invariant Scattering Convolution Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 35, No. 8, pp. 1872-1886. 24 Agenda ▪ Deep Learning – Basic ideas ▪ Deep Learning Model Development for Signals, Time Series, and Text – Data – Processing and transformation – Model design and optimization – Acceleration, prototyping, and deployment ▪ Conclusions 25 Developing Deep Learning Models Design Network Accelerate Training MATLAB as a Train container on NGC Design Optimize Model Exchange Pre-trained Networks Bayesian Hyperparameter Optimization 26 Exchange Models With Deep Learning Frameworks TensorFlow- PyTorch TensorFlow Keras MXNet ONNX MATLAB Core ML Chainer Caffe ONNX = Open Neural Network Exchange Format 27 Exchange Models With Deep Learning Frameworks TensorFlow- PyTorch TensorFlow Deployment Keras Keras importer MXNet Augmentation ONNX MATLAB and Optimization Caffe importer Core ML Chainer Caffe Visualization and Debugging ONNX = Open Neural Network Exchange Format 28 Agenda ▪ Deep Learning – Basic ideas ▪ Deep Learning Model Development for Signals, Time Series, and Text – Data – Processing and transformation – Model design and optimization – Acceleration, prototyping, and deployment ▪ Conclusions 29 Deployment and Scaling for A.I. MATLAB Embedded Devices Enterprise Systems 30 Deploying Deep Learning Models for Inference Intel MKL-DNN Library Application logic NVIDIA Code TensorRT & Generation cuDNN Libraries Auto-generated Code (C/C++/CUDA) ARM Compute Library 31 Enterprise Deployment Deployment to the cloud with MATLAB Compiler and MATLAB Production Server 32 Agenda ▪ Deep Learning – Basic ideas ▪ Deep Learning Model Development for Signals, Time Series, and Text – Data – Processing and transformation – Model design and optimization – Acceleration, prototyping, and deployment ▪ Conclusions 33 Deep Learning Workflow Challenges – Signals and Time Series Data-labeling Apps AugmentationLimited data and Data Labeling Deployment and and Examples simulationavailability algorithms Scaling to various platforms Deep learning models only as good as training data Domain-specific data Limited reference Collaboration processingApplication desirable-specific algorithmsresearch and tools in the AI ecosystem 34 Demo Station Find out more: Deep Learning from Idea to Embedded Hardware Matthias Sommer Technology Showcase 35 Related Trainings ▪ Time series analysis – Signal Preprocessing and Feature Extraction for Data Analytics with MATLAB ▪ Deep-learning (including non-vision applications) – Deep Learning with MATLAB https://nl.mathworks.com/services/training.html 36.