Fpgas and Machine Learning – Intel® FPGA Deep Learning Acceleration Suite – Execution on the FPGA (Model Optimizer & Inference Engine) • Intel® Agilex® FPGA • Oneapi

Intel Network & Custom Logic Group Greg Nash, System Architect – Government, Aerospace & Military Jim Moawad, Technical Solution Specialist – FPGA Acceleration July 29, 2019 - Argonne Training Program on Extreme Scale Computing

Public notices & disclaimers

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No product or component can be absolutely secure. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. For more complete information about performance and benchmark results, visit http://www.intel.com/benchmarks . Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/benchmarks . Intel® Advanced Vector Extensions (Intel® AVX)* provides higher throughput to certain processor operations. Due to varying processor power characteristics, utilizing AVX instructions may cause a) some parts to operate at less than the rated frequency and b) some parts with Intel® Turbo Boost Technology 2.0 to not achieve any or maximum turbo frequencies. Performance varies depending on hardware, software, and system configuration and you can learn more at http://www.intel.com/go/turbo. Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction. Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate. Intel, the Intel logo, Intel Atom, Intel Core, Intel Nervana, Intel Optane, Intel Xeon, Iris, Movidius, OpenVINO, Pentium, Celeron, Stratix, Arria and the Stratix logo and are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as property of others.

• Intel® and AI / Machine Learning • Accelerate Deep Learning Using OpenVINO Toolkit • Deep Learning Acceleration with FPGA – FPGAs and Machine Learning – Intel® FPGA Deep Learning Acceleration Suite – Execution on the FPGA (Model Optimizer & Inference Engine) • Intel® Agilex® FPGA • OneAPI

3 Machine Learning Applies to Nearly Every Market

Agriculture Energy Education Government Finance Health

Achieve higher Maximize Transform Enhance safety, Turn data Revolutionize yields & increase production the learning research, and into valuable patient efficiency and uptime experience more intelligence outcomes Industrial Media Retail Smart Home Telecom Transport

Empower Create Enable homes Drive network truly intelligent thrilling Transform stores that see, hear, and operational Automated Industry 4.0 experiences and inventory and respond efficiency driving

Wide variety of market segments incorporating AI and Deep Learning which is increasing compute demands across the board.

4 © 2019 Intel Corporation AI & Machine Learning is a Vast Field of Study Regression Classification AI Supervised Clustering Learning Decision Trees Machine Data Generation learning Image Processing unSupervised Speech Processing Learning Natural Language Processing Deep Recommender Systems learning Adversarial Networks Reinforcement Reinforcement Learning Learning No one size fits all approach to AI

Data deluge (2019) Analytics Curve Insights

25 GB per month Internet User Act/adapt Cognitive 2 Analytics 50 GB per day Business Smart Car Forecast

2 Prescriptive Analytics 3 TB per day Smart Hospital Foresight 2 Predictive 40 TB per day Analytics Operational Airplane Data insight 2 Diagnostic 1 pB per day Analytics Smart Factory Hindsight 2 Descriptive AI 50 PB per day Analytics Security City Safety Is the driving force

1. Source: http://www.cisco.com/c/en/us/solutions/service-provider/vni-network-traffic-forecast/infographic.html 2. Source: https://www.cisco.com/c/dam/m/en_us/service-provider/ciscoknowledgenetwork/files/547_11_10-15-DocumentsCisco_GCI_Deck_2014-2019_for_CKN__10NOV2015_.pdf 6 © 2019 Intel Corporation Diverse Workloads require CPU GPU AI FPGA DIVERSE architectures

The future is a diverse mix of scalar, vector, matrix, and spatial architectures deployed in CPU, GPU, AI, FPGA and other accelerators Scalar Vector Matrix Spatial

SVMS

7 © 2019 Intel Corporation AI Compute Guide CPU Accelerators FPGA, VPU, GPU NNP-I Inference Highly efficient multi-model Delivering Real time workflows inferencing for cloud, data center with AI for optimized system and appliances at scale performance

Foundation for all workloads including AI Energy efficient Edge AI for Intel® DL Boost for inference Video/Vision & training acceleration NNP-T Training Fastest time-to-train at scale with high bandwidth AI server connections for the most GPU discrete & Integrated, Multi models – high performance across broad set of existing and emerging DL models for applications like optimized for highly parallel Natural Language Processing, Speech & Text, Recommendation Engines, and Video & Image search & persistent, intense usage filtering workload acceleration A+ - Combined workflows where use cases span AI plus additional workloads (Eg - Immersive Media - Media + Graphics in GPU or L3 forwarding in FPGA) 8 © 2019 Intel Corporation Tame your data with a robust data layer

Source(s)? ingest Store Process Analyze Structured? Tool for live File, block or Integration, Applications Volume? streaming data object-based cleaning, in HPC, Big Data, DURABILITY? ingestion from storage solution normalization HPDA, AI & more Streaming? Internet of given cost, and more that have access LOCALITY? Things (IOT) access, volume transformations to a common GOVERNANCE? sensors in and perf on batch and/or compute and Other? endpoint devices requirements streaming data data pool

Visit: intel.com/content/www/us/en/analytics/tame-the-data-deluge.html

*Other names and brands may be claimed as the property of others 9 © 2019 Intel Corporation Visit: software.intel.com/ai/courses Choose any approach from machine to deep learning SVM Un- supervised Class- Learning Machine Supervised ification Learning Learning Nearest Naïve Neighbor Bayes GMM kmeans GPR SVR Discri- minant Linear Clustering Analysis Hier- Regres- archical sion Neural Logistic Decision Nets Trees Reinforce- Hidden Statistical ment Markov Methods Neural Learning Ensemble Models Methods Nets

10 © 2019 Intel Corporation Visit: www.intel.ai/technology Speed up development using open AI software Machine learning Deep learning TOOLKITS App Open source platform for building E2E Analytics & Deep learning inference deployment Open source, scalable, and developers AI applications on Apache Spark* with distributed on CPU/GPU/FPGA/VPU for Caffe*, extensible distributed deep learning TensorFlow*, Keras*, BigDL TensorFlow*, MXNet*, ONNX*, Kaldi* platform built on Kubernetes (BETA)

Python R Distributed Intel-optimized Frameworks libraries * • Scikit- • Cart • MlLib (on Spark) * * And more framework Data learn • Random • Mahout optimizations underway • Pandas Forest including PaddlePaddle*, scientists * * • NumPy • e1071 Chainer*, CNTK* & others

Intel® Intel® Data Analytics Intel® Math Kernel Kernels Distribution Acceleration Library Library Library for Deep Neural for Python* (DAAL) Networks (MKL-DNN) developers Intel distribution High performance machine Open source compiler for deep learning model optimized for learning & data analytics Open source DNN functions for computations optimized for multiple devices (CPU, GPU, machine learning library CPU / integrated graphics NNP) from multiple frameworks (TF, MXNet, ONNX)

1 An open source version is available at: 01.org/openvinotoolkit *Other names and brands may be claimed as the property of others. Developer personas show above represent the primary user base for each row, but are not mutually-exclusive All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. 1111 © 2019 Intel Corporation Optimization Notice Visit: www.intel.ai/technology Deploy AI anywhere with unprecedented hardware choice

D e e C v d l I g o c e u e d If needed Automated Dedicated Flexible Dedicated Dedicated Graphics, Media Driving Media/Vision Acceleration DL Inference DL Training & Analytics

Intel GPU NNP-I NNP-T

All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. 12 © 2019 Intel Corporation Deep learning Deployed Data Tools Training Model Optimization Inference

Cloud Cloud Containers -OR- -OR- -OR-

Multi-Use Cluster Multi-Use Cluster (Spark/HPC/other) Intel-optimized Trained Intel® Distribution (Spark/HPC/other) Frameworks -OR- Model of OpenVINO™ -OR- -OR- Weights Toolkit

-OR- Dedicated Training Edge Cluster Libraries Server Device (with Nauta) End-to-end deep learning on Intel

nd formerly known as 2 generation INTEL® XEON® SCALABLE PROCESSOR Cascade Lake

Drop-in compatible CPU on Intel® Xeon® Scalable platform TCO/Flexibility Performance SecUrity Begin your AI journey efficiently, Built-in Acceleration with Hardware-Enhanced now with even more agility… Intel® Deep Learning Boost… Security…  Intel® Security Essentials  IMT – Intel® Infrastructure Up to deep  Intel® SecL: Intel® Security Management Technologies learning Libraries for Data Center  ADQ – Application Device Queues 30X throughput!1  SST – Intel® Speed Select Technology  TDT – Intel® Threat Detection Throughput (img/s) Technology

1 Based on Intel internal testing: 1X,5.7x,14x and 30x performance improvement based on Intel® Optimization for Café ResNet-50 inference throughput performance on Intel® Xeon® Scalable Processor. See Configuration Details 3 Performance results are based on testing as of 7/11/2017(1x) ,11/8/2018 (5.7x), 2/20/2019 (14x) and 2/26/2019 (30x) and may not reflect all publically available security updates. No product can be absolutely secure. See configuration disclosure for details. , Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit: http://www.intel.com/performance 15 © 2019 Intel Corporation Intel® Deep Learning Boost (DL Boost) featuring Vector Neural Network Instructions (VNNI)

Sign Mantissa INT8 07 06 05 04 03 02 01 00

Current AVX-512 instructions to perform INT8 convolutions: vpmaddubsw, vpmaddwd, vpaddd

INPUT INT8 OUTPUT vpmaddubsw INT16 OUTPUT INPUT vpmaddwd INT32 OUTPUT INT8 vpaddd CONSTANT INT32 INT16 CONSTANT INT32

NEW AVX-512 (VNNI) instruction to accelerate INT8 convolutions: vpdpbusd INPUT INT8 OUTPUT vpdpbusd INT32 INPUT INT8 CONSTANT INT32

Inception v3 1.8x2 Caffe 285x SSD-VGG16 N 2.0x2 N T N 50x Resnet-50 1.9x2

Baseline vs. Baseline vs. Baseline 2 Inception v3 8 1.8x I July 2017 July 2017 Skylake launch February 2019 TensorFlow CASCADE SKYLAKE 2S Intel® Xeon® Scalable Processor (Skylake) LAKE 1 5.7x inference throughput improvement with Intel® Optimizations for Caffe ResNet-50 on Intel® Xeon® Platinum 8180 Processor in Feb 2019 compared to performance at launch in July 2017. See configuration details on Config 1 Performance results are based on testing as of dates shown in configuration and may not reflect all publicly available security updates. 28/24/2018) Results have been estimated using internal Intel analysis or architecture simulation or modeling, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. No product can be absolutely secure. See configuration disclosure for details. Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit: http://www.intel.com/performance 17 Intel® Nervana™ neural network processors (NNP) ¥

Fastest time-to-train with NNP-T high bandwidth AI server DEDICATED connections for the most DL TRAINING persistent, intense usage

Highly-efficient multi-model NNP-I inferencing for cloud, data DEDICATED center and intense appliances DL INFERENCE

‡The Intel® Nervana™ Neural Network Processor is a future product that is not broadly available today All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. 18 © 2019 Intel Corporation Intel® FPGA for AI

FIRST TO MARKET TO ACCELERATE Falcon Mesa Real-time Evolving AI WORKLOADS workloads

Precision Rnn Latency Delivering AI+ for Flexible Lstm Sparsity system level functionality Speech WL Adversarial Networks AI+ I/O Ingest Reinforcement Learning AI+ Networking Neuromorphic Computing Ai+ security … Ai+ pre/post processing … Enabling real-time AI in a wide range of embedded, edge and cloud apps

All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. 19 © 2019 Intel Corporation Intel® Movidius™ Vision processing unit (vPU) DRONES SERVICE ROBOTS • Sense and avoid • Navigation • GPS denied hovering • 3D Vol. mapping • Pixel labeling • Multimodal sensing • Video, image capture AR-VR HMD Surveillance • 6DOF pose, position, mapping • Detection/classification • Gaze, eye tracking • Identification • Gesture tracking, recognition • Multi-nodal systems • See-through camera • Multimodal sensing • Video, image capture SMART HOME • Detection, tracking WEARABLES • Perimeter, presence monitoring • Detection, tracking • Recognition, classification • Recognition • Multi-nodal systems • Video, image, session capture • Multimodal sensing • Video, image capture Power-efficient image processing, computer vision & deep learning for devices

All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. 20 © 2019 Intel Corporation Intel® Gaussian neural accelerator (GNA)

Ample throughput Intel® Try it TODAY! For speech, language, and other sensing inference GNA (IP) Low power <100 mW power consumption for always-on applications Intel® Speech Enabling Flexibility Developer Kit Gaussian mixture model (GMM) and DSP https://software.intel.com/en- neural network inference support us/iot/speech-enabling-dev-kit

Learn more: https://sigport.org/sites/default/files/docs/PosterFinal.pdf

Streaming Co-Processor for Low-Power Audio Inference & More

All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. 21 © 2019 Intel Corporation Intel integrated processor graphics Ubiquity/Scalability Powerful, Flexible Architecture • Shipped in 1 billion+ Intel® SoCs • Rich data type support for 32bitFP, 16bitFP, • Broad choice of performance/power across Intel 32bitInteger, 16bitInteger with SIMD Atom®, Intel® Core™, and Intel® Xeon® processors multiply-accumulate instructions

Media Leadership Memory Architecture • Intel® Quick Sync Video – fixed-function media • Shared memory architecture on die between CPU blocks to improve power and performance and GPU to enable lower latency and power • Intel® Media SDK – API that provides access to hardware-accelerated codecs

MacOS (CoreML and MPS1) Windows O/S (WinML) Hardware integration Software support OpenVINO™ Toolkit (Win, Linux) clDNN

Built-in deep learning inference acceleration

Visit www.mobileye.com

Automated Driving Platform

All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice. 23 © 2019 Intel Corporation Deep learning Software Framework Topology Library Provides a standard way to A set of algorithms modeled Highly optimized functions intended customize and deploy Deep loosely after the human brain to be used in Neural Network Neural Networks that forms a “network” designed implementations to maximize to recognize complex patterns performance Supplies generic functionality Libraries are framework and model which can be selectively extended Also referred to as a network, agnostic, hardware specific with user defined additions to NN, or model. enhance specific tasks MACHINE LEARNING LIBRARIES Python R Distributed • Scikit-learn • Cart • MlLib (on Spark) • Pandas • Random • Mahout • NumPy Forest • e1071 DAAL MKL-DNN clDNN Intel® Data Analytics Open-source deep neural Acceleration Library network functions for (includes machine CPU / integrated graphics learning)

24 Frameworks

Google Amazon Microsoft Facebook

ONNX promises to ease Open Neural Network Exchange (ONNX) interoperability between • Enabling interoperability between different frameworks frameworks. • Streamlining the path from research to production

Caffe2, PyTorch, Microsoft Cognitive Toolkit, Apache MXNet https://github.com/onnx/onnx

25 © 2019 Intel Corporation x1 x2 x3 x4 Convolutional Neural Network (CNN) → Layers of neurons connected by learned weights → Three layer classes: • Convolution Layer • Pooling Layer • Fully connected Layer → Connections between the layers point in one direction (Feed Forward Network) → Used to extract features from images for recognition or detection

y y y Recurrent Neural Network (RNN) 1 2 3 → Allows long-term dependencies to affect the output by relying on internal memory to process inputs = → Types of RNN: • LSTM (Long Short Term Memory) • GRU (Gated Recurrent Unit) → Designed to analyze sequential data (i.e., NLP)

26 IMAGE RECOGNITION TEXT PROCESSING

https://www.microsoft.com/en-us/translator/mt.aspx#nnt http://ieeexplore.ieee.org/document/7124883/ Workload: Neural Machine Translation Workload: ResNet50, InceptionV3, SSD-VGG16

CNN RNN

SPEECH RECOGNITION

https://www.news- medical.net/whitepaper/20170821/Speech-Recognition-in- Healthcare-a-Significant-Improvement-or-Severe- Headache.aspx Workload: DeepSpeech2

Purpose Neural Network Type Neural Network Example Classify image(s) CNN - Convolutional Neural ResNet50, Inception V3, Image Recognition Network Inception_ResV2, MobileNet, Golden SqueezeNet, DenseNet Retrieve r Locate AND classify CNN SSD– Single Shot Detector, R-FCN, Object Detection object(s) in image Yolo_V2, VGG_16, DDRN/D-DBPN Natural Language Extract context & LTSM (RNN)- Long Short Google NMT Processing (NLP) meaning from text Term Memory Convert speech to text RNN - Recursive Neural Baidu DeepSpeech, Transformer Speech Recognition Network

Convert text to speech GAN - Generative Adversarial WaveNet Text-to-Speech Network with CNN/RNN

Recommendation Recommend ads, search, MLP - Multilayer Perceptron Wide & Deep MLP, NCF Systems apps, etc. Create images like GAN with CNN DCGAN, DRAW Data Generation training data

Reinforcement Learning from feedback CNN or CNN+RNN A3C Learning on action

All products, computer systems, dates, and figures are preliminary based on current expectations, and are subject to change without notice.

Single-Crop top-1 validation accuracies versus amount of operation for a Single-Crop top-1 validation accuracies single forward pass AlexNet depth: 8 layers, GoogleNet depth: 22 layers, VGG depth: 16 layers or 19 layers, ResNet depth: 50 layers or 101 layers Legend in gray circles in the image shows number of network parameters

An Analysis of Deep Neural Network Models for Practical Applications Alfredo Canziani & Eugenio Culurciello & Adam Paszke

Deep Learning Development Cycle

Data acquisition and Create models organization

Integrate trained models Adjust models to meet with application code performance and accuracy objectives

Intel® Distribution OpenVINO™ Toolkit Provides Deployment from Intel® Edge to Cloud

Training Did You Know? Human Bicycle Training requires a very large Forward “Strawberry” data set and deep neural network (many layers) to ? “Bicycle” achieve the highest accuracy in most cases Lots of Error Backward Strawberry Labeled Data! Model Weights

Inference Accuracy

Forward “Bicycle”?

Data Set Size ??????

34 © 2019 Intel Corporation What it is A toolkit to accelerate development of high performance computer vision & deep learning into vision applications from device to High performance, perform ai at the edge cloud. It enables deep learning on hardware accelerators and easy deployment across multiple types of Intel® platforms.

Who needs this product? Streamlined & Optimized deep learning inference . Computer vision/deep learning software developers . Data scientists . OEMs, ISVs, System Integrators Heterogeneous, cross-platform flexibility Usages Security surveillance, robotics, retail, healthcare, AI, office automation, Free Download  software.intel.com/openvino-toolkit transportation, non-vision use cases (speech, text) & more. Open Source version  01.org/openvinotoolkit

DEEP LEARNING COMPUTER VISION

OpenCL™ Supports 100+ public Model Inference models, incl. 30+ Computer vision library Optimized media Optimizer Engine pretrained models (kernel & graphic APIs) encode/decode functions

Supports major AI frameworks Cross-platform flexibility High Performance, high Efficiency

Multiple products launched Breadth of product Rapid adoption by developers based on this toolkit portfolio Strong Adoption + Rapidly Expanding Capability software.intel.com/openvino-toolkit

36 © 2019 Intel Corporation Optimization Notice Intel® Deep learning deployment toolkit For Deep Learning Inference – Part of Intel® Distribution of OpenVINO toolkit Model Optimizer Inference Engine . What it is: A Python*-based tool to import trained models . What it is: High-level inference API and convert them to Intermediate representation. . Why important: Interface is implemented as dynamically . Why important: Optimizes for performance/space with loaded plugins for each hardware type. Delivers best conservative topology transformations; biggest boost is performance for each type without requiring users to from conversion to data types matching hardware. implement and maintain multiple code pathways. Trained CPU Plugin Extendibility Models C++ Caffe* GPU Plugin Extendibility Inference OpenCL™ TensorFlow* Engine Model FPGA Plugin Common API MxNet* Optimizer IR Extendibility IR (C++ / Python) NCS Plugin Convert & .data OpenCL™ Load, infer Optimized cross- ONNX* Optimize (Pytorch, Caffe2 & more) platform inference GNA Plugin IR = Intermediate Kaldi* Representation format VAD Plugin

GPU = Intel CPU with integrated graphics processing unit/Intel® Processor Graphics VAD = Vision Accelerator Design Products; includes FPGA and 8 MyriadX versions

37 © 2019 Intel Corporation Optimization Notice OpenCL and the OpenCL logo are trademarks of Apple Inc. used by permission by Khronos AI Software in The big picture Architects Data Engineers Data Scientists App Developers Ingest/Store Prepare Model Deploy/Visualize

Intel® And Unified Workflows DL Studio more…

Collect, Integrate, ETL & ELT Manage Metadata Deep Learning Deploy Inference Open Open (Managed) Proprietary And more…

And more… Visualize And more… And more… Pre-Process Data Store & Manage Big Data Machine Learning & Analytics Open Open (Managed)

Proprietary Proprietary

Proprietary: And more… And more… Library Developers Library IT Systems Management And more…

API’s

And more… And more… Enterprise Applications

Other names and brands may be claimed as the property of others. Note: displayed logos are not a complete list of solutions/providers in each category, personas are not mutually-exclusive by workflow step, and categorization is generalized

Intel® Distribution of OpenVINO™ toolkit Open Source Directory Tool/Component OpenVINO™ toolkit (open source) https://github.com Installer (including necessary drivers) P Intel® Deep Learning Deployment toolkit Model Optimizer P P /opencv/dldt/tree/2018/model-optimizer Inference Engine P P /opencv/dldt/tree/2018/inference-engine

Intel® Math Kernel Library BLAS, Intel® MKL1, Intel CPU plug-in P P /opencv/dldt/tree/2018/inference-engine (Intel® MKL) only1 jit (Intel MKL)

Intel GPU (Intel® Processor Graphics) plug-in P P /opencv/dldt/tree/2018/inference-engine Heterogeneous plug-in P P /opencv/dldt/tree/2018/inference-engine Intel GNA plug-in P Intel® FPGA plug-in P Intel® Neural Compute Stick (1 & 2) VPU plug-in P Intel® Vision Accelerator based on Movidius plug-in P 30+ Pretrained Models - incl. Model Zoo P P /opencv/open_model_zoo (IR models that run in IE + open sources models) Samples (APIs) P P /opencv/dldt/tree/2018/inference-engine Demos P P /opencv/open_model_zoo Traditional Computer Vision OpenCV* P P /opencv/opencv OpenVX (with samples) P Intel® Media SDK P P2 /Intel-Media-SDK/MediaSDK OpenCL™ Drivers & Runtimes P P2 /intel/compute-runtime FPGA RunTime Environment, Deep Learning P Acceleration & Bitstreams (Linux* only) 1Intel MKL is not open source but does provide the best performance 39 © 2019 Intel Corporation 2Refer to readme file for validated versions Speed Deployment with Pre-trained Models & Samples Expedite development, accelerate deep learning inference performance, and speed production deployment. Pretrained Models in Intel® Distribution of OpenVINO™ toolkit . Age & Gender . Text Detection & Recognition . Identify Roadside objects . Face Detection–standard & enhanced . Vehicle Detection . Advanced Roadside Identification . Head Position . Retail Environment . Person Detection & Action Recognition . Human Detection–eye-level . Pedestrian Detection . Person Re-identification–ultra small/ultra fast & high-angle detection . Pedestrian & Vehicle Detection . Face Re-identification . Detect People, Vehicles & Bikes . Person Attributes Recognition Crossroad . Landmarks Regression . License Plate Detection: small & front . Emotion Recognition facing . Smart Classroom Use Cases . Identify Someone from Different Videos– . Vehicle Metadata standard & enhanced . Single image Super Resolution (3 models) . Human Pose Estimation . Facial Landmarks . Instance segmentation . Action recognition – encoder & . Gaze estimation decoder . and more… Binary Models

. Face Detection Binary . Vehicle Detection Binary . ResNet50 Binary . Pedestrian Detection Binary

Use Model Optimizer & Inference Engine for public models & Intel pretrained models

. Object Detection  Hello Infer Classification

. Standard & Pipelined Image Classification  Interactive Face Detection

. Security Barrier  Image Segmentation

. Object Detection SSD  Validation Application

. Neural Style Transfer  Multi-channel Face Detection . Object Detection for Single Shot Multibox Detector using Asynch API+

42 43 Vision Inference requirements Vary

Design Variable Considerations Performance Frames/sec, latency (real-time constraints) Power Power envelope, power per inference Cost Hardware cost, development cost, cost per inference Batch size 1, 8, 32, etc. Precision FP32, FP16, FP11 Image size 720p, 1080p, 4K Camera input Raw video, encoded Network support CNN, RNN, LSTM, custom Operating environment Temperature controlled environment, outdoor use, 24/7 operation Product life Operational lifespan, product availability Regulation Security, privacy

There is no “one-size-fits-all” solution ...... and sometimes requirements change

44 FPGA Overview DSP Block Memory Block • Field Programmable Gate Array (FPGA) – Millions of logic elements – Thousands of embedded memory blocks – Thousands of DSP blocks – Programmable routing – High speed transceivers – Various built-in hardened IP • Used to create Custom Hardware

Logic • Well-suited for Matrix Multiplation Programmable Modules Routing Switch

45 Intel® FPGA –application Accelerationfrom Edge to Cloud Combining Intel FPGA hardware and software to efficiently accelerate workloads for processing-intense tasks Programmable Acceleration Card (PAC) offering

Mustang-F-100-A10* Intel® FPGA PAC Intel® FPGA PAC Intel® FPGA PAC N3000 for networking with Arria®10 GX D5005 for Datacentre Devices / edge Network/NFV Cloud/enterprise

* Other names and brands may be claimed as the property of others. 46 Why FPGA For Deep Learning Inference Key differentiators • Low latency • Low power Vision Acceleration • High throughput (FPS) Design with Intel®

• Flexibility to adapt to new, evolving, and custom networks Arria® 10 FPGA

• Supports large image sizes (e.g., 4K)

• Large networks (up to 4 billion parameters)

• Security / Encryption

• Wide ambient temperature range (-40° C to 105° C)*

• 24/7/365 operation

• Long lifespan (8–10 years)

*-40°C to 105°C temp range is for chip down design. Board temp range is 0° to 65°C.

47 Increase Deep Learning Workload Performance on Public Models using Intel® Distribution of OpenVINO™ toolkit & Intel® Architecture Comparison of Frames per Second (FPS)

19.9x1

Improvement Relative Performance Relative

Standard Caffe* Baseline

Public Models (Batch Size) Optimize with OpenVINO on CPU, Get an even Bigger Performance Boost with Intel® FPGA

1Depending on workload, quality/resolution for FP16 may be marginally impacted. A performance/quality tradeoff from FP32 to FP16 can affect accuracy; customers are encouraged to experiment to find what works best for their situation. Performance results are based on testing as of June 13, 2018 and may not reflect all publicly available security updates. See configuration disclosure for details. No product can be absolutely secure. For more complete information about performance and benchmark results, visit www.intel.com/benchmarks. Configuration: Testing by Intel as of June 13, 2018. Intel® Core™ i7-6700K CPU @ 2.90GHz fixed, GPU GT2 @ 1.00GHz fixed Internal ONLY testing, Test v3.15.21 – Ubuntu* 16.04, OpenVINO 2018 RC4, Intel® Arria® 10 FPGA 1150GX. Tests were based on various parameters such as model used (these are public), batch size, and other factors. Different models can be accelerated with different Intel hardware solutions, yet use the same Intel software tools. Intel’s compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel 48 microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice revision #20110804 Solving Machine Learning Challenges with FPGA

Ease-of-use Real-Time Flexibility software abstraction, deterministic customizable hardware platforms & libraries low latency for next gen DNNarchitectures

Intel FPGA solutions enable Intel FPGA hardware Intel FPGAs can be customized software-defined programming implements a deterministic low to enable advances in machine of customized machine latency data path unlike any learning algorithms. learning accelerator libraries. other competing compute device.

49 Public Intel FPGA Machine Learning Success

Microsoft: “Microsoft has revealed that Altera FPGAs have been installed across every Azure cloud server, creating what the company is calling “the world’s first AI supercomputer.” -Sept 2016

NEC: “To create the NeoFace Accelerator, the engine software IP was integrated into an Intel Arria 10 FPGA, which operate in Xeon processor–based servers. ” -June 2017

JD.COM: “Arria®10 FPGA can achieve 5x improvement in the performance of LSTM accelerator card compared to GPU.” –Apr 2017

Inspur/iFlytech: “Leading server vendor Inspur Group and Altera today launched a speech recognition acceleration solution based on Altera's Arria® 10 FPGAs and DNN algorithm from iFLYTEK.” -Dec 2016

ZTE: “Using Intel’s Arria 10 FPGA, ZTE engineers were able to achieve more than 1000 images per second in facial recognition.” –Jan 2017

50 Why FPGAs for Deep Learning?

feeder PE PE PE PE

feeder feeder feeder feeder

51 Customized Performance

Convolutional Neural Networks are Compute Intensive Convolutional Neural Networks are Compute Intensive Feature Benefit

Highly parallel Facilitates efficient low-batch video architecture stream processing and reduces latency Configurable FP32 9Tflops, FP16, FP11 Distributed Accelerates computation by tuning Floating Point DSP compute performance Blocks Tightly coupled >50TB/s on chip SRAM bandwidth, high-bandwidth random access, reduces latency, memory minimizes external memory access

Fine-grained & low latency Programmable Reduces unnecessary data movement, between compute and memory Data Path improving latency and efficiency

Optional Optional Memory Memory Support for variable precision (trade-off Configurability throughput and accuracy). Future proof

Function 1 Function 2 Function 3 designs, and system connectivity IO Pipeline Parallelism IO

52 FPGAs System Flexibility to Control the Data path

Compute Acceleration/Offload . Workload agnostic compute . FPGAaaS . Virtualization

Intel® Xeon® Processor

Inline Data Flow Processing . Machine learning Storage Acceleration . Object detection and recognition . Advanced driver assistance system (ADAS) . Machine learning . Gesture recognition . Cryptography . Face detection . Compression . Indexing

AlexNet VGG GoogleNet ResNet Many efforts to improve efficiency LeNet XNORNet [IEEE} [ILSVRC’12} [ILSVRC’14} [ILSVRC’14} [ILSVRC’15} – Batching BinaryConnect TernaryConnect – Reduce bit width [NIPS’15] [ICLR’16]

– Sparse weights Spatially SparseCNN Pruning SparseCNN [CIFAR-10 winner ‘14] [NIPS’15] [CVPR’15]

– Sparse activations DeepComp HashedNets [ICLR’16] [ICML’15] – Weight sharing SqueezeNet – Compact network

I W O ··· I 3 2 3 X = ··· 1 1 3 3 3 1 O 2 Shared Weights W

System Latency = I/O Latency + Compute Latency Compute Latency 0.3ms

F X P e G o A n

I/O Latency 0.25ms 1ms

55 © 2019 Intel Corporation Deep Learning Topology Inference Processing Feature vector Intel® FPGA Deep Learning for index “head” Acceleration Suite 1 features image image Tags Re-size / Neural net “head” crop “the body” 2 … Most of the compute is here Object detect “head” 10 Vision: CNNs (ResNet) Pre-processing Similar for Speech and Translation Post-processing

Intel® FPGA Deep Learning Acceleration Suite DDR Features DDR . CNN acceleration engine for common topologies executed in a graph loop Feature Map Cache architecture – AlexNet, GoogleNet, LeNet, SqueezeNet, Memory VGG16, ResNet, Yolo, SSD… Reader . Software Deployment Convolution PE /Writer Array – No FPGA compile required – Run-time reconfigurable . Customized Hardware Development Crossbar – Custom architecture creation w/ Config parameters Engine prim prim prim custom – Custom primitives using OpenCL™ flow

Trained Model . Supports common software frameworks (Caffe*, TensorFlow*) Caffe*, TensorFlow*, etc… . Model Optimizer enhances model for improved execution, storage, and transmission Model Optimizer Deep Learning FP Quantize Deployment Toolkit . Inference Engine optimizes inference execution across Intel® Model Compress hardware solutions using unified deployment API Model Analysis . Intel® FPGA DLA Suite provides turn-key or customized CNN Intermediate acceleration for common topologies Representation

Inference Engine Heterogenous Optimized Acceleration Engine DDR DLA Runtime CPU/FPGA Feature Map Cache MKL-DNN Memor Engine Deployment Pre-compiled Graph Architectures DDR y Reader GoogLeNet Optimized Template Conv DDR /Writer ResNet Optimized Template PE Array SqueezeNet Optimized Template DDR Intel® Intel® VGG Optimized Template Xeon® Crossbar Processor FPGA Additional, Generic CNN Templates Config Engine Hardware Customization Supported

Application Developer Design OpenVINO™ toolkit Run Program

Turnkey Software Deployment Flow

Bitstream Choose from many Neural Library precompiled FPGA Images Net Or FPGA Architecture Development Flow Custom create FPGA bitstream User Customization IP of DLA Suite Intel® FPGA SDK Architect Design Source Code Compile for OpenCL™

Acceleration Stack Platform Solution Edge Computing Solution

ML Framework (Caffe*, Software Stack ML Framework (Caffe*, Software Stack TensorFlow*) Hardware TensorFlow*) Hardware Platform & IP Platform & IP Application Application

DL Deployment Toolkit DL Deployment Toolkit

DLA Runtime Engine DLA Workload DLA Runtime Engine DLA Workload

OpenCL™ Runtime FPGA Interface OpenCL™ Runtime OpenCL™ BSP Acceleration Stack Manager OpenCL™ BSP Driver

Intel® Vision Intel® Intel® Xeon Intel® CPU Accelerator Programmable CPU Design with FPGA Acceleration Card Intel® FPGA Acceleration Hub Intel® Vision Accelerator Design Products

61 Intel® FPGA Deep Learning Acceleration(DLA for OpenVINO)

PreFeature-compiled Map Graph Cache Architecture Example Topologies

DDR Feature Map Cache AlexNet GoogLeNet Tiny Yolo Memory Convolution Memory DDR Processing Reader Reader Conv Element /Writer /WriterDDR VGG16 ResNet 18 SqueezeNet PE Array Array DDR SqueezeNet ResNet 101 ResNet 50 Crossbar SDD Crossbar Configuration Engine CUSTOM* PRIM PRIM PRIM Config MobileNet ResNet SSD …* ReLU Norm MaxPool Engine

*Deeper customization options *More topologies added with every release COMING SOON!

* Other names and brands may be claimed as the property of others. 62 DLA Architecture: Built for Performance

DDR Feature Map Cache . Maximize Parallelism on the FPGA Memory DDR Conv Reader /Writer DDR – Filter Parallelism (Processing Elements) PE Array DDR – Input-Depth Parallelism Crossbar – Winograd Transformation Config ReL Nor Max Engine – Batching U m Pool – Feature Stream Buffer Execute – Filter Cache Stream Buffer . Choosing FPGA Bitstream – Data Type / Design Exploration Convolution / Fully ReLU Norm MaxPool – Primitive Support Connected

Inew 푥 푦 1 1 = Iold 푥 + 푥′ 푦 + 푦′ × F 푥′ 푦′ 푥′=−1 푦′=−1

Filter Input Feature Map (3D Space) (Set of 2D Images)

Output Feature Map Repeat for Multiple Filters to Create Multiple “Layers” of Output Feature Map

AlexNet Graph

Stream Buffer

Convolution / ReLU Norm MaxPool Fully Connected