SUMMIT B E R L I N Apache Mxnet (Incubating) and Gluon: What’S in It for You?

SUMMIT B e r l i n Apache MXNet (Incubating) and Gluon: What’s in it for You?

Constantin Gonzalez Principal Solutions Architect AWS [email protected] | @zalez

Put machine learning in the hands of every developer

Vision Speech Language Chatbots Forecasting Recommendations

NEW NEW NEW A I SERVICES

A m a z o n A m a z o n A m a z o n A m a z o n A m a z o n A m a z o n A m a z o n A m a z o n A m a z o n A m a z o n Rekognition Rekognition T e x t r a c t P o l l y T r a n s c r i b e T r a n s l a t e C o m p r e h e n d L e x F o r e c a s t Personalize I m a g e V i d e o

N o t e b o o k s S u p e r v i s e d L e a r n i n g NEW T r a i n i n g D e p l o y m e n t A m a z o n ML SERVICES Ground Truth A l g o r i t h m s Unsupervised S a g e M a k e r L e a r n i n g NEW NEW NEW Optimization H o s t i n g ( N e o ) Marketplace Reinforcement L e a r n i n g

Frameworks Interfaces Infrastructure NEW NEW ML FRAMEWORKS & INFRASTRUCTURE A m a z o n A m a z o n F P G A s AWS Greengrass A m a z o n A m a z o n E C 2 P 3 NEWE C 2 C 5 E l a s t i c I n f e r e n t i a & P 3 D N I n f e r e n c e

History & Motivation

Being more productive as a developer

Getting more training done, in less time

Deployment: Easy, efficient and scalable

Portability and flexibility: Languages, devices, and frameworks

Getting started and Resources

imperative symbolic

Developer Productivity

Inference Training Efficiency Efficiency

2 푢 푡 + x

푧 휆 k

x x

x y a 1 b 1

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. MXNet computational dependency graph net = mx.sym.Variable('data') net = mx.sym.FullyConnected(net, name='fc1', num_hidden=64) net = mx.sym.Activation(net, name='relu1', act_type="relu") net = mx.sym.FullyConnected(net, name='fc2', num_hidden=10) net = mx.sym.SoftmaxOutput(net, name='softmax')

# create a trainable module on compute context mlp_model = mx.mod.Module(symbol=mlp, context=ctx) mlp_model.fit(train_iter,

eval_data=val_iter,

optimizer='sgd',

optimizer_params={'learning_rate':0.1},

eval_metric='acc',

batch_end_callback = mx.callback.Speedometer(batch_size, 100),

num_epoch=10)

Java Perl Julia Clojure

Python Scala C++ R

Frontend Backend

C++

net = gluon.nn.HybridSequential() with net.name_scope():

net.add(gluon.nn.Dense(units=64, activation='relu'))

net.add(gluon.nn.Dense(units=10)) softmax_cross_entropy = gluon.loss.SoftmaxCrossEntropyLoss() net.initialize(mx.init.Xavier(magnitude=2.24), ctx=ctx, force_reinit=True) trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': 0.02})

cumulative_loss = 0

for i, (data, label) in enumerate(train_data):

data = data.as_in_context(model_ctx).reshape((-1, 784))

label = label.as_in_context(model_ctx)

with autograd.record():

output = net(data)

loss = softmax_cross_entropy(output, label)

loss.backward()

trainer.step(data.shape[0])

Debuggable Flexible Scalable

50+ Pre-trained models, with training scripts, datasets, tutorials https://gluon-cv.mxnet.io

• Track waffle fry freshness • Identify fries that have exceeded hold time • Gluon Computer vision model for object detection and tracking • A team of students with no ML expertise • 12 months from no ML knowledge to completion

• 300+ word embedding pre-trained models • 5 language models • Neural Machine Translation (Google NMT, Transformer) • Flexible data pipeline tools • Public datasets • NLP examples, e.g. sentiment analysis

https://mxnet.incubator.apache.org/tutorials/vision/large_scale_classification.html

• Deep Learning algorithms built with MXNet • Computer vision and driving simulation • MXNet teaches computers to Recognize and track objects • Avoid collisions and prioritize safety • Largest simulation of its kind • Simulated a billion miles of road driving with wide range of variables and driving conditions

1 2 3

Collect and Choose and Set up and Train and Deploy models Scale and manage prepare training optimize your manage Tune ML Models in production the production data ML algorithm environments environment for training

Model Parameters Hyper Parameters

Many Frameworks MXNet

SNPE Caffe2 Many Platforms

ONNX: Common NGraph PyTorch O(n2) Intermediate Representation (IR) Pairs • Open source • Community driven TensorRT TF • Simple CoreML CNTK

• Deep Learning for Humans • 2nd most popular Deep Learning framework • Now Keras users can leverage MXNet’s performance

Training Inference Instance P3.8x Large, P3.16x Large C5.xLarge, C4.8xLarge Network ResNet50v1 ResNet50v1 Batch size 32 * Num of GPUs 32 Image size 3*256*256 3*256*256

ResNet50-Synthetic data 1200 Keras-MXNet Keras-TensorFlow 1000 GPUs [ Image/sec ] [ Image/sec ] Speed Up

800 1 194 184 600 1.05 400 4 764 393 200 1.94

0 1 GPU 4 GPU 8 GPU 8 1068 261 4.09 Keras-MXNet Keras-TensorFlow

ResNet50-ImageNet data GPUs Keras-MXNet Keras-TensorFlow Speed Up 1200 1000 1 135 52 2.59 800

600 4 536 162 3.30

400

200 8 722 211 3.42

0 1 GPU 4 GPU 8 GPU

Keras-MXNet Keras-TensorFlow

ResNet50 - Batch Inference 30

25 20 Instance Keras-MXNet Keras-TensorFlow Speed Up 15 10 C5.X Large 5.79 3.27 1.782 5

0 C5.8X Large 27.9 18.2 1 2 1.53

Keras-MXNet Keras-TensorFlow

Model

Mobile

Model Server Internet Desktop

IoT

Availability

The Undifferentiated Networking

Heavy Lifting of Monitoring Model Serving Model Decoupling

Cross Framework

Cross Platform Model Archive

Trained Network Model Model Archive Signature Model Export Custom CLI Code

MXNet Model Server

MXNet NGINX Pull or Build MMS Container Push MMS Container MMS Container Launch MMS Dockerfile Container Cluster

Model Inference Artifacts InstanceType: c3.4xlarge Endpoint InitialInstanceCount: 3

I ModelName: prod

VariantName: primary ML Hosting Service 30 50 InitialVariantWeight: 50

ProductionVariant

10 10 One-Click!

Inference Image Model versions

EndpointConfiguration

Amazon ECR Amazon Provided Algorithms

Amazon SageMaker

https://aws.amazon.com/blogs/machine-learning/ introducing-nnvm-compiler-a-new-open-end-to-end-compiler-for-ai-frameworks/

Get accuracy Automatic Broad framework Broad hardware and performance optimization support support

KEY FEATURES

Open-source Neo-AI device runtime and compiler under the Apache software license; 1/10th the size of original frameworks

CUDA & CuDNN TensorRT MKL, MKLML & MKLDNN pip install mxnet-cu92 pip install mxnet-tensorrt-cu92 e.g. pip install mxnet-mkl

TVM

TVM ONNX

Model Zoo TensorRT GluonCV Model Server GluonNLP

MXBoard Keras

• Amazon SageMaker: aws.amazon.com/sagemaker • Amazon SageMaker Neo: aws.amazon.com/sagemaker/neo • Amazon SageMaker Reinforcement Learning: aws.amazon.com/about-aws/whats-new/2018/11/amazon-sagemaker- announces-support-for-reinforcement-learning • Amazon Elastic Inference: aws.amazon.com/machine-learning/elastic-inference • AWS IoT Greengrass ML Inference: aws.amazon.com/greengrass/ml • Dynamic Training with Apache MXNet on AWS: https://aws.amazon.com/about-aws/whats-new/2018/11/introducing-dynamic- training-with-apache-mxnet/

• GitHub: github.com/apache/incubator-mxnet • Discussion forum: discuss.mxnet.io • Blog: medium.com/apache-mxnet • SlideShare: slideshare.net/apachemxnet • Twitter: @ApacheMXNet • YouTube : youtube.com/apachemxnet • Reddit: r/mxnet/ • Meetup: meetup.com/pro/deep-learning-with-apache-mxnet

[email protected]

Constantin Gonzalez [email protected] | @zalez