SUMMIT B e r l i n Apache MXNet (Incubating) and Gluon: What’s in it for You?
Constantin Gonzalez Principal Solutions Architect AWS [email protected] | @zalez
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Our mission at AWS
Put machine learning in the hands of every developer
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. The Amazon Machine Learning stack
Vision Speech Language Chatbots Forecasting Recommendations
NEW NEW NEW A I SERVICES
A m a z o n A m a z o n A m a z o n A m a z o n A m a z o n A m a z o n A m a z o n A m a z o n A m a z o n A m a z o n Rekognition Rekognition T e x t r a c t P o l l y T r a n s c r i b e T r a n s l a t e C o m p r e h e n d L e x F o r e c a s t Personalize I m a g e V i d e o
N o t e b o o k s S u p e r v i s e d L e a r n i n g NEW T r a i n i n g D e p l o y m e n t A m a z o n ML SERVICES Ground Truth A l g o r i t h m s Unsupervised S a g e M a k e r L e a r n i n g NEW NEW NEW Optimization H o s t i n g ( N e o ) Marketplace Reinforcement L e a r n i n g
Frameworks Interfaces Infrastructure NEW NEW ML FRAMEWORKS & INFRASTRUCTURE A m a z o n A m a z o n F P G A s AWS Greengrass A m a z o n A m a z o n E C 2 P 3 NEWE C 2 C 5 E l a s t i c I n f e r e n t i a & P 3 D N I n f e r e n c e
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. ML FRAMEWORKS & INFRASTRUCTURE
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. What you’ll learn about today
History & Motivation
Being more productive as a developer
Getting more training done, in less time
Deployment: Easy, efficient and scalable
Portability and flexibility: Languages, devices, and frameworks
Getting started and Resources
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. History
imperative symbolic
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Goals
Developer Productivity
Inference Training Efficiency Efficiency
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Computational dependency graph
3
2 푢 푡 + x
푧 휆 k
x x
x y a 1 b 1
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. MXNet computational dependency graph net = mx.sym.Variable('data') net = mx.sym.FullyConnected(net, name='fc1', num_hidden=64) net = mx.sym.Activation(net, name='relu1', act_type="relu") net = mx.sym.FullyConnected(net, name='fc2', num_hidden=10) net = mx.sym.SoftmaxOutput(net, name='softmax')
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Training import logging logging.getLogger().setLevel(logging.DEBUG) # logging to stdout
# create a trainable module on compute context mlp_model = mx.mod.Module(symbol=mlp, context=ctx) mlp_model.fit(train_iter,
eval_data=val_iter,
optimizer='sgd',
optimizer_params={'learning_rate':0.1},
eval_metric='acc',
batch_end_callback = mx.callback.Speedometer(batch_size, 100),
num_epoch=10)
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. MXNet computational dependency graph net = mx.sym.Variable('data') net = mx.sym.FullyConnected(net, name='fc1', num_hidden=64) net = mx.sym.Activation(net, name='relu1', act_type="relu") net = mx.sym.FullyConnected(net, name='fc2', num_hidden=10) net = mx.sym.SoftmaxOutput(net, name='softmax')
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multi-language support
Java Perl Julia Clojure
Python Scala C++ R
Frontend Backend
C++
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Simple, Easy-to- Flexible, Imperative Dynamic Graphs High Performance Understand Code Structure
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Network definition in Gluon
net = gluon.nn.HybridSequential() with net.name_scope():
net.add(gluon.nn.Dense(units=64, activation='relu'))
net.add(gluon.nn.Dense(units=10)) softmax_cross_entropy = gluon.loss.SoftmaxCrossEntropyLoss() net.initialize(mx.init.Xavier(magnitude=2.24), ctx=ctx, force_reinit=True) trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': 0.02})
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Training in Gluon smoothing_constant = .01 for e in range(10):
cumulative_loss = 0
for i, (data, label) in enumerate(train_data):
data = data.as_in_context(model_ctx).reshape((-1, 784))
label = label.as_in_context(model_ctx)
with autograd.record():
output = net(data)
loss = softmax_cross_entropy(output, label)
loss.backward()
trainer.step(data.shape[0])
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Imperative API
Debuggable Flexible Scalable
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gluon Model Zoo
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. GluonCV: a deep learning toolkit for computer vision
50+ Pre-trained models, with training scripts, datasets, tutorials https://gluon-cv.mxnet.io
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. GluonCV pre-trained models
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. GluonCV example code
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Chick-fil-A keeps waffle fries fresh using MXNet
• Track waffle fry freshness • Identify fries that have exceeded hold time • Gluon Computer vision model for object detection and tracking • A team of students with no ML expertise • 12 months from no ML knowledge to completion
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. GluonNLP: a deep learning toolkit for natural language processing
• 300+ word embedding pre-trained models • 5 language models • Neural Machine Translation (Google NMT, Transformer) • Flexible data pipeline tools • Public datasets • NLP examples, e.g. sentiment analysis
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Training efficiency – 92%
https://mxnet.incubator.apache.org/tutorials/vision/large_scale_classification.html
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. TuSimple uses MXNet for autonomous vehicles
• Deep Learning algorithms built with MXNet • Computer vision and driving simulation • MXNet teaches computers to Recognize and track objects • Avoid collisions and prioritize safety • Largest simulation of its kind • Simulated a billion miles of road driving with wide range of variables and driving conditions
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker: Build, train, and deploy ML models at scale
1 2 3
Collect and Choose and Set up and Train and Deploy models Scale and manage prepare training optimize your manage Tune ML Models in production the production data ML algorithm environments environment for training
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. ONNX
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Portability with ONNX
Model Parameters Hyper Parameters
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Open Neural Network eXchange - Overview
Many Frameworks MXNet
SNPE Caffe2 Many Platforms
ONNX: Common NGraph PyTorch O(n2) Intermediate Representation (IR) Pairs • Open source • Community driven TensorRT TF • Simple CoreML CNTK
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Supported tools
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Supported runtimes
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Supported compilers and visualizers
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. KerasKeras – MXNetMXNet hthttps://github.com/awslabs/kerastps://github.com/awslabs/keras-apacheapache-mxnet
SUMMITSUM M I T © 2019,2019, AmazonAmazon WebWeb Services,Services, Inc.Inc. oror itsits affiliates.affiliates. AllAll rightsrights reserved.reserved. Keras – Apache MXNet
• Deep Learning for Humans • 2nd most popular Deep Learning framework • Now Keras users can leverage MXNet’s performance
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Keras benchmarks Setup: https://github.com/awslabs/keras-apache-mxnet/tree/master/benchmark
Training Inference Instance P3.8x Large, P3.16x Large C5.xLarge, C4.8xLarge Network ResNet50v1 ResNet50v1 Batch size 32 * Num of GPUs 32 Image size 3*256*256 3*256*256
ResNet50-Synthetic data 1200 Keras-MXNet Keras-TensorFlow 1000 GPUs [ Image/sec ] [ Image/sec ] Speed Up
800 1 194 184 600 1.05 400 4 764 393 200 1.94
0 1 GPU 4 GPU 8 GPU 8 1068 261 4.09 Keras-MXNet Keras-TensorFlow
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Keras benchmarks
ResNet50-ImageNet data GPUs Keras-MXNet Keras-TensorFlow Speed Up 1200 1000 1 135 52 2.59 800
600 4 536 162 3.30
400
200 8 722 211 3.42
0 1 GPU 4 GPU 8 GPU
Keras-MXNet Keras-TensorFlow
ResNet50 - Batch Inference 30
25 20 Instance Keras-MXNet Keras-TensorFlow Speed Up 15 10 C5.X Large 5.79 3.27 1.782 5
0 C5.8X Large 27.9 18.2 1 2 1.53
Keras-MXNet Keras-TensorFlow
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Model Server
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. WhatSodoes whata modeldoes aserver deployedlook modellike? looks like?
Model
Mobile
Model Server Internet Desktop
IoT
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Performance
Availability
The Undifferentiated Networking
Heavy Lifting of Monitoring Model Serving Model Decoupling
Cross Framework
Cross Platform Model Archive
Trained Network Model Model Archive Signature Model Export Custom CLI Code
Auxiliary Assets SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Containerization Lightweight virtualization, isolation, runs anywhere
MXNet Model Server
MXNet NGINX Pull or Build MMS Container Push MMS Container MMS Container Launch MMS Dockerfile Container Cluster
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Deployment with Amazon SageMaker
Model Inference Artifacts InstanceType: c3.4xlarge Endpoint InitialInstanceCount: 3
I ModelName: prod
VariantName: primary ML Hosting Service 30 50 InitialVariantWeight: 50
ProductionVariant
10 10 One-Click!
Inference Image Model versions
EndpointConfiguration
Amazon ECR Amazon Provided Algorithms
Amazon SageMaker
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. TVM and NNVM
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Inference efficiency – TVM/NNVM
https://aws.amazon.com/blogs/machine-learning/ introducing-nnvm-compiler-a-new-open-end-to-end-compiler-for-ai-frameworks/
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Neo Train once, run anywhere with 2x the performance
Get accuracy Automatic Broad framework Broad hardware and performance optimization support support
KEY FEATURES
Open-source Neo-AI device runtime and compiler under the Apache software license; 1/10th the size of original frameworks
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Deep Learning acceleration
CUDA & CuDNN TensorRT MKL, MKLML & MKLDNN pip install mxnet-cu92 pip install mxnet-tensorrt-cu92 e.g. pip install mxnet-mkl
TVM
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Apache MXNet ecosystem
TVM ONNX
Model Zoo TensorRT GluonCV Model Server GluonNLP
MXBoard Keras
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Apache MXNet customers
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Getting started with Apache MXNet on AWS
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Using Apache MXNet with AWS ML services
• Amazon SageMaker: aws.amazon.com/sagemaker • Amazon SageMaker Neo: aws.amazon.com/sagemaker/neo • Amazon SageMaker Reinforcement Learning: aws.amazon.com/about-aws/whats-new/2018/11/amazon-sagemaker- announces-support-for-reinforcement-learning • Amazon Elastic Inference: aws.amazon.com/machine-learning/elastic-inference • AWS IoT Greengrass ML Inference: aws.amazon.com/greengrass/ml • Dynamic Training with Apache MXNet on AWS: https://aws.amazon.com/about-aws/whats-new/2018/11/introducing-dynamic- training-with-apache-mxnet/
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Project home page
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Staying in touch
• GitHub: github.com/apache/incubator-mxnet • Discussion forum: discuss.mxnet.io • Blog: medium.com/apache-mxnet • SlideShare: slideshare.net/apachemxnet • Twitter: @ApacheMXNet • YouTube : youtube.com/apachemxnet • Reddit: r/mxnet/ • Meetup: meetup.com/pro/deep-learning-with-apache-mxnet
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Contact us!
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Summary
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. What will you build?
Constantin Gonzalez [email protected] | @zalez
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. What will you build?
Constantin Gonzalez [email protected] | @zalez
SUMMIT © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.