Model Serving for Hagay Lupesko, Amazon AI

©2018 , Inc. or its affiliates, All rights reserved

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Deep Learning is a Big Deal It has a growing impact on our lives

Autonomous Personalization Logistics Voice Vehicles

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Deep Learning is a Big Deal It’s able to do better than humans (and ML)

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. So what does a deployed model looks like?

Model

Mobile

Model Server Internet Desktop

IoT

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Performance

Availability

Networking The Undifferentiated Heavy Lifting of Monitoring

Model Serving Model Decoupling

Cross Framework

Cross Platform

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Apache MXNet - Background

• Apache (incubating) open source project • Framework for building and training DNNs • Created by academia (CMU and UW) • Adopted by AWS as DNN framework of choice, Nov 2016 http://mxnet.io

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Apache MXNet - Highlights

• Imperative, Symbolic and Dynamic APIs Ease of Use • APIs in Python, Scala, C, C++, (and more) • Examples and tutorials

• Optimized for CPU, GPU, ARM (and more) Performance • Highly scalable distributed training • Quantization, Sparse, NCCL, and more…

• Train on cloud, predict on edge Portability • Model serving framework • ONNX support

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. It’s Demo Time!

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. REST and OpenAPI

Model Archive Containerization

ONNX Support Operational Metrics

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Model Archive

Trained Network

Model Model Archive Signature Model Export CLI Custom Code

Auxiliary Assets

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Containerization Lightweight virtualization, isolation, runs anywhere

MXNet Model Server Balancer Load Load MXNet NGINX MMS Container MMSMMS Container Container Docker Run MMS Docker Container Cluster Image

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Operational Metrics

• Requests Metrics • Latencies • Resources

• Model Name Dimensions • Host Name

• Log / CSV Target • AWS CloudWatch

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Back © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Challenges Ahead

• Batching • New players • Ease of use • Caching • ONNX • Internal • JIT • Plugins Amazon dev Compilation tools Platform • Custom code Adoption • Industry • Quantization partners Performance

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Try it out, Star and Contribute! github.com/awslabs/mxnet-model-server

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.