Model Serving for Deep Learning Hagay Lupesko, Amazon AI
©2018 Amazon Web Services, Inc. or its affiliates, All rights reserved
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Deep Learning is a Big Deal It has a growing impact on our lives
Autonomous Personalization Logistics Voice Vehicles
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Deep Learning is a Big Deal It’s able to do better than humans (and ML)
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. So what does a deployed model looks like?
Model
Mobile
Model Server Internet Desktop
IoT
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Performance
Availability
Networking The Undifferentiated Heavy Lifting of Monitoring
Model Serving Model Decoupling
Cross Framework
Cross Platform
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Apache MXNet - Background
• Apache (incubating) open source project • Framework for building and training DNNs • Created by academia (CMU and UW) • Adopted by AWS as DNN framework of choice, Nov 2016 http://mxnet.io
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Apache MXNet - Highlights
• Imperative, Symbolic and Dynamic APIs Ease of Use • APIs in Python, Scala, C, C++, R (and more) • Examples and tutorials
• Optimized for CPU, GPU, ARM (and more) Performance • Highly scalable distributed training • Quantization, Sparse, NCCL, and more…
• Train on cloud, predict on edge Portability • Model serving framework • ONNX support
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. It’s Demo Time!
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. REST and OpenAPI
Model Archive Containerization
ONNX Support Operational Metrics
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Model Archive
Trained Network
Model Model Archive Signature Model Export CLI Custom Code
Auxiliary Assets
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Containerization Lightweight virtualization, isolation, runs anywhere
MXNet Model Server Balancer Load Load MXNet NGINX MMS Container MMSMMS Container Container Docker Run MMS Docker Container Cluster Image
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Operational Metrics
• Requests Metrics • Latencies • Resources
• Model Name Dimensions • Host Name
• Log / CSV Target • AWS CloudWatch
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Back © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Challenges Ahead
• Batching • New players • Ease of use • Caching • ONNX • Internal • JIT • Plugins Amazon dev Compilation tools Platform • Custom code Adoption • Industry • Quantization partners Performance
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Try it out, Star and Contribute! github.com/awslabs/mxnet-model-server
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.