Amazon Elastic Inference Developer Guide Amazon Elastic Inference Developer Guide Amazon Elastic Inference: Developer Guide Copyright © Amazon Web Services, Inc. and/or its affiliates. All rights reserved. Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon. Amazon Elastic Inference Developer Guide Table of Contents What Is Amazon Elastic Inference? ....................................................................................................... 1 Prerequisites .............................................................................................................................. 1 Pricing for Amazon Elastic Inference ............................................................................................. 1 Elastic Inference Uses ................................................................................................................. 1 Elastic Inference Basics ............................................................................................................... 2 Elastic Inference Uses ......................................................................................................... 1 Getting Started .......................................................................................................................... 3 Amazon Elastic Inference Service Limits ................................................................................ 3 Choosing an Instance and Accelerator Type for Your Model ..................................................... 5 Using Amazon Elastic Inference with EC2 Auto Scaling ............................................................ 5 Working with Amazon Elastic Inference ................................................................................................ 6 Setting Up ................................................................................................................................ 6 Configuring Your Security Groups for Elastic Inference ............................................................ 6 Configuring AWS PrivateLink Endpoint Services ..................................................................... 7 Configuring an Instance Role with an Elastic Inference Policy ................................................... 8 Launching an Instance with Elastic Inference ......................................................................... 9 TensorFlow Models ................................................................................................................... 11 Elastic Inference Enabled TensorFlow .................................................................................. 11 Additional Requirements and Considerations ....................................................................... 12 TensorFlow Elastic Inference with Python ............................................................................ 12 TensorFlow 2 Elastic Inference with Python ......................................................................... 21 MXNet Models ......................................................................................................................... 30 More Models and Resources ............................................................................................... 31 MXNet Elastic Inference with Python ................................................................................. 31 MXNet Elastic Inference with Deep Java Library (DJL) ............................................................ 51 PyTorch Models ........................................................................................................................ 62 Compile Elastic Inference-enabled PyTorch models ............................................................... 63 Additional Requirements and Considerations ....................................................................... 64 PyTorch Elastic Inference with Python ................................................................................ 65 Monitoring Elastic Inference Accelerators ..................................................................................... 70 EI_VISIBLE_DEVICES ......................................................................................................... 70 EI Tool ............................................................................................................................ 71 Health Check ................................................................................................................... 74 MXNet Elastic Inference with SageMaker .................................................................................... 75 Using Amazon Deep Learning Containers With Elastic Inference ............................................................. 76 Using Amazon Deep Learning Containers with Amazon Elastic Inference on Amazon EC2 .................... 76 Prerequisites .................................................................................................................... 76 Using TensorFlow Elastic Inference accelerators on EC2 ......................................................... 77 Using MXNet Elastic Inference accelerators on Amazon EC2 ................................................... 78 Using PyTorch Elastic Inference accelerators on Amazon EC2 .................................................. 79 Using Deep Learning Containers with Amazon Deep Learning Containers on Amazon ECS ................... 80 Prerequisites .................................................................................................................... 80 Using TensorFlow Elastic Inference accelerators on Amazon ECS ............................................. 81 Using MXNet Elastic Inference accelerators on Amazon ECS ................................................... 83 Using PyTorch Elastic Inference accelerators on Amazon ECS .................................................. 86 Using Amazon Deep Learning Containers with Elastic Inference on Amazon SageMaker ...................... 89 Security ........................................................................................................................................... 90 Identity and Access Management ............................................................................................... 90 Authenticating With Identities ............................................................................................ 91 Managing Access Using Policies .......................................................................................... 92 Logging and Monitoring ............................................................................................................ 94 Compliance Validation .............................................................................................................. 94 Resilience ................................................................................................................................ 95 Infrastructure Security .............................................................................................................. 95 iii Amazon Elastic Inference Developer Guide Configuration and Vulnerability Analysis ..................................................................................... 95 Using CloudWatch Metrics to Monitor Elastic Inference ......................................................................... 96 Elastic Inference Metrics and Dimensions ..................................................................................... 96 Creating CloudWatch Alarms to Monitor Elastic Inference .............................................................. 98 Troubleshooting ............................................................................................................................... 99 Issues Launching Accelerators .................................................................................................... 99 Resolving Configuration Issues ................................................................................................... 99 Issues Running AWS Batch ........................................................................................................ 99 Resolving Permission Issues ..................................................................................................... 100 Stop and Start the Instance ..................................................................................................... 100 Troubleshooting Model Performance ......................................................................................... 100 Submitting Feedback .............................................................................................................. 100 Amazon Elastic Inference Error Codes ....................................................................................... 101 Document History .......................................................................................................................... 105 AWS glossary ................................................................................................................................. 106 iv Amazon Elastic Inference Developer Guide Prerequisites What Is Amazon Elastic