Amazon Managed Streaming for Apache Kafka Developer Guide Amazon Managed Streaming for Apache Kafka Developer Guide Amazon Managed Streaming for Apache Kafka: Developer Guide Copyright © Amazon Web Services, Inc. and/or its affiliates. All rights reserved. Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon. Amazon Managed Streaming for Apache Kafka Developer Guide Table of Contents What Is Amazon MSK? ........................................................................................................................ 1 Setting Up ........................................................................................................................................ 4 Sign Up for AWS ........................................................................................................................ 4 Download Libraries and Tools ...................................................................................................... 4 Getting Started .................................................................................................................................. 5 Step 1: Create a VPC .................................................................................................................. 5 Step 2: Enable High Availability and Fault Tolerance ....................................................................... 8 Step 3: Create a Cluster .............................................................................................................. 9 Step 4: Create a Client Machine ................................................................................................. 11 Step 5: Create a Topic .............................................................................................................. 11 Step 6: Produce and Consume Data ............................................................................................ 13 Step 7: View Metrics ................................................................................................................. 14 Step 8: Delete the Cluster ......................................................................................................... 15 How It Works ................................................................................................................................... 16 Creating a Cluster .................................................................................................................... 16 Broker types .................................................................................................................... 16 Creating a cluster using the AWS Management Console ......................................................... 17 Creating a cluster using the AWS CLI .................................................................................. 18 Creating a cluster with a custom MSK configuration using the AWS CLI .................................... 19 Creating a cluster using the API ......................................................................................... 19 Deleting a Cluster .................................................................................................................... 19 Deleting a cluster using the AWS Management Console ......................................................... 19 Deleting a cluster using the AWS CLI .................................................................................. 19 Deleting a cluster using the API ......................................................................................... 20 Getting the Apache ZooKeeper Connection String ........................................................................ 20 Getting the Apache ZooKeeper connection string using the AWS Management Console .............. 20 Getting the Apache ZooKeeper connection string using the AWS CLI ....................................... 20 Getting the Apache ZooKeeper connection string using the API .............................................. 21 Getting the Bootstrap Brokers ................................................................................................... 21 Getting the bootstrap brokers using the AWS Management Console ........................................ 21 Getting the bootstrap brokers using the AWS CLI ................................................................. 22 Getting the bootstrap brokers using the API ........................................................................ 22 Listing Clusters ........................................................................................................................ 22 Listing clusters using the AWS Management Console ............................................................ 22 Listing clusters using the AWS CLI ...................................................................................... 22 Listing clusters using the API ............................................................................................. 22 Scaling Up Broker Storage ......................................................................................................... 22 Automatic scaling ............................................................................................................. 23 Manual scaling ................................................................................................................. 24 Updating the Broker Type ......................................................................................................... 25 Updating the broker type using the AWS Management Console .............................................. 25 Updating the broker type using the AWS CLI ....................................................................... 26 Updating the broker type using the API .............................................................................. 27 Updating the Configuration of a Cluster ...................................................................................... 27 Updating the configuration of a cluster using the AWS CLI .................................................... 27 Updating the configuration of a cluster using the API ........................................................... 28 Expanding a Cluster .................................................................................................................. 29 Expanding a cluster using the AWS Management Console ...................................................... 29 Expanding a cluster using the AWS CLI ............................................................................... 29 Expanding a cluster using the API ...................................................................................... 30 Updating Security ..................................................................................................................... 30 Updating a cluster's security settings using the AWS Management Console .............................. 31 Updating a cluster's security settings using the AWS CLI ........................................................ 31 Updating a cluster's security settings using the API ............................................................... 32 iii Amazon Managed Streaming for Apache Kafka Developer Guide Rebooting a Broker for a Cluster ................................................................................................ 32 Rebooting a Broker Using the AWS Management Console ...................................................... 32 Rebooting a Broker Using the AWS CLI ............................................................................... 33 Rebooting a Broker Using the API ...................................................................................... 32 Tagging a Cluster ..................................................................................................................... 34 Tag Basics ....................................................................................................................... 34 Tracking Costs Using Tagging ............................................................................................ 34 Tag Restrictions ................................................................................................................ 34 Tagging Resources Using the Amazon MSK API .................................................................... 35 Configuration ................................................................................................................................... 36 Custom Configurations .............................................................................................................. 36 Dynamic Configuration ...................................................................................................... 41 Topic-Level Configuration .................................................................................................. 41 States ............................................................................................................................. 41 Default Configuration ............................................................................................................... 41 Configuration Operations .........................................................................................................