Amazon Managed Streaming for Apache Kafka Developer Guide Amazon Managed Streaming for Apache Kafka Developer Guide Amazon Managed Streaming for Apache Kafka: Developer Guide Amazon Managed Streaming for Apache Kafka Developer Guide Table of Contents What Is Amazon MSK? ........................................................................................................................ 1 Setting Up ........................................................................................................................................ 4 Sign Up for Amazon ................................................................................................................... 4 Download Libraries and Tools ...................................................................................................... 4 Getting Started .................................................................................................................................. 5 Step 1: Create a VPC .................................................................................................................. 5 Step 2: Enable High Availability and Fault Tolerance ....................................................................... 8 Step 3: Create a Cluster .............................................................................................................. 9 Step 4: Create a Client Machine ................................................................................................. 11 Step 5: Create a Topic .............................................................................................................. 12 Step 6: Produce and Consume Data ............................................................................................ 13 Step 7: View Metrics ................................................................................................................. 14 Step 8: Delete the Cluster ......................................................................................................... 15 How It Works ................................................................................................................................... 16 Creating a Cluster .................................................................................................................... 16 Broker types .................................................................................................................... 16 Creating a cluster using the Amazon Web Services Management Console ................................. 17 Creating a cluster using the Amazon CLI ............................................................................. 18 Creating a cluster with a custom MSK configuration using the Amazon CLI ............................... 19 Creating a cluster using the API ......................................................................................... 19 Deleting a Cluster .................................................................................................................... 19 Deleting a cluster using the Amazon Web Services Management Console ................................. 19 Deleting a cluster using the Amazon CLI ............................................................................. 19 Deleting a cluster using the API ......................................................................................... 20 Getting the Apache ZooKeeper Connection String ........................................................................ 20 Getting the Apache ZooKeeper connection string using the Amazon Web Services Management Console ........................................................................................................................... 20 Getting the Apache ZooKeeper connection string using the Amazon CLI .................................. 20 Getting the Apache ZooKeeper connection string using the API .............................................. 21 Getting the Bootstrap Brokers ................................................................................................... 21 Getting the bootstrap brokers using the Amazon Web Services Management Console ................ 21 Getting the bootstrap brokers using the Amazon CLI ............................................................ 22 Getting the bootstrap brokers using the API ........................................................................ 22 Listing Clusters ........................................................................................................................ 22 Listing clusters using the Amazon Web Services Management Console .................................... 22 Listing clusters using the Amazon CLI ................................................................................. 22 Listing clusters using the API ............................................................................................. 22 Scaling Up Broker Storage ......................................................................................................... 22 Automatic scaling ............................................................................................................. 23 Manual scaling ................................................................................................................. 24 Updating the Broker Type ......................................................................................................... 25 Updating the broker type using the Amazon Web Services Management Console ...................... 25 Updating the broker type using the Amazon CLI .................................................................. 26 Updating the broker type using the API .............................................................................. 27 Updating the Configuration of a Cluster ...................................................................................... 27 Updating the configuration of a cluster using the Amazon CLI ................................................ 27 Updating the configuration of a cluster using the API ........................................................... 29 Expanding a Cluster .................................................................................................................. 29 Expanding a cluster using the Amazon Web Services Management Console .............................. 29 Expanding a cluster using the Amazon CLI .......................................................................... 29 Expanding a cluster using the API ...................................................................................... 30 Updating Security ..................................................................................................................... 30 Updating a cluster's security settings using the Amazon Web Services Management Console ....... 31 Updating a cluster's security settings using the Amazon CLI ................................................... 31 iii Amazon Managed Streaming for Apache Kafka Developer Guide Updating a cluster's security settings using the API ............................................................... 32 Rebooting a Broker for a Cluster ................................................................................................ 32 Rebooting a Broker Using the Amazon Web Services Management Console .............................. 33 Rebooting a Broker Using the Amazon CLI ........................................................................... 33 Rebooting a Broker Using the API ...................................................................................... 32 Tagging a Cluster ..................................................................................................................... 34 Tag Basics ....................................................................................................................... 34 Tracking Costs Using Tagging ............................................................................................ 34 Tag Restrictions ................................................................................................................ 35 Tagging Resources Using the Amazon MSK API .................................................................... 35 Configuration ................................................................................................................................... 36 Custom Configurations .............................................................................................................. 36 Dynamic Configuration ...................................................................................................... 41 Topic-Level Configuration .................................................................................................. 41 States ............................................................................................................................. 41 Default Configuration ............................................................................................................... 41 Configuration Operations .......................................................................................................... 43 Create Configuration ......................................................................................................... 44 To update an MSK configuration ........................................................................................ 44 To delete an MSK configuration ........................................................................................