DOCSLIB.ORG

Amazon Managed Streaming for Apache Kafka Developer Guide Amazon Managed Streaming for Apache Kafka Developer Guide

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Amazon Managed Streaming for Apache Kafka Developer Guide Amazon Managed Streaming for Apache Kafka Developer Guide Amazon Managed Streaming for Apache Kafka Developer Guide Amazon Managed Streaming for Apache Kafka Developer Guide Amazon Managed Streaming for Apache Kafka: Developer Guide Amazon Managed Streaming for Apache Kafka Developer Guide Table of Contents What Is Amazon MSK? ........................................................................................................................ 1 Setting Up ........................................................................................................................................ 4 Sign Up for Amazon ................................................................................................................... 4 Download Libraries and Tools ...................................................................................................... 4 Getting Started .................................................................................................................................. 5 Step 1: Create a VPC .................................................................................................................. 5 Step 2: Enable High Availability and Fault Tolerance ....................................................................... 8 Step 3: Create a Cluster .............................................................................................................. 9 Step 4: Create a Client Machine ................................................................................................. 11 Step 5: Create a Topic .............................................................................................................. 12 Step 6: Produce and Consume Data ............................................................................................ 13 Step 7: View Metrics ................................................................................................................. 14 Step 8: Delete the Cluster ......................................................................................................... 15 How It Works ................................................................................................................................... 16 Creating a Cluster .................................................................................................................... 16 Broker types .................................................................................................................... 16 Creating a cluster using the Amazon Web Services Management Console ................................. 17 Creating a cluster using the Amazon CLI ............................................................................. 18 Creating a cluster with a custom MSK configuration using the Amazon CLI ............................... 19 Creating a cluster using the API ......................................................................................... 19 Deleting a Cluster .................................................................................................................... 19 Deleting a cluster using the Amazon Web Services Management Console ................................. 19 Deleting a cluster using the Amazon CLI ............................................................................. 19 Deleting a cluster using the API ......................................................................................... 20 Getting the Apache ZooKeeper Connection String ........................................................................ 20 Getting the Apache ZooKeeper connection string using the Amazon Web Services Management Console ........................................................................................................................... 20 Getting the Apache ZooKeeper connection string using the Amazon CLI .................................. 20 Getting the Apache ZooKeeper connection string using the API .............................................. 21 Getting the Bootstrap Brokers ................................................................................................... 21 Getting the bootstrap brokers using the Amazon Web Services Management Console ................ 21 Getting the bootstrap brokers using the Amazon CLI ............................................................ 22 Getting the bootstrap brokers using the API ........................................................................ 22 Listing Clusters ........................................................................................................................ 22 Listing clusters using the Amazon Web Services Management Console .................................... 22 Listing clusters using the Amazon CLI ................................................................................. 22 Listing clusters using the API ............................................................................................. 22 Scaling Up Broker Storage ......................................................................................................... 22 Automatic scaling ............................................................................................................. 23 Manual scaling ................................................................................................................. 24 Updating the Broker Type ......................................................................................................... 25 Updating the broker type using the Amazon Web Services Management Console ...................... 25 Updating the broker type using the Amazon CLI .................................................................. 26 Updating the broker type using the API .............................................................................. 27 Updating the Configuration of a Cluster ...................................................................................... 27 Updating the configuration of a cluster using the Amazon CLI ................................................ 27 Updating the configuration of a cluster using the API ........................................................... 29 Expanding a Cluster .................................................................................................................. 29 Expanding a cluster using the Amazon Web Services Management Console .............................. 29 Expanding a cluster using the Amazon CLI .......................................................................... 29 Expanding a cluster using the API ...................................................................................... 30 Updating Security ..................................................................................................................... 30 Updating a cluster's security settings using the Amazon Web Services Management Console ....... 31 Updating a cluster's security settings using the Amazon CLI ................................................... 31 iii Amazon Managed Streaming for Apache Kafka Developer Guide Updating a cluster's security settings using the API ............................................................... 32 Rebooting a Broker for a Cluster ................................................................................................ 32 Rebooting a Broker Using the Amazon Web Services Management Console .............................. 33 Rebooting a Broker Using the Amazon CLI ........................................................................... 33 Rebooting a Broker Using the API ...................................................................................... 32 Tagging a Cluster ..................................................................................................................... 34 Tag Basics ....................................................................................................................... 34 Tracking Costs Using Tagging ............................................................................................ 34 Tag Restrictions ................................................................................................................ 35 Tagging Resources Using the Amazon MSK API .................................................................... 35 Configuration ................................................................................................................................... 36 Custom Configurations .............................................................................................................. 36 Dynamic Configuration ...................................................................................................... 41 Topic-Level Configuration .................................................................................................. 41 States ............................................................................................................................. 41 Default Configuration ............................................................................................................... 41 Configuration Operations .......................................................................................................... 43 Create Configuration ......................................................................................................... 44 To update an MSK configuration ........................................................................................ 44 To delete an MSK configuration ........................................................................................
Load more

Recommended publications
  • Scalable Cloud Computing
    Scalable Cloud Computing Keijo Heljanko Department of Computer Science and Engineering School of Science Aalto University [email protected] 2.10-2013 Mobile Cloud Computing - Keijo Heljanko (keijo.heljanko@aalto.fi) 1/57 Guest Lecturer I Guest Lecturer: Assoc. Prof. Keijo Heljanko, Department of Computer Science and Engineering, Aalto University, I Email: [email protected] I Homepage: https://people.aalto.fi/keijo_heljanko I For more info into today’s topic, attend the course: “T-79.5308 Scalable Cloud Computing” Mobile Cloud Computing - Keijo Heljanko (keijo.heljanko@aalto.fi) 2/57 Business Drivers of Cloud Computing I Large data centers allow for economics of scale I Cheaper hardware purchases I Cheaper cooling of hardware I Example: Google paid 40 MEur for a Summa paper mill site in Hamina, Finland: Data center cooled with sea water from the Baltic Sea I Cheaper electricity I Cheaper network capacity I Smaller number of administrators / computer I Unreliable commodity hardware is used I Reliability obtained by replication of hardware components and a combined with a fault tolerant software stack Mobile Cloud Computing - Keijo Heljanko (keijo.heljanko@aalto.fi) 3/57 Cloud Computing Technologies A collection of technologies aimed to provide elastic “pay as you go” computing I Virtualization of computing resources: Amazon EC2, Eucalyptus, OpenNebula, Open Stack Compute, . I Scalable file storage: Amazon S3, GFS, HDFS, . I Scalable batch processing: Google MapReduce, Apache Hadoop, PACT, Microsoft Dryad, Google Pregel, Spark, ::: I Scalable datastore: Amazon Dynamo, Apache Cassandra, Google Bigtable, HBase,. I Distributed Coordination: Google Chubby, Apache Zookeeper, . I Scalable Web applications hosting: Google App Engine, Microsoft Azure, Heroku, .
    [Show full text]
  • Building a Modern, Scalable Cyber Intelligence Platform with Apache Kafka
    White Paper Information Security | Machine Learning October 2020 IT@Intel: Building a Modern, Scalable Cyber Intelligence Platform with Apache Kafka Our Apache Kafka data pipeline based on Confluent Platform ingests tens of terabytes per day, providing in-stream processing for faster security threat detection and response Intel IT Authors Executive Summary Ryan Clark Advanced cyber threats continue to increase in frequency and sophistication, Information Security Engineer threatening computing environments and impacting businesses’ ability to grow. Jen Edmondson More than ever, large enterprises must invest in effective information security, Product Owner using technologies that improve detection and response times. At Intel, we Dennis Kwong are transforming from our legacy cybersecurity systems to a modern, scalable Information Security Engineer Cyber Intelligence Platform (CIP) based on Kafka and Splunk. In our 2019 paper, Transforming Intel’s Security Posture with Innovations in Data Intelligence, we Jac Noel discussed the data lake, monitoring, and security capabilities of Splunk. This Security Solutions Architect paper describes the essential role Apache Kafka plays in our CIP and its key Elaine Rainbolt benefits, as shown here: Industry Engagement Manager ECONOMIES OPERATE ON DATA REDUCE TECHNICAL GENERATES OF SCALE IN STREAM DEBT AND CONTEXTUALLY RICH Paul Salessi DOWNSTREAM COSTS DATA Information Security Engineer Intel IT Contributors Victor Colvard Information Security Engineer GLOBAL ALWAYS MODERN KAFKA LEADERSHIP SCALE AND REACH ON ARCHITECTURE WITH THROUGH CONFLUENT Juan Fernandez THRIVING COMMUNITY EXPERTISE Technical Solutions Specialist Frank Ober SSD Principal Engineer Apache Kafka is the foundation of our CIP architecture. We achieve economies of Table of Contents scale as we acquire data once and consume it many times.
    [Show full text]
  • Zookeeper Administrator's Guide
    ZooKeeper Administrator's Guide A Guide to Deployment and Administration by Table of contents 1 Deployment........................................................................................................................ 2 1.1 System Requirements....................................................................................................2 1.2 Clustered (Multi-Server) Setup.....................................................................................2 1.3 Single Server and Developer Setup..............................................................................4 2 Administration.................................................................................................................... 4 2.1 Designing a ZooKeeper Deployment........................................................................... 5 2.2 Provisioning.................................................................................................................. 6 2.3 Things to Consider: ZooKeeper Strengths and Limitations..........................................6 2.4 Administering................................................................................................................6 2.5 Maintenance.................................................................................................................. 6 2.6 Supervision....................................................................................................................7 2.7 Monitoring.....................................................................................................................8
    [Show full text]
  • Unravel Data Systems Version 4.5
    UNRAVEL DATA SYSTEMS VERSION 4.5 Component name Component version name License names jQuery 1.8.2 MIT License Apache Tomcat 5.5.23 Apache License 2.0 Tachyon Project POM 0.8.2 Apache License 2.0 Apache Directory LDAP API Model 1.0.0-M20 Apache License 2.0 apache/incubator-heron 0.16.5.1 Apache License 2.0 Maven Plugin API 3.0.4 Apache License 2.0 ApacheDS Authentication Interceptor 2.0.0-M15 Apache License 2.0 Apache Directory LDAP API Extras ACI 1.0.0-M20 Apache License 2.0 Apache HttpComponents Core 4.3.3 Apache License 2.0 Spark Project Tags 2.0.0-preview Apache License 2.0 Curator Testing 3.3.0 Apache License 2.0 Apache HttpComponents Core 4.4.5 Apache License 2.0 Apache Commons Daemon 1.0.15 Apache License 2.0 classworlds 2.4 Apache License 2.0 abego TreeLayout Core 1.0.1 BSD 3-clause "New" or "Revised" License jackson-core 2.8.6 Apache License 2.0 Lucene Join 6.6.1 Apache License 2.0 Apache Commons CLI 1.3-cloudera-pre-r1439998 Apache License 2.0 hive-apache 0.5 Apache License 2.0 scala-parser-combinators 1.0.4 BSD 3-clause "New" or "Revised" License com.springsource.javax.xml.bind 2.1.7 Common Development and Distribution License 1.0 SnakeYAML 1.15 Apache License 2.0 JUnit 4.12 Common Public License 1.0 ApacheDS Protocol Kerberos 2.0.0-M12 Apache License 2.0 Apache Groovy 2.4.6 Apache License 2.0 JGraphT - Core 1.2.0 (GNU Lesser General Public License v2.1 or later AND Eclipse Public License 1.0) chill-java 0.5.0 Apache License 2.0 Apache Commons Logging 1.2 Apache License 2.0 OpenCensus 0.12.3 Apache License 2.0 ApacheDS Protocol
    [Show full text]
  • Talend Open Studio for Big Data Release Notes
    Talend Open Studio for Big Data Release Notes 6.0.0 Talend Open Studio for Big Data Adapted for v6.0.0. Supersedes previous releases. Publication date July 2, 2015 Copyleft This documentation is provided under the terms of the Creative Commons Public License (CCPL). For more information about what you can and cannot do with this documentation in accordance with the CCPL, please read: http://creativecommons.org/licenses/by-nc-sa/2.0/ Notices Talend is a trademark of Talend, Inc. All brands, product names, company names, trademarks and service marks are the properties of their respective owners. License Agreement The software described in this documentation is licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.html. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. This product includes software developed at AOP Alliance (Java/J2EE AOP standards), ASM, Amazon, AntlR, Apache ActiveMQ, Apache Ant, Apache Avro, Apache Axiom, Apache Axis, Apache Axis 2, Apache Batik, Apache CXF, Apache Cassandra, Apache Chemistry, Apache Common Http Client, Apache Common Http Core, Apache Commons, Apache Commons Bcel, Apache Commons JxPath, Apache
    [Show full text]
  • HDP 3.1.4 Release Notes Date of Publish: 2019-08-26
    Release Notes 3 HDP 3.1.4 Release Notes Date of Publish: 2019-08-26 https://docs.hortonworks.com Release Notes | Contents | ii Contents HDP 3.1.4 Release Notes..........................................................................................4 Component Versions.................................................................................................4 Descriptions of New Features..................................................................................5 Deprecation Notices.................................................................................................. 6 Terminology.......................................................................................................................................................... 6 Removed Components and Product Capabilities.................................................................................................6 Testing Unsupported Features................................................................................ 6 Descriptions of the Latest Technical Preview Features.......................................................................................7 Upgrading to HDP 3.1.4...........................................................................................7 Behavioral Changes.................................................................................................. 7 Apache Patch Information.....................................................................................11 Accumulo...........................................................................................................................................................
    [Show full text]
  • Analysis and Detection of Anomalies in Mobile Devices
    Master’s Degree in Informatics Engineering Dissertation Final Report Analysis and detection of anomalies in mobile devices António Carlos Lagarto Cabral Bastos de Lima [email protected] Supervisor: Prof. Dr. Tiago Cruz Co-Supervisor: Prof. Dr. Paulo Simões Date: September 1, 2017 Master’s Degree in Informatics Engineering Dissertation Final Report Analysis and detection of anomalies in mobile devices António Carlos Lagarto Cabral Bastos de Lima [email protected] Supervisor: Prof. Dr. Tiago Cruz Co-Supervisor: Prof. Dr. Paulo Simões Date: September 1, 2017 i Acknowledgements I strongly believe that both nature and nurture playing an equal part in shaping an in- dividual, and that in the end, it is what you do with the gift of life that determines who you are. However, in order to achieve great things motivation alone might just not cut it, and that’s where surrounding yourself with people that want to watch you succeed and better yourself comes in. It makes the trip easier and more enjoyable, and there is a plethora of people that I want to acknowledge for coming this far. First of all, I’d like to thank professor Tiago Cruz for giving me the support, motivation and resources to work on this project. The idea itself started over one of our then semi- regular morning coffee conversations and from there it developed into a full-fledged concept quickly. But this acknowledgement doesn’t start there, it dates a few years back when I first had the pleasure of having him as my teacher in one of the introductory courses.
    [Show full text]
  • Open Source and Third Party Documentation
    Open Source and Third Party Documentation Verint.com Twitter.com/verint Facebook.com/verint Blog.verint.com Content Introduction.....................2 Licenses..........................3 Page 1 Open Source Attribution Certain components of this Software or software contained in this Product (collectively, "Software") may be covered by so-called "free or open source" software licenses ("Open Source Components"), which includes any software licenses approved as open source licenses by the Open Source Initiative or any similar licenses, including without limitation any license that, as a condition of distribution of the Open Source Components licensed, requires that the distributor make the Open Source Components available in source code format. A license in each Open Source Component is provided to you in accordance with the specific license terms specified in their respective license terms. EXCEPT WITH REGARD TO ANY WARRANTIES OR OTHER RIGHTS AND OBLIGATIONS EXPRESSLY PROVIDED DIRECTLY TO YOU FROM VERINT, ALL OPEN SOURCE COMPONENTS ARE PROVIDED "AS IS" AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. Any third party technology that may be appropriate or necessary for use with the Verint Product is licensed to you only for use with the Verint Product under the terms of the third party license agreement specified in the Documentation, the Software or as provided online at http://verint.com/thirdpartylicense. You may not take any action that would separate the third party technology from the Verint Product. Unless otherwise permitted under the terms of the third party license agreement, you agree to only use the third party technology in conjunction with the Verint Product.
    [Show full text]
  • An Empirical Performance Evaluation of Apache Kafka
    How Fast Can We Insert? An Empirical Performance Evaluation of Apache Kafka Guenter Hesse, Christoph Matthies, Matthias Uflacker Hasso Plattner Institute University of Potsdam Germany fi[email protected] about such study results is a prerequisite for making informed Abstract—Message brokers see widespread adoption in modern decisions about whether a system is suitable for the existing IT landscapes, with Apache Kafka being one of the most use cases. Additionally, it is also crucial for finding or fine- employed platforms. These systems feature well-defined APIs for use and configuration and present flexible solutions for tuning appropriate system configurations. various data storage scenarios. Their ability to scale horizontally The contributions of this research are as follows: enables users to adapt to growing data volumes and changing • We propose a user-centered and extensible monitoring environments. However, one of the main challenges concerning framework, which includes tooling for analyzing any message brokers is the danger of them becoming a bottleneck within an IT architecture. To prevent this, knowledge about the JVM-based system. amount of data a message broker using a specific configuration • We present an analysis that highlights the capabilities of can handle needs to be available. In this paper, we propose a Apache Kafka regarding the maximum achievable rate of monitoring architecture for message brokers and similar Java incoming records per time unit. Virtual Machine-based systems. We present a comprehensive • We enable reproducibility of the presented results by performance analysis of the popular Apache Kafka platform 1 using our approach. As part of the benchmark, we study selected making all needed artifacts available online .
    [Show full text]
  • Hortonworks Data Platform Date of Publish: 2018-09-21
    Release Notes 3 Hortonworks Data Platform Date of Publish: 2018-09-21 http://docs.hortonworks.com Contents HDP 3.0.1 Release Notes..........................................................................................3 Component Versions.............................................................................................................................................3 New Features........................................................................................................................................................ 3 Deprecation Notices..............................................................................................................................................4 Terminology.............................................................................................................................................. 4 Removed Components and Product Capabilities.....................................................................................4 Unsupported Features........................................................................................................................................... 4 Technical Preview Features......................................................................................................................4 Upgrading to HDP 3.0.1...................................................................................................................................... 5 Before you begin.....................................................................................................................................
    [Show full text]
  • Talend ESB Development Guide
    Talend ESB Development Guide 7.0.1 Talend ESB Adapted for v7.0.1. Supersedes previous releases. Publication date: April 13, 2018 Copyright © 2018 Talend Inc. All rights reserved. Copyleft This documentation is provided under the terms of the Creative Commons Public License (CCPL). For more information about what you can and cannot do with this documentation in accordance with the CCPL, please read: http://creativecommons.org/licenses/by-nc-sa/2.0/ This document may include documentation produced at The Apache Software Foundation which is licensed under The Apache License 2.0. Notices Talend and Talend ESB are trademarks of Talend, Inc. Apache CXF, CXF, Apache Karaf, Karaf, Apache Cellar, Cellar, Apache Camel, Camel, Apache Maven, Maven, Apache Archiva, Archiva, Apache Syncope, Syncope, Apache ActiveMQ, ActiveMQ, Apache Log4j, Log4j, Apache Felix, Felix, Apache ServiceMix, ServiceMix, Apache Ant, Ant, Apache Derby, Derby, Apache Tomcat, Tomcat, Apache ZooKeeper, ZooKeeper, Apache Jackrabbit, Jackrabbit, Apache Santuario, Santuario, Apache DS, DS, Apache Avro, Avro, Apache Abdera, Abdera, Apache Chemistry, Chemistry, Apache CouchDB, CouchDB, Apache Kafka, Kafka, Apache Lucene, Lucene, Apache MINA, MINA, Apache Velocity, Velocity, Apache FOP, FOP, Apache HBase, HBase, Apache Hadoop, Hadoop, Apache Shiro, Shiro, Apache Axiom, Axiom, Apache Neethi, Neethi, Apache WSS4J, WSS4J are trademarks of The Apache Foundation. Eclipse Equinox is a trademark of the Eclipse Foundation, Inc. SoapUI is a trademark of SmartBear Software. Hyperic is a trademark
    [Show full text]
  • The Geography of Job Tasks
    Working Papers WP 21-27 August 2021 https://doi.org/10.21799/frbp.wp.2021.27 The Geography of Job Tasks Enghin Atalay Federal Reserve Bank of Philadelphia Research Department Sebastian Sotelo University of Michigan-Ann Arbor Daniel Tannenbaum University of Nebraska-Lincoln ISSN: 1962-5361 Disclaimer: This Philadelphia Fed working paper represents preliminary research that is being circulated for discussion purposes. The views expressed in these papers are solely those of the authors and do not necessarily reflect the views of the Federal Reserve Bank of Philadelphia or the Federal Reserve System. Any errors or omissions are the responsibility of the authors. Philadelphia Fed working papers are free to download at: https://philadelphiafed.org/research-and-data/publications/working-papers. The Geography of Job Tasks ∗ Enghin Atalay Sebastian Sotelo Daniel Tannenbaum August 3, 2021 Abstract The returns to skills and the nature of work differ systematically across labor mar- kets of different sizes. Prior research has pointed to worker interactions, technological innovation, and specialization as key sources of urban productivity gains, but has been limited by the available data in its ability to fully characterize work across geographies. We study the sources of geographic inequality and present new facts about the geog- raphy of work using online job ads. We show that the (i) intensity of interactive and analytic tasks, (ii) technological requirements, and (iii) task specialization all increase with city size. The gradient for tasks and technologies exists both across and within occupations. It is also steeper for jobs requiring a college degree and for workers employed in non-tradable industries.
    [Show full text]