• Abstract and Figures
• Public Full-text

Industry Paper: Kafka versus RabbitMQ A comparative study of two industry reference publish/subscribe implementations Philippe Dobbelaere Kyumars Sheykh Esmaili Nokia Bell Labs Nokia Bell Labs Antwerp, Belgium Antwerp, Belgium ABSTRACT 1 INTRODUCTION Publish/subscribe is a distributed interaction paradigm well adapted e Internet has considerably changed the scale of distributed sys- to the deployment of scalable and loosely coupled systems. tems. Distributed systems now involve thousands of entities po- Apache Kaa and RabbitMQ are two popular open-source and tentially distributed all over the world whose location and behav- commercially-supported pub/sub systems that have been around for ior may greatly vary throughout the lifetime of the system. ese almost a decade and have seen wide adoption. Given the popularity constraints underline the need for more exible communication of these two systems and the fact that both are branded as pub/sub models and systems that reect the dynamic and decoupled na- systems, two frequently asked questions in the relevant online ture of the applications. Individual point-to-point and synchronous forums are: how do they compare against each other and which communications lead to rigid and static applications, and make the one to use? development of dynamic large-scale applications cumbersome [14]. In this paper, we frame the arguments in a holistic approach by To reduce the burden of application designers, the glue between establishing a common comparison framework based on the core the dierent entities in such large-scale seings should rather be functionalities of pub/sub systems. Using this framework, we then provided by a dedicated middleware infrastructure, based on an ad- venture into a qualitative and quantitative (i.e. empirical) compari- equate communication scheme. e publish/subscribe interaction son of the common features of the two systems. Additionally, we scheme provides the loosely coupled form of interaction required also highlight the distinct features that each of these systems has. in such large scale seings [14]. Aer enumerating a set of use cases that are best suited for Rab- Apache Kaa [1] and RabbitMQ [4] are two popular open-source bitMQ or Kaa, we try to guide the reader through a determination and commercially-supported pub/sub systems (by Conuent Inc. table to choose the best architecture given his/her particular set of and Pivotal) that have been around for almost a decade and have requirements. seen wide adoption in enterprise companies. Despite commonalities, these two systems have dierent his- tories and design goals, and distinct features. For example, they CCS CONCEPTS follow dierent architectural models: In RabbitMQ, producers pub- •So ware and its engineering →Message oriented middle- lish (batches of) message(s) with a routing key to a network of ware; Publish-subscribe / event-based architectures; •General exchanges where routing decisions happen, ending up in a queue and reference →Evaluation; where consumers can get at messages through a push (preferred) or pull mechanism. In Kaa producers publish (batches of) message(s) KEYWORDS to a disk based append log that is topic specic. Any number of consumers can pull stored messages through an index mechanism. Publish/Subscribe Systems, Message Brokers, Apache Kaa, Rab- Given the popularity of these two systems and the fact that both bitMQ, Log Files, AMQP, Performance, Reliability are branded as pub/sub systems, two frequently asked questions in the relevant online forums are: how do they compare against each ACM Reference format: other and which one to use? Philippe Dobbelaere and Kyumars Sheykh Esmaili. 2017. Industry Paper: While one can nd several ad-hoc recommendations (some based Kaa versus RabbitMQ. In Proceedings of DEBS ’17, Barcelona, Spain, June on pointed benchmarks) on the web, we found these hard to gen- 19-23, 2017, 12 pages. eralize to other applications and believe they do not do justice to DOI: 10.1145/3093742.3093908 the systems under discussion. More specically, (i) the bigger con- text of such analysis, e.g., qualitative comparison or the distinct features of each system is oen overlooked, (ii) due to the fast pace of developments, in particular in the case of Kaa, some of the Permission to make digital or hard copies of all or part of this work for personal or reported results are stale and no longer valid, and (iii) it is dicult classroom use is granted without fee provided that copies are not made or distributed to compare results across dierent experiments. for prot or commercial advantage and that copies bear this notice and the full citation on the rst page. Copyrights for components of this work owned by others than the In this paper, we frame the arguments in a more holistic approach. author(s) must be honored. Abstracting with credit is permied. To copy otherwise, or More concretely, we start (in Section 2) with a brief description republish, to post on servers or to redistribute to lists, requires prior specic permission of the core functionalities of publish/subscribe systems as well as and/or a fee. Request permissions from [email protected]. DEBS ’17, Barcelona, Spain the common quality-of-service guarantees they provide. We then © 2017 Copyright held by the owner/author(s). Publication rights licensed to ACM. (in Section 3) give a high-level description of both Apache Kaa 978-1-4503-5065-5/17/06...$15.00 and RabbitMQ. Based on the framework established in Section 2, DOI: 10.1145/3093742.3093908 227 DEBS ’17, June 19-23, 2017, Barcelona, Spain Philippe Dobbelaere and Kyumars Sheykh Esmaili we venture into a qualitative (in Section 4) and quantitative (in 2.2 ality-of-Service Guarantees Section 5) comparison of the common features of the two systems. In addition to the aforementioned core functionalities of pub/sub In addition to the common features, we list the important features systems, they are also dened by a relatively large set of required that are unique to either of the two in Section 6. We next enumer- and desired guarantees that are generally referred to as ality-of- ate a number of use case classes that are best-suited for Kaa or Service (QoS) guarantees [9, 11, 14]. RabbitMQ, as well as propose options for a combined use of the For sake of simplicity, we have grouped the most important two systems in Section 7. ere, we also propose a determination pub/sub QoS guarantees into ve separate categories and will ex- table to help choose the best architecture when given a particular plain them in the following sections. set of requirements. Finally, we conclude the paper in Section 8. It should be noted that an important assumption in this section is the distributed nature of modern pub/sub systems. Distribution is necessary (but not sucient) to bring scalability. However, it 2 BACKGROUND: PUB/SUB SYSTEMS brings a number of technical problems that make the design and In this section we highlight the main concepts of the publish/subscribe implementation of distributed storage, indexing and computing a paradigm, its required and desired guarantees as well as some of delicate issue [7]. its realizations out there. 2.2.1 Correctness. As proposed in [28], correctness behavior e primary purpose of this section is to establish a common can be dened using three primitives: no-loss, no-duplication, no- framework/language that will be used in the rest of the paper. disorder. Building upon these primitives, the following two criteria Knowledgeable readers may skip it. are relevant in pub/sub systems: • Delivery Guarantees, the three common variants are: ◦ at most once (aka “best eort”; guarantees no-duplicates): 2.1 Core Functionalities in this mode, under normal operating conditions, packets will be Publish/subscribe is a distributed interaction paradigm well adapted delivered, but during failure packet loss might occur. Trying to do to the deployment of scalable and loosely coupled systems. beer than this will always cost system resources, so this mode has Decoupling the publishers and subscribers is arguably the most the best throughput. fundamental functionality of a pub/sub system. Eugster et al. [14] ◦ at least once (guarantees no-loss): in this mode, the system have decomposed the decoupling that the pub/sub coordination will make sure that no packets get lost. Recovery from failures scheme provides along the following three dimensions: might cause duplicate packets to be sent, possibly out-of-order. ◦ exactly once (guarantees no-loss and no-duplicates): this (1) Entity decoupling: publishers and consumers do not requires an expensive end-to-end two phase commit. need to be aware of each other. e pub/sub infrastructure • Ordering Guarantees, the three common variants here are: terminates the interaction in the middle. ◦ no ordering: absence of ordering guarantees is an ideal case (2) Time decoupling: e interacting parties do not need to for performance. be actively participating in the interaction, or even stronger, ◦ partitioned ordering: in this mode, a single partition can be switched on, at the same time. dened for each message ow that needs to be consumed in-order. (3) Synchronization decoupling: the interaction between While more expensive than the previous group, it can possibly have either producer or consumer and the pub/sub infrastruc- high performance implementations. ture does not synchronously need to block the producer ◦ global order: due to the synchronization overhead, impos- or consumer execution threads, allowing maximum usage ing a global ordering guarantee across dierent channels requires of processor resources at producers and consumers alike. signicant additional resources and can severely penalize perfor- mance. Another core functionality of pub/sub systems is routing logic 2.2.2 Availability. Availability is the capacity of a system to (also known as subscription model) which decides if and where a maximize its uptime.

Load more

  12

Copy Link