This Paper Must Be Cited As

Document downloaded from:

http://hdl.handle.net/10251/64607

This paper must be cited as:

Luzuriaga Quichimbo, JE.; Pérez, M.; Boronat, P.; Cano Escribá, JC.; Tavares De Araujo Cesariny Calafate, CM.; Manzoni, P. (2015). A comparative evaluation of AMQP and MQTT protocols over unstable and mobile networks. 12th IEEE Consumer Communications and Networking Conference (CCNC 2015). IEEE. doi:10.1109/CCNC.2015.7158101.

The final publication is available at

http://dx.doi.org/10.1109/CCNC.2015.7158101

Additional Information

© 2015 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. A comparative evaluation of AMQP and MQTT protocols over unstable and mobile networks

Jorge E. Luzuriaga∗, Miguel Perez†, Pablo Boronat†, Juan Carlos Cano∗, Carlos Calafate∗, Pietro Manzoni∗ ∗Department of Computer Engineering Universitat Politecnica` de Valencia,` Valencia, SPAIN [email protected], jucano,calafate,[email protected] †Universitat Jaume I, Castello´ de la Plana, SPAIN [email protected], [email protected]

Abstract—Message oriented middleware (MOM) refers to business application [6]. It works like instant messaging or the software infrastructure supporting sending and receiving email, and the difference towards these available solutions is messages between distributed systems. AMQP and MQTT are the that AMQP comprises both a network protocol, which specifies two most relevant protocols in this context. They are extensively the entities (producer/consumer, broker) to interoperate with used for exchanging messages since they provide an abstraction each other, and a protocol model, which specifies the represen- of the different participating system entities, alleviating their tation of messages, and the commands to interoperate among coordination and simplifying the communication programming details. the entities. Furthermore, AMQP messages are self-contained, and data content in a message is opaque and immutable. Also, These protocols, however, have not been thoroughly tested there is no limit for the size of a message; it can either support in the context of mobile or dynamic networks like vehicular a 4 GByte message or a 4 KByte one, in any case ensuring networks. In this paper we present an experimental evaluation of security, reliability, and performance. both protocols in such scenarios, characterizing their behavior in terms of message loss, latency, jitter and saturation boundary MQTT is a lightweight machine-to-machine messaging values. Based on the results obtained, we provide criteria of protocol [2], and it has a clear focus on the mobile sector. applicability of these protocols, and we assess their performance Its information exchange procedure is resource-efficient, and and viability. This evaluation is of interest for the upcoming it does not specify a particular data format. Additionally, it applications of MOM, especially to systems related to the Internet of Things. provides security that all messages transmitted even if the connection breaks off briefly, solving problems that arise upon I.INTRODUCTION unreliable communications. Modern Internet-based applications are becoming more Both protocols use similar techniques for message delive- oriented towards the interaction of wireless and mobile devices ring. When a message is sent by a client to a message broker, it with cloud resources and services. For many years HTTP has is placed in a queue, and after that, all customers subscribed to been used as the reference communications protocol in this this queue automatically receive the message as a push notifi- context. HTTP is a very wide spread protocol, and APIs for cation [9]. To support a wide variety of usage scenarios, AMQP its use are available basically for every programming language. offers several possibilities for message delivery, namely: point- to-point, store-and-forward. However, the MQTT protocol However, more flexible middleware systems have been provides a basic messaging topic based on specified topics developed to ease the design of cloud-based applications. (subscription’s name) without possibility of store-and-forward Significant research efforts have been dedicated to define new mechanisms to support message delivery [13]. communication systems that connect distributed components via message passing; they are called Message Oriented Middle- Both of these protocols were designed to facilitate the ware (MOM). The basic idea of MOM is that communication dialogue among the components of a system, by simplifying takes place by adding messages to distributed queues, and message exchange independently of their underlying platforms. by getting messages from those queues. Based on the model There are libraries available for the most popular programming of Message Oriented Middleware, many protocols have been languages, and there are implementations for the most common developed, e.g. DDS, STOMP, XMPP. The two most widely operating systems and platforms. In addition, they take into used proposals are: the Advanced Message Queuing Proto- consideration security and confidentiality issues without affec- col (AMQP) and the Message Queuing Telemetry Transport ting significantly the communications performance. (MQTT). AMQP was created in 2003 by John O’Hara of In this paper we evaluate AMQP and MQTT, and we com- JPMorgan Chase and MQTT was originated in 1999 by Andy pare their capabilities and capacities through measurements Stanford-Clark of IBM. Both protocols provide some platform under a mobile or unstable wireless network testbed. We call agnostic methods to improve the communication and ensure unstable networks those in which links can be frequently that information is safely transported between systems. modified or broken without control. Examples of unstable AMQP is an open standard for enterprise messaging, networks are mobile networks or wireless networks in urban designed to support messaging for almost any distributed or environments, like community networks or vehicular networks, which suffer from channel interferences or node blackouts. Our goal is to determine whether these protocols provide a satisfactory service, depending of the applications’ load needs, in terms of message size and communication rates. We present the evaluation results regarding the effect of a mobile producer/publisher changing from one WiFi access point (AP) to another in the same IP network. We developed a synthetic load generator, called amqperf, that sends messages with a sequence number to detect losses or messages delivered in different sending order. The size of messages and the fre- Fig. 1: A picture of the scenario. quency in which they are sent by the producer can be modified. Using a simple scenario composed by one producer/publisher, one consumer/subscriber and one message broker, we observe the effect of network changes on the messages’ jitter, and A study about MQTT, a “light weight” publish-subscribe we detect the saturation boundary values with the specific based messaging protocol, is presented in [7]. The correlation hardware used in the tests. between the end-to-end latency and loss of system messages is studied. Three different QoS levels with different sizes of The rest of the article is organized as follows: Section II payload (from 1 to 16 Kbytes) are tested on a real world presents a literature review related to the topic. Section III scenario with both wired and wireless clients using 3G. They provides a description of the methodology used in this work, prove that there is a strong correlation between these two showing how measurements have been done in order to be variables. reproducible. Section IV presents the results and, finally, Section V concludes the paper, highlighting the next steps to However, few studies have focused on the effectiveness of follow in this research line. both protocols over unstable networks.

III.METHODOLOGYOFTHEEXPERIMENTS II.RELATED WORK The publish/subscribe topology allows us to use a sim- The AMQP and MQTT protocols have proved to be useful ple scenario with decoupled components in order to inter- in production scenarios, and have been used in challenging operate among them. In our experiments, we use a message applications, including Autonomous Computing [4], Cloud producer/publisher client, which at a given frequency, sends computing [14] or in aspects related to the Internet of Things AMQP/MQTT messages of a prefixed size to a message (IoT) [15]. broker. There are several works in which the AMQP and MQTT In the case of AMQP, the message-broker accepts incoming protocols are evaluated separately. In [10] the performance messages from a producer in an exchange (an exchange is of AMQP is assessed using Infiniband and Gigabit Ethernet essentially a router [12]) and, based on a set of criterions networks with Qpid as AMQP middleware. Five simple syn- routes the messages to a specific queue. In the case of MQTT, thetic benchmarks modeled after the OSU Micro-benchmarks the message-broker forward the incoming messages from pu- for MPI were used. They exercise the number of Publishers, blishers directly to the subscribers. A subscription is initially the number of Consumers, and the Exchange type. Each bench- created by a client application with a simple subscription name mark measures performance for data capacity (the amount of or a predefined topic. raw data in MegaBytes per second), message rate (the number of discrete messages transmitted), and speed (average time one In our scenario, the message consumer/subscriber is co- message takes to travel from the publisher to the consumer). nnected to the message-broker, and it is always ready to get or consume messages. The message-broker and the con- In [1] authors present a way to evaluate the performance sumer/subscriber client are executed on the same computer. To of AMQP by using an adapted version of the well-known reach the message-broker, the producer/publisher is connected SPECjms2007 and jms2009-PS benchmarks. This would allow to a WiFi access point within the same network. A picture to compare AMQP with other messaging systems such as JMS of the scenario configuration used in our testbed is shown in (Java Message Service) in terms of performance, stability and Figure 1. scalability. The consumer/subscriber records in a log file the sending In [5] a performance comparison between AMQP and (timestamped by the producer in each message) and reception RESTful web services is presented. Three different tests are times, along with the sequence number. There is not strict performed, which consist of several client applications sending synchronization between the producer and consumer clocks. messages during 30 minutes to the broker or the web server, When there are changes in the producer/publisher link, the respectively; once the messages arrive to the server they are regularity of message reception is affected. Since the inter- stored in a database. Then, the average number of messages message times are modified, message bursts can be delivered per second that have been sent is compared to the total number to the consumer, and even the sending order can be changed. of messages stored in the database. They conclude that, when the AMQP protocol is used to exchange messages, a larger Current implementations of AMQP and MQTT use TCP/IP number of messages per second is supported. connections for communication in order to enhance reliability. Fig. 2: Simulating the node mobility in an indoor scenario.

If the producer/publisher’s connection suffers an interruption, the client (producer’s AMQP or publisher’s MQTT) Fig. 3: Times involved in the experiments. accumulates the messages in its buffer and keeps it for a limited time, waiting till the connection with the message broker is re-established. The problem appears when, in the access point migration for the producer. The obtained values producer/publisher part the storage buffer capacity is depleted, are detailed in the following section. thereby causing message losses. The message broker was created on a server with an AMD A testing application, which we call amqperf, has been 8-core processor and 16GBytes of RAM memory. The client developed to generate a workload for the message queuing sys- had an Atom N450 processor and 1GByte of RAM memory. tem on the producer. Amqperf uses the RabbitMQ library [16], Both of them were running Ubuntu 12.04 GNU/Linux distri- which is an AMQP implementation, and the Paho library, bution. For the wireless network we have used the OpenWRT which is an open source MQTT implementation. Amqperf also GNU/Linux distribution with Attitude Adjustment version on works as a consumer/subscriber. an Alix PC-Enginees (alix2d2) and a Tplink (TL-WDR3600) In our experiments, the duration of the tests was about 20 routers. The tests were run on a dedicated LAN without seconds, which is sufficient to check the access point migration external traffic. of the message producer. During the tests we checked whether there were messages losses or if messages arrived out of order. IV. RESULTS In order to simulate node mobility in an indoor scenario, In this section we present our experimental results. The we used a set of scripts that shut down or activates the routers’ scenario is based on a wireless producer which migrates from radios and disassociate/associate all client devices. A schema one access point to another maintaining the same IP address; of this approach is shown in Figure 2. the TCP connection between the producer/publisher and the Due to the asynchrony among the internal clocks of the di- message broker is not affected. fferent entities of a distributed system, we cannot determine an For the experiments, we have used a completely dedicated exact latency value. Instead, we have calculated the variation network without external traffic. The tests were repeated 100 in the delay of the received messages, i.e., the jitter. times for each combination of inter-message period and me- The nth message inter-arrival time jitter is computed ssage size. We analyse the behaviour for each scenario, and through the following equation: we generalize it through a cumulative distribution function. 0 0 Jn = tn − tn−1 − T A. Behaviour during access point transition 0 AMQP and MQTT protocols ensure that, when a client where tn is the arrival time of message n to the consumer, and T is the (fixed) inter-message production period. T is one reconnects, it does not repeat messages and resumes the of the variables fixed for each experiment. Note that, with previous session with the message broker. this formula, we are not concerned by a possible asynchrony In order to understand the event, when a producer is between the producer and the consumer, which are executed migrating from one access point to another, we provide Fi- in different computers. An example of the timing involved in gure 4. These figures show the typical jitter behaviour for each the experiments can be seen in Figure 3. message received by the consumer/subscriber. The reference for the values used in the test is about 5 Mbps, which is the bandwidth needed, for example, to support high definition video streaming. This value can be reached, for instance, by transmitting messages of 12500B (12.5KBytes) In Figures 4a and 4b, we can see a vertical line close to the every 0.02 seconds. We made some tests to detect the point at message number 500th that reaches 3.3 seconds. These figures which messages start being lost in both cases: with and without were obtained producing and sending, during 20 seconds (a) period = 10ms (b) period = 10ms

Fig. 4: Jitter’s behaviour on the producer migration between access points, while it is sending messages of 512 B using: AMQP (left) and MQTT (right).

messages of 512 Bytes with an inter-message period 10ms. Due is the consequence of access point migration given that the to the high amount of messages, it is difficult to distinguish network had no external traffic, and that the workloads used the transition event. So, in Figures 4c and 4d, we show the in these tests do not saturate the system. profile of the curve with an inter-message period of 500ms in which we can see a positive peak corresponding to the hand-off Using the Cumulative Distribution Function on the set of time, and also jitter with negative values due to the reception experimental data, we have analysed the behaviour of the of a burst of messages which the producer retained during message’s jitter that arrives to the consumer focusing on the the communication’s interruption. As expected, the number of instant when the producer makes an access point migration. messages with negative jitter can be approximated by the peak Figure 5 shows the distribution function of the jitter using positive jitter divided by the message producing period. For a period of 10ms for message production and message sizes instance, in Figure 4d, it is 3000/500 ≈ 6 messages. of 512 Bytes and 6 KBytes for each protocol tested. When Also, Figure 4a shows a small oscillation after the hand- message size is 512 Bytes (Figures 5a and 5b), we observe off peak when using the AMQP protocol. We have found that, that the jitter value is concentrated around 3.3 seconds, with during the message burst, the delivery follows a LIFO (last- sporadic cases of jitters of 7 seconds, without significant input first-output) order, which results in messages consumed differences between these protocols. When the message size is in inverted order. This does not occur with MQTT protocol, bigger (Figures 5c and 5d) about half of the cases present jitter where the delivery is in order. values close to 3.3 seconds while the other half of the cases double this value. We consider that this behavior for bigger B. Jitter analysis messages is due to the fragmentation of their payload by both protocols. We repeated the experiments over the same scenario, with the same combination of message production rates and me- To study the jitter evolution, we have used the statistical ssage sizes. We used sizes between 0.5 KBytes and 6 KBytes, analysis for rounding mode values (rounded to the nearest and periods between message production of 10, 50, 100, 500 hundred) to fit the most representative value instead of using and 1000 ms. We know that the maximum jitter in our tests the value that appears most often in the data sets. We have (a) message size = 512 Bytes (b) message size = 512 Bytes

Fig. 7: Threshold limit of messages losses for different inter- message period and message size.

has a ﬁxed size header of 8 Bytes while MQTT has only a 2 Byte header.

V. CONCLUSIONSANDFUTUREWORK (c) message size = 6 KBytes (d) message size = 6 KBytes In this paper we presented an experimental analysis of the behavior of two application protocols based on Message Fig. 5: Cumulative Distribution Function of the maximum jitter Oriented Middleware standards, namely AMQP and MQTT. with a inter-message production period of 10 ms, using: AMQP We focussed on the information exchange over unstable and (left) and MQTT (right). mobile networks, like in vehicular networks, to provide a characterization of the publish-and-subscribe models in these scenarios. represented the jitter’s mode in two ways: as a function of the We have evaluated scenarios where the producer/publisher message size (Figure 6a) or as function of the inter-message suffers a handover process to measure effects such as the period (Figure 6b). variability of the jitter and the information loss, with a simple workload model of one producer/publisher and one C. Workload boundary consumer/subscriber, in an extended wireless network (i.e. several access points conforming a same Service Set). We have In order to know the capacity of the messaging system to observed that the mean jitter values during transitions tend to handle heavy workloads, we executed the experiments without oscillate between 3 and 6 seconds, although 7 seconds peaks access point migration. There is, therefore, no interruption have also been detected for high transmission rates (e.g., 100 in the wireless link between the producer/publisher and me- messages per second). ssaging broker. Note that these saturation boundary values can be dependent on the platform used, and even on their We have found that, during message bursts, the delivery configuration. follows a LIFO (last-input first-output) order, which results in messages consumed in inverted order. But this is not done A typical user application sends a few messages per second, MQTT protocol, where the delivery is always in order. with average load below 5 Mbps, which is well managed We have demonstrated that there is no information loss both by message protocols and the network. Performing this during the hand-off; then, we can say that the messaging exhaustive delimitation of the workloads, in Figure 7 we show system of these protocols is robust, and that it guarantees an approximation of the capacity of the system in terms of message delivery without losses. Messages losses are present message size and number of messages produced per second. only when, in the producer side the load is higher than its For loads above the lines in each case, the system is system buffer capacity. saturated, causing that a certain proportion of all produced In order to select the right protocol to build systems messages do not arrive to consumers. The limits are close to and applications with mobile communications over unstable 20 Mbps, which is near to the bandwidth that we have obtained network environments, both of these protocols can be used. with the iperf tool for the TCP test. The application/system architect’s decision to choose one of We note that the payload limit of a message in the MQTT them, will be determined according to different criteria, such protocol is greater than for AMQP. We consider that is mainly as security and energy efficiency. AMQP offers more aspects caused by the difference between the frame header: AMQP related to security [17], and MQTT is more energy efficient [7]. (a) message size (b) inter-message period

Fig. 6: Evolution of maximum jitter as a function of (a) and (b)

We recommend the use of AMQP protocol to build reliable, [8] Luzuriaga, J. and Perez, M. and Boronat, P. and Cano, J. C. and Calafate, scalable, and advanced clustering messaging infrastructures C. and Manzoni, P. Testing AMQP Protocol on Unstable and Mobile over an ideal WLAN, and the use of MQTT protocol to support Networks, Internet and Distributed Computing Systems, 7th International Conference, IDCS 2014, Italy, Proceedings Lecture Notes in Computer connections with edge nodes (simple sensors/actuators) under Science, pp. 250-260. constrained environments (low-speed wireless access). [9] O’Hara, John, Toward a Commodity Enterprise Middleware, ACM, Communications Magazine, 2007 As future work, we plan to evaluate these protocols on [10] Subramoni, Hari and Marsh, Gregory and Narravula, Sundeep and Lai, more complex scenarios in which a roaming producer changes Ping and Panda, Dhabaleswar K. Design and evaluation of benchmarks between IP networks and, consequently, the active TCP co- for financial applications using advanced message queuing protocol nnection has to be reset. (AMQP) over infiniband, Workshop on High Performance Computational Finance, WHPCF 2008. [11] Vermesan, O. and Harrison, V. M. and Kalaboukas, H.K. and Tomasella, ACKNOWLEDGMENTS M. and (Eds.) The Internet of Things - Strategic Research Roadmap, Cluster of European Research Projects on the IoT, CERP-IoT, 2009. This work was partially supported by the Ministerio de [12] Vinoski, S, Advanced Message Queuing Protocol, IEEE Internet Com- Ciencia e Innovacion´ , Spain, under Grant TIN2011-27543- puting, vol.10, 2006. C03-01. [13] Cohn, Raphael StormMQ: A Comparison of AMQP and MQTT, Avail- able: www.stormmq.com, 2011. [14] OpenStack, Foundation AMQP and Nova, Available: REFERENCES http://docs.openstack.org/developer/nova/devref/rpc.html, 2014. [1] Appel, Stefan and Sachs, Kai and Buchmann, Alejandro, Towards bench- [15] IMatix, Corporation Security and Robustness, Available: marking of AMQP, ACM, Proceedings of the Fourth ACM International http://zeromq.org/docs:welcome-from-amqp, 2014. Conference on Distributed Event-Based Systems, 99–100, 2010. [16] Pivotal Software, Inc., Messaging that just works, Available: [2] Chen, Whei-Jen and Gupta, Rahul and Lampkin, Valerie and Robertson, https://www.rabbitmq.com/reliability.html, 2014. Dale M. and Subrahmanyam, Nagesh, Responsive Mobile User Experi- [17] OASIS, Standard OASIS Advanced Message Queuing Protocol AMQP ence Using MQTT and IBM MessageSight IBM Corp., 2014. Version 1.0, 2014. [3] Gluhak, A. and Krco, S. and Nati, M. and Pfisterer, D. and Mitton, N. and Razafindralambo, T., A survey on facilities for experimental internet of things research, IEEE, Communications Magazine, pp. 58-67, 2009. [4] Gusmeroli, S. and Piccione, S. and Rotondi, D., IoT@Work automation middleware system design and architecture, IEEE International Confer- ence on Emerging Technologies and Factory Automation ETFA, 2012. [5] Fernandes, J.L. and Lopes, I.C. and Rodrigues, J.J.P.C. and Ullah, S., Performance evaluation of RESTful web services and AMQP protocol, IEEE ICUFN, pp. 810–815, 2013. [6] Kramer, Joshua Advanced Message Queuing Protocol (AMQP), Linux Journal 187, Houston, TX, 2009. [7] Lee, Shinho and Kim, Hyeonwoo and Hong, Dong Kweon and Ju, Hongtaek Correlation analysis of MQTT loss and delay according to QoS level, International Conference on Information Networking, pp. 714-717, 2013.