Journal of Information Processing Vol.25 199–208 (Feb. 2017) [DOI: 10.2197/ipsjjip.25.199] Regular Paper High Throughput Dequeuing Technique in Distributed Message Queues for IoT Masafumi Kinoshita1,2,a) Hiroaki Konoura1 Takafumi Koike1 Kenji Leibnitz2,3 Masayuki Murata2 Received: May 23, 2016, Accepted: November 1, 2016 Abstract: With the dramatic increase in Internet of Things (IoT) related messaging volume, message queue systems are highly required for both interoperability among devices, as well as for control message traffic between devices and heterogeneous back-end systems (BES). When connected BES issue several dequeue requests to the message queue and no message is available, the frequency of missed-dequeues increases, which causes a degradation of the maximum throughput. Therefore, we propose the retry dequeue-request scheduling (RDS) method that decreases the number of dequeue requests from the BES by delaying the replies to the BES when missed-dequeues occur. Simulation and ex- perimental evaluations show that the throughput of the RDS method achieves 180% of that of the conventional dequeue method. Keywords: Internet of Things (IoT), distributed messaging, message queue, key-value store (KVS) cess messages at different rates to achieve maximum throughput 1. Introduction for the individual objectives of the IoT services, such as analysis, In the Internet of Things (IoT) era, the amount of all digital data management of devices, or data visualization. Therefore, to com- in the world created by various devices and sensors is exponen- pensate for the heterogeneity in message traffic between devices tially increasing, and it is predicted to reach 40 ZB by 2020 [1]. and BES, message queue systems use buffering to handle mes- IoT service systems utilizing data from devices typically consist sage traffic from devices. This compensation is achieved through of 3 groups: field devices which send and receive data, back-end distributed message queue systems, which enables the distri- application systems (BES) in a data center/cloud, and the message bution and load-balancing of message processing on multiple queue (or message hub/message bus) systems located between the servers [9]. In the past, the specifications of field devices and BES devices and back-end systems. were defined in advance. However, today’s IoT service systems Message queue systems are widely used for interoperability as well as development and operations (DevOps) trends require and control of the huge message traffic between devices and rapid implementation and continuous modification, additionally BES [2], [3]. Especially, the control of message traffic has be- to the BES also becoming adaptable [10], [11], [12], [13]. Fur- come an important requirement as the volume of IoT messages thermore, progress of distribution platforms such as Spark [14] or has increased dramatically over the past years. There are sev- Storm [15], have dramatically improved the performance of BES. eral solutions such as Kafka [4], Amazon Kinesis [5], Azure IoT In this background, updating the processes or parameter settings Hub [6], etc., following different approaches depending on their of BES can impact the system’s performance. respective objectives. In addition, to satisfy these requirements Actually, when the number of BES connecting to the message as well as obtaining a high availability, such as a short failover queue increases, we can observe that this situation impacts the time of within one second for social infrastructure systems, we performance of our proposed message queue and degrades the proposed a high-throughput and reliable message queue sys- throughput for retrieving messages from the message queue (de- tem based on a distributed in-memory key-value store (KVS) in queue). By analyzing the factor of throughput degradation, we [7], [8], [9]. recognize a large number of missed-dequeues, which means that Here, we will address another issue of traffic control between the lack of messages in the selected queue wastes computational devices and BES for IoT services. Devices transmit messages pe- resources. riodically at their own intervals, such as the period of log collec- Therefore, in this paper, we focus on the dequeue process of tion for their service requirements. On the other hand, BES pro- distributed message queue systems and we propose a method called Retry Dequeue-request Scheduling (RDS) to solve the 1 Hitachi Ltd., Chiyoda, Tokyo 100–8280, Japan throughput degradation problem. We evaluate the RDS method 2 Osaka University, Suita, Osaka 565–0871, Japan by simulation and also prove its advantage in experimental real 3 National Institute of Information and Communications Technology, Suita, Osaka 565–0871, Japan servers. a) [email protected] c 2017 Information Processing Society of Japan 199 Journal of Information Processing Vol.25 199–208 (Feb. 2017) The rest of this paper is organized as follows. The background lect much data from devices and learn through trial and error of and issues of message queues are introduced in Section 2. Sec- data analysis. This approach requires a large data volume for var- tion 3 presents our proposed method and its design. Section 4 ious analyses. Therefore, traffic volume from devices generating shows its performance evaluation by simulation, followed by Sec- periodical message data has become enormous in IoT service sys- tion 5 where we present the experimental evaluation. Section 6 tems. The transmitted data sizes of sensors highly depend on their describes related work and Section 7 concludes this paper. service requirements and protocols, such as MQTT, REST, and Transport Layer Security (TLS) [24], etc. From our past experi- 2. Background ences with specific use cases, such as monitoring or optimization 2.1 Outline of IoT Service System services, we assume in this paper that the data size is 1 KB, which 2.1.1 Message Queue in IoT Service System can be widely applied to IoT services. Figure 1 outlines an example of the system structure in IoT On the other hand, BES collect data for various IoT objectives, services. Message queues are widely used for a large variety such as monitoring and optimization and retrieve messages from of services, e.g., monitoring/optimization of services in indus- the queue at their own timing, which is non-periodic and process- try, smart meter services in electricity companies, connected ve- dependent. These processing times differ by context of message, hicle services, or services of a telecom company collecting data message size, and other related data. To achieve higher through- from smart devices. Message queues are required for the inter- put by fully utilizing computational resources, the BES retrieves operability and abstraction (absorption)ofmessagetrafficofde- messages from the queue with a pull-based method [4]. We de- vices. By supporting IoT protocols, e.g., MQ Telemetry Trans- scribe further details in Section 6. In addition, progress in dis- port (MQTT) [16], Representational State Transfer (REST) [17], tribution platforms, such as Spark, leads to a dramatic change in or Constrained Application Protocol (CoAP) [18], and by making processing time of the BES. devices and BES become loosely coupled (independent), mes- Here it can be seen that while devices send massive amounts of sage queues enable the developer to interoperate between them periodical messages, BES process messages at their own timing rapidly. Message queues buffer messages into a queue on a per- in IoT. Therefore, the control function of the massive and hetero- sistent storage (enqueue) and enable the BES to retrieve the mes- geneous message traffic in the message queue becomes a crucial sages from the queue at their own timing. Under the condi- issue in IoT. In this paper, we are targeting these heterogeneous tion that the message queue has both, sufficient performance to environments in the IoT service system. process messaging traffic from devices and scalability in perfor- mance and storage, the message queue enables the developers of 2.2 Conventional Approach using Distributed Message the BES to design their system without considering the entire vol- Queues ume of the messaging traffic. 2.2.1 Architecture for High Scalability and Availability 2.1.2 Heterogeneity in IoT Message Traffic In our previous work, we proposed a high-throughput and re- It is generally agreed that IoT services require information liable message queue system based on a distributed in-memory from historical or real-time data for their own objectives, such key-value store (KVS) for social infrastructure systems [7], [8], as monitoring and optimization [19], [20], [21], [22], [23]. In [9] (Fig. 2). The proposed messaging system adopts a fabric ar- Refs. [19], [20], [22], IoT service systems are required to man- chitecture with connected full-meshed servers for high scalabil- age the massive volume of data generated by sensors in various ity and availability. The proposed messaging system consists of fields, such as smart grids, connected vehicles, and heavy equip- 3parts:theenqueue controller (E-Ctrl) for receiving and stor- ment, etc. In Ref. [21], optimization at the enterprise level in ing messages in a queue, the distributed queue to the KVS server smart manufacturing results in requiring only periodically col- as persistent storage, and the dequeue controller (D-Ctrl) for re- lected data. In Ref. [23], general smart sensors are organized in ceiving dequeue requests from BES and retrieving the messages simple packages, i.e., they may consist of single chips and gener- from queues. This structure enables eliminating a single point of ate simple periodical data. failure and enhances the horizontal scalability of each part. The general approach in IoT to find values in data is to col- Fig. 2 Overview of distributed message queue system. E-Ctrl and D-Ctrl Fig. 1 Structure of IoT service system. denote enqueue controller and dequeue controller, respectively. c 2017 Information Processing Society of Japan 200 Journal of Information Processing Vol.25 199–208 (Feb. 2017) 2.2.2 Transparency in Distributed Message Queues the expected message traffic volume.