Faculty of Computing and Mathematical Sciences Department of Computer Science Message Queuing for Network Monitoring Anthony James Coddington Report of an Investigation 2012 The University of Waikato, Hamilton, New Zealand Message Queuing for Network Monitoring Anthony James Coddington This report is submitted in partial fulfillment of the requirements for the degree of Bachelor of Computing and Mathematical Sciences with Honours (BCMS(Hons)) at The University of Waikato. COMP520-12C (HAM) © 2012 Anthony James Coddington Abstract Network monitoring systems, such as the AMP project, are more useful when measurements are taken from a diverse range of locations in order to provide information from different parts of networks. Modern messaging protocols, such as those based on AMQP (Advanced Message Queuing Protocol), allow for more flexible, generic and potentially interoperable message routing and more sophisticated test co-ordination than those currently used by AMP; but many are not designed with the requirements of network monitoring in mind. This project was to investigate messaging protocols and their reliability and scalability in wide area networks, particularly with regards to coping when networks may not be behaving correctly (where network monitoring is most useful); and recommend a solution for network monitoring systems. No com- plete existing system that met all of the requirements of AMP was found, so a new client based on the STOMP protocol was implemented, which is compat- ible with a variety of messaging system implementations. This client includes a number of the features required for robust network monitoring, including a client-side persistence system and automatic re-connection to improve robust- ness to network failure. To further improve flexibility and resilience to failure, a system for extending STOMP to support direct peer-to-peer communication between measurement nodes is also proposed. Acknowledgements I would like to thank my supervisor, Richard Nelson for his excellent guidance throughout the project. I would also like to thank Brendon Jones for his helpful explanations of AMP, and in editing this report. Finally, I would like to thank the entire WAND group for their tireless help with my C coding questions throughout the project, I have learned an enormous amount this year. Contents Abstract ii Acknowledgements iii 1 Introduction1 2 Background3 2.1 AMP Overview.........................3 2.2 Requirements..........................5 2.3 Messaging-Oriented Middleware................5 2.4 Types of Message System....................6 2.4.1 Messaging Queuing Systems.................6 2.4.2 Publish-Subscribe Systems..................6 2.5 Message Brokers.........................6 3 Existing Systems8 3.1 Advanced Message Queuing Protocol.............8 3.2 ZeroMQ.............................9 3.3 Log Collection Systems..................... 10 3.3.1 Facebook Scribe....................... 10 3.4 MQ Telemetry Transport.................... 11 3.5 ActiveMQ............................ 12 3.6 Simple (or Streaming) Text Orientated Messaging Protocol (STOMP)............................ 12 3.7 Conclusion............................ 13 4 Architecture 15 4.1 STOMP............................. 15 4.1.1 STOMP Frame Types.................... 16 4.1.2 Extensibility......................... 17 Contents v 4.1.3 Limitations.......................... 17 4.1.4 Existing C/C++ Clients................... 18 4.1.5 STOMP 1.1.......................... 18 4.2 Design.............................. 19 4.3 Security............................. 19 4.4 Persistence............................ 19 4.5 End-to-end Acknowledgements................. 20 4.6 Non-blocking Connections................... 20 5 Implementation 22 5.1 Initial Implementation..................... 22 5.2 Frame Abstraction....................... 22 5.3 Persistence............................ 23 5.4 Non-blocking Connection.................... 23 5.5 Acknowledgements and Re-connection............. 24 6 Evaluation 25 6.1 Testing.............................. 25 6.2 Challenges............................ 25 6.3 Evaluation against Requirements............... 26 6.4 Future Work........................... 27 6.4.1 Completion of Persistence Integration and Reliability... 27 6.4.2 Peer-to-peer.......................... 27 6.4.3 Additional STOMP Features................. 30 6.4.4 Further Investigation of Broker-side Features........ 30 6.4.5 AMP Integration....................... 30 7 Conclusion 32 References 33 List of Figures 2.1 AMP Architecture...........................4 2.2 Broker-centric message-oriented middleware.............7 4.1 A STOMP SEND frame sent by publishers to the broker, and its corresponding MESSAGE frame, sent by the broker to subscribers.. 16 6.1 Peer distribution STOMP with re-sender............... 29 List of Tables 3.1 Surveyed Messaging Systems..................... 14 List of Acronyms AMQP Advanced Message Queuing Protocol MQTT MQ Telemetry Transport STOMP Simple (or Streaming) Text Orientated Messaging Protocol SSL Secure Socket Layer RFC Request for Comment SASL Simple Authentication and Security Layer RPC Remote Procedure Call API Application Programming Interface JMS Java Message Service WAN Wide Area Network CMS C++ Messaging Service APR Apache Portable Runtime HTTP Hypertext Transfer Protocol HTTPS HTTP Secure TCP Transmission Control Protocol WAL Write Ahead Logging I/O Input/Output LDAP Lightweight Directory Access Protocol Chapter 1 Introduction Generic, flexible messaging middleware is popular, with a wide variety of imple- mentations; but many systems have not been designed with the requirements of network monitoring in mind, the most important of which is robustness to network failure. This project was to select and implement a more flexible and generic messaging system suitable for use in the AMP active network monitor- ing system that could be used to allow more sophisticated test co-ordination, and better support for multiple collection servers; with increased reliability. It was considered desirable, but not essential that the system could be used for other messaging in addition to network monitoring applications. The original aim of the project was to investigate and select an existing mes- saging system for network monitoring, and integrate the system into AMP. As the investigation continued, it became clear that there were not any existing messaging systems that met the requirements for AMP, which is written in C and designed to be lightweight with minimal dependencies (WAND Net- work Research Group, n.d.b). Systems were found to be one or more of: too large to run on lightweight nodes (making use of Java on the client-side for example); lacking support for secure authentication of message origin (which is required for deployment in Wide Area Networks (WANs)); or lacking support for client-side persistence. As it became clear that extensive client work would be required, extension of the existing messaging system was considered. It was decided that the use of the simple STOMP protocol described in STOMP (2011), which is supported by many different messaging systems, would bring benefits of interoperability with other systems through a standardised format, while still remaining simple and being more flexible than the current system. Chapter 1 Introduction 2 After determining that existing STOMP C libraries were unsuitable the deci- sion was made to implement a new STOMP 1.1 library, of which none existed for the C language. To achieve this, a basic low-level frame handling layer was written, similar to one of the existing libraries, but with the addition of support for security. Following this, a significantly higher level Application Programming Interface (API) was added, as well as a high-level client-side persistence system that used a mature backing store to facilitate robustness. Finally, an architecture was designed extending STOMP to support a peer-to- peer communication directly between nodes. Chapter2 of this report gives an overview of AMP and messaging-oriented middleware, and details requirements identified for the system. Chapter3 focuses on existing messaging systems surveyed in a major initial investigation at the beginning of the project. Chapter4 discusses the STOMP protocol in more detail, and the design of the system implemented. Chapter5 describes the implementation completed as part of the project. Chapter6 describes the testing undertaken, and an evaluation of the system as currently implemented against the requirements identified in Section 2.2; followed by a discussion of future work that was not included in the implementation, including discussion of approaches designed to extend STOMP to work in a decentralised manner. Finally, Chapter7 is a conclusion summarising the outcomes of the project. Chapter 2 Background This chapter includes an overview of AMP and its messaging system, and details the requirements of any replacement system. An overview of the main types of messaging middleware is then presented. 2.1 AMP Overview The AMP project, developed by the National Laboratory for Applied Network Research (NLANR), now maintained by The University of Waikato WAND research group, is a distributed active probe network monitoring system. At its peak there were at least 130 nodes in operation, mostly located in the United States of America (McGregor, 2002). AMP in New Zealand is now primarily used to monitor Internet service provider and university networks around the country, as well as the REANNZ research network; with approximately