Faculty of Computing and Mathematical Sciences Department of Computer Science

Message Queuing for Network Monitoring

Anthony James Coddington

Report of an Investigation

2012

The University of Waikato, Hamilton, New Zealand

Message Queuing for Network Monitoring

Anthony James Coddington

This report is submitted in partial fulfillment of the requirements for the degree of Bachelor of Computing and Mathematical Sciences with Honours (BCMS(Hons)) at The University of Waikato.

COMP520-12C (HAM)

© 2012 Anthony James Coddington

Abstract

Network monitoring systems, such as the AMP project, are more useful when measurements are taken from a diverse range of locations in order to provide information from different parts of networks. Modern messaging protocols, such as those based on AMQP (Advanced Message Queuing Protocol), allow for more flexible, generic and potentially interoperable message routing and more sophisticated test co-ordination than those currently used by AMP; but many are not designed with the requirements of network monitoring in mind. This project was to investigate messaging protocols and their reliability and scalability in wide area networks, particularly with regards to coping when networks may not be behaving correctly (where network monitoring is most useful); and recommend a solution for network monitoring systems. No com- plete existing system that met all of the requirements of AMP was found, so a new client based on the STOMP protocol was implemented, which is compat- ible with a variety of messaging system implementations. This client includes a number of the features required for robust network monitoring, including a client-side persistence system and automatic re-connection to improve robust- ness to network failure. To further improve flexibility and resilience to failure, a system for extending STOMP to support direct peer-to-peer communication between measurement nodes is also proposed. Acknowledgements

I would like to thank my supervisor, Richard Nelson for his excellent guidance throughout the project. I would also like to thank Brendon Jones for his helpful explanations of AMP, and in editing this report. Finally, I would like to thank the entire WAND group for their tireless help with my C coding questions throughout the project, I have learned an enormous amount this year. Contents

Abstract ii

Acknowledgements iii

1 Introduction1

2 Background3 2.1 AMP Overview...... 3 2.2 Requirements...... 5 2.3 Messaging-Oriented Middleware...... 5 2.4 Types of Message System...... 6 2.4.1 Messaging Queuing Systems...... 6 2.4.2 Publish-Subscribe Systems...... 6 2.5 Message Brokers...... 6

3 Existing Systems8 3.1 Advanced Message Queuing Protocol...... 8 3.2 ZeroMQ...... 9 3.3 Log Collection Systems...... 10 3.3.1 Facebook Scribe...... 10 3.4 MQ Telemetry Transport...... 11 3.5 ActiveMQ...... 12 3.6 Simple (or Streaming) Text Orientated Messaging Protocol (STOMP)...... 12 3.7 Conclusion...... 13

4 Architecture 15 4.1 STOMP...... 15 4.1.1 STOMP Frame Types...... 16 4.1.2 Extensibility...... 17 Contents v

4.1.3 Limitations...... 17 4.1.4 Existing C/C++ Clients...... 18 4.1.5 STOMP 1.1...... 18 4.2 Design...... 19 4.3 Security...... 19 4.4 Persistence...... 19 4.5 End-to-end Acknowledgements...... 20 4.6 Non-blocking Connections...... 20

5 Implementation 22 5.1 Initial Implementation...... 22 5.2 Frame Abstraction...... 22 5.3 Persistence...... 23 5.4 Non-blocking Connection...... 23 5.5 Acknowledgements and Re-connection...... 24

6 Evaluation 25 6.1 Testing...... 25 6.2 Challenges...... 25 6.3 Evaluation against Requirements...... 26 6.4 Future Work...... 27 6.4.1 Completion of Persistence Integration and Reliability... 27 6.4.2 Peer-to-peer...... 27 6.4.3 Additional STOMP Features...... 30 6.4.4 Further Investigation of Broker-side Features...... 30 6.4.5 AMP Integration...... 30

7 Conclusion 32

References 33 List of Figures

2.1 AMP Architecture...... 4 2.2 Broker-centric message-oriented middleware...... 7

4.1 A STOMP SEND frame sent by publishers to the broker, and its corresponding MESSAGE frame, sent by the broker to subscribers.. 16

6.1 Peer distribution STOMP with re-sender...... 29 List of Tables

3.1 Surveyed Messaging Systems...... 14 List of Acronyms

AMQP Advanced Message Queuing Protocol

MQTT MQ Telemetry Transport

STOMP Simple (or Streaming) Text Orientated Messaging Protocol

SSL Secure Socket Layer

RFC Request for Comment

SASL Simple Authentication and Security Layer

RPC Remote Procedure Call

API Application Programming Interface

JMS Java Message Service

WAN Wide Area Network

CMS C++ Messaging Service

APR Apache Portable Runtime

HTTP Hypertext Transfer Protocol

HTTPS HTTP Secure

TCP Transmission Control Protocol

WAL Write Ahead Logging

I/O Input/Output

LDAP Lightweight Directory Access Protocol Chapter 1

Introduction

Generic, flexible messaging middleware is popular, with a wide variety of imple- mentations; but many systems have not been designed with the requirements of network monitoring in mind, the most important of which is robustness to network failure. This project was to select and implement a more flexible and generic messaging system suitable for use in the AMP active network monitor- ing system that could be used to allow more sophisticated test co-ordination, and better support for multiple collection servers; with increased reliability. It was considered desirable, but not essential that the system could be used for other messaging in addition to network monitoring applications.

The original aim of the project was to investigate and select an existing mes- saging system for network monitoring, and integrate the system into AMP. As the investigation continued, it became clear that there were not any existing messaging systems that met the requirements for AMP, which is written in C and designed to be lightweight with minimal dependencies (WAND Net- work Research Group, n.d.b). Systems were found to be one or more of: too large to run on lightweight nodes (making use of Java on the client-side for example); lacking support for secure authentication of message origin (which is required for deployment in Wide Area Networks (WANs)); or lacking support for client-side persistence. As it became clear that extensive client work would be required, extension of the existing messaging system was considered. It was decided that the use of the simple STOMP protocol described in STOMP (2011), which is supported by many different messaging systems, would bring benefits of interoperability with other systems through a standardised format, while still remaining simple and being more flexible than the current system. Chapter 1 Introduction 2

After determining that existing STOMP C libraries were unsuitable the deci- sion was made to implement a new STOMP 1.1 library, of which none existed for the C language. To achieve this, a basic low-level frame handling layer was written, similar to one of the existing libraries, but with the addition of support for security. Following this, a significantly higher level Application Programming Interface (API) was added, as well as a high-level client-side persistence system that used a mature backing store to facilitate robustness. Finally, an architecture was designed extending STOMP to support a peer-to- peer communication directly between nodes.

Chapter2 of this report gives an overview of AMP and messaging-oriented middleware, and details requirements identified for the system. Chapter3 focuses on existing messaging systems surveyed in a major initial investigation at the beginning of the project. Chapter4 discusses the STOMP protocol in more detail, and the design of the system implemented. Chapter5 describes the implementation completed as part of the project. Chapter6 describes the testing undertaken, and an evaluation of the system as currently implemented against the requirements identified in Section 2.2; followed by a discussion of future work that was not included in the implementation, including discussion of approaches designed to extend STOMP to work in a decentralised manner. Finally, Chapter7 is a conclusion summarising the outcomes of the project. Chapter 2

Background

This chapter includes an overview of AMP and its messaging system, and details the requirements of any replacement system. An overview of the main types of messaging middleware is then presented.

2.1 AMP Overview

The AMP project, developed by the National Laboratory for Applied Network Research (NLANR), now maintained by The University of Waikato WAND research group, is a distributed active probe network monitoring system. At its peak there were at least 130 nodes in operation, mostly located in the United States of America (McGregor, 2002). AMP in New Zealand is now primarily used to monitor Internet service provider and university networks around the country, as well as the REANNZ research network; with approximately 25 nodes currently active (WAND Network Research Group, n.d.a). Recently the system has also been installed on more than 130 nodes in a European deployment (Snijders, 2011). AMP nodes run a variety of network tests at scheduled times to monitor network connectivity and health by measuring characteristics such as reachability, delay and network path length. Results are sent to one or more central servers, and can be viewed with a server-side web interface (WAND Network Research Group, n.d.c).

The current AMP messaging system for results is relatively simple. Nodes initiate an Secure Socket Layer (SSL) secured Transmission Control Protocol (TCP) connection to every server they are configured to report to (WAND Network Research Group, n.d.c). Results are sent with a simple system of sequence numbers and sliding window acknowledgements. Result messages Chapter 2 Background 4 consist of a fixed length binary header, and a body consisting of a union of result structures (WAND Network Research Group, n.d.b). In adverse network conditions, where the client cannot connect to the server, the node buffers messages to disk and retries at a later time. Test co-ordination is done by individual tests, which can request a command to be run on a remote node. Test scheduling is handled separately using configuration files. AMP is written in C, and was designed with low computing power monitoring nodes in mind. Resource constraints of modern hardware are significantly less of an issue than at the time AMP was first written, but the system should remain lightweight to allow for the potential to run on low-power low-cost systems such as the Raspberry Pi.

The AMP architecture, as shown in figure 2.1, consists of a client process, measured, that runs tests, plus an additional module, xfer that communicates with an equivalent process on the server in order to transfer results. xfer puts messages in a ring buffer and also persists messages to disk, in case of network or client failure; to ensure results eventually arrive at the collection server. The server-side xferd process saves messages to disk, from which the web interface reads to present results. The current messaging architecture is multi-threaded with a thread per server to be connected to (WAND Network Research Group, n.d.c). The aim of this project was to select and implement a messaging system suitable to replace the xfer and xferd parts of AMP, with a view to replacing the test co-ordination system, and potentially eventually the test scheduling system.

2.2 Requirements

A number of requirements were identified for a messaging system to be suit- able for network monitoring, and AMP in particular. The key requirement is that the system be robust in unreliable network conditions, such as when the network link between a node and the collection server becomes unavailable or unstable; as results from this period can be key in troubleshooting networking issues. This means that some form of client-side persistence (also known as durability) is required to maintain messages if the network, client or server fail. It was also identified that the system should be scalable in WANs, in the sense of not relying on features generally not available in a WAN, such as support for broadcast packets, or assuming a secure network environment. In addition, Chapter 2 Background 5

Figure 2.1: AMP Architecture support for multiple possibly redundant servers is also desirable.

More specifically to the AMP project, it was desirable for the system be lightweight (avoiding large footprint runtimes such as Java), and be capable of running on a variety of architectures. The measurement nodes may be run on low power, low cost devices such as the Raspberry Pi or routers running the OpenWRT custom firmware with few available resources. It was not, however, a requirement to run on extremely constrained devices such as those used for sensor networks. AMP is written in C, and was designed with a minimum of dependencies. It was considered desirable to introduce few additional de- pendencies, and use the C language for any integration work with AMP. It was also desirable to make use of a pre-existing well tested persistence sys- tem, to reduce system complexity and increase robustness. Finally, overall, the system should be more flexible than the current system used by AMP, with the possibility of adding additional features such as more sophisticated test coordination between measurement nodes.

2.3 Messaging-Oriented Middleware

Messaging middleware are generic messaging systems designed for connecting systems within an organisation over a common messaging infrastructure with Chapter 2 Background 6 an application-agnostic protocol. This allows for a common monitoring system, and abstracts the process of message passing from the application developer (Curry, 2005).

2.4 Types of Message System

There are two main types of messaging systems: Message Queuing systems and publish-subscribe systems. Many existing messaging middleware systems support both, but some support only one or the other; and the goals of each type of system are quite different.

2.4.1 Messaging Queuing Systems

According to Curry(2005), message queuing systems involve queuing messages to be sent to exactly one of a set of consumers. Typically producers send a single copy of a message to centralised broker, and the broker puts the mes- sage in a queue. Consumers subscribe to the queue, and the broker distributes the message to one consumer (often reliably exactly one), and the consumer is generally required to acknowledge successful (or unsuccessful) processing of the message. If one consumer fails to process the message, it will usually be delivered to another consumer. Message queuing systems are useful for job processing of various forms, such as used in distributed web backend process- ing. They are not, however, well suited to monitoring systems; as they limit distribution to one of several collection servers as opposed to being delivered to more than one server.

2.4.2 Publish-Subscribe Systems

Publish-subscribe systems, on the other hand, involve a more general concept of topics. Publishers publish messages on particular topics, and subscribers subscribe to topics they are interested in. All subscribers to a given topic receive a copy of the message. This allows for more flexible and generic mes- saging, beyond that of simple queuing. Many systems also support a hierarchy of topics, that allow for subscription to a subset of messages on a more general topic. Chapter 2 Background 7

2.5 Message Brokers

The majority of messaging systems involve the concept of a broker that serves as a centralised point to handle routing messages, as shown in figure 2.2. This means clients can be simpler and do not need to know which consumers or subscribers are available at any given time. Clients register queues and topics with the broker; and send messages to it to be distributed. Once the client has sent messages to the broker, it does not need to worry about ensuring the messages are actually delivered. This has the advantage of simplifying client implementation (often leading to a proliferation of clients for many languages), but introduces a single point of failure. In addition, many systems do not support discover of which topics other clients are subscribed to. To reduce the possibility of failure, many systems support clustering (multiple brokers that act as a single broker) or federation of brokers (a network of brokers that exchange information) (SpringSource, n.d.b).

Figure 2.2: Broker-centric message-oriented middleware

Publisher A publishes to subscribers A and B (publish-subscribe). Producer B sends messages to a queue consumed by C (message queuing). Chapter 3

Existing Systems

A major part of the project was to investigate existing generic messaging sys- tems as the original goal was to select and implement an existing system into AMP. The following is a discussion of the systems that appeared most promis- ing, followed by a brief survey of other systems encountered during the inves- tigation. Finally, an overview of the features and limitations of the systems surveyed is presented.

3.1 Advanced Message Queuing Protocol

The Advanced Message Queuing Protocol (AMQP) was developed “to become the standard for interoperability between all messaging middleware,” with a standardised wire protocol (OASIS, n.d.). The protocol is designed to meet the needs of a variety of applications. AMQP has recently undergone a large shift with protocol version 1.0. According to SpringSource(n.d.c), version 0.9.1 and earlier used a broker-based system of queues, topics, and exchanges; sup- porting both message queuing and publish-subscribe. Exchanges are entities within the broker to which queues are bound, and define the type of messag- ing to employ. Examples of exchange types include Direct (simple queue-based routing), Fanout (distribute messages to all bound queues), Topic and Head- ers (routing based on particular AMQP headers) (SpringSource, n.d.c). Like many other messaging systems, AMQP has the limitation that queues exist on the broker-side; with the additional limitation that subscribers must know the name of the queues to which they wish to subscribe. AMQP supports security through Simple Authentication and Security Layer (SASL) and SSL (SpringSource, n.d.d). Chapter 3 Existing Systems 9

AMQP 1.0 completely changes this architecture however, and operates at a lower level of abstraction involving links between nodes (OASIS, 2011b, pp. 23-24). Despite a more flexible architecture, AMQP 1.0 still does not define a standardised method for inter-broker federation (support for this is planned for AMQP 1.1 (OASIS, 2011a)). AMQP version 1.0 and later are intended to be forwards compatible, however it provides no true backwards compatibility with versions prior to 1.0. Earlier version of AMQP also had issues with backwards compatibility and differing feature sets.

There are currently three main open source implementations of AMQP: Rab- bitMQ, Apache Qpid, and OpenAMQ. RabbitMQ is a very popular imple- mentation, and is written in Erlang, currently supporting AMQP 0.9.1 with some extensions (SpringSource, n.d.a). Apache Qpid was originally developed by Red Hat and currently implements AMQP 0.10 (which is more recent than 0.9.1), and has C++ and Java variants (Apache Software Foundation, n.d.b). These protocol versions are not compatible, and the C++ broker is not in- teroperable with RabbitMQ as it does not support any earlier versions of the protocol (Apache Software Foundation, n.d.c). OpenAMQ was developed by IMatix Corporation, and according to Kramer(2009), was the original im- plementation of AMQP. It was discontinued in 2011 in favour of focusing development on ZeroMQ (Hintjens, 2010).

At the time the decision was made on which system to use, there were no completed open-source implementations of AMQP 1.0. More recently, a new lightweight flexible library that includes a C version and peer to peer support, Apache Qpid Proton, has neared completion and may bear further investiga- tion (Apache Software Foundation, n.d.f). Reasons for rejecting AMQP in- cluded fragmentation, a complex protocol (synchronous control, asynchronous messaging), server-side queues (and lack of persistence in existing client li- braries) and a lack of standardised broker federation. Apache Qpid was con- sidered, with a broker on each node (as it is relatively lightweight, and has a version written in C++), but it was ultimately rejected primarily due to complexity and interoperability issues.

3.2 ZeroMQ

ZeroMQ was developed by iMatix Corporation to resolve many of the perceived issues with AMQP(Hintjens, 2010). ZeroMQ is designed to be similar to the Chapter 3 Existing Systems 10

Berkeley sockets interface, with sockets for a variety of messaging patterns such as publish-subscribe and request-response. This can be used to build a messaging system with or without a broker, and is available for a wide variety of languages. Similar to the Berkeley sockets API, message queues are opaque to the application, with little configurability beyond setting a maximum queue size. This allows for much flexibility in defining messaging patterns, but unfor- tunately complicates implementation of persistence. ZeroMQ is designed to be extremely fast and suitable for use in financial applications, where efficiency is paramount (iMatix Corporation, n.d.b). ZeroMQ includes an extremely de- tailed manual that describes more advanced messaging patterns, as well as a number of ‘Request for Comments (RFCs)’ to define these patterns. Unfor- tunately, while example code is provided for each pattern in the manual, it is generally not production ready; and the RFCs often introduce unnecessary constraints on the messaging architecture. The main issue, however, is that ZeroMQ does not include support for any form of security, and the opaque- ness of messaging sockets would make it difficult to support a protocol such as SSL. One RFC discusses an implementation of SASL authentication, but also introduces a broker-centric architecture (Hintjens, 2011). Previous ver- sions of ZeroMQ supported a mechanism for storing large queues on disk, but this did not survive application restarts and was removed to avoid confusion. It was decided that, for the purposes of network monitoring, there was little advantage to using ZeroMQ over simply designing an entirely new protocol.

3.3 Log Collection Systems

A number of systems primarily intended for log collection were also investigated because they naturally include the concept of persisting to disk locally while upstream servers are not available. These system include Facebook Scribe, Apache Flume and Apache Kafka. The latter two systems were not thoroughly investigated as the systems heavily rely on Java on the client side which did not meet the requirements of suitability for use on lightweight nodes.

3.3.1 Facebook Scribe

Facebook Scribe was developed by Facebook originally for use internally to reliably collect logs from many servers (Johnson, 2008). The system has an in- teresting architecture, which later inspired the proposed peer-to-peer extension Chapter 3 Existing Systems 11 of the final system. Each node connects to a local re-sender, which handles reliably transferring logs to an upstream server, of which there may be a hi- erarchy. Scribe is an extremely simple system, utilising Remote Procedure Calls (RPCs) with the Apache Thrift cross platform service framework, with a simple response from the server that it either received the message or that the call should be retried later. Log messages consist of a category and a message, and log messages can be stored in various ways or forwarded by servers, de- pending on their category, through a server configuration file (Facebook, inc., 2010). Scribe was designed for internal use within data centres, so does not support secure authentication, although Qualtrics Labs Inc.(2011) has imple- mented this separately using SSL. The primary limitation of Scribe is that it is architected for one-way collection of logs; significant modification would be required to make use of it as a general purpose messaging system.

3.4 MQ Telemetry Transport

MQ Telemetry Transport (MQTT) was developed by IBM and Eurotech be- ginning in 1999. It was originally designed for telemetry applications for low power, high latency networks and as such has clients for a variety of embedded platforms (IBM, 2010). Unlike many other protocols, there are some existing MQTT clients that support client-side persistence for reliability. Being de- signed for low power applications, MQTT has relatively low overhead and a simple protocol to fit within very small memory constraints. The protocol itself only supports simple plain-text authentication, but SSL is also supported by some implementations. MQTT is designed as a lightweight publish-subscribe system, with a micro-broker. Messages can have three levels of quality of service with each level offering different delivery guarantees and involving dif- ferent amounts of handshaking traffic (IBM, 2010). Clients can also specify a will message to be delivered when they become unreachable to notify other nodes of the change in availability.

Unfortunately, the open source implementations of MQTT are relatively im- mature as MQTT was only made royalty free in 2010, having long been part of IBM Websphere MQ (IBM, n.d.). An initiative to formally standardise the MQTT protocol began in 2011 (IBM, 2011). According to Light(2011), the main open source micro-broker Mosquito does not currently support SSL. The C version of the Eclipse Paho project (an open source release of the existing Chapter 3 Existing Systems 12

Websphere MQ clients (IBM, 2012)) also does not support SSL, nor does it support client-side persistence (Eclipse Foundation, n.d.). Apache ActiveMQ 5.6.0, as well as the latest version of Apache Apollo (intended to become the eventual successor to ActivMQ) have added support for MQTT since this project began, including support for SSL, but there is still a lack of suitable client libraries.

3.5 ActiveMQ

Apache ActiveMQ is a popular open-source messaging system, with a commer- cially supported version distributed by Red Hat known as the Fuse Message Broker(Red Hat, n.d.). It is primarily designed to be used with OpenWire; an ActiveMQ-specific binary wire format. ActiveMQ is written in Java and supports Java Message Service (JMS), a Java messaging interface that is in- dependent of wire protocol(Curry, 2005). ActiveMQ also supports a variety of other protocols, including STOMP, MQTT (in the most recent version), a RESTful web API and WebSockets (Apache Software Foundation, n.d.c). It supports a number of features including federation, (broker-side) persistence and SSL; as well as a variety of backend authentication mechanisms includ- ing Lightweight Directory Access Protocol (LDAP). The OpenWire protocol is primarily supported for use with Java JMS clients, as well as .NET and a large C++ API, ActiveMQ-CPP, that essentially replicates many of the features of JMS in C++. For other languages, the use of the simple text-based STOMP is recommended (Apache Software Foundation, 2012b). It was decided that the libraries for OpenWire use were too large, complex and dependency in- troducing for suitability for the project; however STOMP was an appealingly simple and flexible protocol; and is supported by a large number of messaging systems in addition to ActiveMQ.

3.6 STOMP

Simple (or Streaming) Text Orientated Messaging Protocol (STOMP) is a simple text-oriented protocol for messaging. It is designed to be simple and easy to implement, and is supported by a variety of messaging systems, often as a secondary protocol (STOMP, n.d.). The architecture of the system is broker centric, and is compatible with the models of AMQP and ActiveMQ. Messages Chapter 3 Existing Systems 13 are sent to a destination, which is implementation dependant and with most implementations allows for queues and topics. The main appeal of STOMP was its simplicity, while being easily extensible through support for arbitrary custom headers. In 2011, version 1.1 of the protocol was released that better defines the frame format as well as defining a few extra features including heart beat messages and negative acknowledgements (STOMP, 2011). STOMP has a number of limitations due to being broker-centric, and lacks specification in some areas such as error codes; however the extensibility of the protocol through custom headers allows working around many of the issues, with the benefit of maintaining some measure of interoperability with many different messaging systems.

STOMP is supported as a secondary protocol by a large number of messaging systems, including ActiveMQ, RabbitMQ, and any JMS provider through the StompConnector JMS bridge; as well as several STOMP-only brokers written in Java, Python and Ruby (STOMP, n.d.). ActiveMQ and its successor Apollo were the only brokers that supported STOMP 1.1 so were the focus of the investigation.

In addition to it’s own OpenWire format, ActiveMQ also supports STOMP. Version 1.1 support was added in version 5.6.0, released in May 2012 (Apache Software Foundation, 2012a). It includes support for SSL, as well as a num- ber of authentication backends for plain-text authentication. It also supports a number of custom headers for mapping to JMS concepts, such as for cor- relating related messages (Apache Software Foundation, n.d.d). ActiveMQ is implemented in Java. Apache Apollo, implemented in Scala, is intended as the eventual successor to ActiveMQ, and is designed to be significantly more efficient (Apache Software Foundation, 2012c). It initially only supported STOMP 1.1, but now also includes support for MQTT.

3.7 Conclusion

No existing system could be found that met the requirements of network mon- itoring, as outlined in Section 2.2. This was extremely surprising as it would at first glance appear to be a common requirement for non-continuously con- nected nodes to reliably transmit results, as with sensor networks or mobile devices for example. The closest system in this respect was MQTT, but the architecture was deemed too restrictive; and the open source implementations Chapter 3 Existing Systems 14 too immature, particularly on the client side in C.

Table 3.1 presents a partial overview of the systems surveyed. The overview includes primary protocols supported (with support for STOMP indicated); whether the system supports multiple brokers (such as through federation), or is peer to peer, and whether the system or clients support persistence. Client authentication support and the language the broker is written in is also included. Many systems support a variety of client languages, support for the C language is indicated. Two systems not already discussed are included in the table. Redis is a distributed key-value store that includes support for non- reliable publish-subscribe (Redis, 2010). IceStorm had a parent architecture that was deemed to complex, ICE; however the ICE-E embedded variant may bear further investigation (ZeroC, n.d.). In addition to those listed, a large number of message queuing only systems were found and discarded. In total over 25 systems were investigated.

It also became clear that a number of vendors had several similar products in the messaging space: Red Hat originally developed Qpid (an AMQP implemen- tation), and also maintains a version of ActiveMQ, as well as the Java-based HornetQ messaging system(JBoss, n.d.). iMatix has developed a number of systems, including OpenAMQ, RestMS (iMatix Corporation, n.d.a) and Ze- roMQ. This suggests that messaging middleware is still an unstable field with a number of standardisation efforts that may or may not see widespread adop- tion.

At the conclusion of the selection process, it was decided to use the STOMP protocol, with some extensions. Despite having similar limitations to AMQP with regards to architecture, it is simple and easily extendible. The STOMP protocol is also supported by many other messaging systems. Using STOMP gives most of the benefits of a generic, interoperable protocol but with the flexibility and simplicity to implement a client with additional features such as client-side persistence and end-to-end acknowledgements. Chapter 3 Existing Systems 15

Table 3.1: Surveyed Messaging Systems Name Protocol Type Persistence Auth Language RabbitMQ AMQP† multi-broker broker secure Erlang* Qpid AMQP multi-broker broker secure C++/Java OpenAMQ AMQP multi-broker no secure C++ ZeroMQ custom flexible no no C Scribe Thrift multi-broker both plain‡ C++ Mosquito MQTT broker both plain‡ C++* ActiveMQ various† multi-broker broker secure Java* HornetQ JMS† multi-broker broker secure Java Redis custom broker broker plain C IceStorm custom RPC flexible both secure various* RestMS1 REST multi-broker? broker secure C

* C/C++ client available † STOMP support ‡ SSL support is being added 1 RestMS appears unmaintained Chapter 4

Architecture

This chapter introduces the chosen protocol, STOMP, in more detail including a brief survey of existing client libraries. Following this is an overview of the design and architecture of a new STOMP client implemented with the require- ments of network monitoring in mind, including secure authentication, end-to- end acknowledgements and client side persistence. Finally, the complications the use of SSL security adds to multiplexing of connections are discussed.

4.1 STOMP

Simple (or Streaming) Text Orientated Messaging Protocol (STOMP), as out- lined in Section 3.6, is a simple text-orientated protocol for messaging. STOMP is a broker-based protocol supported by many messaging systems as an addi- tional protocol. The protocol format is similar to that of Hypertext Transfer Protocol (HTTP), consisting of frames with a command, headers separated by new lines, a new line and then the frame body (which is terminated by a null byte). TCP is used for message transport, with individual frames distinguished by their terminating null. Message frames include a destination header, with the body of the message as the frame body. STOMP has a separate frame type for messages sent from the client to the broker and messages distributed by the broker to the client. Messages are sent using a SEND frame, which is then redistributed by the broker with addition of a unique message identifier as MESSAGE frames, as shown in Figure 4.1. Chapter 4 Architecture 17

SEND MESSAGE destination:/topic/foo message-id:123 content-type:text/plain destination:/topic/foo content-length:5 content-type:text/plain content-length:5 hello\0 hello\0 Figure 4.1: A STOMP SEND frame sent by publishers to the broker, and its correspond- ing MESSAGE frame, sent by the broker to subscribers

4.1.1 STOMP Frame Types

STOMP is a simple protocol and has a small set of frames. In addition to SEND and MESSAGE frames there are a number of control frames for connection, acknowledgement and destination subscription management. The following is a brief outline of frame types most relevant to the project. A full description of the 10 frame types in STOMP 1.1 is available in the protocol specification (STOMP, 2011).

CONNECT Sent by the client to initiate a connection. With STOMP 1.1 this includes a list of accepted versions and a host name (in case the server has multiple virtual hosts).

CONNECTED Returned by the broker to indicate successful connection. In- cludes version negotiated.

SEND A message sent to the broker to be delivered. Includes a destination. May include a receipt identifier requesting a receipt when the broker re- ceives the message. Binary messages (that may contain a null character) are supported by specifying a content-length.

RECEIPT Acknowledgement by the broker of receipt of the message by the broker, if a receipt identifier was specified in the SEND frame.

MESSAGE Message to the receiving client as sent by the broker, includes the headers and body of the original SEND frame, with additional headers including a unique message-id (that is not the same as the receipt iden- tifier).

ACK Acknowledges a message to the broker, using the message-id from the message. The publishing client is not notified of the acknowledgement.

SUBSCRIBE Subscribes to a given destination, includes an identifier for the Chapter 4 Architecture 18

subscription and an acknowledgement mode.

UNSUBSCRIBE Unsubscribes from a destination, using the identifier for the subscription.

DISCONNECT Sent by the client to gracefully disconnect from the broker. May request a receipt, but no further frames may be sent by the client.

ERROR Indicates an error in the previous frame. Includes a short error mes- sage and may include a more detailed error in the body. Includes the receipt-id from the frame that generated the error if applicable.

4.1.2 Extensibility

STOMP is easily extensible, due its support for arbitrary headers. Headers in STOMP are simple key/value strings separated by a delimiter that can encode information about a message, such as it’s destination or a unique identifier for the message. The STOMP specification requires that the broker forward any headers to clients, even those that it does not understand. This means that extensions to the protocol can be made without requiring support from the broker, and that if a client or broker does not support a particular feature the message itself should still be processed.

4.1.3 Limitations

STOMP has a number of limitations due primarily to its simple broker-centric but implementation agnostic model. Firstly, the formatting of the destination header is implementation dependant, but it is common to adopt the ActiveMQ format (which is similar to that of standard JMS)(SpringSource, n.d.e). Sim- ilarly the format of error messages is not defined (STOMP, 2011). The main limitation, however is a strong reliance on the broker for reliability. Message identifiers are assigned by the broker, but not provided to the message pro- ducer, and acknowledgements are sent to the broker rather than the producer. The concept behind this is that the broker handles reliability once the message is handed to it. This would make the protocol largely unsuitable if it were not for the ease of extensibility arbitrary headers provide. Chapter 4 Architecture 19

4.1.4 Existing C/C++ Clients

After choosing STOMP as the protoccol to use for the messaging system, existing STOMP libraries for the C and C++ langugages were explored to see if an existing library could be used (with the addition of client-side persistence). The simplicity of the STOMP protocol also meant that implementing a new client was also feasible, and this was the approach taken after determining that existing clients were unsuitable. libstomp

Libstomp is a simple low-level STOMP library written in C, it makes use of the Apache Portable Runtime (APR) library for platform-independent network sockets (LogicBase, 2005). The use of this library was considered but it does not support SSL, and neither does APR. Due to its extremely small code size and low-level API, it was decided it was easier to write a new library with support for SSL than modify libstomp. This also removes the external dependency on APR.

Apache CMS

Apache C++ Messaging Service (CMS) is the primary library for Apache Ac- tiveMQ for use with C++. It is described as a “JMS-like API for C++” (Apache Software Foundation, n.d.a). ActiveMQ-CPP, the specific implemen- tation of CMS for AcitveMQ, supports both OpenWire and the STOMP pro- tocol, in a generic manner. This means that there is much OpenWire-specific functionality in the library that is not applicable to STOMP, along with many other unnecessary features as JMS is a large system. The primary reason ActiveMQ-CPP was not chosen was its unnecessary complexity for use with the simple STOMP protocol, particularly for introducing additional protocol features.

4.1.5 STOMP 1.1

Version 1.1 of STOMP adds escaping of special characters in headers, negative acknowledgements and heartbeat messages; as well as formalising the specifi- cation. Very few STOMP implementations currently support STOMP 1.1. In order to more future-proof the system, it was decided to include support for at least the version negotiation and escaping in 1.1, so a version 1.1 compatible broker was required for testing; which at the time of writing was limited to Chapter 4 Architecture 20

Apache ActiveMQ and Apollo. ActiveMQ was chosen, as the default configu- ration of Apollo made interactive testing difficult, due to tight message timing constraints before automatic disconnection. Neither broker was suitable for use co-resident on the nodes, as both make use of the Java runtime library.

4.2 Design

Following a survey of existing messaging middleware with respect to their suit- ability for network monitoring, it was decided to implement a new STOMP client with support for security, client-side persistence and end-to-end acknowl- edgements. The library is intended to be lightweight, flexible and generic; and applicable as a general purpose high-level STOMP client implementation, with strong robustness to network failure. The system was designed with ease of use and testability in mind. It uses a layered architecture with separation be- tween persistence, low level frame handling, and a high level API for making connections and sending and receiving messages. An internal representation abstracted from the network protocol was used to represent frames, to keep version-specific functionality to a minimum. The system was designed to make use of non-blocking Input/Output (I/O) multiplexing, as opposed to multiple threads to increase reliability and reduce susceptibility to errors.

4.3 Security

STOMP authentication is plain-text only so OpenSSL was used for security, primarily due to its ubiquity and use already with the existing AMP system. Many STOMP implementations, including ActiveMQ, support SSL for secure messaging (Apache Software Foundation, n.d.d). While encryption of data transferred was not a requirement, SSL provides strong mechanisms for verify- ing the identify of clients and servers through public key authentication. It is also worth noting that SSL has non-encrypted variants, which can be used to authenticate if encryption is not required. OpenSSL also includes a high level socket API, BIO, that almost completely abstracts details of unencrypted and SSL connections to a common interface (OpenSSL, n.d.a). Chapter 4 Architecture 21

4.4 Persistence

Client-side persistence is required to ensure eventual delivery of results when the network link between the client (such as an AMP measurement node) and broker is unavailable. In order to increase reliability, the decision was made early to use an existing, mature, robust persistence system.

SQLite is a lightweight embedded SQL database, used extensively with the Android mobile operating system, among many other places (SQLite, n.d.b). It was chosen primarily due to its small footprint and ubiquity, with some consideration given to its cross-platform nature in case the developed STOMP library was ever used on the Windows platform.

4.5 End-to-end Acknowledgements

STOMP does not natively support end-to-end acknowledgements, yet this is a requirement for reliable test coordination, and is desirable for communi- cating results to a collection server to avoid complete reliance on broker-side persistence and re-delivery. End-to-end acknowledgements were implemented by making use of the correlation-id and reply-to headers optional in the STOMP specification, using the correlation-id as a sequence number. The receiving client can acknowledge a frame end-to-end by sending a message with the same correlation-id to the reply-to topic (which should be unique per client) with an empty body. A sequence number was also necessary to allow for the possibility of the receiving client to detect duplicate messages where the publishing client re-sends messages from it’s client-side persistence store after a network failure (which may occur if the receipt from the broker for the original message is not received by the publishing client). Despite the fact message receipts have an identifier, testing showed that ActiveMQ does not use it for message de-duplication.

4.6 Non-blocking Connections

It was decided to use connection multiplexing as opposed to multiple threads to provide support for multiple simultaneous STOMP connections. This re- quires the use of network sockets that do not block. The use of non-blocking connections with SSL is more complicated than with standard Berkeley sock- Chapter 4 Architecture 22 ets. This is due to the way an SSL connection potentially involves multiple reads or writes to the underlying socket when attempting to read from (or write to) the SSL socket. This means that it is sometimes necessary to wait for the underlying socket to be writeable before re-attempting a read or vice versa (Rescorla, 2002, section 6). The OpenSSL API provides mechanism for checking this after a failed read or write, but it complicates the use of methods normally used to wait on data availability. It also means that a non-blocking connection must be used as waiting on a socket to be readable and performing a blocking read may still block (OpenSSL, n.d.b). To reduce the complexity of dealing with this problem, a function that waited on the appropriate socket availability depending on the state of the connection was written. Chapter 5

Implementation

5.1 Initial Implementation

The first part of the implementation, after researching the STOMP specifica- tion and OpenSSL API, involved building a basic test application that made a simple BIO connection requesting a web page. Following this, the small test application was extended to establish an SSL connection using HTTP Secure (HTTPS). At that point, the test was modified to send a simple STOMP CONNECT frame over an unencrypted connection. Apache ActiveMQ was then set up with STOMP enabled, and an initial STOMP connection handshake achieved. ActiveMQ was then configured to accept anonymous SSL connections, and the test refactored to connect via STOMP over a secure connection. Next, an SSL connection utilising public key authentication was set up. This was found to be complicated, due to the differing key storage formats used by Java and OpenSSL; it was possible but involved using an in- termediate format as the Java keystore does not allow import of a private key (Seifarth, 2007). Once achieved, this process was then documented due to its complexity.

5.2 Frame Abstraction

After the proof of concept of interoperability, work on low-level STOMP frame handling began (independently from network connection handling). Support for both STOMP 1.1 and 1.0 formats was included from the beginning (the essential difference being the escaping of certain special characters in STOMP 1.1 headers). These functions abstracted encoding, decoding and managing Chapter 5 Implementation 24 frames and headers for use by the higher level API. Unit tests were written to test the features of the frame handling, and the test application was extended to make use of the functions for encoding the frame. This test code, loosely based on the sending process used by libstomp, became the basis for the frame sending and receiving part of the high level API.

A high level connection-oriented API was then implemented, utilising the low level frame handling to build common frames to allow for subscription and message sending and retrieval. Handshaking and version negotiation was im- plemented so that all subsequent frames would use the negotiated version.

5.3 Persistence

A client-side persistence system was implemented, using SQLite as a back- end; and exposing a high level API of frame storage and retrieval for a given connection. Internally the system used a single table uniquely identified by connection host name and a numeric message ID (intended to be the same as that used for receipt and end-to-end acknowledgement IDs). The system made use of stored procedures to increase speed, but still took a noticeable amount of time to persist messages to disk due to the atomic transactional nature of SQLite, which are limited by hard drive speed to in the order of 60 per sec- ond (SQLite, n.d.a, section 19). This could be improved by bundling multiple messages into transactions (at the cost of a slight decrease in robustness to client failure), or using Write Ahead Logging (WAL) mode (where changes are appended to a log rather than atomically written directly to the database) (SQLite, n.d.c). As the message frequency for AMP is relatively low this issue should not pose too much of a problem; and if it became a serious issue a drop-in replacement for the persistence system following the same API could be written.

5.4 Non-blocking Connection

Following basic implementation of the persistence system, work began on con- verting the existing frame send and receive functions to allow non-blocking operation. Initially this was written in a complex manner attempting to read and keep track of as much data as possible from the network socket at a time; including attempting to deal with edge cases such as a read spanning multiple Chapter 5 Implementation 25 messages. This was found to be error prone and difficult to write elegantly. It was later discovered that the line-wise buffering abstraction provided by the OpenSSL BIO library worked in a non-blocking manner when the underlying socket was non-blocking, albeit returning partial lines. This allowed for a sig- nificantly more elegant frame receiving architecture with a greatly simplified state machine that would not read beyond the end of a frame. Refactoring of the send frame function was more difficult, as it processed frames in a linear fashion using multiple socket sends. This was reduced to a state machine, with the majority of the function consisting of selecting the next data to send. Wrapper functions that ran these non-blocking functions in a busy wait loop were added to reduce initial code changes required.

5.5 Acknowledgements and Re-connection

Support for acknowledging messages from the broker was added, as well as some preliminary support for sending end-to-end acknowledgements using a custom header (but not receiving them). Following this, a basic function for processing control frames, such as receipts, and passing only MESSAGE frames to the caller was implemented, primarily as a proof of concept. A more flexible subscription management system, with a similar structure to frame headers, was added to allow for the possibility of re-subscribing after a connection failure.

Finally, automatic re-connection after a network failure was implemented. This first involved refactoring connection to be non-blocking (although the initial connection function is currently still blocking). A state machine was used to manage connecting, STOMP handshaking, and then re-subscribing to topics. As a final step, a helper function was written to abstract waiting for data using select() to work with abstract connections, and handle the different waiting requirements of SSL in order to remove busy waiting. Conversion of the remaining functions to be non-blocking was not completed, but this would simply involve tracking when such functions are called a second time and not duplicating frames, instead re-calling the existing non-blocking functions. Chapter 6

Evaluation

6.1 Testing

Unit testing of (non-network related) features of the implementation was in- cluded at every stage, with the unit tests extended to cover any issues found, leading to a reasonably stable core API. Connection-handling functions were tested manually, the complexity of which beyond the low-level frame API was kept to a minimum. Frequent use was made of the STOMP trace feature of ActiveMQ and telnet as a second STOMP client to test individual features. Additional testing of automatic re-connection is required to show its robust- ness. Extensive testing of the reliability and robustness (especially to network failure) of the system was planned; but unfortunately was not undertaken due to time-constraints and the fact the persistence system was not fully integrated with the frame sending and receiving systems within the time available.

6.2 Challenges

A number of challenges were encountered throughout the project. It was ex- tremely surprising that there were no suitable existing systems designed for non-continuously connected nodes. It was also found, with many systems, to be difficult to determine the exact set of features and limitations; through omis- sion, self-promotion and differing target markets. Existing formal research into the area, particularly with regards to commonly available systems was sparse; with some investigation for the project amounting to verifying the accuracy or inaccuracy of discussion describing the benefits of systems from sources with differing requirements. The sparse documentation of OpenSSL was also Chapter 6 Evaluation 27 found to be problematic, especially with regards to the exact behaviour of non-blocking I/O, such as when used with a buffered BIO socket.

Implementation in general was challenging, due to the author’s inexperience with the C programming language at the beginning of the project. Particular difficulty was encountered appropriately managing the size of buffers for arbi- trary length strings; and appropriately estimating the time required to imple- ment features given the additional complexity of the C language. Conversion of the entire system to be non-blocking was also found to be complex. The ini- tial implementation was designed with non-blocking requests in mind, however the need for all message exchanges to be non-blocking (including connection and disconnection) was initially overlooked. This required major refactoring of connection handshaking. Overall, implementation took a significantly longer period of time than anticipated. This meant that rather than implementing the STOMP client library and persistence relatively quickly, and then integrating the system into AMP; not all parts of the library were completed.

6.3 Evaluation against Requirements

The system as currently implemented, meets the requirements of being stan- dard based, supporting secure authentication (through SSL). A system for de-centralised operation was designed and is described in Section 6.4.2. The system is as lightweight and self-contained as possible over and above the de- pendencies of AMP. To this end the only external dependency in addition to OpenSSL (which was already used in the existing messaging system) is SQLite, and SQLite is very small and lightweight with no external dependencies and could freely be included statically if necessary. No architecture dependant features are included, no modification should be needed on any Unix system, and minimal modification would be required for the library to work on Win- dows. The system should also be suitable for wide area networks, as it does not make any assumptions about network topology and uses well standardised and tested SSL and TCP connections. TCP should also provide reasonable re- liability and integrity of messages, although additional mechanisms (a message checksum as an additional header for example) could be added for additional reliability. The system is more flexible and generic than the current system in AMP, through its use of the interoperable STOMP protocol, and support for communication between arbitrary nodes. Chapter 6 Evaluation 28

The system as currently implemented was designed with strong robustness in adverse networking conditions in mind, but does not yet meet this require- ment due to incomplete integration between the persistence system and frame processing. Complications with the use of non-blocking I/O for multiple con- nection support meant there was not enough time to complete both this feature and implement automatic re-connection on connection failure. The latter was (mostly) implemented, as it provided a useful way of testing non-blocking connection. In order to complete integration, support for queuing frames for output is required (which could make use of the existing persistence store and rely on in-memory caching by SQLite). This would also allow properly robust subscription management.

6.4 Future Work

A number of areas of future work, that were originally planned to be included in the project have been identified.

6.4.1 Completion of Persistence Integration and Reliability

While the bulk of the STOMP library was completed, including a high-level API abstraction above STOMP frames, as well as a persistence system using SQLite; the persistence system has not yet been properly integrated with with the STOMP system. It is possible to persist and retrieve frames with the persistence store agnostic API, but this is currently not done automatically. On a related issue, messages are currently not queued for sending (which could use the persistence store, relying on in-memory caching); which also means message receipts are currently not associated with certain frames, such as subscription requests. Automatic re-connection may also need to be triggered in circumstances other than when the underlying connection fails, as the time for this to occur can be very long (iMatix Corporation, n.d.b, Heartbeating); such as through the use of STOMP 1.1 heart beat messages. Finally, processing of end-to-end acknowledgements needs to be implemented, as currently they are only sent. Chapter 6 Evaluation 29

6.4.2 Peer-to-peer

To increase reliability, robustness and to reduce load on a single broker; it was considered desirable for the system to work in a distributed fashion. Almost all systems evaluated, with the notable exception of ZeroMQ, were designed with a broker-centric model (as this allows clients not to need to handle reli- able delivery, or know which other clients will receive the message). STOMP is also heavily broker-centric, however as the protocol is very simple, and changes only slightly on the broker side, along with its support for custom headers; it is possible to adapt the protocol to support peer-to-peer operation while main- taining compatibility with the existing broker-based system. Two proposals have been made to accomplish this.

Client library accepting connections

One possibility for a peer-to-peer extension would be for the client to simply accept connections (and CONNECT messages), and thus act as a very simple lightweight broker. The client could then treat SEND messages from a peer (destined to a topic it is subscribed to) as if they were MESSAGE frames, as the format is almost identical. Such a client could also accept and send ACK frames directly but this could cause compatibility issues.

Client-side Re-sender

An alternative, and likely preferable, option for peer-to-peer STOMP is to implement a very lightweight broker that is co-resident with the client, sim- ilar to the idea of the Scribe ‘re-sender.’ This re-sender could be configured (possibly with a special message from the client, such as a SUBSCRIBE that also specifies a central broker) with one or more centralised standard STOMP brokers to forward messages to. The re-sender could then relay any messages and subscriptions to the central broker(s), essentially acting as a client. This would have the advantage of allowing greater flexibility and modularity, and could be made to work with other existing STOMP client. This approach is very similar to that taken by Apache Qpid for broker federation (Apache Software Foundation, n.d.e). One issue with this approach, however, is that as the re-sender acts as a client it would not be possible to discover if any clients of the broker had subscribed to a particular topic. To resolve this it may be necessary to broadcast messages to all standard brokers. Given the sparseness of lightweight STOMP 1.1 brokers, however, it would be perfectly feasible to Chapter 6 Evaluation 30 extend the re-sender into a fully fledged broker and run it as the centralised broker as well, with only minor modifications. Alternatively, a topic could be set up to announce subscriptions, but this would not be as interoperable with existing STOMP clients.

Peer Distribution

Both proposals would require some mechanism for distributing connection in- formation to peers to enable peer-to-peer operation. This could easily be accomplished using the broker-centric network as an overlay for the first mes- sage, as shown in Figure 6.1, and including a custom header with connection details (such as a reply-address in addition to the semi-standard reply-to topic). This would mean the re-sender (or client, in the case of the first op- tion) could initiate a direct connection to the peer and use that from then on for direct messages. Careful thought would need to be given around whether messages should also be sent via the central broker (in case there was an- other client subscribed at the broker). Consideration would also need to be given to handling of duplicated messages that arrive by both paths, likely with a different message-id assigned by the broker so a mechanism such as the correlation-id header would need to be used to detect duplicate messages. An alternative approach to peer distribution would be to broadcast peer con-

Figure 6.1: Peer distribution STOMP with re-sender Chapter 6 Evaluation 31 tact information over a special topic that all clients subscribe to.

6.4.3 Additional STOMP Features

Many features of STOMP were included in the library, however there are a few features missing. STOMP transactions are not yet implemented, which allow for a number of messages to be distributed (or acknowledged) by the broker atomically. Implementing transactions would be relatively straightforward, as the two additional commands and header are simple. Similarly, the NACK (negative acknowledgement) frame is not implemented, as it was not deemed particularly useful as the behaviour is server dependant (and usually consists of putting messages in a dead letter queue or delivering to another client). Finally, support for the heart-beating introduced in STOMP 1.1 has not been included (STOMP, 2011).

6.4.4 Further Investigation of Broker-side Features

ActiveMQ supports a wide variety of features, some of which may be useful for network monitoring and bear further investigation. For example, in addition to basic broker-side persistence of messages, it has some support for durable subscribers and durable topics (Apache Software Foundation, n.d.g). This is where a client can request the broker to deliver messages missed while it was offline upon re-subscribing to a topic. In order to facilitate interoperability with this, ActiveMQ defines a number of additional STOMP headers (Apache Software Foundation, n.d.d). It would be useful for the STOMP library to support adding headers to subscription and connection frames to allow use of this feature, and this information stored with the connection for use on re- connection. Note that the durability feature does not, however, resolve the issues of persisting messages at the client side before they reach the broker. When implementing support for automatic failover between brokers, it would also be worth investigating interoperating with message delivery failover, also supported by ActiveMQ.

6.4.5 AMP Integration

Originally the intention of the project was to survey existing messaging systems and choose one to integrate into AMP. After it was found that no existing sys- tem was suitable, and the implementation of a new STOMP library began; it Chapter 6 Evaluation 32 quickly became clear that it would not be possible to complete integration into AMP within the time available. Integration could proceed through reimple- mentation of the existing xferd application, or integration directly into AMP. As the message body format of a STOMP frame is not defined, the existing result message format could be used; or a new results format defined using a standard format such as Google Protocol Buffers (Google, n.d.). STOMP includes a content-type header that could be used to differentiate the type if changed at a later time. Chapter 7

Conclusion

This project investigated the suitability of messaging middleware for network monitoring, specifically for the AMP active monitoring system. No existing system was found to be suitable so a solution was implemented involving writ- ing a new STOMP 1.1 client with extensions to support robust communication with client side persistence, end-to-end acknowledgement and support for se- cure authentication. This project also contributes a potential architecture for extending STOMP to support direct communication between nodes.

Implementation of a new STOMP library in C was found to be more difficult and time consuming than originally anticipated. It is possible implementation of a suitable new client for a binary protocol such as MQTT would have been of similar, or lesser complexity. Whether STOMP can be made into a suitable messaging protocol for network monitoring has not yet been proven in practice, as completion of the remaining parts of the system and formal testing for ro- bustness has not yet been undertaken. Further developments since the project began, in particular Apache Qpid Proton, as well as any developments in open source implementations of MQTT may be worth investigating before embark- ing on a similar project; as messaging-orientated middleware is a continuously developing field with many different approaches and implementations. References

Apache Software Foundation (2012a). ActiveMQ 5.6.0 Release. Retrieved from http://activemq.apache.org/activemq-560-release.html.

Apache Software Foundation (2012b). ActiveMQ Features Overview. Retrieved from http://activemq.apache.org/features-overview.html.

Apache Software Foundation (2012c). Apollo. Retrieved from http://activemq. apache.org/apollo/.

Apache Software Foundation (n.d.a). ActiveMQ-CPP. Retrieved from http: //activemq.apache.org/cms/.

Apache Software Foundation (n.d.b). AMQP Compatibility of Qpid releases. Retrieved from http://qpid.apache.org/compatibility.html.

Apache Software Foundation (n.d.c). Apache ActiveMQ - Cross Language Clients. Retrieved from http://activemq.apache.org/cross-language-clients. html.

Apache Software Foundation (n.d.d). Apache ActiveMQ - STOMP. Retrieved from http://activemq.apache.org/stomp.html.

Apache Software Foundation (n.d.e). Apache Qpid: Open Source AMQP Mes- saging - Using Broker Federation. Retrieved from https://cwiki.apache.org/ qpid/using-broker-federation.html.

Apache Software Foundation (n.d.f). Apache Qpid Proton. Retrieved from http://qpid.apache.org/proton/.

Apache Software Foundation (n.d.g). How do durable queues and topics work. Retrieved from http://activemq.apache.org/ how-do-durable-queues-and-topics-work.html. References 35

Curry, E. (2005). Message-Oriented Middleware, pp. 1–28. John Wiley & Sons, Ltd. doi:10.1002/0470862084.ch1.

Eclipse Foundation (n.d.). Paho Download. Retrieved from http://www.eclipse. org/paho/download.php.

Facebook, inc. (2010). Scribe. Retrieved from https://github.com/facebook/ scribe/wiki.

Google (n.d.). protobuf - Protocol Buffers - Google’s data interchange format. Retrieved from https://developers.google.com/protocol-buffers/.

Hintjens, P. (2010). iMatix will end OpenAMQ support by 2011. Re- trieved from http://lists.openamq.org/pipermail/openamq-dev/2010-March/ 001598.html.

Hintjens, P. (2011). 11/MTL - Message Transfer Layer - ZeroMQ Request for Comments. Retrieved from http://rfc.zeromq.org/spec:11.

IBM (2010). MQTT: Lightweight Messaging Transport Protocol. Re- trieved from http://public.dhe.ibm.com/software/dw/webservices/ws-mqtt/ ws-mqtt-pdf.pdf.

IBM (2011). Invitation to participate in MQTT Standardization. Retrieved from http://mqtt.org/new/wp-content/uploads/2011/10/ MQTTStandardizationInvitation.pdf.

IBM (2012). Initial Eclipse Paho contributions completed. Retrieved from http://mqtt.org/2012/03/initial-eclipse-paho-contributions-completed.

IBM (n.d.). MQTT FAQ. Retrieved from http://mqtt.org/faq. iMatix Corporation (n.d.a). RestMS. Retrieved from http://www.restms.org/. iMatix Corporation (n.d.b). ZeroMQ - The Guide. Retrieved from http:// zguide.zeromq.org.

JBoss (n.d.). HornetQ General FAQs. Retrieved from https://community.jboss. org/wiki/HornetQGeneralFAQs.

Johnson, R. (2008). Facebook’s Scribe technology now open source. Retrieved from http://www.facebook.com/note.php?note id=32008268919. References 36

Kramer, J. (2009). Advanced Message Queuing Protocol (AMQP). LinuxJour- nal, (187).

Light, R. (2011). Mosquitto should support encrypted (SSL/TLS) connections. Retrieved from https://bugs.launchpad.net/mosquitto/+bug/903114.

LogicBase (2005). libstomp stomp.c. Retrieved from https://github.com/ a3linux/libstomp/blob/master/src/stomp.c.

McGregor, T. (2002). Quality in measurement: beyond the deployment bar- rier. In Proceedings of the 2002 Symposium on Applications and the Internet (SAINT) Workshops, SAINT-W ’02, pp. 66–. Washington, DC, USA: IEEE Computer Society.

OASIS (2011a). AMQP Features. Retrieved from http://www.amqp.org/ product/features.

OASIS (2011b). AMQP v1.0 - Final. Retrieved from http://www.amqp.org/ sites/amqp.org/files/amqp.pdf.

OASIS (n.d.). AMQP. Retrieved from http://www.amqp.org/.

OpenSSL (n.d.a). BIO. Retrieved from http://www.openssl.org/docs/crypto/ bio.html.

OpenSSL (n.d.b). BIO read. Retrieved from http://www.openssl.org/docs/ crypto/BIO read.html.

Qualtrics Labs Inc. (2011). Pull Request #44: Add support for SSL and Thrift 0.7.0. Retrieved from https://github.com/facebook/scribe/pull/44.

Red Hat (n.d.). Fuse message broker - based on activemq. Retrieved from http://fusesource.com/products/enterprise-activemq/.

Redis (2010). Pub/sub - redis. Retrieved from http://redis.io/topics/pubsub.

Rescorla, E. (2002). An Introduction to OpenSSL Programming (Part II). Retrieved from http://www.rtfm.com/openssl-examples/part2.pdf.

Seifarth, J. (2007). Import private key and certificate into Java Key Store (JKS). Retrieved from http://www.agentbob.info/agentbob/79-AB.html. References 37

Snijders, J. (2011). NLNOG RING: Network debugging never was easier. Retrieved from https://ripe65.ripe.net/presentations/105-RIPE65 NLNOG RING Job Snijders.pdf.

SpringSource (n.d.a). Compatibility and conformance. Retrieved from http: //www.rabbitmq.com/specification.html.

SpringSource (n.d.b). Distributed RabbitMQ Brokers. Retrieved from http: //www.rabbitmq.com/distributed.html.

SpringSource (n.d.c). RabbitMQ - AMQP 0-9-1 Model Explained. Retrieved from http://www.rabbitmq.com/tutorials/amqp-concepts.html.

SpringSource (n.d.d). RabbitMQ - SASL Authentication. Retrieved from http://www.rabbitmq.com/authentication.html.

SpringSource (n.d.e). RabbitMQ Stomp Adapter. Retrieved from http://www. rabbitmq.com/stomp.html.

SQLite (n.d.a). Frequently asked questions. Retrieved from http://www.sqlite. org/faq.html.

SQLite (n.d.b). SQLite. Retrieved from http://www.sqlite.org/.

SQLite (n.d.c). Write-ahead logging. Retrieved from http://www.sqlite.org/ wal.html.

STOMP (2011). STOMP Protocol Specification, Version 1.1. Retrieved from http://stomp.github.com/stomp-specification-1.1.html.

STOMP (n.d.). STOMP - Implementations. Retrieved June 8, 2012 from http://stomp.github.com/implementations.html.

WAND Network Research Group (n.d.a). AMP Measurements. Retrieved from http://erg.wand.net.nz/amp/index.php.

WAND Network Research Group (n.d.b). WAND Network Research Group: AMP. [Software] Available at http://research.wand.net.nz/software/amp.php.

WAND Network Research Group (n.d.c). WAND Network Research Group: AMP. Retrieved from http://research.wand.net.nz/software/amp.php.

ZeroC (n.d.). IceStom. Retrieved from http://www.zeroc.com/icestorm/index. html.