Private Public Partnership Project (PPP) Large-scale Integrated Project (IP) D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) Project acronym: FI-Core Project full title: Future Internet - Core Contract No.: 632893 Strategic Objective: FI.ICT-2011.1.7 Technology foundation: Future Internet Core Platform Project Document Number: ICT-2013-FI-632893-WP11-D.11.8.1 Project Document Date: 2016-05-11 Deliverable Type and Security: PU Author: Riccardo Zanetti Contributors: Engineering Ingegneria Informatica S.p.a Future Internet Core Platform 1. Introduction ............................................................................................................................................... 4 1.1. Attributes of the GEri to be tested: ............................................................................................... 4 1.2. Attributes of the GEri to be integrated in the test: ....................................................................... 4 1.3. Attributes of the testing tools: ...................................................................................................... 4 1.4. Developed testing tools ................................................................................................................. 5 1.5. Non-functional metrics .................................................................................................................. 7 2. Testing Summary ....................................................................................................................................... 8 2.1. GE overview ....................................................................................................................................... 8 2.2. Tested Scenarios ................................................................................................................................ 8 2.3. Results overview ................................................................................................................................ 9 3. Test case 1: Update Context .................................................................................................................... 10 3.1. Test case description ....................................................................................................................... 10 3.2. Test results ...................................................................................................................................... 11 3.2.1. Throughput: ............................................................................................................................. 11 3.2.2. HTTP Responses: ...................................................................................................................... 12 3.2.3. Response Times: ...................................................................................................................... 12 3.2.4. Responses /second: ................................................................................................................. 13 3.2.5. Requests Summary: ................................................................................................................. 14 3.2.6. Monitoring: .............................................................................................................................. 14 4. Test case 2: Query Context ...................................................................................................................... 16 4.1. Test case description ....................................................................................................................... 16 4.2. Test results ...................................................................................................................................... 18 4.2.1. Throughput: ............................................................................................................................. 19 4.2.2. HTTP Responses: ...................................................................................................................... 20 4.2.3. Response Times: ...................................................................................................................... 20 4.2.4. Responses /second: ................................................................................................................. 21 4.2.5. Requests Summary: ................................................................................................................. 22 4.2.6. Monitoring: .............................................................................................................................. 23 5. Test case 3: Subscribe Context ................................................................................................................ 25 5.1. Test case description ....................................................................................................................... 25 5.2. Test results ...................................................................................................................................... 26 5.2.1. Throughput: ............................................................................................................................. 26 5.2.2. HTTP Responses: ...................................................................................................................... 27 5.2.3. Response Times: ...................................................................................................................... 27 D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 2 Future Internet Core Platform 5.2.4. Responses/second: .................................................................................................................. 28 5.2.5. Request Summary:................................................................................................................... 29 5.2.6. Monitoring: .............................................................................................................................. 29 6. Conclusions .............................................................................................................................................. 31 7. Annex ....................................................................................................................................................... 32 7.1. Annex 1: Test environment ......................................................................................................... 32 7.1.1. Attributes of the hosting machine ........................................................................................... 32 7.1.2. Attributes of the hosting machine 2 (Data Consumer + Data Provider servers) ..................... 32 7.1.3. Attributes of the testing (client) machine ............................................................................... 33 7.2. Annex 2: JMeter Results .............................................................................................................. 33 D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 3 Future Internet Core Platform 1. Introduction The purpose of this document is to present the results of performance tests carried out on the Generic Enabler IoT Broker, more specifically on its reference implementation named Aeron. Aeron is a middleware used for setting up and maintaining the data flows in IoT deployments. It is designed to interact with large numbers of IoT data providers and data consumers. On behalf of the consumers, the IoT Broker retrieves, assembles, and processes information from the providers, offering the consumers a simple interface and masking the complexity and heterogeneity of the Internet of Things. For this reason, the IoT Broker GE interacts potentially with a large number of gateways, other backend instances, devices, as well as data consumers that had to be setup for executing tests. The first and essential interaction is with the IoT Discovery GE which is needed to the IoT Broker GE for knowing whether and where contexts are available. In order to reduce complexity of the environment setup, the NEC ConfMan implementation has been adopted (instead of more mature and complete implementation of this GE) and deployed in the same server along with Aeron GEri. Other interactions, such those with data consumers and data providers (devices) have been made feasible thanks to the development of dedicated mock up to simulate these actors’ behaviour. 1.1. Attributes of the GEri to be tested: Attribute Value Generic Enabler IoT Broker Chapter Internet of Things Services Enablement GEri name (implementation) tested Aeron GEri version tested 5.2.3 GEri owner NEC Organisation performing the test Engineering Ingegneria Informatica S.p.a Docker file link N/A 1.2. Attributes of the GEri to be integrated in the test: Attribute Value Generic Enabler IoT Discovery Chapter Internet of Things Services Enablement GEri name (implementation) tested NEC Configuration Management (NEC ConfMan) GEri version tested 5.2.3 GEri owner NEC Organisation performing the test Engineering Ingegneria Informatica S.p.a Docker file link N/A 1.3. Attributes of the testing tools: Attribute Value Load Test application Apache JMeter version v 2.13 System monitoring tool NMON (http://sourceforge.net/projects/nmon/) System Data analyser NMON Visualizer (Version 2015-10-21) Java (JRE used for JMeter and NMON OpenJDK Runtime Environment D.11.8.1: Report on QA testing (AERON IoT Broker stress testing)