Private Public Partnership Project (PPP)
Large-scale Integrated Project (IP)
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing)
Project acronym: FI-Core Project full title: Future Internet - Core Contract No.: 632893 Strategic Objective: FI.ICT-2011.1.7 Technology foundation: Future Internet Core Platform Project Document Number: ICT-2013-FI-632893-WP11-D.11.8.1 Project Document Date: 2016-05-11 Deliverable Type and Security: PU Author: Riccardo Zanetti Contributors: Engineering Ingegneria Informatica S.p.a
Future Internet Core Platform
1. Introduction ...... 4 1.1. Attributes of the GEri to be tested: ...... 4 1.2. Attributes of the GEri to be integrated in the test: ...... 4 1.3. Attributes of the testing tools: ...... 4 1.4. Developed testing tools ...... 5 1.5. Non-functional metrics ...... 7 2. Testing Summary ...... 8 2.1. GE overview ...... 8 2.2. Tested Scenarios ...... 8 2.3. Results overview ...... 9 3. Test case 1: Update Context ...... 10 3.1. Test case description ...... 10 3.2. Test results ...... 11 3.2.1. Throughput: ...... 11 3.2.2. HTTP Responses: ...... 12 3.2.3. Response Times: ...... 12 3.2.4. Responses /second: ...... 13 3.2.5. Requests Summary: ...... 14 3.2.6. Monitoring: ...... 14 4. Test case 2: Query Context ...... 16 4.1. Test case description ...... 16 4.2. Test results ...... 18 4.2.1. Throughput: ...... 19 4.2.2. HTTP Responses: ...... 20 4.2.3. Response Times: ...... 20 4.2.4. Responses /second: ...... 21 4.2.5. Requests Summary: ...... 22 4.2.6. Monitoring: ...... 23 5. Test case 3: Subscribe Context ...... 25 5.1. Test case description ...... 25 5.2. Test results ...... 26 5.2.1. Throughput: ...... 26 5.2.2. HTTP Responses: ...... 27 5.2.3. Response Times: ...... 27 D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 2
Future Internet Core Platform
5.2.4. Responses/second: ...... 28 5.2.5. Request Summary:...... 29 5.2.6. Monitoring: ...... 29 6. Conclusions ...... 31 7. Annex ...... 32 7.1. Annex 1: Test environment ...... 32 7.1.1. Attributes of the hosting machine ...... 32 7.1.2. Attributes of the hosting machine 2 (Data Consumer + Data Provider servers) ...... 32 7.1.3. Attributes of the testing (client) machine ...... 33 7.2. Annex 2: JMeter Results ...... 33
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 3
Future Internet Core Platform
1. Introduction The purpose of this document is to present the results of performance tests carried out on the Generic Enabler IoT Broker, more specifically on its reference implementation named Aeron. Aeron is a middleware used for setting up and maintaining the data flows in IoT deployments. It is designed to interact with large numbers of IoT data providers and data consumers. On behalf of the consumers, the IoT Broker retrieves, assembles, and processes information from the providers, offering the consumers a simple interface and masking the complexity and heterogeneity of the Internet of Things. For this reason, the IoT Broker GE interacts potentially with a large number of gateways, other backend instances, devices, as well as data consumers that had to be setup for executing tests. The first and essential interaction is with the IoT Discovery GE which is needed to the IoT Broker GE for knowing whether and where contexts are available. In order to reduce complexity of the environment setup, the NEC ConfMan implementation has been adopted (instead of more mature and complete implementation of this GE) and deployed in the same server along with Aeron GEri. Other interactions, such those with data consumers and data providers (devices) have been made feasible thanks to the development of dedicated mock up to simulate these actors’ behaviour.
1.1. Attributes of the GEri to be tested: Attribute Value Generic Enabler IoT Broker Chapter Internet of Things Services Enablement GEri name (implementation) tested Aeron GEri version tested 5.2.3 GEri owner NEC Organisation performing the test Engineering Ingegneria Informatica S.p.a Docker file link N/A
1.2. Attributes of the GEri to be integrated in the test: Attribute Value Generic Enabler IoT Discovery Chapter Internet of Things Services Enablement GEri name (implementation) tested NEC Configuration Management (NEC ConfMan) GEri version tested 5.2.3 GEri owner NEC Organisation performing the test Engineering Ingegneria Informatica S.p.a Docker file link N/A
1.3. Attributes of the testing tools: Attribute Value Load Test application Apache JMeter version v 2.13 System monitoring tool NMON (http://sourceforge.net/projects/nmon/) System Data analyser NMON Visualizer (Version 2015-10-21) Java (JRE used for JMeter and NMON OpenJDK Runtime Environment
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 4
Future Internet Core Platform
Visualizer) (IcedTea 2.6.3) (7u91-2.6.3-0ubuntu0.14.04.1) Java (JDK and JRE used for developing and 1.7.0_95 running data provider/consumer mock-up OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95- components) 2.6.4-0ubuntu0.12.04.1) OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode) cURL (command line tool used to register in curl 7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 advance contexts to be found by IoT OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 Discovery during the tests)
1.4. Developed testing tools The main feature of this GE is to be essentially a broker component, as such, it intermediates the communication among different actors. Therefore, a full REST web service had to be developed in order to simulate these involved actors and needed by the individual scenarios. This WS is a mock-up component named iot-broker-tester and it is a Java (JAX-RS) application. Depending on the parameters with which this application is invoked, it can respond to two different incoming POST requests compliant with NGSI-10 protocol that are: updateContextRequest and queryContextRequest. For the latter, with further parameters can also be defined the number of simulated IoT Data Providers, namely the open ports ready to answer to the incoming (QueryContext) requests.
So, for managing the UpdateContextRequest the expected command to run the server is:
java –cp [..] eng.fiware.iotbroker.tester.App data-consumer
which opens the port 8070 waiting for NGSI-10 UpdateContextRequest (from IoTBroker)
and it gives back a message like the following:
On the other hand, for managing the QueryContextRequest the expected command to run the server is:
java –cp [..] eng.fiware.iotbroker.tester.App data-provider 30
which opens 30 ports from 8071 to 8100 waiting for a NGSI-10 queryContextRequest (from IoTBroker)
and each of them is able to answer back a message like the following:
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 5
Future Internet Core Platform
In this case, as can be noted in the xml message above, the response message given back must be coherent with the related request (at least for the entity id and type) in order to have the whole transaction closed properly. Furthermore, IoT Discovery component must be made aware in advance of each available data provider through registering the related context with several NGSI-9 messages like the following:
In the picture above there is an excerpt of a NGSI-9 message that is crafted with a json string sent to IoT Discovery through cURL, the free client-side URL transfer tool. It can be noticed, circled in red, the address to which the message is sent (the name of the server must not be misleading since IoT Broker GE and IoT Discovery are deployed in the same server, as already introduced before). In yellow it is instead circled the address of the data provider, the mock-up application that is able to provide the IoT Broker with context data (data that are potentially integrated with those coming from other providers before to be sent back to the requester - JMeter in the case).
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 6
Future Internet Core Platform
1.5. Non-functional metrics The next table shows the metrics that have been measured during these tests.
Class Metric id Metric Yes/No Performance and stability MPS01 Response times for
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 7
Future Internet Core Platform
2. Testing Summary
2.1. GE overview
The GE IoT Broker implementation Aeron is a backend component that offers the following three main features:
1. It offers a single point of contact to the user, hiding the complexity of the multi-provider nature of the Internet of Things.
2. It collects and aggregates information about thousands of real-world objects on behalf of the user.
3. It provides means to assemble lower-level device information (device-centric access).
Moreover, it has unique properties that cannot be found currently in other IoT platforms such as, on the one hand: information is generated and exchanged only when needed (this is in contrast to the state-of- the-art middleware components where any piece of information - whether needed or not - is stored inside a central repository); on the other hand IoT applications can be written without consideration of the device installation.
2.2. Tested Scenarios
The scenarios defined for stress testing were taken, accordingly with the GE owner, from the most used operations of IoT Broker, namely the updateContext, queryContext and subscriptionContext.
The first scenario aimed to stress the NGSI-10 updateContext operation that represents the push-mode interaction offered by this GE, namely a scenario where no previously subscriptions are needed by the related data flow; in this test, 200 users were simulated by as many JMeter threads which solicited with 8150 requests this operation for a period of half an hour. The main objective was to verify the GE behaviour under a workload of 200 requests/s.
The second test scenario aimed to assess stability of the NGSI-10 queryContext operation that represents the interaction offered by this GE designed for the synchronous retrieval of data. This type of scenario required data to be queried with which the system had to be previously filled in; secondly a light workload of 10 users was simulated through as many JMeter threads which solicited with 24.841 requests the queryContext for a period of 10 hours in a row. Stability was thus assessed under a not too demanding workload which consisted in max 5 queries per seconds with 20 IoT agents previously registered, as suggested by the GE owner.
Finally, the third test scenario aimed to stress the NGSI-10 subscribeContext, an operation that represents the asynchronous data retrieval interaction offered by this GE. For this stress test 30 users were simulated by as many JMeter threads which solicited with 3.890 requests this API method during a period of half an hour. The main objective of this test case was to verify the GE behaviour under a workload of 30 requests/s. Although the reached peak number of sent requests was even lower than one aforementioned, it was still good enough to highlights critical issues.
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 8
Future Internet Core Platform
2.3. Results overview
Results collected and analysed from these three scenarios led to the following main conclusions:
• Initial problems with the installation: only a fixing release could solve them.
• Not satisfactory response times: on average >= 14’’ (<= 1 request/s)
• Error rate: 0% in the updateContext scenario 0,07% in the Query Context 1st execution, but 11% in the 2nd one 2,75% in the SubscribeContext scenario.
• Abnormal resources usage (on AV RAM>500 MB, CPU>70%, 95% peak) also according to the minimum system requirements stated in the GEri documentation.
• Despite previous bullets, GE never crashed during each test.
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 9
Future Internet Core Platform
3. Test case 1: Update Context
3.1. Test case description
This test aimed to stress the NGSI-10 updateContext operation that represents the push-mode interaction offered by this GE. This means that no previously subscriptions are needed by the proper data flow as depicted by the picture below where the same dataflow is summarised. In yellow is circled the starting interaction solicited by client tester VM (JMeter) towards the IoT Broker.
In this data flow are expected 4 main actors which are played respectively by the hosts shown in the following table:
Actor SW involved VM host Role
IoT JMeter Injects load to IoT Broker and it Tester machine Data provider + BeanShell module takes measures
Subject of the test IoT Broker Aeron Hosting machine 1 IoT Provides context availability NEC ConfMan Hosting machine 1 Discovery answers to the IoT Broker Receives the updateContext message Data Consumer iot-broker-tester Hosting machine 2 from Iot Broker to close the loop
For this stress test 200 users were simulated by as many JMeter threads which solicited with 8150 requests the API method updateContext for a period of half an hour. The main objective was to verify the GE behaviour under a workload of 200 requests/s (number suggested by the GE owner). Although the peak number of requests reached by the JMETER tester (24 requests/s) was lower than that one suggested of about an order of magnitude (due to the HW capacity of the tester VM) it was still good enough to verify an unexpected low performance results on IoT Broker side, as reported by the following numbers. The same test was executed two times with different load conditions. Nevertheless, the verified reliability is 100% since in half an hour 8672 updates were performed without any error.
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 10
Future Internet Core Platform
ID GE API method Operation Type Payload Max. Concurrent Threads Header: Accept: application/json Content-Type: application/json
http://
The payload example in the table above is related to the smallest message created by the JMeter test plan that in the present case it has only one attribute. During the test, each message had a random number of attributes between 1 and 20.
3.2. Test results
Update Context Update Context Duration 1st execution 2nd execution Start time 29/03/2016 09:00:20 30/03/2016 21:03:04
Stop time 29/03/2016 09:29:56 30/03/2016 21:32:45
Total time (Stop – Start) 00:29:37 00:29:41
Total sent requests 8.150 8.672
Total threads created 200 400
Threads creation rate 200 threads in 1’’ 400 threads in 1’’
Threads ending rate Simultaneously Simultaneously
3.2.1. Throughput: The following table and chart show respectively the average value of throughput measured by JMeter and the overall number of bytes per second along the time exchanged in total by the Aeron IoT Broker VM.
Throughput (Bytes) Update Context Update Context 1st execution 2nd execution Total 1.610.294 3.389.599 Test duration 1791’’ (29’:51’’) 1900’’ (31’:40’’) Average (per 899,10 Bytes/s 1.783,99 Bytes/s second) 0,9 KB/s 1,8 KB/s
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 11
Future Internet Core Platform
3.2.2. HTTP Responses: Aeron IoT Broker server answered with an HTTP 200 code to all requests as shown by the table below.
HTTP Responses (number) HTTP 200 Update Context Update Context 1st execution 2nd execution Total 8.150 8.672 Average (per second) 4,55 4,56
3.2.3. Response Times: The charts below, along with the related table, shows that the average response time is around 43’’, a high value confirmed also by the median one (39’’), while the “fastest” response, in this condition, took 14’’ to be delivered. Doubling the number of users in the second execution, while the minimum response time remains about the same, max and average values show a worsening, as could be easily expected.
Response times Update Context Update Context (milliseconds) 1st execution 2nd execution Minimum 14.186 14.890 Average 43.514 84.710 Median 39.022 64.178 Maximum 176.441 330.963
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 12
Future Internet Core Platform
ms Response time elapsed 200000 180000 160000 140000 120000 100000 80000 60000 40000 20000
0
9:10:43 9:24:57 9:00:20 9:03:08 9:04:51 9:06:03 9:06:59 9:07:57 9:08:54 9:09:44 9:11:39 9:12:36 9:13:31 9:14:23 9:15:11 9:16:10 9:17:00 9:17:58 9:18:51 9:19:47 9:20:30 9:21:33 9:22:24 9:23:33 9:26:03 9:27:09 9:28:03 9:28:56
3.2.4. Responses /second: The following two charts highlight the pending workload in both test executions, over the time, related to the first 1000 requests injected by JMeter to Aeron IoT Broker and its responses (closed requests). The remaining not depicted requests are not significant as the established trend remained unchanged till the end of the test; moreover, if depicted, the first tens of seconds elapsed to get back the first response from the system couldn’t be visible. It can be noted a constant number of around 200 pending requests that dropped down only near the end of the test.
1200
1000
800
600
400 Open requests 200 Closed requests 0 Pending requests
-200
9:00:24 9:00:25 9:00:27 9:00:30 9:00:36 9:02:55 9:03:04 9:03:09 9:03:13 9:03:16 9:03:35 9:03:56 9:04:24 9:04:32 9:04:51 9:05:00 9:05:06 9:05:09 9:05:20 9:05:34 9:05:40 9:05:55 9:06:12 9:06:15 9:06:20 -400 9:00:20
-600
-800
-1000
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 13
Future Internet Core Platform
1200
1000 Sent requests 800 Closed requests
600 Pending requests 400
200
0
21:03:09 21:03:11 21:03:12 21:03:13 21:03:15 21:03:18 21:03:22 21:03:24 21:03:26 21:03:27 21:03:29 21:03:30 21:03:31 21:03:33 21:03:34 21:03:35 21:03:35 21:03:37 21:03:37 21:05:29 21:05:29 21:05:58 21:05:58 21:06:34 21:06:48 21:06:54 21:06:58 21:07:02 21:07:05 21:07:05 21:07:05 21:07:12 21:07:13 21:07:17 21:07:22 21:07:24 21:07:26 21:07:34 21:08:01 21:08:09 21:08:19 21:08:20 21:08:34 21:08:42 21:08:43 21:08:44 21:08:56 -200 21:03:04
-400
-600
-800
Responses/second (number) Update Context Update Context 1st execution 2nd execution Average (per second) 4,55 4,56
3.2.5. Requests Summary: ..
Requests summary Update Context Update Context (number) 1st execution 2nd execution Successful 8.150 8.672 Failed 0 0 % Error 0% 0%
3.2.6. Monitoring: The charts below highlight a very high system resource usage (average usage of CPU=77,96%, of RAM=1,4 GB -> 70%) during the tested work load conditions of the 1st test execution that can be considered abnormal also according to what is stated by the documentation; in fact, GE Installation and Administration Guide states : “If the IoT Broker process uses more than 500MB and the CPU usage is above 40% in the idle state, this can be considered abnormal” Source: http://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/Backend_IoT_Broker_-_IoT_Broker_- _Installation_and_Administration_Guide
The charts for the 2nd execution are not in this report since they are very similar to the previous ones and can also be easily reproduced by the saved data that are included with this report. Anyway they show an average usage of CPU of 78,56% and an average use of RAM of 1,5 GB which corresponds to a 75% of the total amount) confirming the critical state of the system discovered during the first execution.
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 14
Future Internet Core Platform
3.2.6.1. CPU usage:
3.2.6.2. Memory usage:
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 15
Future Internet Core Platform
4. Test case 2: Query Context
4.1. Test case description
This test aimed to assess stability during a 10 h period of the NGSI-10 queryContext operation that represents the interaction offered by this GE designed for the synchronous retrieval of data. The related dataflow taken into account is depicted and briefly described by the following picture. In this picture can be noted, circled in yellow, the starting interaction that is established by the testing system (JMeter) towards the IoT Broker.
The same Query data flow is also depicted more precisely by the following sequence diagram that is perfectly consistent with the previous block diagram.
It is therefore clear that this scenario is played by four main actors as summarised in the following table:
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 16
Future Internet Core Platform
Actor SW involved VM host Role
Injects load to IoT Broker and it DataConsumer JMeter Tester VM takes measures
IoT Broker Aeron Hosting VM 1 Subject of the test IoT Provides context availability info NEC ConfMan Hosting VM 1 Discovery to the IoT Broker A nr. of IoT Each agent receives query context iot-broker- Data providers Hosting VM 2 requests from the IoT Broker tester (30 IoT Agents) providing back the required Context.
This type of scenario required data to be queried with which the system had to be previously filled in. For this purpose, the NEC Confman (IoT Discovery) component was filled-in with data about provider locations through proper NGSI-9 messages as described earlier at paragraph “Developed testing tools”. On the other hand, a custom tool was developed to simulate the presence and the action of 30 agents able to provide context data that were previously registered in terms of 50 entities for each agent. On the JMeter side a BeanShell PreProcessor was configured based on a Java file named QueryContext.java created for managing the dynamically creation of each query message. In fact this file contains the logic on which is based the creation of the message payload, that is summarised as follows: a NGSI-10 queryContext message is created with a variable number of entities to be queried that will be subsequently answered back by the IoT Broker once it has retrieved and carefully assembled related values from the related agents. The maximum numbers of entities which formed the payload was configured to a number of 20 while the maximum number of agents that could be queried simultaneously was set to 30. An example of the created message can be found in the table below.
ID GE API method Operation Type Payload Max. Concurrent Threads Header: http://
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 17
Future Internet Core Platform
For this test a light workload of 10 users was simulated through as many JMeter threads which solicited with 24.841 requests the API method queryContext for a period of 10 hours in a row. The main objective was to verify the GE stability over the time under a not too demanding workload (max 5 queries per seconds with 20 IoT agents previously registered, as suggested by the GE owner). Although the peak number of sent requests reached by the JMETER tester (0,69 requests/s on average with peaks of 10 requests/s only at the very beginning of the test) was even lower than that one suggested, it was still good enough to verify, on the one hand, the reliability of the system with 24.841 query answered with an error rate of 0,07%; on the other hand this test confirmed that also with not heavy work-load the obtained performance are too low as demonstrated by the numbers in the following tables. The same test was executed two times with the same load conditions to increase reliability of results.
4.2. Test results
Duration Execution nr. 1st 2nd Start time 29/03/2016 18:35:53 31/03/2016 17:55:23
Stop time 30/03/2016 04:35:47 01/04/2016 03:55:21
Total time (Stop – Start) 09:59:55 09:59:58
Total sent requests 24.841 19.914
Total threads created 10 10
Threads creation rate 10 threads in 2’’ 10 threads in 2’’
Threads ending rate Simultaneously Simultaneously
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 18
Future Internet Core Platform
4.2.1. Throughput: The following table and two charts show respectively the average value of throughput measured by JMeter and the overall number of bytes per second along the time exchanged by the Aeron IoT Broker VM in the two different executions.
Throughput (Bytes) Execution nr. 1st 2nd Total 207.370.712 147.745.324 Test duration 36.006’’ (10h 00’ 07’’) 36.007’’ (10h 00’ 07’’) Average (per 5.759,34 Bytes/s 4.103,24 Bytes/s second) 5,8 KB/s 4,1 KB/s
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 19
Future Internet Core Platform
4.2.2. HTTP Responses: Aeron IoT Broker server answered with an HTTP 200 code to all the requests it received as shown by the table below, however part of these responses raised an assertion check failure on the JMeter side due to the missing tag
HTTP Responses (number) HTTP 200 Execution nr. 1st 2nd Total 24.824 19.914 Average (per second) 0,69 0,55
4.2.3. Response Times: The chart below, along with the related table, shows that the average response time is around 14’’ while the fastest response, in this condition, took less than 1’’ to be fulfilled (but this is about to only two cases over 24.841). Repeating once the test with the same conditions, the results found show an average response time increased a bit and a significant worsening of the max response time as well as of the error rate.
Response times Query Context Query Context (milliseconds) 1st execution 2nd execution Minimum 429 210 Average 14.484 18.069 Median 14.343 16.616 Maximum 67.894 104.469
elapsed (1st 300 records) - 1st execution 80000 70000 60000 50000 40000 30000 elapsed 20000 10000
0
18:38:52 18:41:58 18:36:14 18:36:58 18:37:32 18:37:59 18:38:29 18:39:19 18:39:31 18:39:54 18:40:09 18:40:27 18:40:41 18:40:58 18:41:21 18:41:41 18:42:08 18:42:23 18:42:46 18:43:06 18:43:24 18:43:41 18:43:57 18:44:13 18:35:53
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 20
Future Internet Core Platform
elapsed (1st 300 records) - 2nd execution 120000
100000
80000
60000
40000 elapsed
20000
0
18:00:38 18:04:04 17:56:20 17:57:08 17:57:52 17:58:15 17:58:44 17:59:09 17:59:38 18:00:01 18:00:28 18:01:05 18:01:33 18:01:56 18:02:20 18:02:34 18:02:58 18:03:28 18:03:48 18:04:34 18:04:53 18:05:09 18:05:26 18:05:49 17:55:23
4.2.4. Responses /second: The following two charts highlight the pending workload in both test executions, over the time, related to the first 300 requests injected by JMeter to Aeron IoT Broker and its responses (closed requests). The remaining not depicted requests/responses are not significant since the established trend remained unchanged until the end of the test; moreover, if depicted, the first seconds of tests wouldn’t be visible that is where the greatest hike in values was registered.
Requests rate - 1st execution 60 Sent requests
50 Closed requests
Pending requests 40
30
20
10
0
-10
18:36:14 18:37:54 18:35:53 18:35:53 18:35:53 18:35:53 18:35:53 18:36:14 18:36:14 18:36:14 18:36:14 18:36:28 18:36:48 18:36:58 18:37:01 18:37:06 18:37:14 18:37:22 18:37:30 18:37:32 18:37:42 18:37:43 18:37:48 18:37:57 18:37:59 -20
-30
-40
-50
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 21
Future Internet Core Platform
Requests rate - 2nd execution 60 Sent requests
Closed requests 50 Pending requests 40
30
20
10
0
-10
-20
-30
-40
-50
Responses/second (number) Update Context Update Context 1st execution 2nd execution Average (per second) 0,69 0,55
4.2.5. Requests Summary: The following table show that a certain amount of errors occurred in a proportion of responses got back by Aeron IoT Broker that is much greater in the second execution. Since the injected load and test conditions are the same in both the executions, this would require further investigation from a functional point of view that is however not in the scope of these tests. More in detail, as aforementioned at paragraph 4.2.2, all the requests were fulfilled getting back an HTTP-200 code that it is a good result itself, but the assertion configured in JMeter that checked the occurrence of the tag
Requests summary Query Context Query Context (number) 1st execution 2nd execution Successful 24.824 17.674 Failed 17 2.240 % Error 0,07% 11%
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 22
Future Internet Core Platform
4.2.6. Monitoring: The charts below highlight a very high system resource usage (CPU average usage=71,86% with peaks of 99,50%, RAM AV usage=1,40 GB) during the tested work load conditions of the 1st test execution that can be considered abnormal also according to what is stated by the documentation; in fact, GE Installation and Administration Guide states : “If the IoT Broker process uses more than 500MB and the CPU usage is above 40% in the idle state, this can be considered abnormal” Source: http://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/Backend_IoT_Broker_-_IoT_Broker_- _Installation_and_Administration_Guide
The charts for the 2nd execution confirm the critical state of the system discovered in the first one with even higher values in terms of resources consumption. In fact the average usage of CPU is 78,56% and the average use of RAM is 1,57 GB.
4.2.6.1. CPU usage:
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 23
Future Internet Core Platform
4.2.6.2. Memory usage:
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 24
Future Internet Core Platform
5. Test case 3: Subscribe Context
5.1. Test case description
This test aimed to stress the NGSI-10 subscribeContext operation that represents the asynchronous data retrieval interaction offered by this GE. For this test, differently from the previous ones, no further actors were needed besides the essential required for getting from Aeron IoTBroker the acknowledge (subscriptionId) for this case. In fact, the entire complex subscription scenario consists of three parts of which, in the following picture, is depicted the first one, namely that one considered for this test case. In this picture, the interaction directly managed by the test, is highlighted in yellow.
In this data flow are thus expected 3 main actors which are played respectively by the hosts shown in the following table:
Actor SW involved VM host Role IoT JMeter Injects load to IoT Broker and it Data + BeanShell Tester machine takes measures Consumer module
IoT Broker Aeron Hosting machine 1 Subject of the test IoT Provides context availability NEC ConfMan Hosting machine 1 Discovery answers to the IoT Broker
For this stress test 30 users were simulated by as many JMeter threads which solicited with 3.890 requests the API method subscribeContext during a period of half an hour. The main objective of this test case was to verify the GE behaviour under a workload of 30 requests/s (number of subscriptions suggested by the GE owner). Although the peak number of sent requests reached by the JMETER tester (2,16 requests/s on average with peaks of 30 requests/s only at the very beginning of the test) was even lower than that one suggested, it was still good enough to verify also in this case low performance results of this GE implementation, as well as a not negligible error rate of 2,75% that would need to be investigated at development side (e.g. in the confman.log that is part of this report, 107 occurrences of NullPointerException can be found).
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 25
Future Internet Core Platform
ID GE API method Operation Type Payload Max. Concurrent Threads Header: Accept: application/xml Content- Type: application/xml
http://
The payload example in the table above is related to a typical message created by the JMeter test plan. The values of tags
5.2. Test results
Duration Start time 25/03/2016 16:32:09
Stop time 25/03/2016 17:02:07
Total time (Stop – Start) 00:29:59
Total sent requests 3.889
Total threads created 30
Threads creation rate 30 threads in 1’’
5.2.1. Throughput: The following graph shows the number of Bytes per second along the time while the table below shows its average value.
Throughput (Bytes) Subscribe Context Total 2.090.415 Test duration 1802’’ (30’:02’’) Average (per 1160 Bytes/s second) 1,16 KB/s
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 26
Future Internet Core Platform
5.2.2. HTTP Responses: Aeron IoT Broker server answered with an HTTP 200 code to all requests as shown by the table below, however part of these responses raised an assertion check failure on JMeter side due to the missing tag
HTTP Responses (number) HTTP 200 Total 3.889 Average (per second) 2,16
5.2.3. Response Times: The chart below, along with the related above table, shows that the average response time is little lower than 14’’, a high value confirmed also by the median value; on the other hand, the fastest response in this condition took 0.23’’ to be fulfilled which is a misleading value, as it is related to only three samples that got back error responses.
Response times Subscribe Context (milliseconds) Minimum 232 (*) Average 13.887 Median 14.435 Maximum 43.660 (*) Only three response times below or equal to 1’’ are all related to cases of error responses
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 27
Future Internet Core Platform
m elapsed 35000 30000 25000 20000 15000 elapsed 10000 5000
0
16:32:51 16:32:51 16:32:52 16:32:53 16:36:14 16:36:18 16:36:19 16:37:32 16:37:35 16:37:40 16:51:40 16:51:48 16:51:54 16:54:25 16:54:29 16:54:37 16:54:37 16:54:44 16:54:45 16:55:00 16:55:01 16:55:02 16:55:03 16:55:07 16:57:53 16:59:58 16:32:48
5.2.4. Responses/second: The following chart highlights the pending workload over the time, related to the first 100 requests injected by JMeter to Aeron IoT Broker as well as its responses (Closed requests). The remaining not depicted requests are not significant as the established trend remained unchanged till the end of the test; moreover, if depicted taking into account the huge number of samples, it wouldn’t be perceptible the part related to the first tens of seconds elapsed for getting the first response from the system.
Responses/second (number) Subscribe Context Average (per second) 2,16
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 28
Future Internet Core Platform
5.2.5. Request Summary: The following table show that a certain amount of errors occurred in a proportion of responses got back by Aeron IoT Broker. More in detail, as aforementioned at paragraph 5.2.2, all the requests were fulfilled getting back an HTTP-200 code that is a good result itself, but the assertion configured in JMeter that checked the occurrence of the tag
Requests summary Subscribe Context (number) Successful 3.782 Failed 107 % Error 2,75%
5.2.6. Monitoring: The charts below highlight a high system resource usage (average usage of CPU=67%, of RAM=0,93 GB -> 46%) during the work load conditions of the 1st test execution that can be considered abnormal also according to what is stated by the documentation; in fact, GE Installation and Administration Guide states : “If the IoT Broker process uses more than 500MB and the CPU usage is above 40% in the idle state, this can be considered abnormal”
Source: http://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/Backend_IoT_Broker_-_IoT_Broker_- _Installation_and_Administration_Guide
5.2.6.1. CPU usage:
5.2.6.2. Memory usage:
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 29
Future Internet Core Platform
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 30
Future Internet Core Platform
6. Conclusions From the results collected through the four scenarios took into account, can be observed what follows:
Installation of this GE took more time than expected. In fact, it required many failed attempts that required the GE owner to release a new version in order to fix some blocking errors. Error rate was 0% for the updateContext scenario while, for the Query Context, it was 0,07% during first execution but 11% in the second one with the same conditions. SubscribeContext had an error rate of 2,75%. Response times were in the average much above 1 second so they cannot be considered satisfactory. More in detail, in the updateContext scenario, the first execution got an average response time value of 43’’ and 84’’ in the second one. In the queryContext scenario the first execution got an average value of 14’’ and 18’’ in the second one. Finally the subscribeContext scenario got an average value of little less than 14’’. These results were detected during normal operating conditions as well as with moderate work load. Therefore it made no sense to test more stressful conditions. GE never crashed during each test. Detected resources usage is abnormal also according to what is stated in GE documentation (RAM usage>500 MB). In fact, in the first two scenarios RAM usage was always above 1,4 GB while CPU > 70% with peaks of 99,5%. In the third scenario (that one with the lowest stressful conditions) RAM usage was 0,93 GB and CPU 67%). Minimum requirements declared on GE documentation (of which there is an excerpt at the section “Attributes of the hosting machine 1 (IoT Broker server)” of this report) are likely underestimated.
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 31
Future Internet Core Platform
7. Annex
7.1. Annex 1: Test environment A dedicated FIWARE Lab was configured in order to host the implementation of this GE (Generic Enabler). Through this environment, based on the cloud operating system OpenStack, it was possible to easily create and configure all the Virtual Machines needed for the stress tests. In particular the following four virtual machines were set up: one for the deployment of Aeron IoT Broker server, one for JMeter, namely the tool used to inject load and for simulating the client and collecting the results generated by Aeron, one for a light implementation of the IoT Discovery, that is an essential component for having IoT Broker working properly, and finally a server where to host a custom mock-up (WS application) developed to simulate the missing actors needed by some test scenarios.
7.1.1. Attributes of the hosting machine Attribute Value Hypervisor vendor KVM Virtualization type Full Virtualization AMD-V
O.S, version Ubuntu 12.04.5 LTS CPU Architecture X86_64 CPU vendor AMD Opteron 23xx (Gen 3 Class Opteron) CPU nr 1 CPU cores 1 CPU GHz 2,4 CPU L1 cache size 64 KB CPU L2 cache size 512 KB RAM 2 GB HDD Storage 20 GB Sizing of Aeron IoT Broker server VM fulfils the minimum system requirements stated at https://github.com/Aeronbroker/Aeron#minimum-system-requirements which are:
Processor: 1 CPU 1.2 GHZ RAM: 1 GB DISK Space:50 MB JAVA : Java 7 Operating System: 32 or 64-bit version Windows or Linux
7.1.2. Attributes of the hosting machine 2 (Data Consumer + Data Provider servers) Attribute Value Hypervisor vendor KVM Virtualization type full Virtualization AMD-V
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 32
Future Internet Core Platform
O.S, version Ubuntu 12.04.5 LTS CPU Architecture X86_64 CPU vendor AMD Opteron 23xx (Gen 3 Class Opteron) CPU nr 2 CPU cores 2 CPU GHz 2,4 CPU L1 cache size 64 KB CPU L2 cache size 512 KB RAM 4 GB HDD Storage 40 GB
7.1.3. Attributes of the testing (client) machine Attribute Value Hypervisor vendor KVM Virtualization type full Virtualization AMD-V
O.S, version Ubuntu 14.04.1 LTS CPU vendor AMD Opteron 23xx (Gen 3 Class Opteron) CPU nr 1 CPU cores 2 CPU GHz 2,4 CPU L1 cache size 64 KB CPU L2 cache size 512 KB RAM 2 GB HDD Storage 20 GB Infrastructure above is made available by the organisation in charge of performing the tests.
7.2. Annex 2: JMeter Results Test case 1, 1st execution: update_2016-03-29 075955.csv (*), iotbroker433_160329_0959.nmon
Test case 1, 2nd execution: update_2016-03-30 190249. csv (*), iotbroker433_160330_2100.nmon
Test case 2, 1st execution: queryContext_2016-03-29 163549. csv (*), iotbroker433_160329_1834.nmon
Test case 2, 2nd execution: queryContext_2016-03-31 155519. csv (*), iotbroker433_160331_1751.nmon
Test case 3: subscription_2016-03-25 143206. csv (*), iotbroker433_160325_1529.nmon
(*) Timestamps of CSV files generated by JMeter in the client VM are set two hours behind the system time (which is synchronised between client and server VMs). Therefore adjustments have been set, once imported, to this data-source, in order to produce charts that are time comparable with those produced by nmon tool in the server VM.
D.11.8.1: Report on QA testing (AERON IoT Broker stress testing) 33