PC2 V10 PERFORMANCE TEST FRAMEWORK
A Project
Presented to the faculty of the Department of Computer Science
California State University, Sacramento
Submitted in partial satisfaction of the requirements for the degree of
MASTER OF SCIENCE
in
Computer Science
by
Bhushan Arun Ladde
FALL 2018
© 2018
Bhushan Arun Ladde
ALL RIGHTS RESERVED
ii
PC2 V10 PERFORMANCE TEST FRAMEWORK
A Project
by
Bhushan Arun Ladde
Approved by:
______, Committee Chair John Clevenger, Ph.D.
______, Second Reader Doan Nguyen. Ph.D.
______Date
iii
Student: Bhushan Arun Ladde
I certify that this student has met the requirements for format contained in the University format manual, and that this project is suitable for shelving in the Library and credit is to be awarded for the project.
______, Graduate Coordinator ______Jinsong Ouyang, Ph.D. Date
Department of Computer Science
iv
Abstract
of
PC2 V10 PERFORMANCE TEST FRAMEWORK
by
Bhushan Arun Ladde
The PC2 Contest Control System provides for the ability of teams in a programming contest to submit runs – that is, programs which are attempts at solving a contest problem specified by the contest judges. In Version 10 (V10) of PC2, runs are submitted via web pages served by a webserver embedded in a module called the AppServer. On the server- side, AppServer automatically inserts a submitted run into a database. The Dispatcher queues run notifications and makes them available to another module called the Checker.
Checker returns the result to the Dispatcher that whether run correctly solved the problem or not. The Dispatcher in turn forwards the result to the AppServer for subsequent display to the team which submitted the run. There is a Resource Manager which supports the ability of the Contest Administrator to define multiple different contests and to enter contest configuration information for each defined contest. The architecture allows only a single Dispatcher and a single Resource Manager to be running at any given time. Each
V10 module runs on its own separate machine.
v
There are some limitations in the current system. A fundamental premise of the PC2 V10 system is that it is intended to handle large numbers of teams simultaneously participating in multiple different contests. The problem is that there is currently no mechanism for testing the performance of the system under such a large load and verify the performance capabilities of the V10 system.
The proposed solution includes several important points.
A. Build a large collection of programming contest problems and corresponding run submissions.
B. Investigate a variety of available performance and load-testing tools.
C. Implement a framework for submitting the collection of contest problems and corresponding submissions to PC2 V10 and monitoring the resulting performance using the selected tool(s).
D. In the event that Jetty is found to be unable to support the desired load, investigate and propose mechanisms for substituting alternative webservers into V10 (e.g, using the nGinX webserver).
______, Committee Chair John Clevenger, Ph.D.
______Date
vi
ACKNOWLEDGEMENT
I would like to express my gratitude to my project supervisor Dr. John Clevenger for the useful comments, remarks, guidance and engagement through the learning process of this master’s project.
I would also like to thank Dr. Doan Nguyen for taking time to review my report and giving me important feedback.
Added to that, I would like to thank the Department of Computer Science at California
State University, Sacramento for providing an opportunity to pursue my Masters Degree and guiding me all the way to become a successful student.
vii
TABLE OF CONTENTS
Page
Acknowledgement.………...... vii
List of Figures………………………………………………………………………….. x
Chapter
1. INTRODUCTION……………………………………………………………………. 1
1.1 What is a Programming Contest?…………………………………………...... 1
1.2 PC2……………………………………………………………………...... 1
1.3 PC2 V10 Introduction……………………………………………………………3
1.3.1 Main Components of PC2 V10…………………………………………….4
1.3.2 Technology Stack for PC2 V10……………………………………………5
2. EXISTING PROBLEM AND PROPOSED SOLUTION…………………………....6
2.1 Performance Measurement Tool ………………………………………………...6
2.2 Architecture of JMeter…………………..………………………...... 8
3. TESTING TOOLS……………………….…...... 10
3.1 Initial Excel Utils Project Set Up…...…………………………………………..10
3.2 JMeter Installation…………………………...... 12
3.3 JMeter Test Plan Creation ……………………..……………...... 14
viii
3.4 Create Test User in JMeter ………………………………...... 19
3.5 JVM(Java Virtual Machine) Monitor ………...... 20
3.6 Postman ...………………...... 22
4. IMPLEMENTATION STRATEGY ..………………………...... 23
4.1 Contest Data Creation Using PC2 V10 Tool...... 24
4.2 Steps to Implement PC2 V10 Test Plan...... 26
4.3 Measured Points...... 27
5. SIMULATION RESULTS…………………………...... 31
5.1 Load Capacity Result...... 31
5.2 Memory Dump Issue...... 32
5.3 Graphical Analysis of Performance……………………………….…………….32
5.4 Do Not Use XAMPP as a Database...... 33
6. FUTURE SCOPE…………………………...... 36
6.1 Distributed Environment...... 36
6.2 Testing on Different Platforms...... 36
6.3 Security Test...... 36
6.4 Alternative Webservers into V10…………...... 37
References…..….………………………...... 38
ix
LIST OF FIGURES
Figures Page
1. PC2 V10 Architecture………….….………………...... 4
2. JMeter Architecture Block Diagram….…...... 8
3. JMeter Installation Step 1………………...... 12
4. JMeter Installation Step 2………………...... 13
5. JMeter Initial Screen………………...... 13
6. JMeter Test Plan Creation Step 1………………...... 14
7. JMeter Test Plan Creation Step 2………………...... 15
8. JMeter Test Plan Creation Step 3………………...... 15
9. JMeter Test Plan Creation Step 4…………...... 16
10. JMeter Test Plan Creation Step 5…………...... 17
11. JMeter Test Plan Creation Step 6…………...... 18
12. JMeter Test Plan Creation Step 7…………...... 18
13. JMeter Recorder Creation…..…………...... 19
14. JVM Test Monitor - 1….…..…………...... 21
15. JVM Test Monitor - 2….…..…………...... 22
16. JMeter PC2 V10 Login Test Plan Implementation...... 23
17. JMeter PC2 V10 Test Plan Implementation ……………...... 24
18. Contest.yaml Configuration...…………...... 25
x
19. PC2 V10 Tool…………….....…………...... 26
20. Performance Graph of PC2 V10……………………………………………...……33
21. XAMPP…………...………...... 34
xi
1
1. INTRODUCTION
1.1 What is a Programming Contest?
A programming contest is an internet-based contest. There are two type of contests, first one is an individual programming contest where the individual programmer will get a problem statement in which he/she has to provide its individual solution, second one is group programming contest in which there are group of people who will get the problem statement. Users have to provide solution to that problem. A vast majority of problems appear in programming contest are based on mathematics and logical concepts. Once any user has submitted its solution then the automation process verifies the answer. The verification process is performed by the host machine, it is also called as judging the answer. There are a certain number of test cases to those problem statements and the correct solutions are stored in database.
Judges are going to validate the solution from the contestants and compile the code. Once the code is compiled successfully, the code needs to pass all the test cases and generate a valid output. The generated output is verified using the correct solution stored in database.
The output of this verification is informed to the user. User can do multiple submissions depending on the type of contest. If the user has submitted the submission then they have wait to finish it. Once it is finished then they can submit the next submission.
1.2 PC2
PC2 stands for Programming Contest Control System. It is pronounced as P-C-Squared.
There are different versions of PC2. This report is based on PC2 V10 (Version-10). This is
2 a programming contest software similar to some well-known programming contests across the world such as Codechef [6], Facebook Hacker Cup [7], Google Code Jam [8],
HackerRank Week of Code [9], ICFP Programming Contest [10]. PC2 software is operated by ICPC – organized by ACM [11].. The PC2 V10 system is designed to operate on cloud- based servers .The ICPC contest has different levels:
1. College Level – All students at a given college participate
2. Regional Level – Qualified college level competitors can participate in this level
3. World Level (Finals) - Qualified Regional level competitors can participate in this
level
Each contest in an ICPC-like competition involves several categories of participants, as follows:
Administrator:
The administrator is the person who is going to create teams, judges and scoreboard.
The setup work such as deciding the problem statement, finding the correct solution
and validating the answer is done by the Administrator.
Team:
Teams consists of participants of the contest. Teams are responsible for solving the
problem statement and submitting the solution to the software.
3
Judge:
Judges are the automation programs residing on servers. In early version of PC2, the
judgments were manually done by the judges. In version 9, these automation programs
for verifying the code were introduced. The automation program is responsible for
running the submission of code with multiple test cases, generate the output and
compare the output with the correct solution stored in the database to verify the
submission.
1.3 PC2 V10 Introduction
This version is completely moved into a cloud environment. PC2 V10 is a web-based contest management framework. It supports the ability to create and configure contests, control who can participate in defined contests, start/stop contests, and a variety of similar operations[1].
4
1.3.1 Main Components of PC2 V10
Figure 1: PC2 V10 Architecture
Resource Manager:
This is the entry point to the system which is going to start all the server side including
the App Server, Dispatcher, and Checker. According to technical Specs, the Resource
Manager acts as a source of machine resources when other components of the system
need to acquire a resource (such as a machine)[1].
App Server:
It is responsible for communicating with Teams, Judges, and Contest administrator. It
supports REST base communication.
Dispatcher:
It receives notifications of the receipt of a "run" (team submission) and is responsible
5
for directing the run to a resource which can execute it[1].
Checker:
It is a resource capable of compiling, executing, and validating (determining the
correctness of) a run[1].
DBMS:
We are using MySQL database.
1.3.2 Technology Stack for PC2 V10
Logback:
We are using Java logging system to log the error, console message through Java
logger. It is helpful for the developer to easily debug the problem in the code and make
their life easy.
Jetty:
Jetty provides a servlet container and HTTP server. It eliminates the need to deploy an
app to a standalone application server; the server is already embedded into the
application. Starting the application normally from the main method automatically
includes starting the Jetty application server[1].
Jackson:
A set of JSON parsing tools for de/serialization of JSON for REST services[1].
6
2. EXISTING PROBLEM AND PROPOSED SOLUTION
2.1 Performance Measurement Tool
Performance plays an important role in software. It is important for web-based applications as well as for mobile applications. Currently, PC2 is limited to the web-based applications.
As PC2 is a Java based project, a Java framework test tool is required which is also platform compatible. After researching, it was found that there are various tools including HP
LoadRunner, JMeter, NeoLoad, Web Load, LoadUI NG Pro, SmartMeter.io, LoadView,
Appvance, NeoLoad, LoadComplete, WAPT, Loadster, LoadImpact, Rational
Performance Tester, Testing Anywhere, OpenSTA, QEngine (ManageEngine), Loadstorm,
CloudTest, Httperf etc[13].
The project needs a tool which has the ability to put load on the web-based project to check the performance of the software. It is necessary to send the data through a web request which contains the URL query parameter and request post body data. PC2 V10 is using
JMS Queue at three different places. It includes AppServer, Dispatcher, and Checker. It is necessary to put a load on the JMS queue to calculate the performance of PC2 V10.
Open source is also one of the important criteria to select the testing tool. We are using Eclipse as an IDE tool, it is better if we are able to integrate with Eclipse. It should be easy to integrate with any operating system environment including Windows and Linux.
The documentation of the tool should be available easily which makes learning really easy.
7
There are two types of performance testing, load testing and stress testing [3]. Load testing is to see whether the application is capable of handling multiple simultaneous requests from the users of PC2. Load testing gives you system performance. Stress testing is determine by increasing the number of inputs. Stress testing helps you find the upper limits of the software. Both type of testing gives us the threshold value of resources requirement in AppServer, Checker, Dispatcher and Database. It is always better to have a user-friendly tool, user friendly in terms of the GUI from which we can create test cases and it is also easy to understand for a new user who is going to perform the testing. Support part also plays a vital role to create the difficult test cases. PC2 software is a distributed application where Dispatcher, Checker, and AppServer are on the different systems, therefore it is mandatory to test them individually as well as necessary to perform the integration testing. It would be great if we are able to connect to the database via the testing tool. It helps to reduce the overhead work while testing the database transaction because there are large amount programming data inside the PC2 software database. It may slow down the performance of the PC2 V10 software. It will easy if testing software provides direct connection facility. The testing software has to provide the comprehensive documentation. Because of its robust documents, it helps tester to get a clear idea at every step. There are many testing software tools which are provide support for automated recording script of websites. To speed up the testing progress, it would be great if the software is able to give this kind of facility.
8
The final part is to see the report of test plans. It has the report in various formats. It is giving flexibility to the tester to present his work at various points. This makes easier to tester to identify the problem locations. The above are filter criteria we have applied to select the testing tool. Most of the testing software which was considered failed one or more of the above criteria.
The two software which satisfy most of the criteria are HP Load runner and JMeter.
We have compared HP Load runner and JMeter and it is almost giving the same functionality but HP Load runner is paid software and it is going to cost us a lot , thus we have decided to go with JMeter which is really easy to integrate with JAVA applications.
2.2 Architecture of JMeter
How JMeter works is shown in Figure 2,
Figure 2: JMeter Architecture Block Diagram[2]
9
The START is the initial UI block which is providing a user interface for the tester to create different test plans. Create a request and user simulation model is an important role in
JMeter. It is going to create the number of the thread where each thread represents one user. Once you have created the test plan then you just need to click on play button. JMeter is going to send the request to the target software and it will wait to get a response from the server. JMeter performs statistical analysis of response. You can see the request and response data per request in JMeter. It is easy to see if something goes wrong with the request or if it is giving 400 response from the server.
JMeter shows the report in various ways. These includes Aggregate Graph, Aggregate
Report, Assertion Results,Backend Listener, Bean Shell Listener, Comparison Assertion
Visualizer, Generate Summary Results, Graph Results, JSR223 Listener, Mailer
Visualizer, Response Time Graph, Simple Data Writer, Summary Report, View Results in
Table and View Results in Tree.
Each Report has its own specialty. It gives a clear picture of what is going with the request.
It provides detailed analysis which is not only easy to understand but also helpful to recognize the problem.
10
3. TESTING TOOLS
This chapter explains the purpose and configuration of each of the tools using for testing in the project. These include a custom-built Excel-based utility for generating input files for JMeter; JMeter itself for monitoring HTTP requests; JVM Monitor for examing Java
Virtual Machine performance; and Postman, a tool for issuing HTTP requests. Different test scenarios can be tested by providing configuration through external files. There are many file formats in which JMeter is able to read the configuration. File formats are shown on JMeter official site[12]. JMeter test plans help to simulate the HTTP request.
The memory analysis is done using JVM Monitor, It is great tool to measure the memory usage of any software. This chapter will give an idea about Postman as well.
Postman is the software which helps in constructing the request and reading the response.
It provides you a nice GUI which is easy to understand.
3.1 Initial Excel Utils Project Set Up
This project helps to create the sample test data which is require to create a test user. Test users are nothing but thread in the JMeter which we will cover in below topic discussion.
There is a separate project name as Excel-Utils which is going to read the
CDP\\finals2016\\real\\Saved\\eventFeed\\domjudge.xml file from PC2 V10. In this file, you will find all the necessary data for the configuration which includes submission id, problem number, team number and language. Excel-Utils Project is going to fetch the data
11 from this file and create the submission.csv file. This CSV file is input to JMeter and it is going to create the URL request with the help of data in this file. The Excel-Utils project helps to get a respective data (e.g. username, password, run id, submission file name and problem id) from the different location. It creates the resultant Excel file from all that data.
JMeter uses this file to create sample test user request. Below are the steps which are necessary to set up this project.
1) Install Maven [5]. 2) Steps to set up the Excel Utils project 1) Go to the project directory and check whether they have pom.xml file. If pom.xml is present in project directory then perform the below operation. If it is not there then it is not useful to perform below step. 2) Open the command prompt window at project location 3) Run the below commands in a sequence 1) mvn clean install 2) mvn clean eclipse :clean eclipse: eclipse clean install 4) Import the project in an Eclipse. 3) Go to the ReadXMLFile.java and put the path of domjudge.xml file. This is the location where you can select the contest. 4) Go to ExcelWriter.java and run this class. You can put the path in this file where you can to csv resultant file which will get use by JMeter to create the sample test cases. 5) Copy all submission file from finals2016.real.ecs.cdp\submissions to the location where we have kept of csv resultant file. 6) Do the configuration for JMeter and run the test cases.
12
3.2 JMeter Installation
Install JRE and JDK to your system based on O.S. (64bit or 32 bit). Then download the
Apache JMeter from the Apache official site. It is a binary zip file which you need to extract on your local system. You can see the following files as given in Figure 3.
Figure 3: JMeter Installation Step 1
Go into the bin folder.
13
Figure 4: JMeter Installation Step 2
Run the JMETER exe file. Once you will start JMeter then it will look like following window which can be seen in Figure 5.
Figure 5: JMeter Initial Screen
14
3.3 JMeter Test Plan Creation
Follow the below steps to create a test plan.
STEP 1: Click on the file button on the menu bar and it should show following screen as can be seen in Figure 6.
Figure 6: JMeter Test Plan Creation Step 1
STEP 2: select new thread group. Each thread represents one user. So test plan needs to create the thread group which contains multiple thread as shown in Figure 7.
15
Figure 7: JMeter Test Plan Creation Step 2
Give a name to your thread group.
Figure 8: JMeter Test Plan Creation Step 3
16
STEP 3: HTTP is Hyper Text Transfer Protocol. This protocols defines the message communication over the web. PC2 V10 is cloud based solution, It accepts the needed data through HTTP request. So it is required to create a new HTTP request inside the thread group.
Figure 9: JMeter Test Plan Creation Step 4
STEP 4: Enter the name inside the HTTP Request Defaults.
17
Figure 10: JMeter Test Plan Creation Step 5
STEP 5: Results can be seen in various formats. For example in the Figure 11, we see how to view result in tabular formation. Also, request response will be monitored by JMeter and result will be shown in tabular format.
18
Figure 11: JMeter Test Plan Creation Step 6
The result table is shown below in Figure 12.
Figure 12: JMeter Test Plan Creation Step 7
19
JMeter also provides templates for creating HTTP requests, FTP(File Transfer Protocol) and JDBC(Java Database Connectivity) connection requests. There is facility of default template as well. More details can be found in JMeter documentation[12].
Figure 13: JMeter Recorder Creation
3.4 Create Test User in JMeter
Test users are nothing but threads which act like users in test environment.
1) Create a test plan and request inside that test plan.
2) Assign multiple threads to the same request. (Thread act as users).
3) Each thread, Request / Responses can be monitored and filtered using JMeter interface.
20
4) The errored request / response can be easily filtered out and debugged if required. It is marked in red color.
5) Accuracy statistics will also be shown in final report.
6) Developer can easily modify the test plan as per requirements.
3.5 JVM(Java Virtual Machine) Monitor
JVM(Java Virtual Machine) Monitor can be used to monitor CPU, memory consumption and profiling of client code. JVM Monitor can also be integrated with Eclipse IDE to create instant reports by quickly running the client code inside the IDE. All the low-level analysis is done with the help of JVM monitor. It has a Java profiler which is integrated with
Eclipse. It’s helpful to monitor CPU, threads and memory usage of Java products. It is proving the inbuilt facility to find out the running JVM on the local machine and monitor it automatically.
It is the part which is handled by the JVM monitor. You can see in the Figure 14,
21
Figure 14: JVM Test Monitor - 1
1) In left top section: JVM Explorer showing processes that are running through the application.
2) Window shows code in middle.
3) At right bottom, loaded class count can be sync along with heap memory and CPU uses.
4) Memory and CPU uses can further be sliced down to each class if needed.
5) It is easy find the root cause of memory error using the JVM Monitor.
If you will see in the Figure 15 then it is explained you total memory usages and how it got distributed in different areas.
22
Figure 15: JVM Test Monitor - 2
Figure 15 shows heap memory vs non heap memory used by client application. It also shows that how much garbage collector helps to free the memory.
3.6 Postman
Postman is a helpful tool for mocking request headers and query parameters to HTTP server. Request query parameters and body content needs to be matched to pass the validation at server side. Postman created URLS (requests) can be used in JMeter to make final testing job easier. Postman helps to focus time on analysis part rather than spending more time in constructing the requests objects in JMeter [4].
23
4. IMPLEMENTATION STRATEGY
JMeter allows to import configuration through Excel files. For example, different login credentials for any use-case can be provided to a single test plan in JMeter; this avoids the creation of multiple test plans in JMeter. Username and Password can be added inside
EXCEL columns and they can be referred inside the JMeter request body or query / URL string parameters by specifying the $ symbol at the front (e.g. ‘$Username’). The Figure
16 shows the implementation of login test plan in PC2 V10. In Figure 16, it can be seen the username and password is being fetch from the Excel file.
Figure 16: JMeter PC2 V10 Login Test Plan Implementation
24
Furthermore to the above explanation, In Figure 17, JMeter is able to send the request data through Query parameter as well as Payload parameter which is being used in the current implementation of PC2 V10 test plan.
Figure 17: JMeter PC2 V10 Test Plan Implementation
4.1 Contest Data Creation Using PC2 V10 Tool
PC2 V10 contains a tool to create a sample contest database. The tool refers to the
Contest.yml file. PC2 V10 tool is reading all the configuration from Contest.yml.
File is present on location: ‘pc2v10\samps\contests\sumitMTC\config\contest.yaml’.
The CDP folder is the location where you can find all the sample test programs.
25
CDP Location Path: ‘pc2v10\samps\contests\sumitMTC\config’
Figure 18 explains the contest.yml file configuration and it is easy to understand.
Figure 18: Contest.yaml Configuration
In Contest.yml file, mention the problem set and a number of accounts. You can declare the problem set and a number of accounts, required for testing. From this, it is easier to create the number of accounts in the database.
26
Figure 19: PC2 V10 Tool
Once you have provided the necessary configuration then run the PC2 V10 tool, it automatically creates records and inserts data into the database.
4.2 Steps to Implement PC2 V10 Test Plan
Once you have created all the test data into the database, the actual works start. As we have seen in the earlier discussion, it is necessary to do the following set up.
1) Create the request for the module which you want to test in JMeter.
2) Provide a number of requests you want to send at a time in JMeter.
3) Create a request listener where it is showing the result of the request in JMeter.
27
4) Add configurations to create the test data to analyze the performance of the software:
Once we run the JMeter, it is going to read the data one by one from the Excel file then
going to create a request and send it to the PC2 V10. In the PC2 V10, we have integrated
the code to create a performance resultant text file, in this file we are going to record
the total time that which is taken by each module to perform the operation on the
incoming request. This method is giving us detail analysis of the performance of the
PC2 V10 software. We have used the Java System class to get current timing in
milliseconds and used basic mathematical operations to calculate the result.
JMeter is reading the SubmissionID, ProblemNumber, TeamNumber, Language,
LoginName, Password, and RunId which is present in the Excel file. You can modify
to the path to write down this file in your system to get a performance result. This path
is mentioned in the pc2v10.properties file. You can see the parameter as a test result
URL.
4.3 Measured Points
After everything is done, you can send the HTTP requests. In this project, it is measuring the below important points.
1) Response time
When we are sending any request to the AppServers it is going to give us a response.
It is possible to see the total response time of each request in JMeter. It can be seen that
the server log as well in which is easy to understand that how the request is going
28
through the code. If any particular class is taking more time then you can analyze that
class in JUnit test case which is present inside the PC2 V10 software.
2) Memory Analysis
As we have seen in the above chapters, we can do the memory analysis using the JVM
monitor. JVM monitor is elaborating the total part of heap size and stack size gets used
which serving any particular request. With the help of logs, we can find out the number
of classes which are getting used while processing current request. In JVM monitor,
you can see the number of data types (e.g char, string, int) and total memory they have
used. After memory analysis, it is easier for us to reduce bottlenecks in our code so that
memory consumption will get reduce.
3) Load Capacity
This is one of the important tasks of this project. We have sent simultaneously multiple
requests to the PC2 V10 software and then we measure the response time for those
requests. While doing this process, we can find the threshold number above which PC2
V10 software stops responding to your request. Basically, after a certain number of
request at a time, PC2 V10 is stopped responding to our request. Load capacity is a
threshold point from which our software is not able to respond.
4) Overall System Analysis
We have divided system analysis into the five different parts. We have added the code
to measure the performance of each module and used the System.currentTimeMillis()
method to measure the timestamp. For each module, there is starting timestamp and
29 ending timestamp. The difference of ending timestamp and starting timestamp tells the execution time taken by the module. In this way, we have measure the performance of each module.
Part A:
In this part, we are analyzing the performance of the AppServer. AppServer is mainly interacting with the user, where its performance mainly depends on three parts. First, it has to validate the credentials of the users. This means it has to interact with data base(DB). Second, it has to send the data to dispatcher queue if it has valid credentials.
Part B:
We are doing the performance evaluation of the dispatcher queue. It means the total time taken from object inserted from App server to the removal of the object from the dispatcher queue and it is being provided to the Checker. As the number of request gets increased, JMS queue size is getting increased over the time. This will increase the total queue time for each object as the number of request is getting increased.
Part C:
This part measures the checker and database request responses. As checker has to validate the answer against the solution provided by the user, it has to interact with the database. Part C includes the total processing time required to validate the solutions against all the test cases and time taken by the checker to interact with the database.
This is increasing the time requirement as the number of users request gets increased.
30
Part D:
Once checker has completed its analysis, it submits a results object into the checker
JMS queue. From that JMS queue, AppServer is going to fetch the object and serve that object analysis data to the user which we have included in part E. As we have said in part C that if the number of the request is getting increase then it increases the load on the JMS Queue, which automatically increases the processing time for each object.
This is degrading the performance as the number of the request is getting an increase over the time.
Part E:
This is again the evaluation of the AppServer. It has to give the response to the user after successfully serving the request. If the number of users is more, then PC2 V10 will get a slower response from the JMS queue[Part D]. Hence, it is going to serve the response for each request slowly. This is going to affect the overall performance of the system.
31
5. SIMULATION RESULTS
We have built a large collection of programming contest problems and corresponding run submissions using PC2 V10 Tool. Investigated a variety of available performance and load- testing tools as explain in the section 2.1 in the report. We have written the code to measure the performance of each module. Used JMeter performance tool for submitting the collection of contest problems and corresponding submissions to PC2 V10 and monitoring the resulting performance. Below are the important results we have received after carefully analyzing the software.
5.1 Load Capacity Result
We have created 10000 user login credentials. We have made the necessary configuration in JMeter to submit 10000 requests at a time to the PC2 V10. After submitting this many requests, we have identified that PC2 V10 is not responding after the 300 user request at a time. PC2 V10 started failing to give response after 300 requests. Some points came up after the carefully analysis. The first scenario is about the database. We have tried to find bottlenecks in database connection. The code written for PC2 V10 is closing the connection properly so we did not find any problem with DB connections. Secondly, we have tried to find the cause in separate modules (Dispatcher, Checker, and AppServer). We are still not able to find the root cause of this problem.
32
5.2 Memory Dump Issue
This is one of the important analyses we have received in PC2 V10 software. In the analysis, we have sent to the large number of request to the PC2 V10 software. After some time, it stops giving the response. In the backend, we have received the heap space error.
The basic causes of memory dump is, there are total three different servers in PC2 V10 software. It is nothing but Dispatcher, Checker and AppServer. Everything is on the local system. PC2 V10 is using single memory for all server. This is one of the main problem locations. There are resources acquired by each server inside the memory. After submitting a large number of request, it is not able to reutilize the memory. Usage of only one system places a large amount of load on the single machine. Hence, this starts giving the memory dump. In the future, it is mandatory to test this scenario in a distributed environment. The distributed environment is an environment in which AppServer, Dispatcher, and Checker are on different machines. This does not only help to improve the memory utilization but also helps to improve the overall performance of the PC2 V10 software.
5.3 Graphical Analysis of Performance
The overall performance includes sum of the Part A, B, C, D and E. The overall performance of the software degrades as the number of requests increases. After plotting the graph of Run ID with overall performance of the software, it shows decreasing performance as the number of requests increases. Appserver, Dispatcher and Checker uses the JMS queue internally. The main reason behind this is the JMS Queue. The basic nature
33 of the Queue is First in First out. JMS queue is processing the request one by one. Waiting time for each request object gets increase as the incoming request objects are more than outgoing request object. As increasing the number of request will not increase the performance of JMS Queue. It is always executed with its normal execution speed. The result of this is performance delay. This is what happening inside PC2 V10.
Figure 20: Performance Graph of PC2 V10
From the Figure 20 graph, X-axis represents the RunID which is getting added in the queue over the period of time. Y-Axis represent the total response time which is sum of Part A,
Part B, Part C, Part D and Part E. Its response time is increasing in nature which shows that performance degrades as the number of request (Run Id) is increasing.
5.4 Do Not Use XAMPP as a Database
This is one of the important results from our analysis. XAMPP is a software who is
34 providing the common interface for the Apache server, MySql Server, FileZilla, Mercury, and Tomcat[15]. XAMPP is giving you the facility to change the configuration as per the requirement.
Figure 21: XAMPP
XAMPP is providing you with a nice user interface which is easy to handle and easy to manage to see the logs. From the performance point of view, XAMPP tool is not good software if you are using MySQL database service of XAMPP. It runs the Apache service in the background if XAMPP is starting the service for MySQL. Performance testing is done by increasing the request load on the software. XAMPP-MySQL service stops responding if you are doing instantaneous transactions on XAMPP-MySQL database.
35
From the tester point of view, this will create confusion. It is difficult to identify that whether it is the problem of our software or the problem XMAPP software. We did some analysis to understand that this is a problem of XAMPP. After some analysis, we have figured out that there is nothing wrong with PC2 V10 software. We have separately installed MYSQL exe and executed all the SQL commands through SQL command prompt. After this setup, the PC2 V10 is not simulating the problem of memory dump and response failure, so it is concluded that it is not good to use XAMPP as software when we are dealing with performance testing.
36
6. FUTURE SCOPE
There are a number of things which could be done in the future to further test performance issues in the PC2 V10 software. These are outlined below,
6.1 Distributed Environment
It is necessary to test this software in a distributed environment. There are various cases which we need to test in a distributed environment. There are three main modules which include Dispatcher, Checker, and AppServer. Each module has its own Jetty server. The database is one more separate layer. If we put this on the separate machine then we will do the testing for each part. Each one has to be on the separate machine. We will able to test each module separately by putting the load separately on each machine. This will able to give more accurate results.
6.2 Testing on Different Platforms
We have done the testing on Windows-10 operating system. PC2 V10 is necessary to test the software on Linux and Mac operating system. This test is gives the quality assurance.
PC2 V10 is a web base solution which runs on the browser. There are many types of browser currently in the market. Hence, PC2 V10 should get test on various browser. This test ensure us that PC2 V10 website layout not gets disturb on various browser.
6.3 Security Test
This is also one the important aspect of testing. There are various types of attacks happens
37 on website, it includes SQL Injection, DDOS Attack, Cross side scripting attacks etc. These kind of hacks should get tested before website goes to live. This helps to improve the security level of the website.
6.4 Alternative Webservers into V10
Although we have found that PC2 V10 is unable to support the desired load but still we need to investigate and propose mechanisms for substituting alternative webservers into
V10. There are other webserver in the market like nGinX[14]. But it is a good replace jetty with these server and test the PC2 V10 performance.
38
REFERENCES
[1] CSUS Programming Contest Control (PC^2) . “PC2 Technical Documentation”.
[Online]. Available: pc2.ecs.csus.edu/.[Accessed on: Oct 30, 2018].
[2] Why performance testing demands JMeter! (n.d.). [Online]. Available:
https://www.sayonetech.com/blog/why-jmeter-best-companion-performance-test-
engineer/#.WpuUU6inHIU. [Accessed 7 Oct. 2018].
[3] Performance Testing. “Introduction to Performance Testing”. [Online]. Available:
http://guttitech.com/phpfusion/articles.php?article_id=576.[Accessed 7 Oct.
2018].
[4] Postman. “Postman”. [Online]. Available:
www.getpostman.com/. [Accessed 7 Oct. 2018].
[5] JDBC Transaction Example – Mkyong.com. “How to Install Maven on Windows”.
[Online]. Available:
www.mkyong.com/maven/how-to-install-maven-in-windows/.[Accessed 7 Oct.
2018].
[6] CodeChef .“Programming Competition, Programming Contest, Online Computer
Programming.” CodeChef Is a Non-Commercial Competitive Programming
Community, [Online]. Available: www.codechef.com/.[Accessed 7 Oct. 2018].
[7] Coding Competitions. “Facebook Hacker Cup.”
[Online]. Available: www.facebook.com/hackercup/. [Accessed 7 Oct. 2018].
39
[8] Google. “Google”. [Online]. Available: code.google.com/codejam/.
[Accessed 7 Oct. 2018].
[9] HackerRank. “Solve Challenges.”
[Online]. Available: www.hackerrank.com/contests/w35/challenges.
[Accessed 7 Oct. 2018].
[10] ICFP Programming Contest 2018. “ICFP Programming Contest 2018”.
[Online]. Available: icfpcontest2018.github.io/.[Accessed 7 Oct. 2018].
[11] Wikipedia. “ACM International Collegiate Programming Contest”. Wikimedia
Foundation, 28 Sept. 2018. [Online]. Available:
en.wikipedia.org/wiki/ACM_International_Collegiate_Programming_Contest.
[Accessed 7 Oct. 2018].
[12] Apache JMeter. “Apache JMeter™”. [Online]. Available:
jmeter.apache.org/.[Accessed on: Oct 30, 2018].
[13] Pooja, et al. “Top 15 Performance Testing Tools and Load Testing Tools of
2018.” Software Testing Help, 8 June 2018. [Online]. Available:
www.softwaretestinghelp.com/performance-testing-tools-load-testing-tools/.
[Accessed 13 Oct. 2018].
[14] NGINX. “High Performance Load Balancer, Web Server, & Reverse Proxy”.
[Online]. Available: www.nginx.com/.[Accessed 25 Oct. 2018].
[15] XAMPP. “Apache Friends.”
[Online]. Available: www.apachefriends.org/index.html. [Accessed 4 Nov. 2018].