CUBRID 10.1 QA Completion Report

(CUBRID QA Team on July 14, 2017)

Table of Contents

1. Summary ______4

2. Functionality Tests ______6

3. Performance Benchmarks ______7

YCSB ______7 SysBench ______8 TPC-C ______9 TPC-W ______9 HA Replication Performance ______10

4. Stability Test ______10

5. Compatibility Tests ______11

JDBC Driver Compatibility Test ______11 CCI Driver Compatibility Test ______11 Database Volume Compatibility Test ______11

Appendix ______12

Test Categories ______12 Package list of the release candidate build ______13 Test Environments ______14 Functionality Test Scenarios ______15 YCSB Benchmark ______22 SysBench Benchmark ______25 TPC-C Benchmark ______27 TPC-W Benchmark ______29 HA Replication Performance ______32 DOTS Test ______34 Scenario-based Code Coverage Results ______36

CUBRID 10.1 QA Completion Report

1. Summary

The test objective is to ensure that the final release candidate build of CUBRID 10.1 meets the business and user requirements. The tests cover functionality, performance, stability, compatibility. The tested release candidate build is 10.1.0.7663-1ca0ab8. You can find the build packages from Package list of the release candidate build.

After performing the tests, the conclusion is that CUBRID 10.1 satisfies all the release criteria.

Functionality Test 8 functionality tests, including SQL, UTILITY/SHELL, HA, REPLICATION, DRIVERS, RANDOM QUERY,

ISOLATION LEVEL and INSTALLATION tests have all been regressed and passed.

Performance Benchmarks

 YCSB CUBRID 10.1 outperforms CUBRID 10.0, and provides up to a 5% to 26% improvement in throughput operation. In addition, CUBRID 10.1 has a significant advantage compare with MySQL on Workload A and Update operation, which gained 70% and 62% increase in OPS respectively.

 SysBench CUBRID 10.1 significantly improves both TPS and latency of OLTP workload of SysBench benchmark with default isolation level. The number of processed transactions is 1.5 times than CUBRID 10.0, and the average operation latency drops by 37% compared to CUBRID 10.0. Moreover, CUBRID 10.1 also gets 11% increase in TPS compared to MySQL 5.7.

 TPC-C CUBRID 10.1 shows absolute advantages in tpmC and the number of warehouse compared with CUBRID

10.0, which provides up to 79% improvement in tpmC and 1.8x the maximum possible warehouse benefit. Moreover, the elapse time of data/index loading on CUBRID 10.1 is also greatly reduced to

CUBRID 10.0 of 63%, and furthermore, CUBRID 10.1 also get 36% increase in tpmC compare to MySQL

5.7.

 TPC-W

CUBRID 10.1 QA Completion Report

We have measured the maximum number of EB(Emulated Browsers) of 3 TPC-W benchmark profiles(WIPS, WIPSb and WIPSo). Results show WIPS and WIPSb keep same great performance as

CUBRID 10.0. Moreover, WIPSo of CUBRID 10.1 gets 27% improvement in performance compared to

CUBRID 10.0, it satisfies the performance requirement: (EBs / 14) < WIPS < (EBs / 7). The results are close to the maximum possible performance.

 HA Replication Performance According to the results of HA replication performance test, the delay time between master and slave on CUBRID 10.1 is only 8‰ of CUBRID 10.0, and is 2‰ of CUBRID 9.3 under the specific measurement.

Stability Tests During executing DOTS for 24 hours, CPU, Memory and Disk resource usage is still stable. The number of SUCCESS/FAIL queries is normal and notable issues have not been found.

Compatibility Tests We have confirmed that 10.1 JDBC and CCI driver are compatible with 2008 R4.1, 2008 R4.3, 2008 R4.4,

9.2, 9.3 and 10.0 servers; and 10.1 server is also compatible with 2008 R4.1, 2008 R4.3, 2008 R4.4, 9.2,

9.3 and 10.0 drivers.

CUBRID 10.1 QA Completion Report

2. Functionality Tests

This test has been performed to verify all aspects of CUBRID functionalities, including SQL syntax,

CUBRID utilities, HA utilities, replication, random queries, isolation level, JDBC/CCI Driver and installation.

The following table shows the test categories and the pass rate on CUBRID 10.1 based on all the test cases. Please refer to the Functionality Test Scenarios for details.

Table 1. Test results of functionality tests

Test Category Platform Pass Rate CentOS 5.6 64-bit SQL 100% Windows Server 2008 R2 32-bit/64-bit CentOS 5.7 64-bit UTILITY/SHELL 100% Windows Server 2008 R2 32-bit/64-bit HA CentOS 6.6 64-bit 100% REPLICATION CentOS 5.6 64-bit 100% DRIVERS CentOS 5.6 64-bit 100% RANDOM QUERY CentOS 5.6/7.2 64-bit 100% ISOLATION LEVEL CentOS 6.6 64-bit 100% INSTALLATION Refer to INSTALLATION Category 100%

CUBRID 10.1 QA Completion Report

3. Performance Benchmarks

YCSB

We have experimented with four types of YCSB workloads as follows:

 Workload A: Update Heavy, 50/50 reads/writes

 Workload B: Read Heavy, 95/5 reads/writes

 Update: only UPDATE operation

 Mix: 30/10/30/30 reads/scans/inserts/updates

The following table shows the results and YCSB Benchmark does details about the benchmark test.

Table 2. Results of YCSB Benchmark

Throughput (OPS) Workload Ratio 9.3 10.0 10.1 (vs 10.0) Workload A 37,686 48,771 61,583 126% Workload B 59,312 73,099 86,782 119% Update 20,930 33,070 37,112 112% Mix 26,351 33,112 34,724 105%

Table 3. Results of YCSB Benchmark on CUBRID 10.1 and MySQL 5.7

Throughput (OPS) Workload MySQL CUBRID Ratio Workload A 36,279 61,583 170% Workload B 126,536 86,782 69% Update 22,961 37,112 162% Mix 35,045 34,724 99%

CUBRID 10.1 QA Completion Report

SysBench

You can find the results of OLTP workload of SysBench benchmark from the following tables and details from SysBench Benchmark.

Table 4. Results of SysBench Benchmark

Ratio Test Item 9.3 10.0 10.1 (vs 10.0)

The number of read/write requests (per sec) 15,566 12,688 19,962 157%

avg. response time (ms) (per request) 347 426 270 63%

The number of transactions (accumulation) 3,113,233 2,537,968 3,992,818 157%

The number of transactions (per sec) 865 705 1,109 157%

Table 5. Results of SysBench Benchmark between CUBRID 10.1 and MySQL 5.7

Test Item MySQL CUBRID Ratio

The number of read/write requests (per sec) 18,022 19,962 111%

avg. execution time (ms) (per request) 300 270 90%

The number of transactions (accumulation) 3,604,335 3,992,818 111%

The number of transactions (per sec) 1001 1,109 111%

CUBRID 10.1 QA Completion Report

TPC-C

The following tables shows the results of TPC-C Benchmark. Please refer to TPC-C Benchmark for details of the test.

Table 6. Results of TPC-C Benchmark

Ratio Test Item 9.3 10.0 10.1 (vs 10.0) 14,858 14,789 26,420 tpmC (max # of warehouse) 179% (1,000) (1,000) (1,800) Elapse minutes for data loading with 564.12 508.06 186.57 37% 1000 warehouses Elapse minutes for index loading with 302.02 437.63 150.13 34% 1000 warehouses

Table 7. Results of TPC-C Benchmark between CUBRID 10.1 and MySQL 5.7

Test Item MySQL CUBRID Ratio

19,456 26,420 tpmC (max # of warehouse) 136% (1,400) (1,800)

TPC-W

Please find http://www.tpc.org/tpcw/default.asp for TPC-W benchmark and the following table for the results. You can also find the details of the test from TPC-W Benchmark.

Table 8. Results of TPC-W Benchmark

Throughput of max # of EBs Profile 10.0 10.1 Ratio

1,836 1,837 WIPS for Shopping mix 100% (13,000 EB) (13,000 EB) 1,691 1,645 WIPSb for Browsing mix 97% (12,000 EB) (12,000 EB) 4,996 6,323 WIPSo for Ordering mix 127% (36,000 EB) (45,000 EB)

CUBRID 10.1 QA Completion Report

HA Replication Performance

HA replication performance is introduced as a metric to evaluate the performance of replication between

HA master and slave. This test is executed by using the LOAD and MIX scenarios of YCSB with 130 threads, and regarding the result please refer to the following table. You can also find the details of the test scenario from HA Replication Performance.

Table 9. Results of HA Replication Performance

Ratio Test Item 9.3 10.0 10.1 (vs 10.0) Delay Time (MS) 749,039 214,001 1,772 8‰

4. Stability Test

DOTS, a sub-project of an open source project "Linux Test Project", is an open source test tool for the

DBMS testing. CPU, Memory and Disk resource usages and the number of SUCCESS/FAIL queries of

DOTS look quite stable during execution for 24 hours and notable issues have not been found. DOTS

Test shows the details of the long time test.

CUBRID 10.1 QA Completion Report

5. Compatibility Tests

We have confirmed that JDBC and CCI driver of 10.1 are compatible with server of 2008 R4.1, 2008

R4.3, 2008 R4.4, 9.2, 9.3 and 10.0 and 10.1 server is also compatible with the driver of 2008 R4.1, 2008

R4.3, 2008 R4.4, 9.2, 9.3 and 10.0.

JDBC Driver Compatibility Test

Table 10. Test Results of CUBRID 10.1 JDBC Driver with legacy servers

Test 2008 R4.1 2008 R4.3 2008 R4.4 9.2 9.3 10.0 Scenario Component Server Server Server Server Server Server SQL, 10.1 Driver pass pass pass pass pass pass MEDIUM

Table 11. Test Results of CUBRID 10.1 server with legacy drivers

Test 2008 R4.1 2008 R4.3 2008 R4.4 9.2 9.3 10.0 Scenario Component Driver Driver Driver Driver Driver Driver

SQL, 10.1 Server pass pass pass pass pass pass MEDIUM

CCI Driver Compatibility Test

Table 12. Test Results of CUBRID 10.1 CCI Driver with legacy servers

Test 2008 R4.1 2008 R4.3 2008 R4.4 9.2 9.3 10.0 Scenario Component Server Server Server Server Server Server

10.1 Driver CCI SHELL pass pass pass pass pass pass

Table 13. Test Results of CUBRID 10.1 server with legacy drivers

Test 2008 R4.1 2008 R4.3 2008 R4.4 9.2 9.3 10.0 Scenario Component Driver Driver Driver Driver Driver Driver

10.1 Server CCI SHELL pass pass pass pass pass pass

Database Volume Compatibility Test

The database volume of CUBRID 10.1 is incompatible with that of CUBRID 10.0. 11

CUBRID 10.1 QA Completion Report

Appendix

Test Categories

The following figure briefly shows four test categories and groups.

CUBRID Builds

Functionality

SQL UTILITY/SHELL HA REPLICATION

DRIVER RANDOM ISOLATION LEVEL INSTALLATION QUERY

Performance Benchmarks

TPC-C TPC-W YCSB

SysBench HA Replication Performance

Compatibility

JDBC CCI VOLUME Compatibility Compatibility Compatibility

Miscellaneous

DOTS Memory Leak Code Coverage

SIGN OFF

Figure 1. CUBRID Test Categories and Test case Groups

CUBRID 10.1 QA Completion Report

Package list of the release candidate build

QA full regression tests have been performed with the following packages. Four build packages in bold font are builds for daily regression tests.

Package md5sum Size

CUBRID-10.1.0.7663-1ca0ab8-Linux.x86_64.sh 2b062f4ace2a8a5f7df39fa34cd2a804 37M

CUBRID-10.1.0.7663-1ca0ab8-Linux.x86_64.tar.gz f3de50a2d54c43f58b18129ac2063964 37M

CUBRID-10.1.0.7663-1ca0ab8-Linux.x86_64-debug.sh e664dfb315f0b903be3c296ec3e1e3bc 32M

CUBRID-10.1.0.7663-1ca0ab8-Linux.x86_64-debug.tar.gz 5f866c3a1b1268089d66a4a4143f2a78 32M

CUBRID-CCI-10.1.0.7663-1ca0ab8-Linux.x86_64.tar.gz 52cd781f5347af96cae8aaa6e8cc75b1 417K

CUBRID-CCI-Windows-x64-10.1.0.7663-1ca0ab8.zip 2ab1387942706e45e693d96fceaf2187 129K

CUBRID-CCI-Windows-x86-10.1.0.7663-1ca0ab8.zip 97d10ab2722b57577c7838bb26b0121a 108K

CUBRID-CCI-10.1.0.7663-1ca0ab8-Linux.x86_64-debug.tar.gz 5430c7638eb14e29593084aabc0be76e 312K

CUBRID-Windows-x64-10.1.0.7663-1ca0ab8.msi 27cfd6a41b890977016a9d1f403c54fe 31M

CUBRID-Windows-x64-10.1.0.7663-1ca0ab8.zip ca661267340c844b7dd795e84172bf7d 30M

CUBRID-Windows-x86-10.1.0.7663-1ca0ab8.msi bda9144c70abd4c4503f9d924e20c294 30M

CUBRID-Windows-x86-10.1.0.7663-1ca0ab8.zip e9406708b6d2f60394c1997583f48b9f 30M

JDBC-10.1.0.7663-cubrid-src.jar bb04eb15408338486b7909971e16f41d 215K

JDBC-10.1.0.7663-cubrid.jar 773de53df34fad3ebd0ff1584d396381 249K

cubrid-10.1.0.7663-1ca0ab8-Linux.x86_64.rpm 067dbe2a68eb1a66e3b5e4184c0f08c1 35M

cubrid-10.1.0.7663-1ca0ab8.tar.gz e14177f9e675ceb14595b1e2babc032e 44M

cubrid-10.1.0.7663-1ca0ab8.zip 7a552e8258f06fe53989845f420c45bf 47M

CUBRID 10.1 QA Completion Report

Test Environments

Servers for the CUBRID test and their purpose are listed in the table below.

Table 14. Hardware Test Environment

No. OS CPU MEMORY DISK Purpose

Intel(R) Xeon(R) CPU E5645 SQL/MEDIUM/ 1 CentOS 5.6 (64-bit) 32 GB 1.6 TB @ 2.40GHz (12 cores) Memory Leak

Intel(R) Xeon(R) CPU E5645 RQG/ 2 CentOS 5.6 (64-bit) 24 GB 1.1 TB @ 2.40GHz (12 cores) SHELL (long)

Intel(R) Xeon(R) CPU E5645 3 CentOS 5.6 (64-bit) 32 GB 1.1 TB CCI @ 2.40GHz (12 cores)

Intel(R) Xeon(R) CPU E5506 4 CentOS 5.6 (64-bit) 16 GB 692 GB JDBC @ 2.13GHz (8 cores)

Intel(R) Xeon(R) CPU L5520 @ SHELL 5 CentOS 5.7 (64-bit) 8 GB 39 GB 2.27GHz (4 cores) (64-bit Linux)

Intel(R) Xeon(R) CPU E5645 JDBC 6 CentOS 5.9 (64-bit) 24 GB 1.1 TB @ 2.40GHz (12 cores) Compatibility

Intel(R) Xeon(R) CPU E5645 7 CentOS 5.11 (64-bit) 32 GB 1.6 TB SHELL (heavy) @ 2.40GHz (12 cores)

CCI Intel(R) Xeon(R) CPU E5645 8 CentOS 6.3 (64-bit) 24 GB 1.7 TB Compatibility/ @ 2.40GHz (12 cores) HA_REPL

Intel(R) Xeon(R) CPU L5640 @ SysBench/ 9 CentOS 6.3 (64-bit) 48 GB 796 GB 2.27GHz (24 cores) TPC-W

Intel(R) Xeon(R) CPU E5-2630L v2 @ 10 CentOS 6.3 (64-bit) 16GB 2.2TB YCSB/TPC-C 2.40GHz (24 cores)

Intel(R) Xeon(R) CPU L5640 @ ISOLATION/ 11 CentOS 6.6 (64-bit) 4 GB 40 GB 2.27GHz (2 cores) HA SHELL

Intel(R) Xeon(R) CPU E5-2630L 0 @ 12 CentOS 7.2 (64-bit) 32GB 270G DOTS 2.00GHz (24 cores)

Windows Server Intel(R) Xeon(R) CPU E5-2630L 0 13 16 GB 560 GB SQL/MEDIUM 2008 (32-bit) @2.00GHz 2.00GHz (8 cores)

SHELL Windows Server Pentium(R) Dual-Core CPU E6700 @ 14 4 GB 465 GB (32-bit 2008 R2 (64-bit) 3.20GHz 3.20GHz (2 cores) Windows)

Windows Server Intel(R) Xeon(R) CPU E5-2630L 0 15 24 GB 280 GB SQL/MEDIUM 2008 R2 (64-bit) @2.00GHz 2.00GHz (12 cores)

SHELL Windows Server Intel(R) Xeon(R) CPU L5520 @ 16 8 GB 150 GB (64-bit 2008 R2 (64-bit) 2.27GHz 2.27GHz (4 cores) Windows)

CUBRID 10.1 QA Completion Report

Functionality Test Scenarios

 SQL Category - SQL This test has been performed to verify the basic and complex DBMS functionalities by using SQL statements. These SQL statements are executed in both JDBC-based and CCI-based applications.

Total: 15,068 Number of Case Name Description Scenario Files Performs functionality tests of objects supported by object 3,330 CUBRID, and has the largest number of scenarios. Performs functionality tests of user and authorization user_authorization 171 management. Performs tests for the object-oriented concept. CUBRID is object_oriented 197 an object-relational database management system (DBMS). Performs functionality tests of basic functions and operator_function 1,625 operators supported by CUBRID. Performs tests of the insert, update, delete, and select statements, which are the most commonly used SQL manipulation 591 statements in DML. Basic statements, subqueries and various join queries are tested. Performs functionality tests of DCL (Data Control misc 82 Language), including statistics update or other functionalities. javasp 71 Performs functionality tests of Java stored procedures. Performs basic functionality test scenarios of the bigint and 64-bit 170 datetime types Connect_by 84 Performs a test of the hierarchical query feature Performs a test of uncovered codes based on the code Code coverage 9 coverage results. Syntax Extension 275 Performs a test of the syntax extension. Performs a test of known issues, which comes from issue BTS issues 1,567 management system. MySQL compatibility 687 Performs a unit test of the syntax extension 2. FBO 51 Performs a test of the FBO feature. Index enhancement 77 Performs a unit test of the index enhancement. Performs a test of the syntax extension 2. Includes a test of syntax enhancements, system parameters, show SQL Extension 629 statements, date/time functions, string functions, aggregate functions, other functions. Performs a test of the index enhancement. Includes a test of limit optimizing, using index clause enhancement, Index enhancement 202 descending index scan, covering index, ordering index, optimizing group by clause, Index scan with like predicate, next key locking,etc.

SQL Extension 3 336 Performs a unit test of syntax extension 3, performance and Index Enhancement internationalization features. Includes multi-table

CUBRID 10.1 QA Completion Report

Internationalization UPDATE/DELETE, pseudo column, analytic functions, (CUBRID 9.0 Beta unit test) MERGE statements, ENUM type, filtered index, function based index, index skip scan, partition and collation. MySQL compatibility Performs a test of several functions, regular expression and 120 for NEWS service hint rewriting. Performs a test of syntax extension 3, performance and SQL Extension 3 internationalization features, Test of syntax extension 3 includes multi-table UPDATE/DELETE, pseudo column, Index Enhancement analytic functions, MERGE statements, ENUM type, and 2,001 Internationalization other functions, Test of performance includes filtered index, (CUBRID 9.0 Beta QA function based index, index skip scan and partition scenario) enhancement. Test of internationalization includes tests of 11 languages. Performs a test of syntax extension, internationalization features. Test for syntax extension includes TRUNC, SQL Extension 3 WIDTH_BUCKEY, ROUND, NTILE functions, LEAD analytic Internationalization 304 function, and direct access to partitions in INSERT/UPDATE (CUBRID 9.1 QA scenario) statements. Test for internationalization includes collation per table, SHOW COLLATION, COLLATE modifier applied to expressions, etc. Performs a test of alter table to add columns when table 844 feature enhancement 6 already contains data Performs a test of sql extension, internationalization features. SQL Extension Test for sql extension includes NULLS order syntax, and Internationalization 181 some functions: FIRST_VALUE, LAST_VALUE, NTH_VALUE, (CUBRID 9.2 QA scenario) CUME_DIST, PERCENT_RANK, MEDIAN. Test for internationalization includes new collations added, hash partition on columns with any collation, etc. Performs a test of sql extension, optimization and other features. SQL Extension Test for sql extension includes encryption/decryption functions, SELECT ... FOR UPDATE, DROP SERIAL IF Optimization 58 EXISTS, Collation inference for host variables etc. (CUBRID 9.3 QA scenario) Test for optimization includes Hash Aggregate Evaluation, Analytic Functions, Loose Index Scan, Statistics improvement, etc. Performs a test of sql extension which includes SQL Extension PERCENTILE, CREATE TABLE IF NOT EXISTS, COMMENT, 2,068 (CUBRID 10.0 QA scenario) INDEX Improvement and Optimization, Time Zone, BINARY Charset, etc.

SQL Extension Performs a test of sql extension which includes CTE 176 support, limit clause extension and default clause (CUBRID 10.1 QA scenario) extension, etc

CUBRID 10.1 QA Completion Report

 SQL Category – MEDIUM This test has been performed to verify the basic and complex DBMS functionalities by using SQL statements. These SQL statements were executed in JDBC-based applications.

Total: 970 Number of Case Name Description Scenario Files Performs regression test scenarios for bug fixes that have 01_fixed 113 been implemented since the initial version. Performs test scenarios for functionalities supported by 02_xtests 441 CUBRID, but not by other DBMSs. Performs test scenarios for sequential/index scan queries 03_full_mdb 20 with an index. Performs test scenarios that include testing queries for limit 04_full 12 values of CUBRID. Performs negative test scenarios for functionalities that are 05_err_x 252 supported by CUBRID, but not by other DBMSs. 06_fulltests 11 Performs test scenarios for search queries with OIDs. Includes a query that gives various conditions to a WHERE 07_mc_dep 54 clause in the SELECT query, and tests whether or not a correct result has been selected. Includes scenarios that test queries performing schema 08_mc_ind 67 change.

 UTILITY Category – SHELL This test has been performed to verify the basic DBMS functionalities using shell scripts. In particular, this test is also to verify CUBRID utilities that cannot be tested by SQL statements. Scenarios of shell scripts were executed to verify DBMS conformity.

Total: 2,846 Number of Case Name Description Scenario Files Includes a script that tests the database management utility 222 commands supported by CUBRID. Includes scenarios that change the configuration of Sqlx_init 9 CUBRID DBMS parameters, and checks whether they are working correctly. Includes scenarios that verify there is no regression by Itrack 56 checking the bug fixes in CUBRID, and stores scenarios that cannot be tested by SQL. Includes miscellaneous items, such as JDBC cache, query Miscellaneous 43 cache and async commit test Includes scenarios added to improve code coverage and addition 142 mainly tests the options of CUBRID utilities.

CUBRID 10.1 QA Completion Report

Includes scenarios that verify there is no regression by BTS issues 1518 checking the bug fixes in CUBRID, and stores scenarios that cannot be tested by SQL. Includes scenarios that verify next key lock and change the configuration of CUBRID DBMS related to index Index enhancement 5 enhancement, which has been added in CUBRID 2008 R4.0 Beta. 64bit scenario 4 Includes file size on Linux 64 bit Includes shell cases to improve coverage, all the cases are Improve coverage scenario 4 related to the system parameter test xa datasource 9 Includes scenarios to cover xa DataSource features MySQL service Includes scenarios to test CUBRID compatibility with 8 compatibility MySQL service Includes scenarios that verify syntax extension, which has MySQL compatibility 4 been added in CUBRID 2008 R3.1. CUBRID 9.0 Beta Includes scenarios that verify CUBRID 9.0 Beta functions 52 QA such as i18n, enum, etc. Unstable 7 Includes scenarios that are not very stable Includes scenarios that added by QA to verify CUBRID 9.0 CUBRID 9.0 Beta QA 77 Beta functions such as i18n, cursor holdability, etc. Includes scenarios that added by QA to verify prefix key, CUBRID 9.1 QA 57 enum, collation of CUBRID 9.1 i18n function. New feature and feature Includes error message enhancement, server statistic 39 enhancement update and query profiling features Performs a test for sql extension, internationalization SQL Extension features. Test for sql extension includes NULLS order Internationalization syntax, and some functions: FIRST_VALUE, LAST_VALUE, 14 (CUBRID 9.2 QA shells NTH_VALUE, CUME_DIST, PERCENT_RANK and MEDIAN. script scenario) Test of internationalization includes new collations added, hash partition on columns with any collation, etc. Performs a test for sql extension, optimization and SQL Extension diagnostics statements features. Test for sql extension Optimization includes SELECT … FOR UPDATE, schema locking, base64 44 function. Test of optimization includes hash aggregation, Diagnostics statements statistics, etc. In addition, provide SHOW statements for (CUBRID 9.3 QA scenario) diagnostics: show log header, show volume header, show heap header, show slotted page, show access, etc. Performs a test for sql extension, internationalization and diagnostics statements features. Test for sql extension SQL Extension includes CREATE OR REPLACE, COMMENT, kill statement, Internationalization CRC32 function, etc. Test of internationalization includes 64 Diagnostics statements time zone, make_tz utility, BINARY charset, etc. In addition, provide more SHOW statements for diagnostics: (CUBRID 10.0 QA scenario) show index header, show job queue, show threads, show tran tables, etc. SQL Extension CTE Query support Performs a test for sql extension such as CTE query Limit clause, Default clause 4 support, limit clause extension, default clause extension and string compression and string compression, etc. (CUBRID 10.1 QA scenario) Test cases under this folder require high performance test Heavy shell 32 machines in execution.

CUBRID 10.1 QA Completion Report

Test cases under this folder require long time to execute. Long time shell 49 They are executed weekly. All the manual test cases which can’t be automated or need Manual shell 383 long time to regress.

 HA Category – HA SHELL This test has been performed to verify HA features including both HA utilities and HA replication.

Total: 15,347 Number of Case Name Description Scenario Files Includes scenarios that verify there is no regression by Bug regression 279 checking the HA bug fixes in CUBRID Migrated existing SQL suite into HA environment. Execute Test Cases migrated from 15,068 them on master node, then check whether be replicated to SQL suite slave or not.

 DRIVER Category – CCI SHELL This test has been performed to verify if all the CCI APIs of CUBRID work well as described in the

CUBRID manual.

Total: 313 Number of Case Name Description Scenario Files Which contains CCI all APIs, each APIs are mentioned in Features test 313 manual are tested in shell scripts

 DRIVER Category - JDBC This test has been performed to verify if all the JDBC APIs of CUBRID work well as described in the

CUBRID manual. Test cases are written towards to unit test.

Total: 2,299 Number of Case Name Description Scenario Files Which include unit test for JDBC, JDBC spec 3.0 test, and Features test 2,299 other open source databases JDBC case migration.

 RANDOM QUERY Category – RQG This test has been performed to verify concurrent execution with plentiful SQL syntax. Additionally, it also checks the recovery of database and somewhat consistence of data.

CUBRID 10.1 QA Completion Report

Total: 98 Number of Case Name Description Scenario Files Performs a test for MVCC feature which includes plentiful MVCC 98 SQL syntax, with or without index, recovery and vacuum, etc.

 ISOLATION LEVEL Category – ISOLATION LEVEL This test has been performed to verify the lock behaviors with fewer threads. Test cases are designed with diverse isolations.

Total: 6,249 Number of Case Name Description Scenario Files Two or more clients with read committed isolation execute Read committed isolation 2,772 in parallel to check lock behaviors and data consistency. Two or more clients with repeatable read isolation execute Repeatable read isolation 1,366 in parallel to check lock behaviors and data consistency. A mix of repeatable read Two or more clients with mixed isolations of repeatable and read committed 2,111 read and read committed isolation execute in parallel to isolations. check lock behaviors and data consistency.

 INSTALLATION Category - Installation The test has been performed to verify whether all kinds of packages work well with various platforms, not only focus on the installation step. According to the test results, CUBRID 10.1 installation feature works well as expectation completely. Please refer to the table as below for Installation test scenarios:

 Install and uninstall package

 Start and stop service/server/broker and manager

 Create and delete database

 Execute a basic query in csql client

 Execute negative compatibility scenarios

CUBRID 10.1 QA Completion Report

Package Test OS

cubrid-10.1.0.7663-1ca0ab8-Linux.x86_64.rpm CentOS 5.11/6.9/7.3 (64-bit) CUBRID-10.1.0.7663-1ca0ab8-linux.x86_64.sh Debian 8.8/Ubuntu 17.04/Fedora 25 (64-bit) CUBRID-10.1.0.7663-1ca0ab8-linux.x86_64-debug.sh

CUBRID-10.1.0.7663-1ca0ab8-linux.x86_64.tar.gz CentOS 5.11/6.9/7.3 (64-bit) CUBRID-10.1.0.7663-1ca0ab8-linux.x86_64-debug.tar.gz

CUBRID-Windows-x86-10.1.0.7663-1ca0ab8.msi Windows server 2008, Windows 7/10 (32-bit)

Windows server 2008, Windows 7/10 (32-bit)

CUBRID-Windows-x86-10.1.0.7663-1ca0ab8.zip Windows server 2008 R2/2012 R2/2016,

Windows 10/7 (64-bit)

CUBRID-Windows-x64-10.1.0.7663-1ca0ab8.msi Windows server 2008 R2/2012 R2/2016,

CUBRID-Windows-x64-10.1.0.7663-1ca0ab8.zip Windows 7/10 (64-bit)

CUBRID 10.1 QA Completion Report

YCSB Benchmark

As a framework for benchmarking system, YCSB is popular and widely used in the world nowadays (see https://github.com/brianfrankcooper/YCSB/wiki).

 Test Machines

Items CUBRID Server Machine CUBRID Broker /YCSB Machine

Intel(R) Xeon(R) CPU Intel(R) Xeon(R) CPU CPU E5-2630L v2 @ 2.40GHz (24 cores) E5-2630L v2 @ 2.40GHz (24 cores)

Memory 16G 16G

Storage 2.2T (SSD) 2.2T (SSD)

OS CentOS 6.3 (64bit) CentOS 6.3 (64bit)

java version " 1.7.0_01" Others YCSB 0.4.0

 CUBRID Configuration  cubrid_broker.conf (Workload A/B):

MIN_NUM_APPL_SERVER =120 MAX_NUM_APPL_SERVER =120 SQL_LOG =OFF

 cubrid_broker.conf (Update/Mix):

MIN_NUM_APPL_SERVER =320 MAX_NUM_APPL_SERVER =320 SQL_LOG =OFF

 cubrid.conf (Workload A/B):

data_buffer_size=4G log_buffer_size=2G max_clients=200 vacuum_worker_count=50 data_buffer_neighbor_flush_pages=0

 cubrid.conf (Update/Mix):

data_buffer_size=4G log_buffer_size=2G max_clients=500 vacuum_worker_count=50

CUBRID 10.1 QA Completion Report

 CUBRID database Volume Configuration

Space description for database 'ycsb' with pagesize 4.0K. (log pagesize: 4.0K)

type purpose volume_count used_size free_size total_size PERMANENT PERMANENT DATA 4 14.8 M 15.5 G 15.5 G PERMANENT TEMPORARY DATA 1 0.2 M 2.0 G 2.0 G TEMPORARY TEMPORARY DATA 0 0.0 M 0.0 M 0.0 M - - 5 15.0 M 17.5 G 17.5 G

Detailed space description for files: data_type file_count used_size file_table_size reserved_size total_size INDEX 18 0.1 M 0.1 M 4.3 M 4.5 M HEAP 28 0.3 M 0.1 M 6.6 M 7.0 M SYSTEM 9 0.3 M 0.0 M 1.9 M 2.2 M TEMP 0 0.0 M 0.0 M 0.0 M 0.0 M - 55 0.8 M 0.2 M 12.8 M 13.8 M  MySQL Configuration  my.cnf

innodb_buffer_pool_size = 4G innodb_io_capacity = 800 innodb_adaptive_hash_index = OFF max_connections = 200 (Workload A / B) max_connections = 500 (Update / Mix)  YCSB Workload Configuration  Workload A

recordcount=10000000 operationcount=200000000 maxexecutiontime=1200 threads=100

 Workload B

recordcount=10000000 operationcount=200000000 maxexecutiontime=1200 threads=100

 Update

recordcount=10000000 operationcount=10000000 workload=com.yahoo.ycsb.workloads.CoreWorkload 23

CUBRID 10.1 QA Completion Report

readallfields=true readproportion=0 updateproportion=1 scanproportion=0 insertproportion=0 requestdistribution=zipfian fieldlength=10 threads=300

 Mix

recordcount=10000000 operationcount=10000000 workload=com.yahoo.ycsb.workloads.CoreWorkload readallfields=true readproportion=0.3 updateproportion=0.3 scanproportion=0.1 insertproportion=0.3 maxscanlength=200 requestdistribution=zipfian fieldlength=10 threads=300

CUBRID 10.1 QA Completion Report

SysBench Benchmark

SysBench is a modular, cross-platform and multi-threaded benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load (see https://github.com/akopytov/sysbench/tree/1.0). SysBench runs a specified number of threads which could execute requests in parallel. The actual workload produced by requests depends on the specified test mode. You can limit either the total number of requests or the total time for the benchmark, or both.

Available test modes are implemented by compiled-in modules and SysBench was designed to make adding new test modes an easy task. Each test mode may have additional (or workload-specific) options.

 Test Machine

Items Test Machine

Intel(R) Xeon(R) CPU CPU L5640@ 2.27GHz (24 cores)

Memory 48G

Storage 800G

OS CentOS 6.3 (64bit)

Others gcc (GCC) 4.4.6 20120305 (Red Hat 4.4.6-4)

 CUBRID Configuration  cubrid_broker.conf:

MIN_NUM_APPL_SERVER =320 MAX_NUM_APPL_SERVER =320 SQL_LOG=OFF  cubrid.conf:

data_buffer_size=4G log_buffer_size=2G max_clients=500 vacuum_worker_count=50 error_log_level="syntax" error_log_warning=n error_log_size=2147483647

CUBRID 10.1 QA Completion Report

 CUBRID database volume configuration

Space description for database 'sysbench' with pagesize 16.0K. (log pagesize: 16.0K)

type purpose volume_count used_size free_size total_size PERMANENT PERMANENT DATA 5 57.0 M 8.4 G 8.5 G PERMANENT TEMPORARY DATA 1 1.0 M 2.0 G 2.0 G TEMPORARY TEMPORARY DATA 0 0.0 M 0.0 M 0.0 M - - 6 58.0 M 10.4 G 10.5 G

Detailed space description for files: data_type file_count used_size file_table_size reserved_size total_size INDEX 17 0.3 M 0.3 M 16.5 M 17.0 M HEAP 27 0.6 M 0.4 M 26.0 M 27.0 M SYSTEM 8 0.4 M 0.1 M 7.5 M 8.0 M TEMP 0 0.0 M 0.0 M 0.0 M 0.0 M - 52 1.2 M 0.8 M 49.9 M 52.0 M  SysBench Configuration

num-threads=300 max-time=3600 oltp-table-size=10000000 oltp-tables-count=20 test=oltp.lua  MySQL Configuration  my.conf:

innodb_buffer_pool_size = 4G innodb_io_capacity = 800 innodb_adaptive_hash_index = OFF max_connections = 1000

CUBRID 10.1 QA Completion Report

TPC-C Benchmark

TPC Benchmark C, approved in July of 1992, is an on-line transaction processing (OLTP) benchmark.

TPC-C (see http://www.tpc.org/tpcc/) is more complex than the previous OLTP benchmarks such as

TPC-A because of its multiple transaction types, complicated database and overall execution structure.

TPC-C involves a mix of five concurrent transactions of different types and complexity either executed on-line or queued for deferred execution. The database is comprised of nine types of tables with a wide range of record and population sizes. TPC-C is measured in transactions per minute (tpmC).

 Test Machines

Items CUBRID Server Machine CUBRID Broker /BenchmarkSQL Machine

Intel(R) Xeon(R) CPU Intel(R) Xeon(R) CPU CPU E5-2630L v2 @ 2.40GHz (24 cores) E5-2630L v2 @ 2.40GHz (24 cores)

Memory 16G 16G

Storage 2.2T (SSD) 2.2T (SSD)

OS CentOS 6.3 (64bit) CentOS 6.3 (64bit)

BenchmarkSQL Others Java version: 1.6.0_22

 CUBRID Configuration  cubrid_broker.conf:

MIN_NUM_APPL_SERVER =1000 MAX_NUM_APPL_SERVER =1000 SQL_LOG =OFF  cubrid.conf:

data_buffer_size=10G log_buffer_size=2G max_clients=1010 checkpoint_interval=30min error_log_size=2000000000 data_buffer_neighbor_flush_pages=0 vacuum_worker_count=50  cubrid.conf (data/index loading)

data_buffer_size=10G log_buffer_size=2G

Note: SA mode is used when loading index for CUBRID 10.0.  CUBRID database volume configuration

CUBRID 10.1 QA Completion Report

Space description for database 'tpcdb5' with pagesize 4.0K. (log pagesize: 4.0K)

type purpose volume_count used_size free_size total_size PERMANENT PERMANENT DATA 14 16.8 M 260.5 G 260.5 G PERMANENT TEMPORARY DATA 1 0.2 M 10.0 G 10.0 G TEMPORARY TEMPORARY DATA 0 0.0 M 0.0 M 0.0 M - - 15 17.0 M 270.5 G 270.5 G

Detailed space description for files: data_type file_count used_size file_table_size reserved_size total_size INDEX 17 0.1 M 0.1 M 4.1 M 4.2 M HEAP 27 0.3 M 0.1 M 6.3 M 6.8 M SYSTEM 9 0.3 M 0.0 M 1.9 M 2.2 M TEMP 0 0.0 M 0.0 M 0.0 M 0.0 M - 53 0.7 M 0.2 M 12.3 M 13.2 M  TPC-C Benchmark C Configuration  Configurations

dbPoolSize=1000 warehouses=1800 runTxnsPerTerminal=0 measureInterval=1800 rampUp=1200 rampDown=300  MySQL Configuration  my.conf:

innodb_buffer_pool_size = 10G innodb_io_capacity = 800 innodb_adaptive_hash_index = OFF max_connections = 1010

CUBRID 10.1 QA Completion Report

TPC-W Benchmark

TPC Benchmark™ W (TPC-W, please see http://www.tpc.org/tpcw/) is a transactional web benchmark.

The workload is performed in a controlled internet commerce environment that simulates the activities of a business oriented transactional web server.

 Test Machines

Items CUBRID Server Machine CUBRID Broker Machine TPC-W/Tomcat Machine

Intel(R) Xeon(R) CPU Intel(R) Xeon(R) CPU Intel(R) Xeon(R) CPU CPU [email protected]*1(24 cores) [email protected]*1(24 cores) [email protected]*1(24 cores)

Memory 48G 48G 48G

Storage 800G 800G 800G

OS CentOS 6.3 (64bit) CentOS 6.3 (64bit) CentOS 6.6 (64bit)

Java version "1.8.0_121" Others Tomcat 6.0.28

 CUBRID Configuration  cubrid_broker.conf(WIPSb/WIPS):

MIN_NUM_APPL_SERVER =1000 MAX_NUM_APPL_SERVER =1000 SQL_LOG =OFF

 cubrid_broker.conf(WIPSo):

MIN_NUM_APPL_SERVER =1500 MAX_NUM_APPL_SERVER =1500 SQL_LOG =OFF

 cubrid.conf:

data_buffer_size=4G log_buffer_size=2G max_clients=2000 checkpoint_interval=30min error_log_size=2000000000 vacuum_worker_count=50

CUBRID 10.1 QA Completion Report

 CUBRID database volume configuration(WIPSb/WIPS)

Space description for database 'tpcwdb' with pagesize 16.0K. (log pagesize: 16.0K)

type purpose volume_count used_size free_size total_size PERMANENT PERMANENT DATA 10 62.0 M 90.4 G 90.5 G PERMANENT TEMPORARY DATA 1 1.0 M 10.0 G 10.0 G TEMPORARY TEMPORARY DATA 0 0.0 M 0.0 M 0.0 M - - 11 63.0 M 100.4 G 100.5 G

 CUBRID database volume configuration(WIPSo)

Space description for database 'tpcwdb' with pagesize 16.0K. (log pagesize: 16.0K)

type purpose volume_count used_size free_size total_size PERMANENT PERMANENT DATA 15 67.0 M 270.4 G 270.5 G PERMANENT TEMPORARY DATA 1 1.0 M 10.0 G 10.0 G TEMPORARY TEMPORARY DATA 0 0.0 M 0.0 M 0.0 M - - 16 68.0 M 280.4 G 280.5 G

 Tomcat Configuration  Server.xml

connectionTimeout="20000" maxtheads="5000" minSpareThreads="254" maxSpareThreads="3500"

CUBRID 10.1 QA Completion Report

acceptCount="41000" maxKeepAliveRequests="1"  content.xml(WIPSb/WIPS)

maxActive="1000" maxIdle="1000" maxWait="60000" poolPreparedStatements="true" maxOpenPreparedStatements="100" initialSize="1000"  content.xml(WIPSo)

maxActive="1500" maxIdle="1500" maxWait="60000" poolPreparedStatements="true" maxOpenPreparedStatements="100" initialSize="1500"

CUBRID 10.1 QA Completion Report

HA Replication Performance

HA replication performance use YCSB benchmark test as workload on master node to evaluate the performance of replication between master and slave node.

 Test Machines

CUBRID Broker/YCSB Items CUBRID Master Machine CUBRID Slave Machine Machine Intel(R) Xeon(R) CPU Intel(R) Xeon(R) CPU Intel(R) Xeon(R) CPU CPU [email protected] (24 cores) [email protected] (24 cores) [email protected] (24 cores)

Memory 48G 48G 48G

Storage 800G 800G 800G

OS CentOS 6.3 (64bit) CentOS 6.3 (64bit) CentOS 6.6 (64bit)

java version " 1.6.0_22" Others YCSB 0.1

 CUBRID Configuration  cubrid_broker.conf:

MIN_NUM_APPL_SERVER =150 MAX_NUM_APPL_SERVER =150 SQL_LOG =OFF

 cubrid.conf:

data_buffer_size=5G log_buffer_size=2G max_clients=500 vacuum_worker_count=50 ha_mode=on

 cubrid_ha.conf:

ha_copy_sync_mode=sync:sync

 CUBRID database Volume Configuration

Space description for database 'ycsb' with pagesize 16.0K. (log pagesize: 16.0K)

type purpose volume_count used_size free_size total_size PERMANENT PERMANENT DATA 4 56.0 M 15.4 G 15.5 G PERMANENT TEMPORARY DATA 1 1.0 M 2.0 G 2.0 G

CUBRID 10.1 QA Completion Report

TEMPORARY TEMPORARY DATA 0 0.0 M 0.0 M 0.0 M - - 5 57.0 M 17.4 G 17.5 G

recordcount=20000000 operationcount=10000000 readproportion=0 updateproportion=0 scanproportion=0 insertproportion=1 threads=130 fieldlength=10  Mix

recordcount=20000000 operationcount=10000000 readproportion=0 insertproportion=0.6 updateproportion=0.3 deleteproportion=0.1 fieldlength=10 maxscanlength=200 threads=130

CUBRID 10.1 QA Completion Report

DOTS Test

DOTS, a sub-project of an open project called "Linux Test Project", is an open test tool for testing the DBMS.

 Test Machine

Items CUBRID Server Machine

CPU Intel(R) Xeon(R) CPU E5-2630L 0 @ 2.00GHz (24 cores)

Memory 32G

Storage 270G

OS CentOS 7.2 (64bit)

Others java version "1.6.0_13"

 CUBRID Configuration  cubrid_broker.conf

MIN_NUM_APPL_SERVER=100 MAX_NUM_APPL_SERVER=100 SQL_LOG =OFF  cubrid.conf

data_buffer_size=1G max_clients=200

 CUBRID database volume configuration

Space description for database 'TESTDB' with pagesize 16.0K. (log pagesize: 16.0K)

type purpose volume_count used_size free_size total_size PERMANENT PERMANENT DATA 1 53.0 M 459.0 M 512.0 M PERMANENT TEMPORARY DATA 0 0.0 M 0.0 M 0.0 M TEMPORARY TEMPORARY DATA 0 0.0 M 0.0 M 0.0 M - - 1 53.0 M 459.0 M 512.0 M

CUBRID 10.1 QA Completion Report

 DOTS Configuration

DURATION=24:00 CONCURRENT_CONNECTIONS= 20 CPU_TARGET = 75 SUMMARY_INTERVAL = 5 MAX_ROWS= 900000000 CREATIONINTERVAL = 1 MAX_CONNECTION=80

CUBRID 10.1 QA Completion Report

Scenario-based Code Coverage Results