CUBRID 10.1 QA Completion Report
Total Page:16
File Type:pdf, Size:1020Kb
CUBRID 10.1 QA Completion Report (CUBRID QA Team on July 14, 2017) CUBRID 10.1 QA Completion Report Table of Contents 1. Summary ____________________________________________________________________ 4 2. Functionality Tests ___________________________________________________________ 6 3. Performance Benchmarks ____________________________________________________ 7 YCSB ___________________________________________________________________________________________ 7 SysBench _______________________________________________________________________________________ 8 TPC-C __________________________________________________________________________________________ 9 TPC-W _________________________________________________________________________________________ 9 HA Replication Performance ___________________________________________________________________ 10 4. Stability Test _________________________________________________________________ 10 5. Compatibility Tests ___________________________________________________________ 11 JDBC Driver Compatibility Test ________________________________________________________________ 11 CCI Driver Compatibility Test __________________________________________________________________ 11 Database Volume Compatibility Test __________________________________________________________ 11 Appendix _________________________________________________________________________ 12 Test Categories ________________________________________________________________________________ 12 Package list of the release candidate build ____________________________________________________ 13 Test Environments _____________________________________________________________________________ 14 Functionality Test Scenarios ___________________________________________________________________ 15 YCSB Benchmark _______________________________________________________________________________ 22 SysBench Benchmark __________________________________________________________________________ 25 TPC-C Benchmark ______________________________________________________________________________ 27 TPC-W Benchmark _____________________________________________________________________________ 29 HA Replication Performance ___________________________________________________________________ 32 DOTS Test _____________________________________________________________________________________ 34 Scenario-based Code Coverage Results _______________________________________________________ 36 2 CUBRID 10.1 QA Completion Report 3 CUBRID 10.1 QA Completion Report 1. Summary The test objective is to ensure that the final release candidate build of CUBRID 10.1 meets the business and user requirements. The tests cover functionality, performance, stability, compatibility. The tested release candidate build is 10.1.0.7663-1ca0ab8. You can find the build packages from Package list of the release candidate build. After performing the tests, the conclusion is that CUBRID 10.1 satisfies all the release criteria. Functionality Test 8 functionality tests, including SQL, UTILITY/SHELL, HA, REPLICATION, DRIVERS, RANDOM QUERY, ISOLATION LEVEL and INSTALLATION tests have all been regressed and passed. Performance Benchmarks YCSB CUBRID 10.1 outperforms CUBRID 10.0, and provides up to a 5% to 26% improvement in throughput operation. In addition, CUBRID 10.1 has a significant advantage compare with MySQL on Workload A and Update operation, which gained 70% and 62% increase in OPS respectively. SysBench CUBRID 10.1 significantly improves both TPS and latency of OLTP workload of SysBench benchmark with default isolation level. The number of processed transactions is 1.5 times than CUBRID 10.0, and the average operation latency drops by 37% compared to CUBRID 10.0. Moreover, CUBRID 10.1 also gets 11% increase in TPS compared to MySQL 5.7. TPC-C CUBRID 10.1 shows absolute advantages in tpmC and the number of warehouse compared with CUBRID 10.0, which provides up to 79% improvement in tpmC and 1.8x the maximum possible warehouse benefit. Moreover, the elapse time of data/index loading on CUBRID 10.1 is also greatly reduced to CUBRID 10.0 of 63%, and furthermore, CUBRID 10.1 also get 36% increase in tpmC compare to MySQL 5.7. TPC-W 4 CUBRID 10.1 QA Completion Report We have measured the maximum number of EB(Emulated Browsers) of 3 TPC-W benchmark profiles(WIPS, WIPSb and WIPSo). Results show WIPS and WIPSb keep same great performance as CUBRID 10.0. Moreover, WIPSo of CUBRID 10.1 gets 27% improvement in performance compared to CUBRID 10.0, it satisfies the performance requirement: (EBs / 14) < WIPS < (EBs / 7). The results are close to the maximum possible performance. HA Replication Performance According to the results of HA replication performance test, the delay time between master and slave on CUBRID 10.1 is only 8‰ of CUBRID 10.0, and is 2‰ of CUBRID 9.3 under the specific measurement. Stability Tests During executing DOTS for 24 hours, CPU, Memory and Disk resource usage is still stable. The number of SUCCESS/FAIL queries is normal and notable issues have not been found. Compatibility Tests We have confirmed that 10.1 JDBC and CCI driver are compatible with 2008 R4.1, 2008 R4.3, 2008 R4.4, 9.2, 9.3 and 10.0 servers; and 10.1 server is also compatible with 2008 R4.1, 2008 R4.3, 2008 R4.4, 9.2, 9.3 and 10.0 drivers. 5 CUBRID 10.1 QA Completion Report 2. Functionality Tests This test has been performed to verify all aspects of CUBRID functionalities, including SQL syntax, CUBRID utilities, HA utilities, replication, random queries, isolation level, JDBC/CCI Driver and installation. The following table shows the test categories and the pass rate on CUBRID 10.1 based on all the test cases. Please refer to the Functionality Test Scenarios for details. Table 1. Test results of functionality tests Test Category Platform Pass Rate CentOS 5.6 64-bit SQL 100% Windows Server 2008 R2 32-bit/64-bit CentOS 5.7 64-bit UTILITY/SHELL 100% Windows Server 2008 R2 32-bit/64-bit HA CentOS 6.6 64-bit 100% REPLICATION CentOS 5.6 64-bit 100% DRIVERS CentOS 5.6 64-bit 100% RANDOM QUERY CentOS 5.6/7.2 64-bit 100% ISOLATION LEVEL CentOS 6.6 64-bit 100% INSTALLATION Refer to INSTALLATION Category 100% 6 CUBRID 10.1 QA Completion Report 3. Performance Benchmarks YCSB We have experimented with four types of YCSB workloads as follows: Workload A: Update Heavy, 50/50 reads/writes Workload B: Read Heavy, 95/5 reads/writes Update: only UPDATE operation Mix: 30/10/30/30 reads/scans/inserts/updates The following table shows the results and YCSB Benchmark does details about the benchmark test. Table 2. Results of YCSB Benchmark Throughput (OPS) Workload Ratio 9.3 10.0 10.1 (vs 10.0) Workload A 37,686 48,771 61,583 126% Workload B 59,312 73,099 86,782 119% Update 20,930 33,070 37,112 112% Mix 26,351 33,112 34,724 105% Table 3. Results of YCSB Benchmark on CUBRID 10.1 and MySQL 5.7 Throughput (OPS) Workload MySQL CUBRID Ratio Workload A 36,279 61,583 170% Workload B 126,536 86,782 69% Update 22,961 37,112 162% Mix 35,045 34,724 99% 7 CUBRID 10.1 QA Completion Report SysBench You can find the results of OLTP workload of SysBench benchmark from the following tables and details from SysBench Benchmark. Table 4. Results of SysBench Benchmark Ratio Test Item 9.3 10.0 10.1 (vs 10.0) The number of read/write requests (per sec) 15,566 12,688 19,962 157% avg. response time (ms) (per request) 347 426 270 63% The number of transactions (accumulation) 3,113,233 2,537,968 3,992,818 157% The number of transactions (per sec) 865 705 1,109 157% Table 5. Results of SysBench Benchmark between CUBRID 10.1 and MySQL 5.7 Test Item MySQL CUBRID Ratio The number of read/write requests (per sec) 18,022 19,962 111% avg. execution time (ms) (per request) 300 270 90% The number of transactions (accumulation) 3,604,335 3,992,818 111% The number of transactions (per sec) 1001 1,109 111% 8 CUBRID 10.1 QA Completion Report TPC-C The following tables shows the results of TPC-C Benchmark. Please refer to TPC-C Benchmark for details of the test. Table 6. Results of TPC-C Benchmark Ratio Test Item 9.3 10.0 10.1 (vs 10.0) 14,858 14,789 26,420 tpmC (max # of warehouse) 179% (1,000) (1,000) (1,800) Elapse minutes for data loading with 564.12 508.06 186.57 37% 1000 warehouses Elapse minutes for index loading with 302.02 437.63 150.13 34% 1000 warehouses Table 7. Results of TPC-C Benchmark between CUBRID 10.1 and MySQL 5.7 Test Item MySQL CUBRID Ratio 19,456 26,420 tpmC (max # of warehouse) 136% (1,400) (1,800) TPC-W Please find http://www.tpc.org/tpcw/default.asp for TPC-W benchmark and the following table for the results. You can also find the details of the test from TPC-W Benchmark. Table 8. Results of TPC-W Benchmark Throughput of max # of EBs Profile 10.0 10.1 Ratio 1,836 1,837 WIPS for Shopping mix 100% (13,000 EB) (13,000 EB) 1,691 1,645 WIPSb for Browsing mix 97% (12,000 EB) (12,000 EB) 4,996 6,323 WIPSo for Ordering mix 127% (36,000 EB) (45,000 EB) 9 CUBRID 10.1 QA Completion Report HA Replication Performance HA replication performance is introduced as a metric to evaluate the performance of replication between HA master and slave. This test is executed by using the LOAD and MIX scenarios of YCSB with 130 threads, and regarding the result please refer to the following table. You can also find the details of the test scenario from HA Replication Performance. Table 9. Results of HA Replication Performance Ratio Test Item 9.3 10.0 10.1 (vs 10.0) Delay Time (MS) 749,039 214,001 1,772 8‰ 4. Stability Test DOTS, a sub-project of an open source