Using SQLancer to test ClickHouse and other database systems
Manuel Rigger Ilya Yatsishin
@RiggerManuel qoega Plan
▎ What is ClickHouse and why do we need good testing?
▎ How do we test ClickHouse and what problems do we have to solve?
▎ What is SQLancer and what are the ideas behind it?
▎ How to add support for yet another DBMS to SQLancer?
2 ▎ Open Source analytical DBMS for BigData with SQL interface.
• Blazingly fast • Scalable • Fault tolerant 2021 2013 Project 2016 Open ★ started Sourced 15K GitHub
https://github.com/ClickHouse/clickhouse 3 Why do we need good CI? In 2020:
361 4081 261 11 <15
Contributors Merged Pull New Features Releases Core Team Requests
4 Pull Requests exponential growth
5 All GitHub data available in ClickHouse
https://gh.clickhouse.tech/explorer/ https://gh-api.clickhouse.tech/play
6 How is ClickHouse tested?
7
ClickHouse Testing
• Style • Flaky check for new or changed tests • Unit • All tests are run in with sanitizers • Functional (address, memory, thread, undefined • Integrational behavior) • Performance • Thread fuzzing (switch running threads • Stress randomly) • Static analysis (clang-tidy, PVSStudio) • Coverage • Compatibility with OS versions • Fuzzing
9
100K tests is not enough Fuzzing
▎ libFuzzer to test generic data inputs – formats, schema etc.
▎ Thread Fuzzer – randomly switch threads to trigger races and dead locks › https://presentations.clickhouse.tech/cpp_siberia_2021/
▎ AST Fuzzer – mutate queries on AST level. › Use SQL queries from all tests as input. Mix them. › High level mutations. Change query settings › https://clickhouse.tech/blog/en/2021/fuzzing-clickhouse/
12 Test development steps
Common tests Instrumentation Fuzzing The next step
Unit, Find bugs earlier. Improve ??? Functional, Sanitizers: coverage. Integration, • Address Stress, • Memory Performance • Undefined Behavior etc. • Thread
13 In search for correctness
▎ Test cases for correctness require developer or QA specialist to write it manually
▎ Additional tests mostly check if DBMS crashes or stops working
▎ It is hard to find reference system to validate results with:
• SQL syntax differs between systems • No SQL conformance testing suite • All SQL standard extensions can't be tested this way
14 How we can check correctness?
▎ SQL has some invariants and basic assumptions
▎ DBMS can help us!
There should be some solutions that already exploit that
15 SQLancer
https://github.com/sqlancer
SQLancer is an effective and widely-used automatic testing tool to find logic bugs in DBMSs
16 Facts
• Written in Java
17 Facts
• Written in Java
$ git clone https://github.com/sqlancer/sqlancer $ cd sqlancer $ mvn package -DskipTests $ cd target $ java -jar sqlancer-*.jar sqlite3
18 Facts
• Written in Java
$ git clone https://github.com/sqlancer/sqlancer $ cd sqlancer $ mvn package -DskipTests $ cd target $ java -jar sqlancer-*.jar sqlite3
You can quickly try out SQLancer on the embedded DBMSs SQLite, H2, and DuckDB without setting up a connection
19 Facts
• Written in Java • Permissive license (MIT License)
20 Facts
• Written in Java • Permissive license (MIT License) • 43,000 LOC
21 Facts
• Written in Java • Permissive license (MIT License) • 43,000 LOC
22 Facts
• Written in Java • Permissive license (MIT License) • 43,000 LOC • 9 contributors
23 Contributors Nazli Ugur Koyluoglu
https://www.citusdata.com/blog/2020/09/04/mining-for-logic-bugs-in-citus-with-sqlancer/
24 Contributors
Patrick Stäuble
25 Contributors Ilya Yatsishin
26 Facts
• Written in Java • Permissive license (MIT License) • 43,000 LOC • 9 contributors • >= 10 supported DBMSs
27 Supported DBMSs
PostgreSQL
28 SQLancer
https://github.com/sqlancer
SQLancer is an effective and widely-used automatic testing tool to find logic bugs in DBMSs
29 More than 450 Bugs
PostgreSQL
I used SQLancer to find over 450 unique, previously unknown bugs in widely-used DBMSs
30 More than 450 Bugs
# Bugs DBMS Logic Error Crash CockroachDB 16 46 5 DuckDB 29 13 31 H2 2 15 1 MariaDB 5 0 1 MySQL 21 9 1 PostgreSQL 1 11 5 SQLite 93 44 42 TiDB 30 27 4 Sum 197 165 90 https://github.com/sqlancer/bugs 31 SQLancer
https://github.com/sqlancer
SQLancer is an effective and widely-used automatic testing tool to find logic bugs in DBMSs
32 Adoption
33 Adoption
Yandex uses SQLancer to test every commit
34 Adoption
DuckDB runs SQLancer on every pull request
35 Adoption
“With the help of SQLancer, an automatic DBMS testing tool, we have been able to identify >100 potential problems in corner cases of the SQL processor.”
https://www.monetdb.org/blog/faster-robuster-and-feature-richer-monetdb-in-2020-and-beyond 36 Adoption
PingCAP implemented a tool go-sqlancer
https://github.com/chaos-mesh/go-sqlancer 37 Adoption
https://www.citusdata.com/blog/2020/09/04/mining-for-logic-bugs-in-citus-with-sqlancer/ 38 SQLancer
https://github.com/sqlancer
SQLancer is an effective and widely-used automatic testing tool to find logic bugs in DBMSs
39 Database Management Systems (DBMS)
Structured Query Language (SQL)
Interact with Database Management System
40 Database Management Systems (DBMS)
CREATE DATABASE db;
Database Management System
41 Database Management Systems (DBMS)
CREATE DATABASE db;
Database Management System
Database db
42 Database Management Systems (DBMS)
CREATE TABLE t0(c0 INTEGER);
Database Management System
Database db
43 Relational Model
44 Relational Model
Table
45 Relational Model
Column
46 Relational Model
Row
47 Database Management Systems (DBMS)
CREATE TABLE t0(c0 INTEGER);
Database Management System
Database db
48 Database Management Systems (DBMS)
CREATE TABLE t0(c0 INTEGER);
Database Management System t0: (c0: INT) Database db
49 Database Management Systems (DBMS)
INSERT INTO t0(c0) VALUES (0), (1), (2);
Database Management System t0: (c0: INT) Database db
50 Database Management Systems (DBMS)
INSERT INTO t0(c0) VALUES (0), (1), (2);
Database Management System t0: (c0: INT) Database db 0 φ 1 φ 2 ¬φ 51 Database Management Systems (DBMS)
SELECT * FROM t0 WHERE p;
Database Management System t0: (c0: INT) Database db 0 φ 1 φ 2 ¬φ 52 Database Management Systems (DBMS)
SELECT * FROM t0 WHERE p;
Database Management System t0: (c0: INT) Database db 0 p 1 p 2 ¬p 53 Database Management Systems (DBMS)
SELECT * FROM t0 WHERE p; Result set 0 p Database Management System 1 p t0: (c0: INT) Database db 0 p 1 p 2 ¬p 54 SQLancer
https://github.com/sqlancer
SQLancer is an effective and widely-used automatic testing tool to find logic bugs in DBMSs
55 Goal: Find Logic Bugs
Bugs that cause the DBMS to return an incorrect result set
56 Motivating Example
t0 t1 c0 c0
0.0 -0.0
SELECT * FROM t0, t1 WHERE t0.c0 = t1.c0; ?
57 Motivating Example
t0 t1 c0 c0
0.0 -0.0
SELECT * FROM t0, t1 WHERE t0.c0 = t1.c0; ?
It might seem disputable whether the predicate should evaluate to true
58 Motivating Example 0 0 0 0 0 … 0 0 0 0 t0 t1 -0 1 0 0 0 … 0 0 0 0 c0 c0
0.0 -0.0 Different binary representation
SELECT * FROM t0, t1 WHERE t0.c0 = t1.c0;
59 Motivating Example 0 0 0 0 0 … 0 0 0 0 t0 t1 false? -0 1 0 0 0 … 0 0 0 0 c0 c0
0.0 -0.0 Different binary representation
SELECT * FROM t0, t1 WHERE t0.c0 = t1.c0; t0.c0 t1.c0
60 Motivating Example
t0 t1 Evaluates to true for most true? c0 c0 programming languages
0.0 -0.0
SELECT * FROM t0, t1 t0.c0 t1.c0 WHERE t0.c0 = t1.c0; 0.0 -0.0
61 Motivating Example
t0 t1 c0 c0
0.0 -0.0
SELECT * FROM t0, t1 t0.c0 t1.c0 WHERE t0.c0 = t1.c0; 0.0 -0.0 ✓
62 Motivating Example
t0 t1 The latest version of MySQL that c0 c0 we tested failed to fetch the row
0.0 -0.0
SELECT * FROM t0, t1 WHERE t0.c0 = t1.c0; t0.c0 t1.c0
https://bugs.mysql.com/bug.php?id=99122 63 Motivating Example
t0 t1 c0 c0
0.0 -0.0
SELECT * FROM t0, t1 WHERE t0.c0 = t1.c0; t0.c0 t1.c0
https://bugs.mysql.com/bug.php?id=99122 64 Motivating Example
t0 t1 We could automatically find the bug without c0 c0 having an accurate understanding ourselves 0.0 -0.0
SELECT * FROM t0, t1 WHERE t0.c0 = t1.c0; t0.c0 t1.c0
https://bugs.mysql.com/bug.php?id=99122 65 SQLancer
https://github.com/sqlancer
SQLancer is an effective and widely-used automatic testing tool to find logic bugs in DBMSs
66 Automatic Testing Core Challenges
Generate a Generate a Validate the Database Query Query’s Result
67 Automatic Testing Core Challenges
CREATE TABLE t0(c0 DOUBLE); CREATE TABLE t1(c0 DOUBLE); INSERT …
Generate a Generate a Validate the Database Query Query’s Result
68 Automatic Testing Core Challenges
SELECT * FROM t0, t1 WHERE t0.c0 = t1.c0;
Generate a Generate a Validate the Database Query Query’s Result
69 Automatic Testing Core Challenges
t0.c0 t1.c0 t0.c0 t1.c0 0 -0 ✓
Generate a Generate a Validate the Database Query Query’s Result
70 Automatic Testing Core Challenges
SQLancer does not require any user interaction for any of the steps
Generate a Generate a Validate the Database Query Query’s Result
71 Automatic Testing Core Challenges
1. Effective test case
Generate a Generate a Validate the Database Query Query’s Result
72 Automatic Testing Core Challenges Manually implemented heuristic database and query generators 1. Effective test case
Generate a Generate a Validate the Database Query Query’s Result
73 SQLsmith
SQLancer’s random database and query generation works similar to existing tools
https://github.com/anse1/sqlsmith 74 Automatic Testing Core Challenges
1. Effective test case 2. Test oracle
Generate a Generate a Validate the Database Query Query’s Result
75 Automatic Testing Core Challenges
Test oracles are the “secret sauce” of SQLancer 1. Effective test case 2. Test oracle
Generate a Generate a Validate the Database Query Query’s Result
76 Test Oracle Incorrect result!
“a test oracle (or just oracle) is a mechanism for determining whether a test has passed or failed”
https://en.wikipedia.org/wiki/Test_oracle 77 What existing test oracles could we use to find bugs in DBMSs?
78 Differential Testing
System 1
Test Case System 2 Generator
System 3
79 Differential Testing
System 1 R1
Test Case System 2 R Generator 2
System 3 R3
80 Differential Testing
System 1 R1
Test Case System 2 R Generator 2
R1 = R2 = R3?
System 3 R3
81 Differential Testing: RAGS (Slutz, VLDB 1998)
82 Differential Testing: RAGS (Slutz, VLDB 1998)
Query Generator
83 Differential Testing: RAGS (Slutz, VLDB 1998)
RS1
Query RS Generator 2
RS3
84 Differential Testing: RAGS (Slutz, VLDB 1998)
RS1
Query RS Generator 2
RS1 = RS2 = RS3?
RS3
85 Differential Testing: RAGS (Slutz, VLDB 1998)
RS1
Query RS = Generator 2 ✓
RS1 = RS2 = RS3?
RS3
86 Differential Testing: RAGS (Slutz, VLDB 1998)
RS1
Query RS ≠ Generator 2 RS1 = RS2 = RS3?
RS3
87 Differential Testing: RAGS (Slutz, VLDB 1998)
DBMS- specific SQL
Common SQL Core
88 Differential Testing: RAGS (Slutz, VLDB 1998)
DBMS- specific SQL “We are unable to use Postgres as an oracle because CockroachDB has slightly different semantics and SQL support, and generating queries that execute identically on both is tricky […].” – Cockroach Labs Common SQL Core
89 What oracles were introduced in SQLancer?
90 Finding Logic Bugs in DBMSs
Ternary Logic SQLancer provides Partitioning three test oracles OOPSLA ‘20
Non-optimizing Pivoted Query Reference Engine Synthesis Construction OSDI ‘20 ESEC/FSE ‘20
91 Finding Logic Bugs in DBMSs
Ternary Logic Partitioning OOPSLA ‘20
Non-optimizing Pivoted Query Reference Engine Synthesis Construction OSDI ‘20 ESEC/FSE ‘20
92 Query Partitioning
Ternary Logic Partitioning (TLP) is based on a conceptual framework called Query Partitioning
93 Query Partitioning
Query Q Generator RS(Q)
94 Query Partitioning
original query
Query Q Generator RS(Q)
95 Query Partitioning
Q’1 RS(Q’1)
Q’2 Query Q RS(Q’2) RS(Q) Q’3 Generator RS(Q’3)
Q’n RS(Q’n)
Partition the result set
96 Query Partitioning Partitioning queries
Q’1 RS(Q’1)
Q’2 Query Q RS(Q’2) RS(Q) Q’3 Generator RS(Q’3)
Q’n RS(Q’n)
97 Query Partitioning Partitions
Q’1 RS(Q’1)
Q’2 Query Q RS(Q’2) RS(Q) Q’3 Generator RS(Q’3)
Q’n RS(Q’n)
98 Query Partitioning
Q’1 RS(Q’1)
Q’2 RS(Q’ ) Query Q 2 Q’ RS(Q) Q’3 RS(Q’) Generator RS(Q’3)
Q’n RS(Q’n)
99 Query Partitioning
Composition operator
Q’1 RS(Q’1)
Q’2 RS(Q’ ) Query Q 2 Q’ RS(Q) Q’3 RS(Q’) Generator RS(Q’3)
Q’n RS(Q’n)
100 Query Partitioning
Q’1 RS(Q’1)
Q’2 RS(Q’ ) Query Q 2 Q’ RS(Q) Q’3 RS(Q’) Generator RS(Q’3)
Q’n RS(Q’n)
Combine the results so that RS(Q)=RS(Q’)
101 Query Partitioning
Q’1 RS(Q’1)
Q’2 RS(Q’ ) Query Q 2 Q’ RS(Q) Q’3 RS(Q’) Generator RS(Q’3)
Q’n RS(Q’n)
= ✓ 102 Query Partitioning
Q’1 RS(Q’1)
Q’2 RS(Q’ ) Query Q 2 Q’ RS(Q) Q’3 RS(Q’) Generator RS(Q’3)
Q’n RS(Q’n) ≠
103 Query Partitioning
Q’1 RS(Q’1)
Q’2 RS(Q’ ) Query Q 2 Q’ RS(Q) Q’3 RS(Q’) Generator RS(Q’3)
Q’n RS(Q’n)
In contrast to differential testing, we need only a single DBMS
104 How to Realize This Idea?
How can we partition the result set?
105 Scenario: Coffee Kitchen
106 Tangerines vs. Clementines
107 Tangerines vs. Clementines
tangerines
108 Tangerines vs. Clementines
clementine
109 Tangerines vs. Clementines
110 Tangerines vs. Clementines
I can never tell different citrus fruits apart
111 Tangerines vs. Clementines
I can!
112 Tangerines vs. Clementines
Show me!
113 TangerinesHow I test Ilya’s vs. understanding Clementines without knowing the differences myself?
114 Tangerines vs. Clementines
Please bring me all clementines
115 Tangerines vs. Clementines
Please bring me all clementines
2 fruits
116 Tangerines vs. Clementines
Please bring me all clementines
117 Tangerines vs. Clementines
Please bring me all clementines
118 Tangerines vs. Clementines
Please bring me all fruits that are not clementines
119 Tangerines vs. Clementines
Please bring me all fruits that are not clementines
4 fruits
120 Tangerines vs. Clementines
2 fruits 4 fruits 6 fruits
121 Tangerines vs. Clementines
5 fruits 2 fruits 4 fruits 6 fruits
122 Tangerines vs. Clementines
You likely classified a fruit as both a tangerine and a clementine!
5 fruits 2 fruits 4 fruits 6 fruits
123 Insight
Insight: Every object in a (mathematical) universe is either a tangerine or not a tangerine
124 How can we apply this idea to find bugs in DBMSs?
125 Ternary Logic
Consider a predicate p and a given row r. Exactly one of the following must hold: • p • NOT p • p IS NULL
126 Ternary Logic
Consider a predicate p and a given row r. Exactly one of the following must hold: • p • NOT p • p IS NULL
127 Ternary Logic
Consider a predicate p and a given row r. Exactly one of the following must hold: • p
• NOT p p • p IS NULL NOT p
p IS NULL
128 Motivating Example
t0 t1 How did this insight allow c0 c0 us to detect this bug? 0 -0
SELECT * FROM t0, t1 WHERE t0.c0 = t1.c0; t0.c0 t1.c0
https://bugs.mysql.com/bug.php?id=99122 129 Example: MySQL
SELECT * FROM t0, t1;
https://bugs.mysql.com/bug.php?id=99122 130 Example: MySQL
SELECT * FROM t0, t1;
t0.c0 t1.c0
0 -0
https://bugs.mysql.com/bug.php?id=99122 131 Example: MySQL
SELECT * FROM t0, t1; SELECT * FROM t0, t1 WHERE t0.c0=t1.c0 UNION ALL SELECT * FROM t0, t1 WHERE NOT (t0.c0=t1.c0) UNION ALL SELECT * FROM t0, t1 WHERE (t0.c0=t1.c0) IS NULL;
t0.c0 t1.c0
0 -0
https://bugs.mysql.com/bug.php?id=99122 132 Example: MySQL p
SELECT * FROM t0, t1; SELECT * FROM t0, t1 WHERE t0.c0=t1.c0 UNION ALL SELECT * FROM t0, t1 WHERE NOT (t0.c0=t1.c0) UNION ALL SELECT * FROM t0, t1 WHERE (t0.c0=t1.c0) IS NULL;
t0.c0 t1.c0
0 -0
https://bugs.mysql.com/bug.php?id=99122 133 Example: MySQL p
SELECT * FROM t0, t1; SELECT * FROM t0, t1 WHERE t0.c0=t1.c0 UNION ALL SELECT * FROM t0, t1 WHERE NOT (t0.c0=t1.c0) UNION ALL SELECT * FROM t0, t1 WHERE (t0.c0=t1.c0) IS NULL;
t0.c0 t1.c0
0 -0 t0.c0 t1.c0
https://bugs.mysql.com/bug.php?id=99122 134 Example: MySQL p
SELECT * FROM t0, t1; SELECT * FROM t0, t1 WHERE t0.c0=t1.c0 UNION ALL SELECT * FROM t0, t1 WHERE NOT (t0.c0=t1.c0) UNION ALL SELECT * FROM t0, t1 WHERE (t0.c0=t1.c0) IS NULL;
t0.c0 t1.c0
0 -0 t0.c0 t1.c0 ≠ https://bugs.mysql.com/bug.php?id=99122 135 Insight
The DBMS is more likely to process the partitioning queries incorrectly due to their higher complexity
136 To what kind of features can we apply the Query Partitioning testing oracle?
137 Scope
• WHERE • GROUP BY • HAVING • DISTINCT queries • Aggregate functions
138 Testing WHERE Clauses
Q Q’ptern Composition Operator SELECT
WHERE ptern
139 Testing WHERE Clauses
Q Q’ptern Composition Operator SELECT
WHERE ptern
140 Testing WHERE Clauses
Q Q’ptern Composition Operator SELECT
WHERE ptern
141 Testing WHERE Clauses
Q Q’ptern Composition Operator SELECT
WHERE ptern
142 Testing WHERE Clauses
Q Q’ptern Composition Operator SELECT
WHERE ptern
UNION ALL keeps duplicate rows
143 Scope
• WHERE • GROUP BY • HAVING • DISTINCT queries • Aggregate functions
144 Testing DISTINCT Clauses
Q Q’ptern Composition operator SELECT DISTINCT SELECT UNION
WHERE ptern;
145 Testing DISTINCT Clauses
Q Q’ptern Composition operator SELECT DISTINCT SELECT UNION
WHERE ptern;
UNION removes duplicate rows
146 Scope
• WHERE • GROUP BY • HAVING • DISTINCT queries • Aggregate functions
147 Testing Self-decomposable Aggregate Functions
Q Q’ptern Composition operator SELECT MAX(
WHERE ptern;
148 Testing Self-decomposable Aggregate Functions
Q Q’ptern Composition operator SELECT MAX(
WHERE ptern;
A partition is an intermediate result, rather than a subset of the result set
149 Bug Example: CockroachDB
SET vectorize=experimental_on; CREATE TABLE t0(c0 INT); CREATE TABLE t1(c0 BOOL) INTERLEAVE IN PARENT t0(rowid); INSERT INTO t0(c0) VALUES (0); INSERT INTO t1(rowid, c0) VALUES(0, TRUE);
150 Bug Example: CockroachDB
SET vectorize=experimental_on; CREATE TABLE t0(c0 INT); CREATE TABLE t1(c0 BOOL) INTERLEAVE IN PARENT t0(rowid); INSERT INTO t0(c0) VALUES (0); INSERT INTO t1(rowid, c0) VALUES(0, TRUE);
SELECT MAX(t1.rowid) FROM t1;
NULL
151 Bug Example: CockroachDB
SET vectorize=experimental_on; CREATE TABLE t0(c0 INT); CREATE TABLE t1(c0 BOOL) INTERLEAVE IN PARENT t0(rowid); INSERT INTO t0(c0) VALUES (0); INSERT INTO t1(rowid, c0) VALUES(0, TRUE);
SELECT MAX(t1.rowid) SELECT MAX(aggr) FROM ( FROM t1; SELECT MAX(t1.rowid) as aggr FROM t1 WHERE '+' >= t1.c0 UNION ALL SELECT MAX(t1.rowid) as aggr FROM t1 WHERE NOT('+' >= t1.c0) UNION ALL SELECT MAX(t1.rowid) as aggr FROM t1 WHERE ('+' >= t1.c0) IS NULL );
NULL 0 ≠ 152 Bug Example: CockroachDB
SET vectorize=experimental_on; CREATE TABLE t0(c0 INT); CREATE TABLE t1(c0 BOOL) INTERLEAVE IN PARENT t0(rowid); INSERT INTO t0(c0) VALUES (0); INSERT INTO t1(rowid, c0) VALUES(0, TRUE);
SELECT MAX(t1.rowid) SELECT MAX(aggr) FROM ( FROM t1; SELECT MAX(t1.rowid) as aggr FROM t1 WHERE '+' >= t1.c0 UNION ALL SELECT MAX(t1.rowid) as aggr FROM t1 WHERE NOT('+' >= t1.c0) UNION ALL SELECT MAX(t1.rowid) as aggr FROM t1 WHERE ('+' >= t1.c0) IS NULL );
NULL 0 ≠ 153 Bug Example: CockroachDB
SET vectorize=experimental_on; CREATE TABLE t0(c0 INT); CREATE TABLE t1(c0 BOOL) INTERLEAVE IN PARENT t0(rowid); INSERT INTO t0(c0) VALUES (0); INSERT INTO t1(rowid, c0) VALUES(0, TRUE);
SELECT MAX(t1.rowid) SELECT MAX(aggr) FROM ( FROM t1; SELECT MAX(t1.rowid) as aggr FROM t1 WHERE '+' >= t1.c0 UNION ALL SELECT MAX(t1.rowid) as aggr FROM t1 WHERE NOT('+' >= t1.c0) UNION ALL SELECT MAX(t1.rowid) as aggr FROM t1 WHERE ('+' >= t1.c0) IS NULL );
NULL 0 ≠ 154 Testing Decomposable Aggregate Functions
Q Q’ptern Composition operator SELECT AVG(
155 Testing Decomposable Aggregate Functions
Q Q’ptern Composition operator SELECT AVG(
A single value to represent a partition is insufficient
156 What bugs did you find in ClickHouse?
157 Bug example: ClickHouse
158 Bug example: ClickHouse
Incorrect Query should return an error, but not incorrect answer.
Each column reference directly contained in the search condition shall be one of the following: a) An unambiguous reference to a column that is functionally dependent on the set consisting of every column referenced by a column reference contained in group by clause. ...
159 How general is the technique? Can I apply it to other domains?
160 Metamorphic Testing
SELECT * FROM t0, t1 t0.c0 t1.c0 WHERE t0.c0 = t1.c0; 0.0 -0.0
Derive
161 Metamorphic Testing
SELECT * FROM t0, t1 t0.c0 t1.c0 WHERE t0.c0 = t1.c0; 0.0 -0.0
Derive SELECT * FROM t0, t1 WHERE t0.c0=t1.c0 UNION ALL SELECT * FROM t0, t1 WHERE NOT (t0.c0=t1.c0) UNION ALL SELECT * FROM t0, t1 WHERE (t0.c0=t1.c0) IS NULL;
162 Metamorphic Testing
Execute Test Case Result
Derive SELECT * FROM t0, t1 WHERE t0.c0=t1.c0 UNION ALL SELECT * FROM t0, t1 WHERE NOT (t0.c0=t1.c0) UNION ALL SELECT * FROM t0, t1 WHERE (t0.c0=t1.c0) IS NULL;
163 Metamorphic Testing
Execute Test Case Result
Derive Derived Test Case
164 Metamorphic Testing
Execute Test Case Result
Derive Derived Test Case
This technique is known as metamorphic testing 165 Equivalence Modulo Inputs (EMI) for Testing Compilers
https://people.inf.ethz.ch/suz/emi/index.html 166 Equivalence Modulo Inputs (EMI) for Testing Compilers
EMI’s idea is to create programs that produce the same output for a given input https://people.inf.ethz.ch/suz/emi/index.html 167 Can we test not only DBMS?
Approach is quite generic.
We can try to use it in 1. Generic API with filtering or grouping 2. Regular expressions library 3. Event processing 4. Image recognition 5. Your ideas
168 What about other metamorphic test oracles for DBMSs?
169 Finding Logic Bugs in DBMSs
Ternary Logic Partitioning OOPSLA ‘20
Non-optimizing Pivoted Query Reference Engine Synthesis Construction OSDI ‘20 ESEC/FSE ‘20
170 Goal: Find Logic Bugs
Optimization bugs: logic bugs in the query optimizer
171 Motivating Example
t0 c0 CREATE TABLE t0(c0 UNIQUE); INSERT INTO t0 VALUES (-1) ; -1 SELECT * FROM t0 WHERE t0.c0 GLOB '-*';
-1 ✓
https://www.sqlite.org/src/tktview?name=0f0428096f 172 Motivating Example
t0 c0 CREATE TABLE t0(c0 UNIQUE); INSERT INTO t0 VALUES (-1) ; -1 SELECT * FROM t0 WHERE t0.c0 GLOB '-*';
Optimizer
{}
https://www.sqlite.org/src/tktview?name=0f0428096f 173 Motivating Example
t0 c0 CREATE TABLE t0(c0 UNIQUE); INSERT INTO t0 VALUES (-1) ; -1 SELECT * FROM t0 WHERE t0.c0 GLOB '-*';
The LIKE optimization malfunctionedOptimizer for non-text columns and a pattern prefix of “-” {}
https://www.sqlite.org/src/tktview?name=0f0428096f 174 Differential Testing
-O3 Optimizer
SELECT * FROM t0 WHERE t0.c0 GLOB '-*'; Optimizer
-O0
175 Differential Testing
-O3 Optimizer {} SELECT * FROM t0 WHERE t0.c0 GLOB '-*'; Optimizer
-O0
176 Differential Testing
-O3 Optimizer {} SELECT * FROM t0 WHERE t0.c0 GLOB '-*'; Optimizer -1 -O0
177 Differential Testing
-O3 Optimizer {} SELECT * FROM t0 ≠ WHERE t0.c0 GLOB '-*'; Optimizer -1 -O0
178 Differential Testing
https://www.sqlite.org/pragma.html 179 Differential Testing
PRAGMA case_sensitive_like = boolean;
https://www.sqlite.org/pragma.html 180 Differential Testing
PRAGMA case_sensitive_like = boolean;
DBMSs typically provide only very limited control over optimizations
https://www.sqlite.org/pragma.html 181 NoREC
Idea: Rewrite the query so that the DBMS cannot optimize it
182 Idea Optimized Query Optimizer {} Query Generator
183 Idea Optimized Query Optimizer {} Query Generator Translation Optimizer Step
Unoptimized Query
184 Idea Optimized Query Optimizer {}
Query ≠ Generator Translation Optimizer Step -1
Unoptimized Query
185 Idea Optimized Query Optimizer {}
Query ≠ Generator Translation Optimizer Step -1
Unoptimized Query We want to create a “non-optimizing reference engine”
186 Given Query
Consider the following format for the optimized query:
SELECT * FROM t0 WHERE p;
187 Given Query
Consider the following format for the optimized query:
SELECT * FROM t0 WHERE p; t0.c0 GLOB '-*'
188 Given Query
Consider the following format for the optimized query:
SELECT * FROM t0 WHERE p; It is unobvious how we could t0.c0 GLOB '-*' derive an unoptimized query
189 Insight
First Insight: The predicate p must always evaluate to the same value, irrespective of its context
190 Translation Step (Correct Case)
SELECT * FROM t0 WHERE p; -1
191 Translation Step (Correct Case)
SELECT * FROM t0 WHERE p; -1
192 Translation Step (Correct Case)
SELECT * FROM t0 WHERE p; -1
SELECT p FROM t0;
193 Translation Step (Correct Case)
SELECT * FROM t0 WHERE p; -1
SELECT p FROM t0; TRUE
194 Translation Step (Correct Case)
p evaluates SELECT * FROM t0 to TRUE for WHERE p; -1 one row
SELECT p FROM t0; TRUE p evaluates to TRUE for one row
195 Insights
Second Insight: DBMSs focus their optimizations on reducing the amount of data that is processed
196 Translation Step
SELECT * FROM t0 WHERE p;
197 Translation Step
Optimizer
SELECT * FROM t0 QUERY PLAN WHERE p; `--SEARCH TABLE t0 USING COVERING INDEX sqlite_autoindex_t0_1 (c0>? AND c0
198 Translation Step
Optimizer
SELECT * FROM t0 QUERY PLAN WHERE p; `--SEARCH TABLE t0 USING COVERING INDEX sqlite_autoindex_t0_1 (c0>? AND c0
Optimizer
SELECT p QUERY PLAN FROM t0; `--SCAN TABLE t0
199 Translation Step
Optimizer
SELECT * FROM t0 WHERE p; {}
Optimizer
SELECT p FROM t0; TRUE
200 Translation Step
Optimizer
SELECT * FROM t0 WHERE p; {}
Result should Optimizer contain one row SELECT p FROM t0; TRUE
201 Translation Step
Optimizer
SELECT * FROM t0 WHERE p; {}
Result should Optimizer contain one row SELECT p FROM t0; TRUE
202 Counting Implementation
Optimizer
SELECT COUNT(*) FROM … WHERE p
Optimizer
SELECT SUM(count) FROM ( SELECT p IS TRUE as count FROM
203 Counting Implementation
Optimizer
SELECT COUNT(*) FROM … 0 WHERE p
Optimizer ≠ SELECT SUM(count) FROM ( SELECT p IS TRUE 1 as count FROM
204 What about other, non-metamorphic test oracles?
205 Finding Logic Bugs in DBMSs
Ternary Logic Partitioning OOPSLA ‘20
Non-optimizing Pivoted Query Reference Engine Synthesis Construction OSDI ‘20 ESEC/FSE ‘20
206 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1
2
NULL
https://sqlite.org/src/tktview/80256748471a01 207 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 IS NOT is a “null-safe” 2 comparison operator NULL
https://sqlite.org/src/tktview/80256748471a01 208 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1
2
NULL
https://sqlite.org/src/tktview/80256748471a01 209 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 0 2
NULL TRUE
https://sqlite.org/src/tktview/80256748471a01 210 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 0 2
NULL TRUE
0
https://sqlite.org/src/tktview/80256748471a01 211 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 1 2
NULL FALSE
0
https://sqlite.org/src/tktview/80256748471a01 212 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 2 2
NULL TRUE
0
2
https://sqlite.org/src/tktview/80256748471a01 213 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 NULL 2
NULL TRUE
0
2
NULL https://sqlite.org/src/tktview/80256748471a01 214 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 NULL 2
NULL TRUE
0
2
NULL https://sqlite.org/src/tktview/80256748471a01 215 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 NULL 2
NULL TRUE 0 NULL was not contained 2 in the result set!
NULL https://sqlite.org/src/tktview/80256748471a01 216 PQS Idea t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (3), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1
2 Validate the result set based on NULL one randomly-selected row
217 PQS Idea t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (3), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1
2 Validate the result set based on NULL Pivot row one randomly-selected row
218 PQS Idea t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (3), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 NULL 2
NULL TRUE
Generate a query that is guaranteed to at least fetch the pivot row 219 PQS Idea t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (3), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 0 2 2 NULL If the pivot row is missing from the result set a bug has been detected
220 Approach
Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained
221 Approach
Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained
222 Approach
Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained
One random row from multiple tables and views
223 Approach
Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained
SELECT c0 FROM t0 WHERE
Generate predicates that evaluate to TRUE for the pivot row and use them in JOIN and WHERE clauses
224 Random Expression Generation
Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained
t0.c0 IS NOT 1; IS NOT
t0.c0 1
225 Random Expression Generation
Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained
t0.c0 IS NOT 1; IS NOT We implemented an expression evaluator for each node t0.c0 1
226 Random Expression Generation
Randomly Generate Validate that Select t0 generate query for the the pivot row pivot row database pivot row is contained c0
0
1 IS Evaluate the tree based NOT 2 on the pivot row NULL
t0.c0 1
227 Random Expression Generation
Randomly Generate Validate that Select t0 generate query for the the pivot row pivot row database pivot row is contained c0
0 Column references return the 1 IS values from the pivot row NOT 2 NULL
t0.c0 1
228 Random Expression Generation
Randomly Generate Validate that Select t0 generate query for the the pivot row pivot row database pivot row is contained c0
0 Column references return the 1 IS values from the pivot row NOT 2 NULL NULL
t0.c0 1
229 Random Expression Generation
Randomly Generate Validate that Select t0 generate query for the the pivot row pivot row database pivot row is contained c0
0
1 IS Constant nodes return their NOT assigned literal values 2 NULL NULL
t0.c0 1
230 Random Expression Generation
Randomly Generate Validate that Select t0 generate query for the the pivot row pivot row database pivot row is contained c0
0
1 IS Constant nodes return their NOT assigned literal values 2 NULL 1 NULL
t0.c0 1
231 Random Expression Generation
Randomly Generate Validate that Select t0 generate TRUE query for the the pivot row pivot row database pivot row is contained c0
0
1 IS Compound nodes NOT compute their result 2 NULL 1 based on their children NULL
t0.c0 1
232 Random Expression Generation
Randomly Generate Validate that Select t0 generate TRUE query for the the pivot row pivot row database pivot row is contained c0
0 TRUE 1 IS Compound nodes NOT compute their result 2 NULL 1 based on their children NULL
t0.c0 1
233 Query Synthesis
Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained
SELECT c0 c0 FROM t0 WHERE t0.c0 IS NOT 1;
234 Query Synthesis
Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained
SELECT c0 c0 FROM t0 WHERE t0.c0 IS NOT 1;
What if the expression does not evaluate to TRUE?
235 Random Expression Rectification
Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained
switch (result) { case TRUE: result = randexpr; case FALSE: result = NOT randexpr; case NULL: result = randexpr IS NULL; } 236 Random Expression Rectification
Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained
Alternatively, we could switch (result) { validate that the pivot row is case TRUE: expectedly not fetched result = randexpr; case FALSE: result = NOT randexpr; case NULL: result = randexpr IS NULL; } 237 Approach
Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained
SELECT (NULL) INTERSECT SELECT c0 FROM t0 WHERE NULL IS NOT 1;
Rely on the DBMS to check whether the row is contained
238 How do the techniques compare to each other?
239 Comparison
Property PQS NoREC TLP WHERE ✓ ✓ ✓ Additional SQL features ✓ Ground truth ✓ No domain knowledge required ✓ ✓ Implementation effort Moderate Very Low Low
240 Comparison
Property PQS NoREC TLP WHERE ✓ ✓ ✓ Additional SQL features ✓ Ground truth ✓ No domain knowledge required ✓ ✓ Implementation effort Moderate Very Low Low
TLP is applicable to testing a wider range of features
241 Comparison
Property PQS NoREC TLP WHERE ✓ ✓ ✓ Additional SQL features ✓ Ground truth ✓ No domain knowledge required ✓ ✓ Implementation effort Moderate Very Low Low
Both NoREC and TLP are metamorphic testing approaches 242 Proposed Testing Strategy
Quickly find the optimization bugs
NoREC TLP PQS
243 Proposed Testing Strategy
Test a wider range of features
NoREC TLP PQS
244 Proposed Testing Strategy
Comprehensively test the DBMS’ core functionality
NoREC TLP PQS
245 What implementation strategy did you use for ClickHouse?
246 SQLancer from developer point of view
▎ Natural and intuitive approach
▎ Fuzzing meets correctness check
▎ Pluggable(but you need to write code on Java)
▎ You can incorporate approach in other software
186 How to make integration with your DBMS
1. Import Java or ODBC driver if you have one 2. Teach SQLancer to make connection, create database and make a generic query 3. A bit more to prepare: Create table with schema and insert generator 4. Implement expression generator 5. Implement oracle 6. Expected error handling 7. RUN!
187 Add your driver
188 Make connection, database and query
189 Create table with Your DBMS can has specific SQL extensions. schema Implement subset of them
190 Expression generator
191 Implement Oracle
Almost the same for different DBMS
192 Test How to deal with Failed results? Is syntax No Mute correct error
Yes ▎ Query can be incorrect Mute errors
▎ You don't need to implement ideal query generator – bad queries also test your system
▎ You can mute known bugs and find more
194 Test How to deal with Failed results? Is syntax No Mute correct error
Yes ▎ Query can be incorrect Result No DBMS obtained crashed? ▎ DMBS can crash Yes No Yes
Probably Critical a bug Bug
Can be found by fuzzing Test How to deal with Failed results? Is syntax No Mute correct error
Yes ▎ Query can be incorrect Result No DBMS obtained crashed? ▎ DMBS can crash Yes No Yes
▎ If results differ – you found a SQLancer found Probably Critical correctness bug correctness bug! a bug Bug
Can be found by fuzzing Add Oracles to your Fuzzer
If you already have a query fuzzer or expression gererator you can implement oracles trivially. You can try implementing NoREC or TLP Where logic by yourself: 1) Take expr from your generator 2) Add some logic to compare results:
SELECT * FROM t0 SELECT * FROM t; WHERE expr; SELECT * FROM t WHERE expr UNION ALL SELECT expr SELECT * FROM t WHERE NOT expr FROM t0; UNION ALL SELECT * FROM t WHERE expr IS NULL; How to contribute?
• Add a new DBMS • Automatic reduction of test cases (#333) • New test oracles • Modularization, plugin system for DBMS support (#8) • Blog posts, tutorials, …
259 Summary
260 Q&A Session
• By the way, you can contribute to SQLancer and ClickHouse https://github.com/sqlancer/sqlancer
https://github.com/ClickHouse/ClickHouse
261