Using SQLancer to test ClickHouse and other systems

Manuel Rigger Ilya Yatsishin

@RiggerManuel qoega Plan

▎ What is ClickHouse and why do we need good testing?

▎ How do we test ClickHouse and what problems do we have to solve?

▎ What is SQLancer and what are the ideas behind it?

▎ How to add support for yet another DBMS to SQLancer?

2 ▎ Open Source analytical DBMS for BigData with SQL interface.

• Blazingly fast • Scalable • Fault tolerant 2021 2013 Project 2016 Open ★ started Sourced 15K GitHub

https://github.com/ClickHouse/clickhouse 3 Why do we need good CI? In 2020:

361 4081 261 11 <15

Contributors Merged Pull New Features Releases Core Team Requests

4 Pull Requests exponential growth

5 All GitHub data available in ClickHouse

https://gh.clickhouse.tech/explorer/ https://gh-api.clickhouse.tech/play

6 How is ClickHouse tested?

7

ClickHouse Testing

• Style • Flaky check for new or changed tests • Unit • All tests are run in with sanitizers • Functional (address, memory, thread, undefined • Integrational behavior) • Performance • Thread fuzzing (switch running threads • Stress randomly) • Static analysis (clang-tidy, PVSStudio) • Coverage • Compatibility with OS versions • Fuzzing

9

100K tests is not enough Fuzzing

▎ libFuzzer to test generic data inputs – formats, schema etc.

▎ Thread Fuzzer – randomly switch threads to trigger races and dead locks › https://presentations.clickhouse.tech/cpp_siberia_2021/

▎ AST Fuzzer – mutate queries on AST level. › Use SQL queries from all tests as input. Mix them. › High level mutations. Change query settings › https://clickhouse.tech/blog/en/2021/fuzzing-clickhouse/

12 Test development steps

Common tests Instrumentation Fuzzing The next step

Unit, Find bugs earlier. Improve ??? Functional, Sanitizers: coverage. Integration, • Address Stress, • Memory Performance • Undefined Behavior etc. • Thread

13 In search for correctness

▎ Test cases for correctness require developer or QA specialist to write it manually

▎ Additional tests mostly check if DBMS crashes or stops working

▎ It is hard to find reference system to validate results with:

• SQL syntax differs between systems • No SQL conformance testing suite • All SQL standard extensions can't be tested this way

14 How we can check correctness?

▎ SQL has some invariants and basic assumptions

▎ DBMS can help us!

There should be some solutions that already exploit that

15 SQLancer

https://github.com/sqlancer

SQLancer is an effective and widely-used automatic testing tool to find logic bugs in DBMSs

16 Facts

• Written in Java

17 Facts

• Written in Java

$ git clone https://github.com/sqlancer/sqlancer $ cd sqlancer $ mvn package -DskipTests $ cd target $ java -jar sqlancer-*.jar sqlite3

18 Facts

• Written in Java

$ git clone https://github.com/sqlancer/sqlancer $ cd sqlancer $ mvn package -DskipTests $ cd target $ java -jar sqlancer-*.jar sqlite3

You can quickly try out SQLancer on the embedded DBMSs SQLite, H2, and DuckDB without setting up a connection

19 Facts

• Written in Java • Permissive license (MIT License)

20 Facts

• Written in Java • Permissive license (MIT License) • 43,000 LOC

21 Facts

• Written in Java • Permissive license (MIT License) • 43,000 LOC

22 Facts

• Written in Java • Permissive license (MIT License) • 43,000 LOC • 9 contributors

23 Contributors Nazli Ugur Koyluoglu

https://www.citusdata.com/blog/2020/09/04/mining-for-logic-bugs-in-citus-with-sqlancer/

24 Contributors

Patrick Stäuble

25 Contributors Ilya Yatsishin

26 Facts

• Written in Java • Permissive license (MIT License) • 43,000 LOC • 9 contributors • >= 10 supported DBMSs

27 Supported DBMSs

PostgreSQL

28 SQLancer

https://github.com/sqlancer

SQLancer is an effective and widely-used automatic testing tool to find logic bugs in DBMSs

29 More than 450 Bugs

PostgreSQL

I used SQLancer to find over 450 unique, previously unknown bugs in widely-used DBMSs

30 More than 450 Bugs

# Bugs DBMS Logic Error Crash CockroachDB 16 46 5 DuckDB 29 13 31 H2 2 15 1 MariaDB 5 0 1 MySQL 21 9 1 PostgreSQL 1 11 5 SQLite 93 44 42 TiDB 30 27 4 Sum 197 165 90 https://github.com/sqlancer/bugs 31 SQLancer

https://github.com/sqlancer

SQLancer is an effective and widely-used automatic testing tool to find logic bugs in DBMSs

32 Adoption

33 Adoption

Yandex uses SQLancer to test every commit

34 Adoption

DuckDB runs SQLancer on every pull request

35 Adoption

“With the help of SQLancer, an automatic DBMS testing tool, we have been able to identify >100 potential problems in corner cases of the SQL processor.”

https://www.monetdb.org/blog/faster-robuster-and-feature-richer-monetdb-in-2020-and-beyond 36 Adoption

PingCAP implemented a tool go-sqlancer

https://github.com/chaos-mesh/go-sqlancer 37 Adoption

https://www.citusdata.com/blog/2020/09/04/mining-for-logic-bugs-in-citus-with-sqlancer/ 38 SQLancer

https://github.com/sqlancer

SQLancer is an effective and widely-used automatic testing tool to find logic bugs in DBMSs

39 Database Management Systems (DBMS)

Structured Query Language (SQL)

Interact with Database Management System

40 Database Management Systems (DBMS)

CREATE DATABASE db;

Database Management System

41 Database Management Systems (DBMS)

CREATE DATABASE db;

Database Management System

Database db

42 Database Management Systems (DBMS)

CREATE TABLE t0(c0 INTEGER);

Database Management System

Database db

43 Relational Model

44 Relational Model

Table

45 Relational Model

Column

46 Relational Model

Row

47 Database Management Systems (DBMS)

CREATE TABLE t0(c0 INTEGER);

Database Management System

Database db

48 Database Management Systems (DBMS)

CREATE TABLE t0(c0 INTEGER);

Database Management System t0: (c0: INT) Database db

49 Database Management Systems (DBMS)

INSERT INTO t0(c0) VALUES (0), (1), (2);

Database Management System t0: (c0: INT) Database db

50 Database Management Systems (DBMS)

INSERT INTO t0(c0) VALUES (0), (1), (2);

Database Management System t0: (c0: INT) Database db 0 φ 1 φ 2 ¬φ 51 Database Management Systems (DBMS)

SELECT * FROM t0 WHERE p;

Database Management System t0: (c0: INT) Database db 0 φ 1 φ 2 ¬φ 52 Database Management Systems (DBMS)

SELECT * FROM t0 WHERE p;

Database Management System t0: (c0: INT) Database db 0 p 1 p 2 ¬p 53 Database Management Systems (DBMS)

SELECT * FROM t0 WHERE p; Result set 0 p Database Management System 1 p t0: (c0: INT) Database db 0 p 1 p 2 ¬p 54 SQLancer

https://github.com/sqlancer

SQLancer is an effective and widely-used automatic testing tool to find logic bugs in DBMSs

55 Goal: Find Logic Bugs

Bugs that cause the DBMS to return an incorrect result set

56 Motivating Example

t0 t1 c0 c0

0.0 -0.0

SELECT * FROM t0, t1 WHERE t0.c0 = t1.c0; ?

57 Motivating Example

t0 t1 c0 c0

0.0 -0.0

SELECT * FROM t0, t1 WHERE t0.c0 = t1.c0; ?

It might seem disputable whether the predicate should evaluate to true

58 Motivating Example 0 0 0 0 0 … 0 0 0 0 t0 t1 -0 1 0 0 0 … 0 0 0 0 c0 c0

0.0 -0.0 Different binary representation

SELECT * FROM t0, t1 WHERE t0.c0 = t1.c0;

59 Motivating Example 0 0 0 0 0 … 0 0 0 0 t0 t1 false? -0 1 0 0 0 … 0 0 0 0 c0 c0

0.0 -0.0 Different binary representation

SELECT * FROM t0, t1 WHERE t0.c0 = t1.c0; t0.c0 t1.c0

60 Motivating Example

t0 t1 Evaluates to true for most true? c0 c0 programming languages

0.0 -0.0

SELECT * FROM t0, t1 t0.c0 t1.c0 WHERE t0.c0 = t1.c0; 0.0 -0.0

61 Motivating Example

t0 t1 c0 c0

0.0 -0.0

SELECT * FROM t0, t1 t0.c0 t1.c0 WHERE t0.c0 = t1.c0; 0.0 -0.0 ✓

62 Motivating Example

t0 t1 The latest version of MySQL that c0 c0 we tested failed to fetch the row

0.0 -0.0

SELECT * FROM t0, t1 WHERE t0.c0 = t1.c0; t0.c0 t1.c0 

https://bugs.mysql.com/bug.php?id=99122 63 Motivating Example

t0 t1 c0 c0

0.0 -0.0

SELECT * FROM t0, t1 WHERE t0.c0 = t1.c0; t0.c0 t1.c0 

https://bugs.mysql.com/bug.php?id=99122 64 Motivating Example

t0 t1 We could automatically find the bug without c0 c0 having an accurate understanding ourselves 0.0 -0.0

SELECT * FROM t0, t1 WHERE t0.c0 = t1.c0; t0.c0 t1.c0 

https://bugs.mysql.com/bug.php?id=99122 65 SQLancer

https://github.com/sqlancer

SQLancer is an effective and widely-used automatic testing tool to find logic bugs in DBMSs

66 Automatic Testing Core Challenges

Generate a Generate a Validate the Database Query Query’s Result

67 Automatic Testing Core Challenges

CREATE TABLE t0(c0 DOUBLE); CREATE TABLE t1(c0 DOUBLE); INSERT …

Generate a Generate a Validate the Database Query Query’s Result

68 Automatic Testing Core Challenges

SELECT * FROM t0, t1 WHERE t0.c0 = t1.c0;

Generate a Generate a Validate the Database Query Query’s Result

69 Automatic Testing Core Challenges

t0.c0 t1.c0 t0.c0 t1.c0  0 -0 ✓

Generate a Generate a Validate the Database Query Query’s Result

70 Automatic Testing Core Challenges

SQLancer does not require any user interaction for any of the steps

Generate a Generate a Validate the Database Query Query’s Result

71 Automatic Testing Core Challenges

1. Effective test case

Generate a Generate a Validate the Database Query Query’s Result

72 Automatic Testing Core Challenges Manually implemented heuristic database and query generators 1. Effective test case

Generate a Generate a Validate the Database Query Query’s Result

73 SQLsmith

SQLancer’s random database and query generation works similar to existing tools

https://github.com/anse1/sqlsmith 74 Automatic Testing Core Challenges

1. Effective test case 2. Test oracle

Generate a Generate a Validate the Database Query Query’s Result

75 Automatic Testing Core Challenges

Test oracles are the “secret sauce” of SQLancer 1. Effective test case 2. Test oracle

Generate a Generate a Validate the Database Query Query’s Result

76 Test Oracle Incorrect result!

“a test oracle (or just oracle) is a mechanism for determining whether a test has passed or failed”

https://en.wikipedia.org/wiki/Test_oracle 77 What existing test oracles could we use to find bugs in DBMSs?

78 Differential Testing

System 1

Test Case System 2 Generator

System 3

79 Differential Testing

System 1 R1

Test Case System 2 Generator 2

System 3 R3

80 Differential Testing

System 1 R1

Test Case System 2 R Generator 2

R1 = R2 = R3?

System 3 R3

81 Differential Testing: RAGS (Slutz, VLDB 1998)

82 Differential Testing: RAGS (Slutz, VLDB 1998)

Query Generator

83 Differential Testing: RAGS (Slutz, VLDB 1998)

RS1

Query RS Generator 2

RS3

84 Differential Testing: RAGS (Slutz, VLDB 1998)

RS1

Query RS Generator 2

RS1 = RS2 = RS3?

RS3

85 Differential Testing: RAGS (Slutz, VLDB 1998)

RS1

Query RS = Generator 2 ✓

RS1 = RS2 = RS3?

RS3

86 Differential Testing: RAGS (Slutz, VLDB 1998)

RS1

Query RS ≠ Generator 2  RS1 = RS2 = RS3?

RS3

87 Differential Testing: RAGS (Slutz, VLDB 1998)

DBMS- specific SQL

Common SQL Core

88 Differential Testing: RAGS (Slutz, VLDB 1998)

DBMS- specific SQL “We are unable to use Postgres as an oracle because CockroachDB has slightly different semantics and SQL support, and generating queries that execute identically on both is tricky […].” – Cockroach Labs Common SQL Core

89 What oracles were introduced in SQLancer?

90 Finding Logic Bugs in DBMSs

Ternary Logic SQLancer provides Partitioning three test oracles OOPSLA ‘20

Non-optimizing Pivoted Query Reference Engine Synthesis Construction OSDI ‘20 ESEC/FSE ‘20

91 Finding Logic Bugs in DBMSs

Ternary Logic Partitioning OOPSLA ‘20

Non-optimizing Pivoted Query Reference Engine Synthesis Construction OSDI ‘20 ESEC/FSE ‘20

92 Query Partitioning

Ternary Logic Partitioning (TLP) is based on a conceptual framework called Query Partitioning

93 Query Partitioning

Query Q Generator RS(Q)

94 Query Partitioning

original query

Query Q Generator RS(Q)

95 Query Partitioning

Q’1 RS(Q’1)

Q’2 Query Q RS(Q’2) RS(Q) Q’3 Generator RS(Q’3)

Q’n RS(Q’n)

Partition the result set

96 Query Partitioning Partitioning queries

Q’1 RS(Q’1)

Q’2 Query Q RS(Q’2) RS(Q) Q’3 Generator RS(Q’3)

Q’n RS(Q’n)

97 Query Partitioning Partitions

Q’1 RS(Q’1)

Q’2 Query Q RS(Q’2) RS(Q) Q’3 Generator RS(Q’3)

Q’n RS(Q’n)

98 Query Partitioning

Q’1 RS(Q’1)

Q’2 RS(Q’ ) Query Q 2 Q’ RS(Q) Q’3 RS(Q’) Generator RS(Q’3)

Q’n RS(Q’n)

99 Query Partitioning

Composition operator

Q’1 RS(Q’1)

Q’2 RS(Q’ ) Query Q 2 Q’ RS(Q) Q’3 RS(Q’) Generator RS(Q’3)

Q’n RS(Q’n)

100 Query Partitioning

Q’1 RS(Q’1)

Q’2 RS(Q’ ) Query Q 2 Q’ RS(Q) Q’3 RS(Q’) Generator RS(Q’3)

Q’n RS(Q’n)

Combine the results so that RS(Q)=RS(Q’)

101 Query Partitioning

Q’1 RS(Q’1)

Q’2 RS(Q’ ) Query Q 2 Q’ RS(Q) Q’3 RS(Q’) Generator RS(Q’3)

Q’n RS(Q’n)

= ✓ 102 Query Partitioning

Q’1 RS(Q’1)

Q’2 RS(Q’ ) Query Q 2 Q’ RS(Q) Q’3 RS(Q’) Generator RS(Q’3)

Q’n RS(Q’n) ≠

103 Query Partitioning

Q’1 RS(Q’1)

Q’2 RS(Q’ ) Query Q 2 Q’ RS(Q) Q’3 RS(Q’) Generator RS(Q’3)

Q’n RS(Q’n)

In contrast to differential testing, we need only a single DBMS

104 How to Realize This Idea?

How can we partition the result set?

105 Scenario: Coffee Kitchen

106 Tangerines vs. Clementines

107 Tangerines vs. Clementines

tangerines

108 Tangerines vs. Clementines

clementine

109 Tangerines vs. Clementines

110 Tangerines vs. Clementines

I can never tell different citrus fruits apart

111 Tangerines vs. Clementines

I can!

112 Tangerines vs. Clementines

Show me!

113 TangerinesHow I test Ilya’s vs. understanding Clementines without knowing the differences myself?

114 Tangerines vs. Clementines

Please bring me all clementines

115 Tangerines vs. Clementines

Please bring me all clementines

2 fruits

116 Tangerines vs. Clementines

Please bring me all clementines

117 Tangerines vs. Clementines

Please bring me all clementines

118 Tangerines vs. Clementines

Please bring me all fruits that are not clementines

119 Tangerines vs. Clementines

Please bring me all fruits that are not clementines

4 fruits

120 Tangerines vs. Clementines

2 fruits 4 fruits 6 fruits

121 Tangerines vs. Clementines

5 fruits 2 fruits 4 fruits 6 fruits

122 Tangerines vs. Clementines

You likely classified a fruit as both a tangerine and a clementine!

5 fruits 2 fruits 4 fruits 6 fruits

123 Insight

Insight: Every object in a (mathematical) universe is either a tangerine or not a tangerine

124 How can we apply this idea to find bugs in DBMSs?

125 Ternary Logic

Consider a predicate p and a given row r. Exactly one of the following must hold: • p • NOT p • p IS NULL

126 Ternary Logic

Consider a predicate p and a given row r. Exactly one of the following must hold: • p • NOT p • p IS NULL

127 Ternary Logic

Consider a predicate p and a given row r. Exactly one of the following must hold: • p

• NOT p p • p IS NULL NOT p

p IS NULL

128 Motivating Example

t0 t1 How did this insight allow c0 c0 us to detect this bug? 0 -0

SELECT * FROM t0, t1 WHERE t0.c0 = t1.c0; t0.c0 t1.c0 

https://bugs.mysql.com/bug.php?id=99122 129 Example: MySQL

SELECT * FROM t0, t1;

https://bugs.mysql.com/bug.php?id=99122 130 Example: MySQL

SELECT * FROM t0, t1;

t0.c0 t1.c0

0 -0

https://bugs.mysql.com/bug.php?id=99122 131 Example: MySQL

SELECT * FROM t0, t1; SELECT * FROM t0, t1 WHERE t0.c0=t1.c0 UNION ALL SELECT * FROM t0, t1 WHERE NOT (t0.c0=t1.c0) UNION ALL SELECT * FROM t0, t1 WHERE (t0.c0=t1.c0) IS NULL;

t0.c0 t1.c0

0 -0

https://bugs.mysql.com/bug.php?id=99122 132 Example: MySQL p

SELECT * FROM t0, t1; SELECT * FROM t0, t1 WHERE t0.c0=t1.c0 UNION ALL SELECT * FROM t0, t1 WHERE NOT (t0.c0=t1.c0) UNION ALL SELECT * FROM t0, t1 WHERE (t0.c0=t1.c0) IS NULL;

t0.c0 t1.c0

0 -0

https://bugs.mysql.com/bug.php?id=99122 133 Example: MySQL p

SELECT * FROM t0, t1; SELECT * FROM t0, t1 WHERE t0.c0=t1.c0 UNION ALL SELECT * FROM t0, t1 WHERE NOT (t0.c0=t1.c0) UNION ALL SELECT * FROM t0, t1 WHERE (t0.c0=t1.c0) IS NULL;

t0.c0 t1.c0

0 -0 t0.c0 t1.c0

https://bugs.mysql.com/bug.php?id=99122 134 Example: MySQL p

SELECT * FROM t0, t1; SELECT * FROM t0, t1 WHERE t0.c0=t1.c0 UNION ALL SELECT * FROM t0, t1 WHERE NOT (t0.c0=t1.c0) UNION ALL SELECT * FROM t0, t1 WHERE (t0.c0=t1.c0) IS NULL;

t0.c0 t1.c0

0 -0 t0.c0 t1.c0 ≠ https://bugs.mysql.com/bug.php?id=99122  135 Insight

The DBMS is more likely to process the partitioning queries incorrectly due to their higher complexity

136 To what kind of features can we apply the Query Partitioning testing oracle?

137 Scope

• WHERE • GROUP BY • HAVING • DISTINCT queries • Aggregate functions

138 Testing WHERE Clauses

Q Q’ptern Composition Operator SELECT SELECT UNION ALL FROM FROM [] []

WHERE ptern

139 Testing WHERE Clauses

Q Q’ptern Composition Operator SELECT SELECT UNION ALL FROM FROM [] []

WHERE ptern

140 Testing WHERE Clauses

Q Q’ptern Composition Operator SELECT SELECT UNION ALL FROM FROM [] []

WHERE ptern

141 Testing WHERE Clauses

Q Q’ptern Composition Operator SELECT SELECT UNION ALL FROM FROM [] []

WHERE ptern

142 Testing WHERE Clauses

Q Q’ptern Composition Operator SELECT SELECT UNION ALL FROM FROM [] []

WHERE ptern

UNION ALL keeps duplicate rows

143 Scope

• WHERE • GROUP BY • HAVING • DISTINCT queries • Aggregate functions

144 Testing DISTINCT Clauses

Q Q’ptern Composition operator SELECT DISTINCT SELECT UNION FROM FROM [] []

WHERE ptern;

145 Testing DISTINCT Clauses

Q Q’ptern Composition operator SELECT DISTINCT SELECT UNION FROM FROM [] []

WHERE ptern;

UNION removes duplicate rows

146 Scope

• WHERE • GROUP BY • HAVING • DISTINCT queries • Aggregate functions

147 Testing Self-decomposable Aggregate Functions

Q Q’ptern Composition operator SELECT MAX() SELECT MAX() MAX FROM FROM [] []

WHERE ptern;

148 Testing Self-decomposable Aggregate Functions

Q Q’ptern Composition operator SELECT MAX() SELECT MAX() MAX FROM FROM [] []

WHERE ptern;

A partition is an intermediate result, rather than a subset of the result set

149 Bug Example: CockroachDB

SET vectorize=experimental_on; CREATE TABLE t0(c0 INT); CREATE TABLE t1(c0 BOOL) INTERLEAVE IN PARENT t0(rowid); INSERT INTO t0(c0) VALUES (0); INSERT INTO t1(rowid, c0) VALUES(0, TRUE);

150 Bug Example: CockroachDB

SET vectorize=experimental_on; CREATE TABLE t0(c0 INT); CREATE TABLE t1(c0 BOOL) INTERLEAVE IN PARENT t0(rowid); INSERT INTO t0(c0) VALUES (0); INSERT INTO t1(rowid, c0) VALUES(0, TRUE);

SELECT MAX(t1.rowid) FROM t1;

NULL

151 Bug Example: CockroachDB

SET vectorize=experimental_on; CREATE TABLE t0(c0 INT); CREATE TABLE t1(c0 BOOL) INTERLEAVE IN PARENT t0(rowid); INSERT INTO t0(c0) VALUES (0); INSERT INTO t1(rowid, c0) VALUES(0, TRUE);

SELECT MAX(t1.rowid) SELECT MAX(aggr) FROM ( FROM t1; SELECT MAX(t1.rowid) as aggr FROM t1 WHERE '+' >= t1.c0 UNION ALL SELECT MAX(t1.rowid) as aggr FROM t1 WHERE NOT('+' >= t1.c0) UNION ALL SELECT MAX(t1.rowid) as aggr FROM t1 WHERE ('+' >= t1.c0) IS NULL );

NULL 0 ≠ 152 Bug Example: CockroachDB

SET vectorize=experimental_on; CREATE TABLE t0(c0 INT); CREATE TABLE t1(c0 BOOL) INTERLEAVE IN PARENT t0(rowid); INSERT INTO t0(c0) VALUES (0); INSERT INTO t1(rowid, c0) VALUES(0, TRUE);

SELECT MAX(t1.rowid) SELECT MAX(aggr) FROM ( FROM t1; SELECT MAX(t1.rowid) as aggr FROM t1 WHERE '+' >= t1.c0 UNION ALL SELECT MAX(t1.rowid) as aggr FROM t1 WHERE NOT('+' >= t1.c0) UNION ALL SELECT MAX(t1.rowid) as aggr FROM t1 WHERE ('+' >= t1.c0) IS NULL );

NULL 0 ≠ 153 Bug Example: CockroachDB

SET vectorize=experimental_on; CREATE TABLE t0(c0 INT); CREATE TABLE t1(c0 BOOL) INTERLEAVE IN PARENT t0(rowid); INSERT INTO t0(c0) VALUES (0); INSERT INTO t1(rowid, c0) VALUES(0, TRUE);

SELECT MAX(t1.rowid) SELECT MAX(aggr) FROM ( FROM t1; SELECT MAX(t1.rowid) as aggr FROM t1 WHERE '+' >= t1.c0 UNION ALL SELECT MAX(t1.rowid) as aggr FROM t1 WHERE NOT('+' >= t1.c0) UNION ALL SELECT MAX(t1.rowid) as aggr FROM t1 WHERE ('+' >= t1.c0) IS NULL );

NULL 0 ≠ 154 Testing Decomposable Aggregate Functions

Q Q’ptern Composition operator SELECT AVG() SELECT SUM() as s, FROM COUNT() as c SUM(s) []; FROM SUM(c) [];

155 Testing Decomposable Aggregate Functions

Q Q’ptern Composition operator SELECT AVG() SELECT SUM() as s, FROM COUNT() as c SUM(s) []; FROM SUM(c) [];

A single value to represent a partition is insufficient

156 What bugs did you find in ClickHouse?

157 Bug example: ClickHouse

158 Bug example: ClickHouse

Incorrect Query should return an error, but not incorrect answer.

Each column reference directly contained in the search condition shall be one of the following: a) An unambiguous reference to a column that is functionally dependent on the set consisting of every column referenced by a column reference contained in group by clause. ...

159 How general is the technique? Can I apply it to other domains?

160 Metamorphic Testing

SELECT * FROM t0, t1 t0.c0 t1.c0 WHERE t0.c0 = t1.c0; 0.0 -0.0

Derive

161 Metamorphic Testing

SELECT * FROM t0, t1 t0.c0 t1.c0 WHERE t0.c0 = t1.c0; 0.0 -0.0

Derive SELECT * FROM t0, t1 WHERE t0.c0=t1.c0 UNION ALL SELECT * FROM t0, t1 WHERE NOT (t0.c0=t1.c0) UNION ALL SELECT * FROM t0, t1 WHERE (t0.c0=t1.c0) IS NULL;

162 Metamorphic Testing

Execute Test Case Result

Derive SELECT * FROM t0, t1 WHERE t0.c0=t1.c0 UNION ALL SELECT * FROM t0, t1 WHERE NOT (t0.c0=t1.c0) UNION ALL SELECT * FROM t0, t1 WHERE (t0.c0=t1.c0) IS NULL;

163 Metamorphic Testing

Execute Test Case Result

Derive Derived Test Case

164 Metamorphic Testing

Execute Test Case Result

Derive Derived Test Case

This technique is known as metamorphic testing 165 Equivalence Modulo Inputs (EMI) for Testing Compilers

https://people.inf.ethz.ch/suz/emi/index.html 166 Equivalence Modulo Inputs (EMI) for Testing Compilers

EMI’s idea is to create programs that produce the same output for a given input https://people.inf.ethz.ch/suz/emi/index.html 167 Can we test not only DBMS?

Approach is quite generic.

We can try to use it in 1. Generic API with filtering or grouping 2. Regular expressions library 3. Event processing 4. Image recognition 5. Your ideas

168 What about other metamorphic test oracles for DBMSs?

169 Finding Logic Bugs in DBMSs

Ternary Logic Partitioning OOPSLA ‘20

Non-optimizing Pivoted Query Reference Engine Synthesis Construction OSDI ‘20 ESEC/FSE ‘20

170 Goal: Find Logic Bugs

Optimization bugs: logic bugs in the query optimizer

171 Motivating Example

t0 c0 CREATE TABLE t0(c0 UNIQUE); INSERT INTO t0 VALUES (-1) ; -1 SELECT * FROM t0 WHERE t0.c0 GLOB '-*';

-1 ✓

https://www.sqlite.org/src/tktview?name=0f0428096f 172 Motivating Example

t0 c0 CREATE TABLE t0(c0 UNIQUE); INSERT INTO t0 VALUES (-1) ; -1 SELECT * FROM t0 WHERE t0.c0 GLOB '-*';

Optimizer

{} 

https://www.sqlite.org/src/tktview?name=0f0428096f 173 Motivating Example

t0 c0 CREATE TABLE t0(c0 UNIQUE); INSERT INTO t0 VALUES (-1) ; -1 SELECT * FROM t0 WHERE t0.c0 GLOB '-*';

The LIKE optimization malfunctionedOptimizer for non-text columns and a pattern prefix of “-” {} 

https://www.sqlite.org/src/tktview?name=0f0428096f 174 Differential Testing

-O3 Optimizer

SELECT * FROM t0 WHERE t0.c0 GLOB '-*'; Optimizer

-O0

175 Differential Testing

-O3 Optimizer {} SELECT * FROM t0 WHERE t0.c0 GLOB '-*'; Optimizer

-O0

176 Differential Testing

-O3 Optimizer {} SELECT * FROM t0 WHERE t0.c0 GLOB '-*'; Optimizer -1 -O0

177 Differential Testing

-O3 Optimizer {} SELECT * FROM t0 ≠ WHERE t0.c0 GLOB '-*'; Optimizer  -1 -O0

178 Differential Testing

https://www.sqlite.org/pragma.html 179 Differential Testing

PRAGMA case_sensitive_like = boolean;

https://www.sqlite.org/pragma.html 180 Differential Testing

PRAGMA case_sensitive_like = boolean;

DBMSs typically provide only very limited control over optimizations

https://www.sqlite.org/pragma.html 181 NoREC

Idea: Rewrite the query so that the DBMS cannot optimize it

182 Idea Optimized Query Optimizer {} Query Generator

183 Idea Optimized Query Optimizer {} Query Generator Translation Optimizer Step

Unoptimized Query

184 Idea Optimized Query Optimizer {}

Query ≠ Generator Translation Optimizer  Step -1

Unoptimized Query

185 Idea Optimized Query Optimizer {}

Query ≠ Generator Translation Optimizer  Step -1

Unoptimized Query We want to create a “non-optimizing reference engine”

186 Given Query

Consider the following format for the optimized query:

SELECT * FROM t0 WHERE p;

187 Given Query

Consider the following format for the optimized query:

SELECT * FROM t0 WHERE p; t0.c0 GLOB '-*'

188 Given Query

Consider the following format for the optimized query:

SELECT * FROM t0 WHERE p; It is unobvious how we could t0.c0 GLOB '-*' derive an unoptimized query

189 Insight

First Insight: The predicate p must always evaluate to the same value, irrespective of its context

190 Translation Step (Correct Case)

SELECT * FROM t0 WHERE p; -1

191 Translation Step (Correct Case)

SELECT * FROM t0 WHERE p; -1

192 Translation Step (Correct Case)

SELECT * FROM t0 WHERE p; -1

SELECT p FROM t0;

193 Translation Step (Correct Case)

SELECT * FROM t0 WHERE p; -1

SELECT p FROM t0; TRUE

194 Translation Step (Correct Case)

p evaluates SELECT * FROM t0 to TRUE for WHERE p; -1 one row

SELECT p FROM t0; TRUE p evaluates to TRUE for one row

195 Insights

Second Insight: DBMSs focus their optimizations on reducing the amount of data that is processed

196 Translation Step

SELECT * FROM t0 WHERE p;

197 Translation Step

Optimizer

SELECT * FROM t0 QUERY PLAN WHERE p; `--SEARCH TABLE t0 USING COVERING INDEX sqlite_autoindex_t0_1 (c0>? AND c0

198 Translation Step

Optimizer

SELECT * FROM t0 QUERY PLAN WHERE p; `--SEARCH TABLE t0 USING COVERING INDEX sqlite_autoindex_t0_1 (c0>? AND c0

Optimizer

SELECT p QUERY PLAN FROM t0; `--SCAN TABLE t0

199 Translation Step

Optimizer

SELECT * FROM t0 WHERE p; {}

Optimizer

SELECT p FROM t0; TRUE

200 Translation Step

Optimizer

SELECT * FROM t0 WHERE p; {}

Result should Optimizer contain one row SELECT p FROM t0; TRUE

201 Translation Step

Optimizer

SELECT * FROM t0 WHERE p; {}

Result should Optimizer contain  one row SELECT p FROM t0; TRUE

202 Counting Implementation

Optimizer

SELECT COUNT(*) FROM … WHERE p

Optimizer

SELECT SUM(count) FROM ( SELECT p IS TRUE as count FROM );

203 Counting Implementation

Optimizer

SELECT COUNT(*) FROM … 0 WHERE p

Optimizer ≠ SELECT SUM(count) FROM (  SELECT p IS TRUE 1 as count FROM );

204 What about other, non-metamorphic test oracles?

205 Finding Logic Bugs in DBMSs

Ternary Logic Partitioning OOPSLA ‘20

Non-optimizing Pivoted Query Reference Engine Synthesis Construction OSDI ‘20 ESEC/FSE ‘20

206 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1

2

NULL

https://sqlite.org/src/tktview/80256748471a01 207 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 IS NOT is a “null-safe” 2 comparison operator NULL

https://sqlite.org/src/tktview/80256748471a01 208 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1

2

NULL

https://sqlite.org/src/tktview/80256748471a01 209 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 0 2

NULL TRUE

https://sqlite.org/src/tktview/80256748471a01 210 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 0 2

NULL TRUE

0

https://sqlite.org/src/tktview/80256748471a01 211 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 1 2

NULL FALSE

0

https://sqlite.org/src/tktview/80256748471a01 212 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 2 2

NULL TRUE

0

2

https://sqlite.org/src/tktview/80256748471a01 213 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 NULL 2

NULL TRUE

0

2

NULL https://sqlite.org/src/tktview/80256748471a01 214 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 NULL 2

NULL TRUE

0

2

NULL https://sqlite.org/src/tktview/80256748471a01 215 Example: SQLite3 Bug t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 NULL 2

NULL TRUE 0 NULL was not contained  2 in the result set!

NULL https://sqlite.org/src/tktview/80256748471a01 216 PQS Idea t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (3), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1

2 Validate the result set based on NULL one randomly-selected row

217 PQS Idea t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (3), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1

2 Validate the result set based on NULL Pivot row one randomly-selected row

218 PQS Idea t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (3), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 NULL 2

NULL TRUE

Generate a query that is guaranteed to at least fetch the pivot row 219 PQS Idea t0 CREATE TABLE t0(c0); c0 CREATE INDEX i0 ON t0(1) WHERE c0 NOT NULL; 0 INSERT INTO t0 (c0) VALUES (0), (1), (2), (3), (NULL); SELECT c0 FROM t0 WHERE t0.c0 IS NOT 1; 1 0 2 2 NULL If the pivot row is missing from the result set a bug has  been detected

220 Approach

Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained

221 Approach

Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained

222 Approach

Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained

One random row from multiple tables and views

223 Approach

Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained

SELECT c0 FROM t0 WHERE

Generate predicates that evaluate to TRUE for the pivot row and use them in JOIN and WHERE clauses

224 Random Expression Generation

Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained

t0.c0 IS NOT 1; IS NOT

t0.c0 1

225 Random Expression Generation

Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained

t0.c0 IS NOT 1; IS NOT We implemented an expression evaluator for each node t0.c0 1

226 Random Expression Generation

Randomly Generate Validate that Select t0 generate query for the the pivot row pivot row database pivot row is contained c0

0

1 IS Evaluate the tree based NOT 2 on the pivot row NULL

t0.c0 1

227 Random Expression Generation

Randomly Generate Validate that Select t0 generate query for the the pivot row pivot row database pivot row is contained c0

0 Column references return the 1 IS values from the pivot row NOT 2 NULL

t0.c0 1

228 Random Expression Generation

Randomly Generate Validate that Select t0 generate query for the the pivot row pivot row database pivot row is contained c0

0 Column references return the 1 IS values from the pivot row NOT 2 NULL NULL

t0.c0 1

229 Random Expression Generation

Randomly Generate Validate that Select t0 generate query for the the pivot row pivot row database pivot row is contained c0

0

1 IS Constant nodes return their NOT assigned literal values 2 NULL NULL

t0.c0 1

230 Random Expression Generation

Randomly Generate Validate that Select t0 generate query for the the pivot row pivot row database pivot row is contained c0

0

1 IS Constant nodes return their NOT assigned literal values 2 NULL 1 NULL

t0.c0 1

231 Random Expression Generation

Randomly Generate Validate that Select t0 generate TRUE query for the the pivot row pivot row database pivot row is contained c0

0

1 IS Compound nodes NOT compute their result 2 NULL 1 based on their children NULL

t0.c0 1

232 Random Expression Generation

Randomly Generate Validate that Select t0 generate TRUE query for the the pivot row pivot row database pivot row is contained c0

0 TRUE 1 IS Compound nodes NOT compute their result 2 NULL 1 based on their children NULL

t0.c0 1

233 Query Synthesis

Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained

SELECT c0 c0 FROM t0 WHERE t0.c0 IS NOT 1;

234 Query Synthesis

Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained

SELECT c0 c0 FROM t0 WHERE t0.c0 IS NOT 1;

What if the expression does not evaluate to TRUE?

235 Random Expression Rectification

Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained

switch (result) { case TRUE: result = randexpr; case FALSE: result = NOT randexpr; case NULL: result = randexpr IS NULL; } 236 Random Expression Rectification

Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained

Alternatively, we could switch (result) { validate that the pivot row is case TRUE: expectedly not fetched result = randexpr; case FALSE: result = NOT randexpr; case NULL: result = randexpr IS NULL; } 237 Approach

Randomly Generate Validate that Select generate query for the the pivot row pivot row database pivot row is contained

SELECT (NULL) INTERSECT SELECT c0 FROM t0 WHERE NULL IS NOT 1;

Rely on the DBMS to check whether the row is contained

238 How do the techniques compare to each other?

239 Comparison

Property PQS NoREC TLP WHERE ✓ ✓ ✓ Additional SQL features   ✓ Ground truth ✓   No domain knowledge required  ✓ ✓ Implementation effort Moderate Very Low Low

240 Comparison

Property PQS NoREC TLP WHERE ✓ ✓ ✓ Additional SQL features   ✓ Ground truth ✓   No domain knowledge required  ✓ ✓ Implementation effort Moderate Very Low Low

TLP is applicable to testing a wider range of features

241 Comparison

Property PQS NoREC TLP WHERE ✓ ✓ ✓ Additional SQL features   ✓ Ground truth ✓   No domain knowledge required  ✓ ✓ Implementation effort Moderate Very Low Low

Both NoREC and TLP are metamorphic testing approaches 242 Proposed Testing Strategy

Quickly find the optimization bugs

NoREC TLP PQS

243 Proposed Testing Strategy

Test a wider range of features

NoREC TLP PQS

244 Proposed Testing Strategy

Comprehensively test the DBMS’ core functionality

NoREC TLP PQS

245 What implementation strategy did you use for ClickHouse?

246 SQLancer from developer point of view

▎ Natural and intuitive approach

▎ Fuzzing meets correctness check

▎ Pluggable(but you need to write code on Java)

▎ You can incorporate approach in other software

186 How to make integration with your DBMS

1. Import Java or ODBC driver if you have one 2. Teach SQLancer to make connection, create database and make a generic query 3. A bit more to prepare: Create table with schema and insert generator 4. Implement expression generator 5. Implement oracle 6. Expected error handling 7. RUN!

187 Add your driver

188 Make connection, database and query

189 Create table with Your DBMS can has specific SQL extensions. schema Implement subset of them

190 Expression generator

191 Implement Oracle

Almost the same for different DBMS

192 Test How to deal with Failed results? Is syntax No Mute correct error

Yes ▎ Query can be incorrect Mute errors

▎ You don't need to implement ideal query generator – bad queries also test your system

▎ You can mute known bugs and find more

194 Test How to deal with Failed results? Is syntax No Mute correct error

Yes ▎ Query can be incorrect Result No DBMS obtained crashed? ▎ DMBS can crash Yes No Yes

Probably Critical a bug Bug

Can be found by fuzzing Test How to deal with Failed results? Is syntax No Mute correct error

Yes ▎ Query can be incorrect Result No DBMS obtained crashed? ▎ DMBS can crash Yes No Yes

▎ If results differ – you found a SQLancer found Probably Critical correctness bug correctness bug! a bug Bug

Can be found by fuzzing Add Oracles to your Fuzzer

If you already have a query fuzzer or expression gererator you can implement oracles trivially. You can try implementing NoREC or TLP Where logic by yourself: 1) Take expr from your generator 2) Add some logic to compare results:

SELECT * FROM t0 SELECT * FROM t; WHERE expr; SELECT * FROM t WHERE expr UNION ALL SELECT expr SELECT * FROM t WHERE NOT expr FROM t0; UNION ALL SELECT * FROM t WHERE expr IS NULL; How to contribute?

• Add a new DBMS • Automatic reduction of test cases (#333) • New test oracles • Modularization, plugin system for DBMS support (#8) • Blog posts, tutorials, …

259 Summary

260 Q&A Session

• By the way, you can contribute to SQLancer and ClickHouse https://github.com/sqlancer/sqlancer

https://github.com/ClickHouse/ClickHouse

261