Currently Atmosfair.De)

More Than a Query Language: SQL in the 21st Century

@MarkusWinand • @ModernSQL

http://www.almaden.ibm.com/cs/people/chamberlin/sequel-1974.pdf More Than a Query Language: SQL in the 21st Century

@MarkusWinand • @ModernSQL

http://www.almaden.ibm.com/cs/people/chamberlin/sequel-1974.pdf Safe Harbour Statement … Instead of a Safe Harbour Statement … Instead of a Safe Harbour Statement … Take this Safe the Planet Statement: I’m traveling a lot for business and always aim for the most environmental friendly way to do so.

If I cannot avoid ﬂying, I offset its climate impact at a transparent climate protection company. (currently atmosfair.de)

Picture: https://upload.wikimedia.org/wikipedia/commons/2/2c/North_America_from_low_orbiting_satellite_Suomi_NPP.jpg

1974 1992 SQL-92 — Tied to the Relational Idea SQL-92 — Tied to the Relational Idea

Relational Data Model ‣ “Atomic” types (domain)

SQL-92 — Tied to the Relational Idea

Relational Data Model ‣ “Atomic” types (domain)

Atom image: https://commons.wikimedia.org/wiki/File:Stylised_atom_with_three_Bohr_model_orbits_and_stylised_nucleus.png SQL-92 — Tied to the Relational Idea

Relational Data Model ‣ “Atomic” types (domain)

A B C

Atom image: https://commons.wikimedia.org/wiki/File:Stylised_atom_with_three_Bohr_model_orbits_and_stylised_nucleus.png SQL-92 — Tied to the Relational Idea

Relational Data Model ‣ “Atomic” types (domain)

A B C SQL-92 — Tied to the Relational Idea

Relational Data Model ‣ “Atomic” types (domain) ‣ Schema independent of processing purposes ‣ “Normalization”

A B C SQL-92 — Tied to the Relational Idea

Relational Data Model ‣ “Atomic” types (domain) ‣ Schema independent of processing purposes ‣ “Normalization”

A B C C D B E SQL-92 — Tied to the Relational Idea

Relational Data Model Relational Operations ‣ “Atomic” types (domain) ‣ Transform data for ‣ Schema independent of each particular processing purposes processing purposes ‣ “Normalization” ‣ JOIN, UNION, nesting, …

A B C C D B E A B C D E SQL-92 — Tied to the Relational Idea

A B C D E A B C C D B E A B E SQL-92 — Tied to the Relational Idea

A B C D E A B C C D B E C D E

A B E SQL-92 — Tied to the Relational Idea

A B C D E A B C C D B E

A B E

C D E 1992 1999 https://www.wiscorp.com/DBMS_-_GreatNews-TheRelationalModelIsDead_-_paper_-_sam.pdf SQL:1999 — Escaping the Relational Cage

To say that these SQL:1999 extensions are mere “extended interpretations” of the relational data model is like saying that an intercontinental ballistic missile is merely an “extended interpretation” of a spear.

https://www.wiscorp.com/DBMS_-_GreatNews-TheRelationalModelIsDead_-_paper_-_sam.pdf SQL:1999 — Escaping the Relational Cage

With SQL/99 you can get the best of both worlds and of course, you can get the worst of both worlds. It’s up to the database practitioners to do the right thing. https://www.wiscorp.com/DBMS_-_GreatNews-TheRelationalModelIsDead_-_paper_-_sam.pdf SQL:1999 — Escaping the Relational Cage SQL:1999 — Escaping the Relational Cage

Relational Model?

SQL:1999 — Escaping the Relational Cage

Relational Model?

I was as confused as anyone else

Date on Database: Writings 2000-2006 Chris Date SQL:1999 — Escaping the Relational Cage

Relational Model? ? I was as confused as anyone else

Date on Database: Writings 2000-2006 Chris Date SQL:1999 — Escaping the Relational Cage

Relational Model?

I was as confused as anyone else ? By the early 1990s, however, I’d seen the light

Date on Database: Writings 2000-2006 Chris Date SQL:1999 — Escaping the Relational Cage

Relational Model?

I was as confused as anyone else ? By the early 1990s, however, I’d seen the light Domains Can Contain Anything! Date on Database: Writings 2000-2006 Chris Date SQL:1999 — Escaping the Relational Cage

Relational Model? ‣ Introduced rich types

Relational Model? A ‣ Introduced rich types

Relational Model? A B ‣ Introduced rich types [ , ] ‣ arrays [ ] [] I was as confused as anyone else ? By the early 1990s, however, I’d seen the light Domains Can Contain Anything! Date on Database: Writings 2000-2006 Chris Date SQL:1999 — Escaping the Relational Cage

Relational Model? A B C ‣ Introduced rich types [ , ] C D

‣ arrays C D ‣ Nested tables (multiset) [ ] [] C D I was as confused as anyone else ? By the early 1990s, however, I’d seen the light Domains Can Contain Anything! Date on Database: Writings 2000-2006 Chris Date SQL:1999 — Escaping the Relational Cage

Relational Model? A B C D ‣ Introduced rich types C D {x: , [ , ] y: } ‣ arrays C D {x: , ‣ Nested tables (multiset) [ ] y: } C D ‣ composite types (objects) {x: , [] y: } I was as confused as anyone else ? By the early 1990s, however, I’d seen the light Domains Can Contain Anything! Date on Database: Writings 2000-2006 Chris Date SQL:1999 — Escaping the Relational Cage

Relational Model? Non-Relational Operations ‣ Introduced rich types ‣ Introduced recursive ‣ arrays queries that process ‣ Nested tables (multiset) their own output ‣ composite types (objects) ‣ Transitive closure

I was as confused as anyone else ? By the early 1990s, however, I’d seen the light Domains Can Contain Anything! Date on Database: Writings 2000-2006 Chris Date SQL:1999 — Recursion CREATE TABLE t ( id INTEGER, parent INTEGER, ) SQL:1999 — Recursion CREATE TABLE t ( id INTEGER, parent INTEGER, ) SQL:1999 — Recursion CREATE TABLE t ( id INTEGER, parent INTEGER, ) SQL:1999 — Recursion SQL:1999 — Recursion

SQL:1999 — Recursion

SELECT t.id, t.parent FROM t WHERE t.id = ?

SQL:1999 — Recursion

SELECT t.id, t.parent FROM t WHERE t.id = ? UNION ALL SELECT t.id, t.parent FROM t WHERE t.parent = ? SQL:1999 — Recursion

WITH RECURSIVE prev (id, parent) AS ( SELECT t.id, t.parent FROM t WHERE t.id = ? UNION ALL

SELECT t.id, t.parent FROM t JOIN prev ON t.parent = prev.id ) SELECT * FROM prev SQL:1999 — Recursion

WITH RECURSIVE prev (id, parent) AS ( SELECT t.id, t.parent FROM t WHERE t.id = ? UNION ALL

SELECT t.id, t.parent FROM t JOIN prev ON t.parent = prev.id ) SELECT * FROM prev SQL:1999 — Recursion

WITH RECURSIVE prev (id, parent) AS ( SELECT t.id, t.parent FROM t WHERE t.id = ? UNION ALL

SELECT t.id, t.parent FROM t JOIN prev ON t.parent = prev.id ) SELECT * FROM prev SQL:1999 — Recursion

1999 2001 2003 2005 2007 2009 2011 2013 2015 2017 5.1 10.2 MariaDB 8.0 MySQL 8.4 PostgreSQL

1.0 3.8.3 SQLite 7.0 DB2 LUW 11gR2 Oracle 2005 SQL Server

LATERAL SQL:1999 — LATERAL

Select-list sub-queries must be scalar[0]: (return no more than one domain value)

SELECT … , (SELECT column_1 FROM t1 WHERE t1.x = t2.y ) AS c FROM t2 … SQL:1999 — LATERAL

Select-list sub-queries must be scalar[0]: (return no more than one domain value)

SELECT … , (SELECT column_1 , column_2 FROM t1 ✗ WHERE t1.x = t2.y ) AS c More than FROM t2 one column? … ⇒Syntax error SQL:1999 — LATERAL

Select-list sub-queries must be scalar[0]: (return no more than one domain value)

SELECT … More than one row? , (SELECT column_1 , column_2 ⇒Runtime error! FROM t1 ✗ WHERE t1.x = t2.y } ) AS c More than FROM t2 one column? … ⇒Syntax error SQL:1999 — LATERAL

Lateral derived tables lift both limitations and can be correlated:

SELECT … , ldt.* FROM t2 CROSS JOIN LATERAL (SELECT column_1, column_2 FROM t1 WHERE t1.x = t2.y ) AS ldt … SQL:1999 — LATERAL

Lateral derived tables lift both limitations and can be correlated:

SELECT … “Derived table” means

, ldt.* it’s in the FROM t2 FROM/JOIN clause CROSS JOIN LATERAL (SELECT column_1, column_2 FROM t1 WHERE t1.x = t2.y ) AS ldt … SQL:1999 — LATERAL

Lateral derived tables lift both limitations and can be correlated:

SELECT … “Derived table” means

, ldt.* it’s in the FROM t2 FROM/JOIN clause CROSS JOIN LATERAL (SELECT column_1, column_2 FROM t1 WHERE t1.x = t2.y ) AS ldt … Still “correlated” SQL:1999 — LATERAL

Lateral derived tables lift both limitations and can be correlated:

SELECT … “Derived table” means

, ldt.* it’s in the FROM t2 FROM/JOIN clause CROSS JOIN LATERAL (SELECT column_1, column_2 FROM t1 WHERE Regular join t1.x = t2.y semantics ) AS ldt … Still “correlated” SQL:1999 — LATERAL

SQL:1999 — LATERAL

‣ Top-N per group

inside a lateral derived table FETCH FIRST (or LIMIT, TOP) applies per row from left tables. SQL:1999 — LATERAL

‣ Top-N per group FROM t

inside a lateral derived table JOIN LATERAL (SELECT … FETCH FIRST (or LIMIT, TOP) FROM … WHERE t.c=… applies per row from left tables. ORDER BY … LIMIT 10 ) derived_table SQL:1999 — LATERAL Add proper index

for Top-N query ‣ Top-N per group FROM t

inside a lateral derived table JOIN LATERAL (SELECT … FETCH FIRST (or LIMIT, TOP) FROM … WHERE t.c=… applies per row from left tables. ORDER BY … LIMIT 10 ) derived_table

https://use-the-index-luke.com/sql/partial-results/top-n-queries SQL:1999 — LATERAL Add proper index

for Top-N query ‣ Top-N per group FROM t

inside a lateral derived table JOIN LATERAL (SELECT … FETCH FIRST (or LIMIT, TOP) FROM … WHERE t.c=… applies per row from left tables. ORDER BY … LIMIT 10 Also useful to ﬁnd most recent ‣ ) derived_table news from several subscribed topics (“multi-source top-N”). https://use-the-index-luke.com/sql/partial-results/top-n-queries SQL:1999 — LATERAL Add proper index

for Top-N query ‣ Top-N per group FROM t

‣ Table function arguments FROM t (TABLE often implies LATERAL) JOIN TABLE (your_func(t.c)) SQL:1999 — LATERAL

1999 2001 2003 2005 2007 2009 2011 2013 2015 2017 5.1 MariaDB 8.0.14 MySQL 9.3 PostgreSQL

1.0 SQLite 9.1 DB2 LUW 11gR1[0] 12cR1 Oracle 2005[1] SQL Server [0]Undocumented. Requires setting trace event 22829. [1]LATERAL is not supported as of SQL Server 2016 but [CROSS|OUTER] APPLY can be used for the same effect. 1999 2003 http://www.acm.org:80/sigmod/record/issues/0206/standard.pdf (via Wayback machine) SQL:2003 — Schemaless & Analytical

Schemaless ‣ Introduced XML ‣ Non-uniform documents in a single column SQL:2003 — Schemaless & Analytical

Schemaless ‣ Introduced XML ‣ Non-uniform documents in a single column

Later: ‣ JSON added with SQL:2016 ‣ Proprietary JSON support: ‣ 2012: PostgreSQL ‣ 2014: Oracle ‣ 2015: MySQL ‣ 2016: SQL Server SQL:2003 — Schemaless & Analytical

Schemaless Analytical ‣ Introduced XML ‣ Introduced ‣ Non-uniform window functions documents in ‣ Accessing other rows a single column of the current result

Later: ‣ JSON added with SQL:2016 ‣ Proprietary JSON support: ‣ 2012: PostgreSQL ‣ 2014: Oracle ‣ 2015: MySQL ‣ 2016: SQL Server SQL:2003 — Schemaless & Analytical

Schemaless Analytical ‣ Introduced XML ‣ Introduced ‣ Non-uniform window functions documents in ‣ Accessing other rows a single column of the current result

Later: Later: ‣ JSON added with SQL:2016 ‣ Extended in SQL:2011 ‣ Proprietary JSON support: ‣ Popular among “New SQLs” ‣ 2012: PostgreSQL ‣ 2013: BigQuery, Hive ‣ 2014: Oracle ‣ 2014: Impala ‣ 2015: MySQL ‣ 2015: Spark SQL ‣ 2016: SQL Server ‣ 2016: NuoDB, MemSQL, Cockroach DB, VoltDB SQL:2003 — Analytical

SELECT id, value id value 1 +10 2 +20 3 -10 4 +50 5 -30 6 -20 FROM t SQL:2003 — Analytical

SELECT id, value id value bal 1 +10 2 +20 3 -10 4 +50 5 -30 6 -20 FROM t SQL:2003 — Analytical

SELECT id, value id value bal 1 +10 +10 2 +20 3 -10 4 +50 5 -30 6 -20 FROM t SQL:2003 — Analytical

SELECT id, value id value bal 1 +10 +10 2 +20 +30 3 -10 +20 4 +50 +70 5 -30 +40 6 -20 +20 FROM t SQL:2003 — Analytical

SELECT id, value id value bal , SUM(value) 1 +10 +10 OVER ( 2 +20 +30 3 -10 +20 4 +50 +70 5 -30 +40 ) bal 6 -20 +20 FROM t SQL:2003 — Analytical

SELECT id, value id value bal , SUM(value) 1 +10 +10 OVER ( 2 +20 +30 ORDER BY id 3 -10 +20 4 +50 +70 5 -30 +40 ) bal 6 -20 +20 FROM t SQL:2003 — Analytical

SELECT id, value id value bal , SUM(value) 1 +10 +10 OVER ( 2 +20 +30 ORDER BY id 3 -10 +20 ROWS BETWEEN 4 +50 +70 UNBOUNDED PRECEDING 5 -30 +40 ) bal 6 -20 +20 FROM t SQL:2003 — Analytical

SELECT id, value id value bal , SUM(value) 1 +10 +10 OVER ( 2 +20 +30 ORDER BY id 3 -10 +20 ROWS BETWEEN 4 +50 +70 UNBOUNDED PRECEDING 5 -30 +40 AND CURRENT ROW ) bal 6 -20 +20 FROM t SQL:2003 — Analytical

1999 2001 2003 2005 2007 2009 2011 2013 2015 2017 5.1 10.2 MariaDB 8.0 MySQL 8.4 PostgreSQL

1.0 3.25.0 SQLite 7.0 DB2 LUW 8i Oracle 2005[0] 2012 SQL Server [0]Without framing OVER SQL:2011 groups option SQL:2011 — OVER GROUPS option

ORDER BY x between 1 preceding and 1 following

SQL:2011 — OVER GROUPS option rows, range ORDER BY x between 1 preceding and 1 following

SQL:2011 — OVER GROUPS option rows, range ORDER BY x between 1 preceding and 1 following x CURRENT ROW 1 3 3.5 3.5 4 SQL:2011 — OVER GROUPS option rows, range ORDER BY x between 1 preceding and 1 following x CURRENT ROW 1 3 rows count(*) 3.5 3.5 4 SQL:2011 — OVER GROUPS option rows, range ORDER BY x between 1 preceding and 1 following x CURRENT ROW 1 3 rows count(*) range 3.5 x between current_row.x - 1 3.5 and current_row.x + 1 4 SQL:2011 — OVER GROUPS option rows, New in ORDER BY x range SQL:2011 between 1 preceding groups and 1 following x CURRENT ROW 1 3 rows count(*) range 3.5 x between current_row.x - 1 3.5 and current_row.x + 1 4 SQL:2011 — OVER GROUPS option rows, New in ORDER BY x range SQL:2011 between 1 preceding groups and 1 following x CURRENT ROW 1 3 rows count(*) groups range 3.5 count(distinct x) x between current_row.x - 1 3.5 and current_row.x + 1 4 SQL:2011 — OVER GROUPS option

1999 2001 2003 2005 2007 2009 2011 2013 2015 2017 5.1 MariaDB MySQL 11 PostgreSQL

3.28.0 SQLite DB2 LUW Oracle SQL Server OVER SQL:2003 frame exclusion SQL:2003 — frame exclusion

OVER (ORDER BY … BETWEEN … exclude [ no others | current row | group | ties ] )

SQL:2003 — frame exclusion

OVER (ORDER BY default … BETWEEN … exclude [ no others | current row no others | group | ties ] )

SQL:2003 — frame exclusion

OVER (ORDER BY default … BETWEEN … exclude [ no others | current row no others | group | ties ] ) x 1 2 2 2 3 SQL:2003 — frame exclusion

OVER (ORDER BY default … BETWEEN … exclude [ no others | current row current row | group | ties ] ) x 1 2 current row 2 2 3 SQL:2003 — frame exclusion

OVER (ORDER BY default … BETWEEN … exclude [ no others | current row | group | ties ] group ) x 1 2 group current row 2 x = current_row 2 3 SQL:2003 — frame exclusion

OVER (ORDER BY default … BETWEEN … exclude [ no others | current row | group | ties ] ties ) x 1 2 group current row 2 ties x = current_row 2 3 SQL:2003 — frame exclusion

1999 2001 2003 2005 2007 2009 2011 2013 2015 2017 5.1 MariaDB MySQL 11 PostgreSQL

3.28.0 SQLite DB2 LUW Oracle SQL Server Inverse Distribution Functions (percentiles) SQL:2003 — Analytical (Median)

SQL:2003 — Analytical (Median)

SELECT d1.val FROM data d1 JOIN data d2 ON (d1.val < d2.val OR (d1.val=d2.val AND d1.id

SELECT d1.val Number rows FROM data d1 JOIN data d2 ON (d1.val < d2.val OR (d1.val=d2.val AND d1.id

SELECT d1.val Number rows FROM data d1 All employees must JOIN data d2 ON (d1.val < d2.val wash hands OR (d1.val=d2.val AND d1.id

SELECT PERCENTILE_DISC(0.5) WITHIN GROUP (ORDER BY val) FROM data SQL:2003 — Analytical (Median) Median SELECT PERCENTILE_DISC(0.5) WITHIN GROUP (ORDER BY val) FROM data SQL:2003 — Analytical (Median) Median SELECT PERCENTILE_DISC(0.5) WITHIN GROUP (ORDER BY val) FROM data Which value? SQL:2003 — Analytical (Median)

SELECT PERCENTILE_DISC(0.5) WITHIN GROUP (ORDER BY val) FROM data SQL:2003 — Analytical (Median)