Modern SQL: Evolution of a dinosaur Markus Winand Kraków, 9-11 May 2018 Still using Windows 3.1? So why stick with SQL-92? @ModernSQL - https://modern-sql.com/ @MarkusWinand SQL:1999 WITH (Common Table Expressions) WITH (non-recursive) The Problem Nested queries are hard to read: SELECT … FROM (SELECT … FROM t1 Understand JOIN (SELECT … FROM … this first ) a ON (…) ) b JOIN (SELECT … FROM … ) c ON (…) WITH (non-recursive) The Problem Nested queries are hard to read: SELECT … FROM (SELECT … FROM t1 Then this... JOIN (SELECT … FROM … ) a ON (…) ) b JOIN (SELECT … FROM … ) c ON (…) WITH (non-recursive) The Problem Nested queries are hard to read: SELECT … FROM (SELECT … FROM t1 JOIN (SELECT … FROM … ) a ON (…) ) b JOIN (SELECT … FROM … Then this... ) c ON (…) WITH (non-recursive) The Problem Nested queries are hard to read: SELECT … Finally the first line makes sense FROM (SELECT … FROM t1 JOIN (SELECT … FROM … ) a ON (…) ) b JOIN (SELECT … FROM … ) c ON (…) WITH (non-recursive) Since SQL:1999 CTEs are statement-scoped views: Keyword WITH a (c1, c2, c3) AS (SELECT c1, c2, c3 FROM …), b (c4, …) AS (SELECT c4, … FROM t1 JOIN a ON (…) ), c (…) WITH (non-recursive) Since SQL:1999 CTEs are statement-scoped views: Name of CTE and (here WITH optional) column names a (c1, c2, c3) AS (SELECT c1, c2, c3 FROM …), b (c4, …) AS (SELECT c4, … FROM t1 JOIN a ON (…) ), c (…) WITH (non-recursive) Since SQL:1999 CTEs are statement-scoped views: WITH a (c1, c2, c3) Definition AS (SELECT c1, c2, c3 FROM …), b (c4, …) AS (SELECT c4, … FROM t1 JOIN a ON (…) ), c (…) WITH (non-recursive) Since SQL:1999 CTEs are statement-scoped views: WITH Introduces a (c1, c2, c3) another CTE AS (SELECT c1, c2, c3 FROM …), b (c4, …) Don't repeat AS (SELECT c4, … WITH FROM t1 JOIN a ON (…) ), c (…) WITH (non-recursive) Since SQL:1999 CTEs are statement-scoped views: WITH a (c1, c2, c3) AS (SELECT c1, c2, c3 FROM …), b (c4, …) AS (SELECT c4, … FROM t1 May refer to JOIN a previous CTEs ON (…) ), c (…) WITH a (c1, c2, c3) WITH (non-recursive)AS (SELECT c1, c2, c3 FROM …), Since SQL:1999 b (c4, …) AS (SELECT c4, … FROM t1 JOIN a ON (…) ), Third CTE c (…) AS (SELECT … FROM …) SELECT … FROM b JOIN c ON (…) WITH a (c1, c2, c3) WITH (non-recursive)AS (SELECT c1, c2, c3 FROM …), Since SQL:1999 b (c4, …) AS (SELECT c4, … FROM t1 JOIN a ON (…) ), c (…) AS (SELECT … FROM …) No comma! SELECT … FROM b JOIN c ON (…) WITH a (c1, c2, c3) WITH (non-recursive)AS (SELECT c1, c2, c3 FROM …), Since SQL:1999 b (c4, …) AS (SELECT c4, … FROM t1 JOIN a ON (…) ), c (…) AS (SELECT … FROM …) Main query SELECT … FROM b JOIN c ON (…) WITH (non-recursive) CTEs are statement-scoped views: Since SQL:1999 WITH a (c1, c2, c3) AS (SELECT c1, c2, c3 FROM …), b (c4, …) AS (SELECT c4, … FROM t1 JOIN a Read ON (…) top down ), c (…) AS (SELECT … FROM …) SELECT … FROM b JOIN c ON (…) WITH (non-recursive) Use-Cases ‣ Literate SQL https://modern-sql.com/use-case/literate-sql Organize SQL code to improve maintainability ‣ Assign column names https://modern-sql.com/use-case/naming-unnamed-columns to tables produced by values or unnest. ‣ Overload tables (for testing) with queries hide tables https://modern-sql.com/use-case/unit-tests-on-transient-data of the same name. WITH (non-recursive) In a Nutshell WITH are the "private methods" of SQL WITH is a prefix to SELECT WITH queries are only visible in the SELECT they precede WITH in detail: https://modern-sql.com/feature/with WITH (non-recursive) Availability 1999 2001 2003 2005 2007 2009 2011 2013 2015 2017 5.1 10.2 MariaDB 8.0 MySQL 8.4 PostgreSQL 3.8.3[0] SQLite 7.0 DB2 LUW 9iR2 Oracle 2005[1] SQL Server [0]Only for top-level SELECT statements [1]Only allowed at the very begin of a statement. E.g. WITH...INSERT...SELECT. WITH RECURSIVE (Common Table Expressions) WITH RECURSIVE The Problem (This page is intentionally left blank) WITH RECURSIVE The Problem Coping with hierarchies in the Adjacency List Model[0] CREATE TABLE t ( id NUMERIC NOT NULL, parent_id NUMERIC, … PRIMARY KEY (id) ) [0] Hierarchies implemented using a “parent id” — see “Joe Celko’s Trees and Hierarchies in SQL for Smarties” WITH RECURSIVE The Problem Coping with hierarchies in the Adjacency List Model[0] SELECT * FROM t AS d0 LEFT JOIN t AS d1 WHERE d0.id = ? ON (d1.parent_id=d0.id) LEFT JOIN t AS d2 ON (d2.parent_id=d1.id) [0] Hierarchies implemented using a “parent id” — see “Joe Celko’s Trees and Hierarchies in SQL for Smarties” WITH RECURSIVE The Problem Coping with hierarchies in the Adjacency List Model[0] SELECT * FROM t AS d0 LEFT JOIN t AS d1 ON (d1.parent_id=d0.id) LEFT JOIN t AS d2 ON (d2.parent_id=d1.id) WHERE d0.id = ? [0] Hierarchies implemented using a “parent id” — see “Joe Celko’s Trees and Hierarchies in SQL for Smarties” WITH RECURSIVE Since SQL:1999 WITH RECURSIVE d (id, parent, …) AS (SELECT id, parent, … SELECT * FROM tbl FROM t AS d0 WHERE id = ? LEFT JOIN t AS d1 UNION ALL ON (d1.parent_id=d0.id) SELECT id, parent, … LEFT JOIN t AS d2 FROM d ON (d2.parent_id=d1.id) JOIN tbl WHERE d0.id = ? ON (tbl.parent=d.id) ) SELECT * FROM subtree WITH RECURSIVE Since SQL:1999 Recursive common table expressions may refer to themselves in a leg of a UNION [ALL]: Keyword WITH RECURSIVE cte (n) AS (SELECT 1 UNION ALL SELECT n+1 FROM cte WHERE n < 3) SELECT * FROM cte WITH RECURSIVE Since SQL:1999 Recursive common table expressions may refer to themselves in a leg of a UNION [ALL]: WITH RECURSIVE cte (n) Column list AS (SELECT 1 mandatory here UNION ALL SELECT n+1 FROM cte WHERE n < 3) SELECT * FROM cte WITH RECURSIVE Since SQL:1999 Recursive common table expressions may refer to themselves in a leg of a UNION [ALL]: WITH RECURSIVE cte (n) AS (SELECT 1 Executed first UNION ALL SELECT n+1 FROM cte WHERE n < 3) SELECT * FROM cte WITH RECURSIVE Since SQL:1999 Recursive common table expressions may refer to themselves in a leg of a UNION [ALL]: WITH RECURSIVE cte (n) Result sent there AS (SELECT 1 UNION ALL SELECT n+1 FROM cte WHERE n < 3) SELECT * FROM cte WITH RECURSIVE Since SQL:1999 Recursive common table expressions may refer to themselves in a leg of a UNION [ALL]: WITH RECURSIVE cte (n) AS (SELECT 1 UNION ALL SELECT n+1 Result FROM cte visible WHERE n < 3) twice SELECT * FROM cte WITH RECURSIVE Since SQL:1999 Recursive common table expressions may refer to themselves in a leg of a UNION [ALL]: WITH RECURSIVE cte (n) AS (SELECT 1 UNION ALL Once it becomes n SELECT n+1 part of --- FROM cte the final 1 WHERE n < 3) result 2 SELECT * FROM cte 3 (3 rows) WITH RECURSIVE Since SQL:1999 Recursive common table expressions may refer to themselves in a leg of a UNION [ALL]: WITH RECURSIVE cte (n) AS (SELECT 1 UNION ALL Second n SELECT n+1 leg of --- FROM cte UNION 1 WHERE n < 3) is 2 SELECT * FROM cte executed 3 (3 rows) WITH RECURSIVE Since SQL:1999 Recursive common table expressions may refer to themselves in a leg of a UNION [ALL]: WITH RECURSIVE cte (n) AS (SELECT 1 UNION ALL It's a n SELECT n+1 loop! --- FROM cte 1 WHERE n < 3) 2 SELECT * FROM cte 3 (3 rows) WITH RECURSIVE Since SQL:1999 Recursive common table expressions may refer to themselves in a leg of a UNION [ALL]: WITH RECURSIVE cte (n) AS (SELECT 1 UNION ALL It's a n SELECT n+1 loop! --- FROM cte 1 WHERE n < 3) 2 SELECT * FROM cte 3 (3 rows) WITH RECURSIVE Since SQL:1999 Recursive common table expressions may refer to themselves in a leg of a UNION [ALL]: WITH RECURSIVE cte (n) AS (SELECT 1 UNION ALL It's a n SELECT n+1 loop! --- FROM cte 1 WHERE n < 3) 2 SELECT * FROM cte 3 (3 rows) WITH RECURSIVE Since SQL:1999 Recursive common table expressions may refer to themselves in a leg of a UNION [ALL]: WITH RECURSIVE cte (n) AS (SELECT 1 UNION ALL n SELECT n+1 n=3 --- FROM cte doesn't 1 WHERE n < 3) match 2 SELECT * FROM cte 3 (3 rows) WITH RECURSIVE Since SQL:1999 Recursive common table expressions may refer to themselves in a leg of a UNION [ALL]: WITH RECURSIVE cte (n) AS (SELECT 1 UNION ALL n SELECT n+1 n=3 --- FROM cte doesn't 1 WHERE n < 3) match 2 SELECT * FROM cte Loop 3 terminates (3 rows) WITH RECURSIVE Availability 1999 2001 2003 2005 2007 2009 2011 2013 2015 2017 5.1 10.2 MariaDB 8.0 MySQL 8.4 PostgreSQL 3.8.3[0] SQLite 7.0 DB2 LUW 11gR2 Oracle 2005 SQL Server [0]Only for top-level SELECT statements SQL:2003 OVER and PARTITION BY OVER (PARTITION BY) The Problem Two distinct concepts could not be used independently: ‣ Merge rows with the same key properties ‣ GROUP BY to specify key properties ‣ DISTINCT to use full row as key ‣ Aggregate data from related rows ‣ Requires GROUP BY to segregate the rows ‣ COUNT, SUM, AVG, MIN, MAX to aggregate grouped rows OVER (PARTITION BY) The Problem No ⇠ Aggregate ⇢ Yes SELECT c1 No , c2 ⇢ FROM t SELECT DISTINCT SELECT c1 Merge rows Merge c1 , SUM(c2) tot ⇠ , c2 FROM t Yes Yes FROM t GROUP BY c1 OVER (PARTITION BY) The Problem No ⇠ Aggregate ⇢ Yes SELECT c1 SELECT c1 No , c2 , c2 , tot ⇢ FROM t FROM t SELECT c1 , SUM(c2) tot FROM t JOIN ( ) ta GROUP BY c1 ON (t.c1=ta.c1) SELECT DISTINCT SELECT c1 Merge rows Merge c1 , SUM(c2) tot ⇠ , c2 FROM t Yes Yes FROM t GROUP BY c1 OVER (PARTITION BY) The Problem No ⇠ Aggregate ⇢ Yes SELECT c1 SELECT c1 No , c2 , c2 , tot ⇢ FROM t FROM t SELECT c1 , SUM(c2) tot FROM t JOIN ( ) ta GROUP BY c1 ON (t.c1=ta.c1) SELECT DISTINCT SELECT c1 Merge rows Merge c1 , SUM(c2) tot ⇠ , c2 FROM t Yes Yes FROM t GROUP BY c1 OVER (PARTITION BY) Since SQL:2003 No ⇠ Aggregate ⇢ Yes SELECT c1 SELECT c1 No , c2 , c2 ⇢ FROM t FROM t, SUM(c2) OVER (PARTITION BY c1) FROM t SELECT DISTINCT SELECT c1 Merge rows Merge c1 , SUM(c2) tot ⇠ , c2 FROM t Yes Yes FROM t GROUP BY