MERGE SQL Statement: Lesser Known Facets

Andrej Pashchenko

@Andrej_SQL blog.sqlora.com About me • Working at Trivadis Germany, Düsseldorf • Focusing on Oracle: • Data Warehousing • Application Development • Application Performance • Course instructor „Oracle New Features for Developers“

@Andrej_SQL blog.sqlora.com

• MERGE is a part of SQL 2003 and has been introduced in Oracle 9i

• Since then, the MERGE has been well adopted and widely used, but sometimes still has some confusing or unexpected behavior ORA-30926 ORA-30926

• ORA-30926 is definitely the most confusing error related to MERGE • The error description is somewhat confusing too:

• One of the reasons is clearly documented: ORA-30926 - Example

• Create an „overlaping“ bonus list and try to merge it in employee table

INSERT INTO scott.bonus (ename, job, sal, comm) SELECT ename, job, sal, sal*0.2 comm Sales department 20% FROM scott.emp WHERE deptno = 30;

INSERT INTO scott.bonus (ename, job, sal, comm) SELECT ename, job, sal, sal*0.1 comm Each manager 10% FROM scott.emp WHERE job = 'MANAGER';

SQL> MERGE INTO scott.emp e 2 USING scott.bonus b But SALES also has a manager: 3 ON (b.ename = e.ename) BLAKE will be updated twice 4 WHEN MATCHED THEN UPDATE set e.comm = b.comm; USING scott.bonus b * ERROR at line 2: ORA-30926: unable to get a stable set of rows in the source tables ORA-30926 - How to find the problem?

• Check the duplicates in the source with respect to the ON-keys:

SQL> MERGE INTO scott.emp e SQL> SELECT ename 2 USING scott.bonus b 2 FROM scott.bonus b 3 ON (b.ename = e.ename) 3 GROUP BY ename 4 WHEN MATCHED THEN 4 HAVING COUNT(*) > 1; 5 UPDATE SET e.comm = b.comm; USING scott.bonus b ENAME * ------ERROR at line 2: BLAKE ORA-30926: unable to get a stable set of rows in the source tables

• Find the correct way to avoid duplicates. Often this is a business question, e.g. use MAX or SUM: SQL>SELECT ename, MAX(comm) comm FROM scott.bonus b GROUP BY ename; ORA-30926 Fixing the problem

• Fix the problem in the source data or directly in your query:

SQL> MERGE INTO scott.emp e 2 USING (SELECT ename, MAX(comm) comm 3 FROM scott.bonus b 4 GROUP BY ename) b 5 ON (b.ename = e.ename) 6 WHEN MATCHED THEN UPDATE set e.comm = b.comm;

8 rows merged.

• What does the documentation say about ORA-30926:

ORA-30926: unable to get a stable set of rows in the source tables Cause: A stable set of rows could not be got because of large dml activity or a non- deterministic where clause. Action: Remove any non-deterministic where clauses and reissue the dml. The whole execution three times?

• Have you noticed that the execution takes much longer if you get ORA-30926?

SQL> MERGE INTO scott.emp e 2 USING scott.bonus b 3 ON (b.ename = e.ename) 4 WHEN MATCHED THEN UPDATE SET e.comm = b.comm; USING scott.bonus b * ERROR at line 2: ORA-30926: unable to get a stable set of rows in the source tables

------| Id | Operation | Name | Starts | E-Rows | A-Rows | ------| 0 | MERGE STATEMENT | | 3 | | 0 | | 1 | MERGE | EMP | 3 | | 0 | | 2 | VIEW | | 3 | | 21 | |* 3 | HASH JOIN | | 3 | 9 | 21 | | 4 | TABLE ACCESS FULL| BONUS | 3 | 9 | 27 | | 5 | TABLE ACCESS FULL| EMP | 3 | 14 | 26 | ------Write Consistency and DML restarts Write Consistency and DML Restarts (UPDATE)

SQL> SELECT ename, sal, comm ENAME SAL COMM FROM emp WHERE sal < 1000; ------SMITH 800 JAMES 950 Session 1 Session 2

SQL> UPDATE emp Cannot update JAMES‘s row an waits t1 2 SET sal = 1000 3 WHERE ename = 'JAMES'; SQL> UPDATE emp t 2 SET comm = nvl(comm,0) + 1000 2 3 WHERE sal < 1000; COMMIT; t3 Session 2 can now update JAMES, but JAMES‘s salary is not less than 1000 anymore

t4 ENAME SAL COMM ------SMITH 800 1000 JAMES 1000 Write Consistency and DML Restart

Identify rows to be updated Get row in Get SCN1 in consistent mode (per SCN1) current mode

No Request new Rollback all changes tracked columns SCN made so far unchanged?

Identify rows to be updated Yes Up to 5000 in consistent mode (new SCN) times Update the row SELECT FOR UPDATE (current mode) Tracked: columns in No Yes WHERE and :OLD, :NEW tracked columns Update locked rows unchanged? values in BEFORE EACH ROW trigger Write Consistency and DML Restarts (MERGE)

Session 1 Session 2 SQL> UPDATE emp Waits for locked row 2 SET sal = 1000 t1 3 WHERE ename = 'JAMES'; SQL> MERGE INTO emp t 2 USING (SELECT 1000 comm FROM dual) q t 3 ON (t.sal < 1000) 2 4 WHEN MATCHED THEN UPDATE 5 SET t.comm = nvl(t.comm,0)+q.comm; COMMIT; t3 Session 2 can now update JAMES, but JAMES‘s salary is not less than 1000 anymore t4 ENAME SAL COMM ------SMITH 800 1000 JAMES 1000 1000 Write Consistency and DML Restarts (MERGE)

t3 COMMIT;

ENAME SAL COMM t4 ------SMITH 800 1000 JAMES 1000 Write Consistency and DML Restarts (MERGE)

Session 1 Session 2 SQL> UPDATE emp Waits for locked row 2 SET comm = 500 t1 3 WHERE ename = 'JAMES'; SQL> MERGE INTO emp t 2 USING (SELECT 1000 comm FROM dual) q t 3 ON (t.sal < 1000) 2 4 WHEN MATCHED THEN UPDATE 5 SET t.comm = nvl(t.comm,0)+ q.comm;

t3 COMMIT;

ENAME SAL COMM t4 ------SMITH 800 1000 JAMES 950 1500 Write Consistency and DML Restart

• MERGE can show a different behavior regarding DML restarts • There was a bug until 18c about not tracking ON-columns, but it is still there even in 19c in some cases • But MERGE is tracking columns in SET clause thus preventing “lost updates” during running statements (no replace for locking strategy in your app) • In case of DML restart triggers can fire multiple times, so avoid any non- transactional logic or autonomous transactions in triggers! ORA-30926 revisited Write Consistency and DML Restart

• Obviously Oracle is using the same mechanism of mini rollbacks as with write consistency also to ensure its deterministic behavior • SET-columns are tracked • Even within the same session updating the column to another value will be detected • The MERGE statement will be restarted • As a result, we can observe the mentioned triple effort: MERGE and than rollback, SELECT FOR UPDATE and then MERGE again ORA-38104 ORA-38104

• Oracle doesn’t allow to update columns used in ON clause • If you feel like you have to do this, verify your requirements carefully • Sometimes useful for ad hoc data manipulation, fixing erroneous data, etc.

EMP • Assign the role EMPNO ENAME ACCOUNTING to each EMP_ROLES employee of deptno=10 7839 KING ENAME ROLE_NAME 7782 CLARK • If an employee is MERGE? KING ACCOUNTING assigned the DEFAULT 7934 MILLER CLARK SALES role, overwrite this assignment. EMP_ROLES CLARK ACCOUNTING ENAME ROLE_NAME MILLER ACCOUNTING KING DEFAULT CLARK SALES ORA-38104

• The straightforward approach doesn’t work:

SQL> MERGE INTO emp_roles t 2 USING (SELECT empno, 'ACCOUNTING' role_name, 'DEFAULT' old_role 3 FROM emp 4 WHERE deptno = 10 5 ) q 6 ON (t.empno = q.empno and t.role_name = q.old_role) 7 WHEN MATCHED THEN UPDATE SET t.role_name = q.role_name 8 WHEN NOT MATCHED THEN INSERT VALUES (q.empno, q.role_name) ; ON (t.empno = q.empno and t.role_name = q.old_role) * ERROR at line 6: ORA-38104: Columns referenced in the ON Clause cannot be updated: "T"."ROLE_NAME" ORA-38104

• Using WHERE clause instead of ON only seems to work, because the result is wrong: no new role assignment for MILLER (7782) SQL> MERGE INTO emp_roles t 2 USING (SELECT empno, 'ACCOUNTING' role_name EMP_ROLES 3 FROM emp 4 WHERE deptno =10 ENAME ROLE_NAME 5 ) q KING ACCOUNTING 6 ON (t.empno = q.empno) 7 WHEN MATCHED THEN UPDATE CLARK SALES 8 SET role_name = q.role_name 9 WHERE t.role_name = 'DEFAULT' MILLER ACCOUNTING 10 WHEN NOT MATCHED THEN INSERT (empno, role_name) 11 VALUES (q.empno, q.role_name) ;

2 rows merged. ORA-38104 – Using ROWID

• Doing the whole logic in USING subquery and merging on ROWID • At the price of increased complexity and performance penalty of one more join

SQL> MERGE INTO emp_roles t 2 USING (SELECT r.rowid rid, 'ACCOUNTING' new_role_name 3 , e.empno EMP_ROLES 4 FROM emp e LEFT JOIN emp_roles r 5 ON e.empno = r.empno ENAME ROLE_NAME 6 AND r.role_name = 'DEFAULT' KING ACCOUNTING 7 WHERE e.deptno = 10 8 ) q CLARK SALES 9 ON (t.rowid = q.rid ) 10 WHEN MATCHED THEN UPDATE CLARK ACCOUNTING 11 SET role_name = q.new_role_name 12 WHEN NOT MATCHED THEN INSERT (empno, role_name) MILLER ACCOUNTING 13 VALUES (q.empno, q.new_role_name) ;

3 rows merged. ORA-38104 – Fooling the Parser

• Using a subquery

SQL> MERGE INTO emp_roles t 2 USING (SELECT empno, 'ACCOUNTING' role_name 3 , 'DEFAULT' old_role EMP_ROLES 4 FROM emp 5 WHERE deptno = 10 ENAME ROLE_NAME 6 ) q KING ACCOUNTING 7 ON ( t.empno = q.empno 8 AND (SELECT t.role_name FROM dual) = q.old_role ) CLARK SALES 9 WHEN MATCHED THEN UPDATE SET role_name = q.role_name 10 WHEN NOT MATCHED THEN INSERT (empno, role_name) CLARK ACCOUNTING 11 VALUES (q.empno, q.role_name) ; MILLER ACCOUNTING 3 rows merged.

Idee: Blog Lukas Eder ORA-38104 – Fooling the Parser

• Using a view as a merge target and hiding a column inside NVL()

SQL> MERGE INTO 2 (SELECT empno, role_name 3 , nvl(role_name,'NEVER') check_role_name EMP_ROLES 4 FROM emp_roles) t 5 USING (SELECT empno, 'ACCOUNTING' role_name ENAME ROLE_NAME 6 , 'DEFAULT' old_role KING ACCOUNTING 7 FROM emp 8 WHERE deptno = 10 ) q CLARK SALES 9 ON ( t.empno = q.empno 10 AND t.check_role_name = q.old_role ) CLARK ACCOUNTING 11 WHEN MATCHED THEN UPDATE SET role_name = q.role_name 12 WHEN NOT MATCHED THEN INSERT (empno, role_name) MILLER ACCOUNTING 13 VALUES (q.empno, q.role_name) ;

3 rows merged.

Idee: Blog Lukas Eder ORA-38104 – Fooling the Parser

• Using row value expressions

SQL> MERGE INTO emp_roles t 2 USING (SELECT empno, 'ACCOUNTING' role_name 3 , 'DEFAULT' old_role EMP_ROLES 4 FROM emp 5 WHERE deptno = 10 ) q ENAME ROLE_NAME 6 ON ( 1=2 OR KING ACCOUNTING 7 (t.empno, t.role_name) = ((q.empno, q.old_role)) ) 8 WHEN MATCHED THEN UPDATE SET role_name = q.role_name CLARK SALES 9 WHEN NOT MATCHED THEN INSERT (empno, role_name) 10 VALUES (q.empno, q.role_name) ; CLARK ACCOUNTING

3 rows merged. MILLER ACCOUNTING

Idee: Blog Lukas Eder ORA-38104 – Can be executed multiple times

• All previous examples could only be executed once! This one is “multi-executable” SQL> MERGE INTO emp_roles t 2 USING ( 3 SELECT DISTINCT EMP_ROLES 4 FIRST_VALUE(r.rowid) 5 OVER(PARTITION BY e.empno ENAME ROLE_NAME 6 ORDER BY DECODE(r.role_name, 7 'ACCOUNTING',1,2)) rid KING ACCOUNTING 8 , e.empno, 'ACCOUNTING' new_role_name 9 FROM emp e LEFT JOIN emp_roles r CLARK SALES 10 ON e.empno = r.empno CLARK ACCOUNTING 11 AND r.role_name in ('DEFAULT', 'ACCOUNTING') 12 WHERE e.deptno = 10 MILLER ACCOUNTING 13 ) q 14 ON (t.rowid = q.rid ) 15 WHEN MATCHED THEN UPDATE SET role_name = q.new_role_name 16 WHEN NOT MATCHED THEN INSERT (empno, role_name) 17 VALUES (q.empno, q.new_role_name) ;

3 rows merged. Direct-Path Insert & Parallel DML Direct Path and MERGE?

SQL> MERGE /*+ append */ SQL> INSERT /*+ append*/ INTO tx2 2 INTO tx2 USING tx ON (tx.n = tx2.n) 2 SELECT n FROM tx 3 WHEN NOT MATCHED THEN INSERT (N) 3 WHERE n NOT IN (SELECT n FROM tx2); VALUES (tx.n) ; 10,000 rows inserted. 10,000 rows merged. ------| Id | Operation | Name | Id | Operation | Name | ------| 0 | INSERT STATEMENT | | 0 | MERGE STATEMENT | | | 1 | LOAD AS SELECT | TX2 | 1 | MERGE | TX2 | | 2 | OPTIMIZER STATISTICS GATHERING | | 2 | VIEW | | |* 3 | HASH JOIN RIGHT ANTI NA | |* 3 | HASH JOIN RIGHT OUTER| | | 4 | TABLE ACCESS FULL | TX2 | 4 | TABLE ACCESS FULL | TX2 | | 5 | TABLE ACCESS FULL | TX | 5 | TABLE ACCESS FULL | TX | ------SQL> SELECT count(*) FROM tx2; ORA-12838: cannot read/modify an object after modifying it in parallel

direct path write has happend Space Management with PDML and MERGE?

SQL> MERGE /*+ append parallel*/ SQL> INSERT /*+ append parallel */ 2 INTO t_tgt_join t0 2 INTO t_tgt_join t0 3 USING ( SELECT ... 3 SELECT ...

------| Id | Operation | |Id | Operation ------| 0 | MERGE STATEMENT | | 0 | INSERT STATEMENT | 1 | PX COORDINATOR | | 1 | PX COORDINATOR | 2 | PX SEND QC (RANDOM) | | 2 | PX SEND QC (RANDOM) | 3 | MERGE | | 3 | LOAD AS SELECT (HIGH WATER MARK BROKERED) | 4 | PX RECEIVE | | 4 | OPTIMIZER STATISTICS GATHERING

SEGMENT_NAME BLOCKS CNT SEGMENT_NAME BLOCKS CNT ------T_TGT_JOIN 8 1024 T_TGT_JOIN 8 1088 T_TGT_JOIN 128 4216 ... 7 rows ... T_TGT_JOIN 256 20 T_TGT_JOIN 128 4647 T_TGT_JOIN 384 2 ... 20 rows ... T_TGT_JOIN 512 8 T_TGT_JOIN 1024 34 T_TGT_JOIN 640 8 T_TGT_JOIN 768 4 30 rows selected. T_TGT_JOIN 896 1 T_TGT_JOIN 1024 134 Conclusion

• It‘s easy to find out the reason for ORA-30926 • But be careful when fixing it only by technical methods. Another discussion with business stakeholders may be necessary. • Don’t execute heavy batch DML in environments with large user activity. • Don‘t use any non-transactional logic inside the triggers • Be careful overcoming ORA-38104 restriction. Rethink your use case if possible. • Keep in mind a different behavior for PDML and Online Statistics Gathering Links

• Oracle documentation, MERGE • Tom Kyte about DML restarts I and II • Ruslan Dautkhanov, Oracle’s Write Consistency • Lukas Eder about ORA-38104 • MERGE and ORA-30926 Besuch uns am Stand auf Ebene 3 ▪ Trivadis Barista (guter Kaffee von morgens bis abends) ▪ Geburtstagstorte (täglich ab 14:00 Uhr) ▪ Speed-Sessions am Stand mit Gewinnspiel: Di 14:45: Martin Berger "Oracle Cloud Free Tier – eben mal kurz ausprobiert…" Mi 10:45: Guido Schmutz "Warum ich Apache Kafka als IT-Spezialist kennen sollte!" Mi 14:45: Stefan Oehrli "PDB Sicherheit und Isolation, Herausforderungen von Oracle Container DB‘s" Do 10:45: Chris Antognini "Welche Query-Optimizer Funktionalitäten sind nur in Autonomous Database verfügbar? “ ▪ Teilnahme an unserem DOAG Gewinnspiel ▪ Networking und Austausch mit unseren Referenten ▪ Party mit Getränken und Snacks im Anschluss an die Sessions am Dienstag 17:30 Uhr