How to Get Data from Oracle to Postgresql and Vice Versa Who We Are

Home , Null (SQL), Select (SQL), SymmetricDS

How to get data from Oracle to PostgreSQL and vice versa Who we are

The Company > Founded in 2010 > More than 70 specialists > Specialized in the Middleware Infrastructure > The invisible part of IT > Customers in Switzerland and all over Europe

Our Offer > Consulting > Service Level Agreements (SLA) > Trainings > License Management

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 2 About me

Daniel Westermann Principal Consultant Open Infrastructure Technology Leader

+41 79 927 24 46 daniel.westermann[at]dbi-services.com @westermanndanie Daniel Westermann

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 3 How to get data from Oracle to PostgreSQL and vice versa Before we start

We have a PostgreSQL user group in Switzerland! > https://www.swisspug.org

Consider supporting us!

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 4 How to get data from Oracle to PostgreSQL and vice versa Before we start

We have a PostgreSQL meetup group in Switzerland! > https://www.meetup.com/Switzerland-PostgreSQL-User-Group/

Consider joining us!

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 5 Agenda

1.Past, present and future 2.SQL/MED 3.Foreign data wrappers 4.Demo 5.Conclusion

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 6 Disclaimer

This session is not about logical replication! If you are looking for this: > Data Replicator from DBPLUS > https://blog.dbi-services.com/real-time-replication-from-oracle-to-postgresql-using-data-replicator-from-dbplus/ > EDB xDB replication server > https://blog.dbi-services.com/replicating-from-oracle-to-postgresql-with-edb-replication-server-1-a-simple-replication- setup/ > Oracle GoldenGate > https://blog.dbi-services.com/?s=goldengate > SymmetricDS > https://www.symmetricds.org/ > (many more)

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 7 Past, present and future

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 8 Past, present and future How did the database landscape look in the past?

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 9 Past, present and future How did the database landscape look in the past?

In the past, the database landscape was quite simple > You either had this:

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 10 Past, present and future How did the database landscape look in the past?

In the past, the database landscape was quite simple > Or this:

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 11 Past, present and future How did the database landscape look in the past?

In the past, the database landscape was quite simple > Or even this:

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 12 Past, present and future How does it look today?

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 13 Past, present and future How does the database landscape look today?

Today the situation is quite different

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 14 Past, present and future What will the future look like?

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 15 Past, present and future How does the database landscape look today?

It will be even more challenging in the future

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 16 SQL/MED

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 17 SQL/MED The SQL standard

ISO/IEC 9075-9:2016 > Information technology - Database languages - SQL - Part 9: Management of External Data (SQL/MED)

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 18 SQL/MED The SQL standard

SQL/MED provides extensions to SQL > Foreign data wrappers > Datalink types > a special SQL type intended to store URLs in database

Provides the ability to deploy federated database systems > Using foreign data wrappers one system can manage data in many external systems

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 19 SQL/MED The SQL standard

SQL Server SQL/MED API Foreign data wrapper Implementation dependent API

... n

Foreign data Foreign server (Foreign tables)

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 20 Foreign data wrappers

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 21 Foreign data wrappers Why all this logos?

There are foreign data wrappers for PostgreSQL for all those!

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 22 Foreign data wrappers Why all this logos?

... and actually there are much more > https://wiki.postgresql.org/wiki/Foreign_data_wrappers

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 23 Foreign data wrappers Accessing foreign data

For accessing foreign data there needs to be a foreign data wrapper > A foreign data wrapper is a library that communicates with an external data source

To access foreign data you need to create a foreign server > Defines how to connect to an external data source

External data is available over foreign tables > Foreign tables are defined over foreign servers

If the external data source requires authentication > A user mapping needs to be created

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 24 Foreign data wrappers Accessing foreign data

You can, of course, also write your own foreign data wrapper > https://www.postgresql.org/docs/current/fdwhandler.html

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 25 Demo

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 26 Demo The target setup

oracle_fdw

CentOS 8 stream Oracle Linux 7.7 PostgreSQL 12.1 Oracle EE 19.3

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 27 Demo Before we start

To give you an idea about foreign data wrappers > file_fdw is there by default > Can be used to access files with plain SQL > Similar to Oracle external tables postgres=# create extension file_fdw; CREATE EXTENSION postgres=# create table file ( dummy text ); CREATE TABLE postgres=# insert into file select i::text||','||i::text||','||i::text||','||i::text||','||i::text from generate_series(1,1000000) i; INSERT 0 1000000 postgres=# copy file to '/var/tmp/dummy.csv'; COPY 1000000 postgres=# \! head -5 /var/tmp/dummy.csv 1,1,1,1,1 …

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 28 Demo Before we start

To give you an idea about foreign data wrappers postgres =# create server srv_file_fdw foreign data wrapper file_fdw; CREATE SERVER postgres=# create foreign table exttab ( a int, b int, c int, d int, e int) server srv_file_fdw options ( filename '/var/tmp/dummy.csv', format 'csv', header 'false' ); CREATE FOREIGN TABLE postgres=# select * from exttab limit 5; a | b | c | d | e ---+---+---+---+--- 1 | 1 | 1 | 1 | 1 2 | 2 | 2 | 2 | 2 3 | 3 | 3 | 3 | 3 4 | 4 | 4 | 4 | 4 5 | 5 | 5 | 5 | 5 (5 rows)

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 29 Demo Steps to do

The procedure for getting the oracle_fdw up and runnig is > (PostgreSQL should already be up and running) > Download the Oracle instant client zip files > Configure and test Oracle instant client > Download oracle_fdw > Build oracle_fdw against the Oracle instant client libraries > Teach PostgreSQL where to find the Oracle libraries > Create a foreign server > Create a user mapping > Create a schema > Import the foreign schema > Have fun ...

Don't worry, all the steps showed live, are in the slides as well!

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 30 Demo Oracle instant client

The oracle_fdw will need the Oracle client libraries > Oracle instant client is perfect for that > https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html

What you'll need to replay the demo is > Basic Package (ZIP) > SQL*Plus Package (ZIP), for testing the connection to Oracle > SDK Package (ZIP), the header files required for building oracle_fdw

We will use the zip distribution so the same can be replayed on Debian based distributions > Works the same on Ubuntu, Debian, SUSE or any other Linux distribution

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 31 Demo Oracle instant client

The Oracle instant client zip files postgres@pg-fdw-oracle:/home/postgres/ [PG1] pwd /home/postgres postgres@pg-fdw-oracle:/home/postgres/ [PG1] ls -l total 75220 -rw-rw-r--. 1 postgres postgres 14:44 instantclient-basic-linux.x64-19.5.0.0.0dbru.zip -rw-rw-r--. 1 postgres postgres 14:44 instantclient-sdk-linux.x64-19.5.0.0.0dbru.zip -rw-rw-r--. 1 postgres postgres 14:44 instantclient-sqlplus-linux.x64-19.5.0.0.0dbru.zip postgres@pg-fdw-oracle:/home/postgres/ [PG1]

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 32 Demo Oracle instant client

Preparing Oracle instant client postgres@pg-fdw-oracle: [PG1] unzip instantclient-basic-linux.x64-19.5.0.0.0dbru.zip postgres@pg-fdw-oracle: [PG1] unzip instantclient-sdk-linux.x64-19.5.0.0.0dbru.zip postgres@pg-fdw-oracle: [PG1] unzip instantclient-sqlplus-linux.x64-19.5.0.0.0dbru.zip postgres@pg-fdw-oracle: [PG1] ls instantclient_19_5/ adrci libclntsh.so libipc1.so libocci.so.12.1 libsqlplusic.so SDK_README xstreams.jar BASIC_LICENSE libclntsh.so.10.1 libmql1.so libocci.so.18.1 libsqlplus.so Sqlplus BASIC_README libclntsh.so.11.1 libnnz19.so libocci.so.19.1 network SQLPLUS_LICENSE genezi libclntsh.so.12.1 libocci.so libociei.so ojdbc8.jar SQLPLUS_README glogin.sql libclntsh.so.18.1 libocci.so.10.1 libocijdbc19.so sdk ucp.jar libclntshcore.so.19.1 libclntsh.so.19.1 libocci.so.11.1 liboramysql19.so SDK_LICENSE uidrvci postgres@pg-fdw-oracle: [PG1] mkdir -p instantclient_19_5/network/admin

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 33 Demo tnsnames.ora

Preparing Oracle instant client, tnsnames.ora postgres@pg-fdw-oracle: [PG1] cat > ~/instantclient_19_5/network/admin/tnsnames.ora << EOF ORATARGET.WORLD = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.22.213)(PORT = 1521)) ) (CONNECT_DATA = (SERVICE_NAME = db1.it.dbi-services.com) ) ) EOF

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 34 Demo sqlnet.ora

Preparing Oracle instant client, sqlnet.ora postgres@pg-fdw-oracle: [PG1] cat > ~/instantclient_19_5/network/admin/sqlnet.ora << EOF NAMES.DIRECTORY_PATH= (LDAP, TNSNAMES, HOSTNAME) NAMES.DEFAULT_DOMAIN = WORLD TRACE_LEVEL_CLIENT = OFF SQLNET.EXPIRE_TIME = 30 EOF

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 35 Demo Setting the environment

Preparing Oracle instant client, environment variables postgres@pg-fdw-oracle: [PG1] export PATH=$PATH:~/instantclient_19_5/ postgres@pg-fdw-oracle: [PG1] export ORACLE_HOME=~/instantclient_19_5/ postgres@pg-fdw-oracle: [PG1] export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME:$ORACLE_HOME/sdk/include/ postgres@pg-fdw-oracle: [PG1] export TNS_ADMIN=$ORACLE_HOME/network/admin

Testing sqlplus postgres@pg-fdw-oracle: [PG1] sqlplus /nolog sqlplus: error while loading shared libraries: libnsl.so.1: cannot open shared object file: No such file or directory

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 36 Demo Installing the missing package

Preparing Oracle instant client, installing the missing package postgres@pg-fdw-oracle: [PG1] sudo dnf install -y libnsl

Testing sqlplus, once again postgres@pg-fdw-oracle: [PG1] sqlplus /nolog

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Feb 7 15:51:19 2020 Version 19.5.0.0.0

SQL>

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 37 Demo Testing the connection to Oracle

Preparing Oracle instant client, testing the connection to the target postgres@pg-fdw-oracle: [PG1] sqlplus system/manager@ORATARGET

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Feb 7 17:03:42 2020 Version 19.5.0.0.0

Last Successful login time: Mon Feb 03 2020 11:27:11 +01:00

Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.3.0.0.0

SQL>

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 38 Demo Getting oracle_fdw

Installing oracle_fdw, getting the source postgres@pg-fdw-oracle: [PG1] git clone https://github.com/laurenz/oracle_fdw.git postgres@pg-fdw-oracle: [PG1] ls oracle_fdw/ CHANGELOG Makefile oracle_fdw--1.1--1.2.sql oracle_fdw.control oracle_utils.c sql expected msvc oracle_fdw--1.2.sql oracle_fdw.h README.md TODO LICENSE oracle_fdw--1.0--1.1.sql oracle_fdw.c oracle_gis.c README.oracle_fdw

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 39 Demo Compiling and installing oracle_fdw

Installing oracle_fdw, compiling and installing postgres@pg-fdw-oracle: [PG1] cd oracle_fdw/ postgres@pg-fdw-oracle: [PG1] make gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -fPIC - I/home/postgres/instantclient_19_5//sdk/include - I/home/postgres/instantclient_19_5//oci/include - I/home/postgres/instantclient_19_5//rdbms/public -I/usr/include/oracle/19.3/client - I/usr/include/oracle/19.3/client64 -I/usr/include/oracle/18.5/client - I/usr/include/oracle/18.5/client64 -I/usr/include/oracle/18.3/client ... postgres@pg-fdw-oracle: [PG1] make install /usr/bin/mkdir -p '/u01/app/postgres/product/12/db_1/lib' /usr/bin/mkdir -p '/u01/app/postgres/product/12/db_1/share/extension' /usr/bin/mkdir -p '/u01/app/postgres/product/12/db_1/share/extension'

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 40 Demo The extension

Creating the oracle_fdw extension in PostgreSQL postgres@pg-fdw-oracle: [PG1] psql postgres psql (12.1 dbi services build) Type "help" for help.

postgres=# create extension oracle_fdw; ERROR: could not load library "/u01/app/postgres/product/12/db_1/lib/oracle_fdw.so": libclntsh.so.19.1: cannot open shared object file: No such file or directory postgres=#

> PostgreSQL needs to find the libraries

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 41 Demo Creating the extension

Creating the oracle_fdw extension in PostgreSQL postgres@pg-fdw-oracle: [PG1] pg_ctl stop postgres@pg-fdw-oracle: [PG1] echo $LD_LIBRARY_PATH :/home/postgres/instantclient_19_5/:/home/postgres/instantclient_19_5/sdk/include/:/home /postgres/instantclient_19_5/:/home/postgres/instantclient_19_5/sdk/include/ postgres@pg-fdw-oracle: [PG1] pg_ctl start postgres@pg-fdw-oracle: [PG1] psql postgres psql (12.1 dbi services build) Type "help" for help.

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 42 Demo Creating the foreign server

Creating the foreign server in PostgreSQL postgres@pg-fdw-oracle: [PG1] psql postgres psql (12.1 dbi services build) Type "help" for help.

postgres=# create server oratarget foreign data wrapper oracle_fdw options (dbserver '//192.168.22.213:1521/db1.it.dbi-services.com'); CREATE SERVER postgres=# \des+ List of foreign servers Name | Foreign-data wrapper | FDW Optopns oratarget | oracle_fdw | (dbserver '//192.168.22.213:1521/db1.it.dbi- services.com') |

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 43 Demo The user mapping

Creating the user mapping in PostgreSQL postgres@pg-fdw-oracle: [PG1] psql postgres psql (12.1 dbi services build) Type "help" for help.

postgres=# create user mapping for postgres server oratarget options (user 'system', password 'manager'); CREATE USER MAPPING

postgres=# \deu+ List of user mappings Server | User name | FDW options ------+------+------oratarget | postgres | ("user" 'system', password 'manager') (1 row)

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 44 Demo A dedicated schema

Creating a dedicated schema postgres@pg-fdw-oracle: [PG1] psql postgres psql (12.1 dbi services build) Type "help" for help.

postgres=# create schema oracle; CREATE SCHEMA postgres=# \dn List of schemas Name | Owner ------+------oracle | postgres public | postgres (2 rows)

postgres=#

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 45 Demo A dedicated schema

Creating a foreign table postgres@pg-fdw-oracle: [PG1] psql postgres psql (12.1 dbi services build) Type "help" for help.

postgres=# create foreign table oracle.sales ( prod_id integer options (key 'true') not null , cust_id integer options (key 'true') not null , time_id timestamp options (key 'true') not null , channel_id integer options (key 'true') not null , promo_id integer options (key 'true') not null , quantity_sold numeric not null , amount_sold numeric not null ) server oratarget options (schema 'SH', table 'SALES');

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 46 Demo A dedicated schema

Getting data from Oracle postgres@pg-fdw-oracle: [PG1] psql postgres psql (12.1 dbi services build) Type "help" for help.

postgres=# select count(*) from oracle.sales; count ------918843 (1 row)

postgres=# select prod_id,cust_id from oracle.sales limit 5; prod_id | cust_id ------+------13 | 987 13 | 1660 13 | 1762 13 | 1843 13 | 1948

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 47 Demo A dedicated schema

Creating a dedicated schema postgres@pg-fdw-oracle: [PG1] psql postgres psql (12.1 dbi services build) Type "help" for help.

postgres=# create schema oratarget; CREATE SCHEMA postgres=# \dn List of schemas Name | Owner ------+------oracle | postgres oratarget | postgres public | postgres (2 rows)

postgres=#

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 48 Demo Importing the definitions

Importing the foreign schema postgres@pg-fdw-oracle: [PG1] psql postgres psql (12.1 dbi services build) Type "help" for help.

postgres=# import foreign schema "SH" from server oratarget into oratarget options (case 'lower'); IMPORT FOREIGN SCHEMA postgres=# select a.relname , b.ftserver , b.ftoptions from pg_class a , pg_foreign_table b where b.ftrelid = a.oid order by a.relname;

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 49 Demo Accessing foreign data

Querying data from Oracle postgres@pg-fdw-oracle: [PG1] psql postgres psql (12.1 dbi services build) Type "help" for help.

postgres=# select prod_id,cust_id,time_id,amount_sold from oratarget.sales limit 5;

prod_id | cust_id | time_id | amount_sold ------+------+------+------13 | 987 | 1998-01-10 00:00:00 | 1232.16 13 | 1660 | 1998-01-10 00:00:00 | 1232.16 13 | 1762 | 1998-01-10 00:00:00 | 1232.16 13 | 1843 | 1998-01-10 00:00:00 | 1232.16 13 | 1948 | 1998-01-10 00:00:00 | 1232.16 (5 rows)

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 50 Demo Creating a view

Locally accessingdata from Oracle, VIEWS postgres@pg-fdw-oracle: [PG1] psql postgres psql (12.1 dbi services build) Type "help" for help.

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 51 Demo Creating a materialized view

Locally accessingdata from Oracle, MATERIALIZED VIEWS postgres@pg-fdw-oracle: [PG1] psql postgres psql (12.1 dbi services build) Type "help" for help.

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 52 Demo Statistics on foreign tables

Gathering statistics on foreign tables postgres@pg-fdw-oracle: [PG1] psql postgres psql (12.1 dbi services build) Type "help" for help.

postgres=# analyze verbose oratarget.customers; INFO: analyzing "oratarget.customers" INFO: "customers": table contains 55500 rows; 30000 rows in sample ANALYZE postgres=# select attname,avg_width,n_distinct from pg_stats where tablename = 'customers' and schemaname = 'oratarget'; attname | avg_width | n_distinct ------+------+------cust_id | 6 | -1 cust_first_name | 6 | 1254 cust_last_name | 7 | 885 cust_gender | 2 | 2

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 53 Demo Explain plans

Explain plans on foreign tables postgres@pg-fdw-oracle: [PG1] psql postgres psql (12.1 dbi services build) Type "help" for help.

postgres=# explain (analyze,verbose) select cust_first_name from oratarget.customers where cust_id = 6783; QUERY PLAN ------Foreign Scan on oratarget.customers (cost=10000.00..10010.00 rows=1 width=6) (actual time=81.749..81.793 rows=1 loops=1) Output: cust_first_name Oracle query: SELECT /*9900a130314c93ead5f39565e35aef4b*/ r1."CUST_ID", r1."CUST_FIRST_NAME" FROM "SH"."CUSTOMERS" r1 WHERE (r1."CUST_ID" = 6783) Oracle plan: SELECT STATEMENT Oracle plan: TABLE ACCESS BY INDEX ROWID CUSTOMERS Oracle plan: INDEX UNIQUE SCAN CUSTOMERS_PK (condition "R1"."CUST_ID"=6783)

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 54 Demo Push down of joins

Push down of joins (result on next slide) postgres@pg-fdw-oracle: [PG1] psql postgres psql (12.1 dbi services build) Type "help" for help.

postgres=# explain (analyze,verbose) select oratarget.customers.cust_first_name , oratarget.customers.cust_last_name , oratarget.sales.prod_id , oratarget.sales.time_id from oratarget.customers join oratarget.sales on oratarget.customers.cust_id = oratarget.sales.cust_id where oratarget.customers.cust_id = 6783;

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 55 Demo Push down of joins

Push down of joins, the resulting explain plan (only one statement on the Oracle side) Foreign Scan (cost=10000.00..20000.00 rows=1000 width=53) Output: customers.cust_first_name, customers.cust_last_name, sales.prod_id, sales.time_id Oracle query: SELECT /*6bc19f5c70777a6c7dd703f7f9cf9dfc*/ r1."CUST_FIRST_NAME", r1."CUST_LAST_NAME", r2."PROD_ID", r2."TIME_ID" FROM ("SH"."CUSTOMERS" r1 INNER JOIN "SH"."SALES" r> Oracle plan: SELECT STATEMENT Oracle plan: HASH JOIN (condition "R1"."CUST_ID"="R2"."CUST_ID") Oracle plan: NESTED LOOPS Oracle plan: NESTED LOOPS Oracle plan: STATISTICS COLLECTOR Oracle plan: TABLE ACCESS FULL CUSTOMERS Oracle plan: PARTITION RANGE ALL Oracle plan: BITMAP CONVERSION TO ROWIDS Oracle plan: BITMAP INDEX SINGLE VALUE SALES_CUST_BIX (condition "R1"."CUST_ID"="R2"."CUST_ID") Oracle plan: TABLE ACCESS BY LOCAL INDEX ROWID SALES Oracle plan: PARTITION RANGE ALL Oracle plan: TABLE ACCESS FULL SALES

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 56 Demo Select based foreign tables

Creating foreign tables over a select statement postgres@pg-fdw-oracle: [PG1] psql postgres psql (12.1 dbi services build) Type "help" for help.

postgres=# create foreign table my_ftab ( cust_first_name text , cust_last_name text , prod_id int , time_id timestamptz ) server oratarget options ( table '(SELECT CUSTOMERS.CUST_FIRST_NAME , CUSTOMERS.CUST_LAST_NAME , SALES.PROD_ID , SALES.TIME_ID FROM SH.CUSTOMERS JOIN SH.SALES ON SH.CUSTOMERS.CUST_ID = SH.SALES.CUST_ID)' ); postgres=# select count(*) from my_ftab;

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 57 Demo Pre-fetching

Controlling pre-fetching (default 200) postgres=# alter foreign table oratarget.customers options ( prefetch '1000' ); ALTER FOREIGN TABLE postgres=# \d oratarget.customers Foreign table "oratarget.customers" Column | Type | Collation | Nullable | ------+------+------+------+ cust_id | numeric | | not null | cust_first_name | character varying(20) | | not null | cust_last_name | character varying(40) | | not null | cust_gender | character(1) | | not null … cust_valid | character varying(1) | | | Server: oratarget FDW options: (schema 'SH', "table" 'CUSTOMERS', prefetch '1000')

postgres=#

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 58 Demo Writing data to Oracle

Writing data back to Oracle postgres@pg-fdw-oracle: [PG1] psql postgres psql (12.1 dbi services build) Type "help" for help.

postgres=# insert into oratarget.customers values ( 1111111,'soug','soug','F',2020,'married','soug',2800 , 'bern',51442,'bern',52610,52770,'519-236-6123' , 'G: 130,000 - 149,999',1500,'[email protected]' , 'total',52772,null,'1998-01-01 00:00:00',null,'I'); INSERT 0 1 postgres=# select count(*) from oratarget.customers where cust_id = 1111111; count ------1 (1 row)

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 59 Demo Read only foreign tables

Setting foreign tables "readonly" postgres@pg-fdw-oracle: [PG1] psql postgres psql (12.1 dbi services build) Type "help" for help.

postgres=# alter foreign table oratarget.customers options(readonly 'true'); ALTER FOREIGN TABLE postgres=# update oratarget.customers set cust_first_name = 'soug2' where cust_id = 1111111; ERROR: foreign table "customers" does not allow updates postgres=#

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 60 Demo Reporting issues

Problem reporting postgres=# SELECT oracle_diag('oratarget'); oracle_diag ------oracle_fdw 2.3.0devel, PostgreSQL 12.1 dbi services build, Oracle client 19.6.0.0.0, Oracle server 19.0.0.0.0 (1 row) postgres=# SELECT oracle_diag(); oracle_diag ------oracle_fdw 2.3.0devel, PostgreSQL 12.1 dbi services build, Oracle client 19.6.0.0.0, ORACLE_HOME=/home/postgres/instantclient_19_5/, TNS_ADMIN=/home/postgres/instantclient_19_5/network/admin

> https://github.com/laurenz/oracle_fdw/issues

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 61 Conclusion

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 62 Conclusion

PostgreSQL comes with a partial implementation of SQL/MED > Management of external data

The foreign data wrappers provided by default are postgres_fdw and file_fdw > Other foreign data wrappers, like oracle_fdw, need to be installed seperately

Foreign data wrappers are implemented as extensions > Many foreign data wrappers do already exist > https://wiki.postgresql.org/wiki/Foreign_data_wrappers

This makes PostgreSQL a great choice as a data integration platform

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 63 Conclusion

Connecting PostgreSQL and Oracle gives you the best of both worlds > Use Oracle where it fits your needs > Use PostgreSQL where it fits your needs

Pushing and getting data becomes quite easy with foreign data wrappers > But, using foreign data wrappers is not a logical replication solution > You will need to refresh/push data for getting it locally > Otherwise you will always go over the network

The reality today is, that you need to deal with various data sources

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 64 Conclusion

Further references > Connecting PostgreSQL to Microsoft SQL Server > https://blog.dbi-services.com/connecting-your-postgresql-instance-to-a-microsoft-sql-server-instance/ > Connecting PostgreSQL to MySQL or MariaDB > https://blog.dbi-services.com/connecting-your-postgresql-instance-to-a-mariadbmysql-instance/ > External tables in PostgreSQL using file_fdw > https://blog.dbi-services.com/external-tables-in-postgresql/ > Connecting PostgreSQL to RSS feeds > https://blog.dbi-services.com/feeding-blogsrrs-items-directly-into-your-postgresql-database/

The complete demo

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 65 Basel Delémont Zürich

Bern Nyon Any questions?

Please do ask! We would love to boost your IT-Infrastructure How about you?

How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 66