How to get data from Oracle to PostgreSQL and vice versa Who we are The Company > Founded in 2010 > More than 70 specialists > Specialized in the Middleware Infrastructure > The invisible part of IT > Customers in Switzerland and all over Europe Our Offer > Consulting > Service Level Agreements (SLA) > Trainings > License Management How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 2 About me Daniel Westermann Principal Consultant Open Infrastructure Technology Leader +41 79 927 24 46 daniel.westermann[at]dbi-services.com @westermanndanie Daniel Westermann How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 3 How to get data from Oracle to PostgreSQL and vice versa Before we start We have a PostgreSQL user group in Switzerland! > https://www.swisspug.org Consider supporting us! How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 4 How to get data from Oracle to PostgreSQL and vice versa Before we start We have a PostgreSQL meetup group in Switzerland! > https://www.meetup.com/Switzerland-PostgreSQL-User-Group/ Consider joining us! How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 5 Agenda 1.Past, present and future 2.SQL/MED 3.Foreign data wrappers 4.Demo 5.Conclusion How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 6 Disclaimer This session is not about logical replication! If you are looking for this: > Data Replicator from DBPLUS > https://blog.dbi-services.com/real-time-replication-from-oracle-to-postgresql-using-data-replicator-from-dbplus/ > EDB xDB replication server > https://blog.dbi-services.com/replicating-from-oracle-to-postgresql-with-edb-replication-server-1-a-simple-replication- setup/ > Oracle GoldenGate > https://blog.dbi-services.com/?s=goldengate > SymmetricDS > https://www.symmetricds.org/ > (many more) How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 7 Past, present and future 1 2 3 4 5 How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 8 Past, present and future How did the database landscape look in the past? How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 9 Past, present and future How did the database landscape look in the past? In the past, the database landscape was quite simple > You either had this: How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 10 Past, present and future How did the database landscape look in the past? In the past, the database landscape was quite simple > Or this: How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 11 Past, present and future How did the database landscape look in the past? In the past, the database landscape was quite simple > Or even this: How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 12 Past, present and future How does it look today? How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 13 Past, present and future How does the database landscape look today? Today the situation is quite different How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 14 Past, present and future What will the future look like? How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 15 Past, present and future How does the database landscape look today? It will be even more challenging in the future How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 16 SQL/MED 1 2 3 4 5 How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 17 SQL/MED The SQL standard ISO/IEC 9075-9:2016 > Information technology - Database languages - SQL - Part 9: Management of External Data (SQL/MED) How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 18 SQL/MED The SQL standard SQL/MED provides extensions to SQL > Foreign data wrappers > Datalink types > a special SQL type intended to store URLs in database Provides the ability to deploy federated database systems > Using foreign data wrappers one system can manage data in many external systems How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 19 SQL/MED The SQL standard 1 SQL Server SQL/MED API Foreign data wrapper Implementation dependent API ... n Foreign data Foreign server (Foreign tables) How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 20 Foreign data wrappers 1 2 3 4 5 How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 21 Foreign data wrappers Why all this logos? There are foreign data wrappers for PostgreSQL for all those! How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 22 Foreign data wrappers Why all this logos? ... and actually there are much more > https://wiki.postgresql.org/wiki/Foreign_data_wrappers How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 23 Foreign data wrappers Accessing foreign data For accessing foreign data there needs to be a foreign data wrapper > A foreign data wrapper is a library that communicates with an external data source To access foreign data you need to create a foreign server > Defines how to connect to an external data source External data is available over foreign tables > Foreign tables are defined over foreign servers If the external data source requires authentication > A user mapping needs to be created How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 24 Foreign data wrappers Accessing foreign data You can, of course, also write your own foreign data wrapper > https://www.postgresql.org/docs/current/fdwhandler.html How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 25 Demo 1 2 3 4 5 How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 26 Demo The target setup oracle_fdw CentOS 8 stream Oracle Linux 7.7 PostgreSQL 12.1 Oracle EE 19.3 How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 27 Demo Before we start To give you an idea about foreign data wrappers > file_fdw is there by default > Can be used to access files with plain SQL > Similar to Oracle external tables postgres=# create extension file_fdw; CREATE EXTENSION postgres=# create table file ( dummy text ); CREATE TABLE postgres=# insert into file select i::text||','||i::text||','||i::text||','||i::text||','||i::text from generate_series(1,1000000) i; INSERT 0 1000000 postgres=# copy file to '/var/tmp/dummy.csv'; COPY 1000000 postgres=# \! head -5 /var/tmp/dummy.csv 1,1,1,1,1 … How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 28 Demo Before we start To give you an idea about foreign data wrappers postgres =# create server srv_file_fdw foreign data wrapper file_fdw; CREATE SERVER postgres=# create foreign table exttab ( a int, b int, c int, d int, e int) server srv_file_fdw options ( filename '/var/tmp/dummy.csv', format 'csv', header 'false' ); CREATE FOREIGN TABLE postgres=# select * from exttab limit 5; a | b | c | d | e ---+---+---+---+--- 1 | 1 | 1 | 1 | 1 2 | 2 | 2 | 2 | 2 3 | 3 | 3 | 3 | 3 4 | 4 | 4 | 4 | 4 5 | 5 | 5 | 5 | 5 (5 rows) How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 29 Demo Steps to do The procedure for getting the oracle_fdw up and runnig is > (PostgreSQL should already be up and running) > Download the Oracle instant client zip files > Configure and test Oracle instant client > Download oracle_fdw > Build oracle_fdw against the Oracle instant client libraries > Teach PostgreSQL where to find the Oracle libraries > Create a foreign server > Create a user mapping > Create a schema > Import the foreign schema > Have fun ... Don't worry, all the steps showed live, are in the slides as well! How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 30 Demo Oracle instant client The oracle_fdw will need the Oracle client libraries > Oracle instant client is perfect for that > https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html What you'll need to replay the demo is > Basic Package (ZIP) > SQL*Plus Package (ZIP), for testing the connection to Oracle > SDK Package (ZIP), the header files required for building oracle_fdw We will use the zip distribution so the same can be replayed on Debian based distributions > Works the same on Ubuntu, Debian, SUSE or any other Linux distribution How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 31 Demo Oracle instant client The Oracle instant client zip files postgres@pg-fdw-oracle:/home/postgres/ [PG1] pwd /home/postgres postgres@pg-fdw-oracle:/home/postgres/ [PG1] ls -l total 75220 -rw-rw-r--. 1 postgres postgres 14:44 instantclient-basic-linux.x64-19.5.0.0.0dbru.zip -rw-rw-r--. 1 postgres postgres 14:44 instantclient-sdk-linux.x64-19.5.0.0.0dbru.zip -rw-rw-r--. 1 postgres postgres 14:44 instantclient-sqlplus-linux.x64-19.5.0.0.0dbru.zip postgres@pg-fdw-oracle:/home/postgres/ [PG1] How to get data from Oracle to PostgreSQL and vice versa 19.06.2020 Page 32 Demo Oracle instant client Preparing Oracle instant client postgres@pg-fdw-oracle: [PG1] unzip instantclient-basic-linux.x64-19.5.0.0.0dbru.zip postgres@pg-fdw-oracle: [PG1] unzip instantclient-sdk-linux.x64-19.5.0.0.0dbru.zip postgres@pg-fdw-oracle: [PG1] unzip instantclient-sqlplus-linux.x64-19.5.0.0.0dbru.zip postgres@pg-fdw-oracle: [PG1] ls instantclient_19_5/ adrci libclntsh.so libipc1.so libocci.so.12.1 libsqlplusic.so SDK_README xstreams.jar BASIC_LICENSE libclntsh.so.10.1 libmql1.so libocci.so.18.1 libsqlplus.so Sqlplus BASIC_README libclntsh.so.11.1