Plan of Action

LowMart Corporation

Oracle 11g Downstream Capture/Warehousing Using Streams GROUP 4 Minh Vo, Susan Champigny, Ganapathi S. Santhana

12/8/2012

1 University Avenue Lowell, MA 01854

Phone: 978 934-4000 LowMart Corporation

25 Plan of Action

. PREPARATION o Disable Oracle Database Vault

. CREATE DATA WAREHOUSE DATABASE: ORA11B o Adding a Warehouse Database with DBCA o Spotcheck Before Proceeding with Streams Setup

. DOWNSTREAM CAPTURE/APPLY SETUP: ORACLE STREAMS o Setting System Parameters, Creating Streams Tablespace, and Creating STRMADMIN User o Configuring STREAMS_QUEUE, CAPTURE, PROPAGATION, and APPLY o Adding Another Apply Stream to Grab Selective Columns o Transformations!!!

. APPENDIX o Created Complaints table in SYSDBA instead of HR o Ensure that strmadmin has all the Necessary Privileges o Run the Health Check Script o Things to look out for / Useful stuff: o Mike's Document

o LowMart Corporation

25 Preparation

DISABLE ORACLE DATABASE VAULT

In your database, as sysdba, enter:

SQL> SELECT * FROM V$OPTION WHERE PARAMETER = 'Oracle Database Vault';

If TRUE, we have Data Vault installed. If FALSE, continue to the next section as it will not disrupt streams implementation.

If TRUE, we’ll need to disable it: a. shutdown everything (asm, databases, listener) b. perform (as oracle user): $ cd $ORACLE_HOME/rdbms/lib $ make -f ins_rdbms.mk dv_off ioracle c. restart ASM, databases, listener

Create Data Warehouse Database: Ora11b

ADDING A WAREHOUSE DATABASE WITH DBCA

On the VirtualBox machine where we installed the ASM implementation of Ora11, start dbca (Database Configuration Assistant) and create the database ora11b.

LowMart Corporation

The warehouse database (ora11b) has been created. If we omitted the sample schemas option during setup, we’ll have to create the HR user to proceed.

SPOTCHECK BEFORE PROCEEDING WITH STREAMS SETUP

Let’s doublecheck /etc/oratab so we can start setting . oraenv:

Agent has already set this up for us. Wonderful. We should also check our listener and tnsnames (done later). LowMart Corporation

Downstream Capture/Apply: Oracle Streams

SETTING SYSTEM PARAMETERS, CREATING STREAMS TABLESPACE, AND CREATING STRMADMIN USER

On both databases (ora11 and ora11b) set the job_queue_processes = 1000, aq_tm_processes = 4 and add supplemental log data. Be sure to create the complaints table in the HR schema as well. LowMart Corporation

On the production machine (ora11), create the streams_tbs tablespace and create the user strmadmin. Grant access to strmadmin. LowMart Corporation

For the Warehousing database, also create the streams_tbs tablespace, the strmadmin user, and grant privileges to strmadmin. Make sure the DATAFILE location is specified correctly.

CONFIGURING STREAMS_QUEUE, CAPTURE, PROPAGATION, AND APPLY

Now as the strmadmin user on the source database, run the DBMS_STREAMS_ADM.REMOVE_QUEUE procedure to clear out old queue if one exists. LowMart Corporation

Now create a new streams queue in ora11:

Setup the source database to capture changes from the targeted table:

Drop the database link if one exists, then create the link to ora11b: LowMart Corporation

Drop existing propagation, just to be safe.

And now, (re)create the propagation. MAKE SURE YOU’RE STILL THE STRMADMIN USER: LowMart Corporation

25 By now, the source database ora11 has been set up to capture changes in the hr.complaints table and send them over to the streams_queue@ora11b_link. Next, we’ll work to start the Apply process on the target database.

On the TARGET database ora11b, connect as strmadmin/oracle and remove and recreate the queue.

Create the apply process:

And link the apply process to the queue and table (hr.complaints). LowMart Corporation

Still on the Target database (ora11b) as strmadmin, drop and (re)create the database link to ora11 source database.

We tested the link with query below, but complained that the listener was not aware of the service requested.

SQL> select * from dual@ora11_link;

So, we’ll append to listener.ora under grid_home: LowMart Corporation

TNSNAMES.ORA already set correctly.

Restart the listener service (lsnrctl stop/start) and retest the link: LowMart Corporation

Awesome! Links are working both to and from ora11/ora11b. While still on the Target machine, get the SCN number and instantiate the table with the SCN number:

Now instantiate to the newest SCN:

Start the Apply process with

SQL> EXEC DBMS_APPLY_ADM.START_APPLY('SYNC_APPLY'); LowMart Corporation

From this point on, all changes in source table hr.complaints (ora11) are propagated to the target table hr.complaints (ora11b).

To test, we’ll insert a record into source (ora11) hr.complaints and monitor the alert logs of both databases:

SQL> insert into hr.complaints values (9,'rogers','roy','phone','does not work','yes');

Adding a record to the source db hr.complaints applies it to the target hr.complaints! LowMart Corporation

ADDING ANOTHER APPLY STREAM TO GRAB SELECTIVE COLUMNS

Now let’s configure for selective columns. We’ll need to create the warehouse table DEFECT_TRENDS with the columns PRODUCT and DEFECT on the target DB (ora11b).

Create new Apply process named “DEFECT_APPLY” so we can attach rules to it without breaking our replication to hr.complaints on target.

BEGIN LowMart Corporation

25 dbms_apply_adm.create_apply( queue_name => 'strmadmin.streams_queue', apply_name => 'defect_apply', apply_captured => false); END; /

Link the Apply process to the queue and table:

BEGIN dbms_streams_adm.add_table_rules( table_name => 'hr.complaints', streams_type => 'APPLY', streams_name => 'DEFECT_APPLY', queue_name => 'strmadmin.streams_queue', include_dml => true, include_ddl => false, include_tagged_lcr => false, source_database => 'ora11'); END; /

Find the RULE_NAME to extend with declarative transformations:

SQL> select streams_type, streams_name, rule_owner, rule_name from dba_streams_rules;

We see that we’ll need to extend STRMADMIN.COMPLAINTS25 since it belongs to the DEFECT_APPLY stream.

TRANSFORMATIONS!!!

We’ve done our sleuth-work. Now it’s time to add transformations on the rows being sent to us.

BEGIN DBMS_STREAMS_ADM.RENAME_TABLE( LowMart Corporation

25 rule_name => ‘STRMADMIN.COMPLAINTS25’, from_table_name => ‘hr.complaints’, to_table_name => ‘hr.defect_trends’, step_number => 0, operation => 'ADD'); END; /

BEGIN DBMS_STREAMS_ADM.DELETE_COLUMN( rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.complaints', column_name => 'CUSTNO', value_type => '*', step_number => 1, operation => 'ADD'); DBMS_STREAMS_ADM.DELETE_COLUMN( rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.complaints', column_name => 'CUSTLNAME', value_type => '*', step_number => 1, operation => 'ADD'); DBMS_STREAMS_ADM.DELETE_COLUMN( rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.complaints', column_name => 'CUSTFNAME', value_type => '*', step_number => 1, operation => 'ADD'); DBMS_STREAMS_ADM.DELETE_COLUMN( rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.complaints', column_name => 'COMPLAINT', value_type => '*', step_number => 1, operation => 'ADD'); END; LowMart Corporation

25 /

To display declarative transformations, run:

COLUMN RULE_OWNER HEADING 'Rule Owner' FORMAT A15 COLUMN RULE_NAME HEADING 'Rule Name' FORMAT A15 COLUMN DECLARATIVE_TYPE HEADING 'Declarative|Type' FORMAT A15 COLUMN PRECEDENCE HEADING 'Precedence' FORMAT 99999 COLUMN STEP_NUMBER HEADING 'Step Number' FORMAT 99999

SELECT RULE_OWNER, RULE_NAME, DECLARATIVE_TYPE, PRECEDENCE, STEP_NUMBER FROM DBA_STREAMS_TRANSFORMATIONS WHERE TRANSFORM_TYPE = 'DECLARATIVE TRANSFORMATION';

Match up the rows:

Start DEFECT_APPLY with LowMart Corporation

BEGIN DBMS_APPLY_ADM.START_APPLY( apply_name => 'defect_apply'); END; /

Doesn’t appear like it’s working yet, though… getting error:

Due to the step numbers assigned, turns out the ordering of my declarative rule-based transformations changed the name of the table first, and then tried to delete columns off the original name, which no longer exists. Removing the DELETE_COLUMN transformations and adding them back again with the proper table name (hr.defect_trends) worked great!

25 table_name => 'hr.complaints', column_name => 'CUSTFNAME', value_type => '*', step_number => 1, operation => 'REMOVE'); DBMS_STREAMS_ADM.DELETE_COLUMN( rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.complaints', column_name => 'COMPLAINT', value_type => '*', step_number => 1, operation => 'REMOVE'); END; /

Then finally:

BEGIN DBMS_STREAMS_ADM.DELETE_COLUMN( rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.defect_trends', column_name => 'CUSTNO', value_type => '*', step_number => 1, operation => 'ADD'); DBMS_STREAMS_ADM.DELETE_COLUMN( rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.defect_trends', column_name => 'CUSTLNAME', value_type => '*', step_number => 1, operation => 'ADD'); DBMS_STREAMS_ADM.DELETE_COLUMN( rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.defect_trends', column_name => 'CUSTFNAME', value_type => '*', step_number => 1, LowMart Corporation

25 operation => 'ADD'); DBMS_STREAMS_ADM.DELETE_COLUMN( rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.defect_trends', column_name => 'COMPLAINT', value_type => '*', step_number => 1, operation => 'ADD'); END; /

Inserting records to ora11 hr.complaints now successfully ships data to ora11b hr.complaints AND hr.defect_trends.

For good measure, run the streams health check script after uploading it to /tmp: LowMart Corporation

25 $ sqlplus /nolog

SQL> spool /tmp/mysite.html SQL> @/tmp/streams_hc_11_2_0_3.sql SQL> exit

And open the generated mysite.html for review.

Fin. Done. LowMart Corporation

25 Appendix

CREATED COMPLAINTS TABLE IN SYSDBA INSTEAD OF HR

Pretty self-explanatory… create the complaints table in HR schema instead.

ENSURE THAT STRMADMIN HAS ALL THE NECESSARY PRIVILEGES

SQL> connect / as sysdba Connected. SQL> GRANT SELECT ON CHANGE_SETS to strmadmin; Grant succeeded. SQL> GRANT EXECUTE on DBMS_CDC_PUBLISH to strmadmin; Grant succeeded. SQL> GRANT SELECT ANY TABLE to strmadmin; Grant succeeded. SQL> GRANT CREATE SESSION to strmadmin; Grant succeeded. SQL> GRANT CREATE TABLE to strmadmin; Grant succeeded. SQL> GRANT CREATE TABLESPACE to strmadmin; Grant succeeded. SQL> GRANT UNLIMITED TABLESPACE to strmadmin; Grant succeeded. SQL> GRANT SELECT_CATALOG_ROLE to strmadmin; Grant succeeded. SQL> GRANT EXECUTE_CATALOG_ROLE to strmadmin; Grant succeeded. SQL> GRANT CREATE SEQUENCE to strmadmin; Grant succeeded. SQL> GRANT DBA to strmadmin; Grant succeeded. SQL> GRANT EXECUTE on DBMS_CDC_PUBLISH to strmadmin; Grant succeeded. SQL> GRANT SELECT ANY TABLE to strmadmin; Grant succeeded. SQL> EXECUTE DBMS_STREAMS_AUTH.GRANT_ADMIN_PRIVILEGE(GRANTEE => 'strmadmin'); PL/SQL procedure successfully completed. LowMart Corporation

25 LowMart Corporation

25 RUN THE HEALTH CHECK SCRIPT

View the output: LowMart Corporation

At this point, it seems my capture setup is not correct. STRMADMIN_CAPTURE. Is this on Prod? Or Warehouse?

THINGS TO LOOK OUT FOR/USEFUL STUFF:

DBA_ALERT_HISTORY exec dbms_streams_adm.remove_streams_configuration; select propagation_name, error_message from dba_propagation;

COLUMN OWNER HEADING 'Owner' FORMAT A10 COLUMN NAME HEADING 'Queue Name' FORMAT A28 COLUMN QUEUE_TABLE HEADING 'Queue Table' FORMAT A22 COLUMN USER_COMMENT HEADING 'Comment' FORMAT A15

SELECT q.OWNER, q.NAME, t.QUEUE_TABLE, q.USER_COMMENT FROM DBA_QUEUES q, DBA_QUEUE_TABLES t WHERE t.OBJECT_TYPE = 'SYS.ANYDATA' AND q.QUEUE_TABLE = t.QUEUE_TABLE AND q.OWNER = t.OWNER;

Owner Queue Name Queue Table Comment ------LowMart Corporation

25 STRMADMIN AQ$_STREAMS_QUEUE_TABLE_E STREAMS_QUEUE_TABLE exception queue STRMADMIN STREAMS_QUEUE STREAMS_QUEUE_TABLE exec DBMS_STREAMS_ADM.REMOVE_QUEUE(‘STREAMS_QUEUE’,TRUE);

Alert log of ora11b:

On target:

SQL> select apply_name, queue_name from dba_apply;

APPLY_NAME QUEUE_NAME ------SYNC_APPLY STREAMS_QUEUE

BEGIN DBMS_APPLY_ADM.ALTER_APPLY( apply_name => 'SYNC_APPLY', apply_user => 'HR' ); END; / LowMart Corporation

25 MIKE’S DOCUMENT

In your database, as sysdba, enter: SELECT * FROM V$OPTION WHERE PARAMETER = 'Oracle Database Vault';

If TRUE, you have DV installed. If FALSE, you are good and no concern with data vault in the process of implementing streams

If TRUE, you need to disable that before proceeding with implementing streams. Here is how to do that: a. shutdown everything (asm, databases, listener) b. perform (as oracle user): $ cd $ORACLE_HOME/rdbms/lib $ make -f ins_rdbms.mk dv_off ioracle c. restart ASM, databases, listener

You will also need to deal with log miner. the docs will tell you more on that but in general, streams plsql procedures will call upon logminer procedures and if not there, streams procedures will fail. create table complaints ( custno number(3) not null, custlname varchar2(20), custfname varchar2(20), product varchar2(10), complaint varchar2(100), defect varchar2(30));

1. created a 2nd database called ora11b, used regular datafiles (non-asm database). Created an HR user in ora11b, granted connect, resource to HR. 2. ensured that the listener (which is controlled by the grid infrastructure) could listen for both ora11 and ora11b 3. saved off the entire server to ensure i had a good fail back server to go to after steps 1 and 2 were done. 4. setup streams requirements for the source database (ora11): setup streams admin user, streams ts, queues, etc. (per oracle documentation) 5. setup streams requirements for the target database (ora11b): same as above, per oracle documentation 6. much trial and error. Oracle documentation talks about using global names, logminer, etc. to set up the streams. That is true but you can also avoid global names altogether (smart idea) and use db_links as the method of moving replicated streams data over from source to target table. That is the route I suggest to consider.

Here is a sample insert record I used for the source table (hr.complaints): insert into hr.complaints values (1,'johnston','dave','phone','does not work','yes');

The idea is to have this newly inserted record in the hr.complaints table replicate over to the ora11b database and into the hr.complaints table there. That's the goal. Sounds easy enough right? Hmmm... for the new tablespace needed, as well as the streams administrator user, here is some handy code for that purpose: for source db (ora11 on asm) CREATE TABLESPACE streams_tbs DATAFILE '+DATA/ora11/datafile/streams' SIZE 25M AUTOEXTEND ON MAXSIZE 256M; CREATE USER strmadmin IDENTIFIED BY oracle DEFAULT TABLESPACE streams_tbs QUOTA UNLIMITED ON streams_tbs; grant connect, dba, imp_full_database, exp_full_database, aq_administrator_role to stradmin; for target db (ora11b: non asm): CREATE TABLESPACE streams_tbs DATAFILE '/u02/oradata/ora11b/streams.dbf' SIZE 25M AUTOEXTEND ON MAXSIZE 256M; LowMart Corporation

25 CREATE USER strmadmin IDENTIFIED BY oracle DEFAULT TABLESPACE streams_tbs QUOTA UNLIMITED ON streams_tbs; grant connect, dba, imp_full_database, exp_full_database, aq_administrator_role to stradmin;

At this point, you must setup queues, capture rules, propagation rules on the source database then do the same on the target database (per the documentation). Once done, and database links are built and working correctly, starting the apply process should render decent results (one hopes). This is usually where the troubleshooting begins and the time required to get this working gets more involved.

To setup and test your database links from each database, you might like using this code:

-- link in source database to connect to the target database CREATE DATABASE LINK ora11b_link CONNECT TO strmadmin IDENTIFIED BY oracle USING 'ora11b';

-- test link select * from dual@ora11b_link;

Well, since I spent around 8 hours straight today working on the solution for the project, I will assume many of you did not do the same. So I will let you continue the effort given the above information as a decent road map on this project and see how you fair with it. http://blog.nominet.org.uk/tech/2011/06/03/how-to-corrupt-your-data-dictionary-with-oracle-streams/

I hit this very same issue yesterday as I was working on the implementation. This is a classic case of 'listening' to the documentation and then getting burned by accident. Obviously in this case, I reloaded my server and started the process again but the 2nd time around, I left those logminer objects right where they always live. In fact, since I mentioned to you about using database links to move the data, the logminer facility is not even used to accomplish the replication of the table data. So avoid, at all costs, the part where Oracle tells you to create a logminer tablespace and then move logminer objects into it. If you should ever drop that tablespace, you will wipe your database out pretty nicely. Not a good thing, as the blogger suggested, if you were implementing a streams solution on a production database.