Developing Integration Projects with Oracle Data Integrator
Total Page:16
File Type:pdf, Size:1020Kb
Oracle® Fusion Middleware Developing Integration Projects with Oracle Data Integrator 12c (12.2.1.4.0) E95627-02 July 2021 Oracle Fusion Middleware Developing Integration Projects with Oracle Data Integrator, 12c (12.2.1.4.0) E95627-02 Copyright © 2010, 2021, Oracle and/or its affiliates. Primary Author: Oracle Corporation This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable: U.S. GOVERNMENT END USERS: Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs) and Oracle computer documentation or other Oracle data delivered to or accessed by U.S. Government end users are "commercial computer software" or "commercial computer software documentation" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, reproduction, duplication, release, display, disclosure, modification, preparation of derivative works, and/or adaptation of i) Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs), ii) Oracle computer documentation and/or iii) other Oracle data, is subject to the rights and limitations specified in the license contained in the applicable contract. The terms governing the U.S. Government’s use of Oracle cloud services are defined by the applicable contract for such services. No other rights are granted to the U.S. Government. This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. Intel and Intel Inside are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Epyc, and the AMD logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. This software or hardware and documentation may provide access to or information about content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services unless otherwise set forth in an applicable agreement between you and Oracle. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services, except as set forth in an applicable agreement between you and Oracle. Contents Preface Audience xviii Documentation Accessibility xviii Related Documents xviii Conventions xix What's New In Oracle Data Integrator? Part I Introduction to Developing with Oracle Data Integrator 1 Overview of an Integration Project Oracle Data Integrator Project Quick Start List 1-1 2 Overview of Oracle Data Integrator Topology Introduction to the Oracle Data Integrator Topology 2-1 Physical Architecture 2-1 Contexts 2-2 Logical Architecture 2-2 Agents 2-3 Languages 2-5 Repositories 2-5 Part II Managing and Reverse-Engineering Metadata 3 Creating and Using Data Models and Datastores Introduction to Models 3-1 Datastores 3-1 Data Integrity 3-2 iii Reverse-engineering 3-2 Changed Data Capture 3-3 Creating and Reverse-Engineering a Model 3-3 Creating a Model 3-3 Creating a Model and Topology Objects 3-4 Reverse-engineering a Model 3-5 Creating and Reverse-Engineering a Datastore 3-7 Creating a Datastore 3-7 Reverse-Engineering File Datastores 3-8 Reverse-Engineering Fixed Files 3-8 Reverse-Engineering Delimited Files 3-9 Reverse-Engineering COBOL Files 3-9 Adding and Deleting Datastore Attributes 3-9 Adding and Deleting Constraints and Filters 3-10 Keys 3-10 References 3-10 Conditions 3-11 Mandatory Attributes 3-11 Filter 3-12 Editing and Viewing a Datastore's Data 3-12 Using Partitioning 3-13 Manually Defining Partitions and Sub-Partitions of Model Datastores 3-13 Checking Data Quality in a Model 3-14 Introduction to Data Integrity 3-14 Checking a Constraint 3-14 Perform a Static Check on a Model, Sub-Model or Datastore 3-15 Reviewing Erroneous Records 3-15 4 Using Journalizing Introduction to Changed Data Capture 4-1 The Journalizing Components 4-2 Simple vs. Consistent Set Journalizing 4-2 Setting up Journalizing 4-3 Setting up and Starting Journalizing 4-3 Journalizing Infrastructure Details 4-7 Journalizing Status 4-8 Using Changed Data 4-8 Viewing Changed Data 4-8 Using Changed Data: Simple Journalizing 4-8 Using Changed Data: Consistent Set Journalizing 4-9 iv Journalizing Tools 4-11 Package Templates for Using Journalizing 4-11 5 Creating Data Models with Common Format Designer Introduction to Common Format Designer 5-1 What is a Diagram? 5-1 Why assemble datastores and attributes from other models? 5-1 Graphical Synonyms 5-2 Using the Diagram 5-2 Creating a New Diagram 5-2 Create Datastores and Attributes 5-2 Creating Graphical Synonyms 5-3 Creating and Editing Constraints and Filters 5-3 Printing a Diagram 5-4 Generating DDL scripts 5-5 Generating Mapping IN/OUT 5-6 Part III Developing Integration Projects 6 Creating an Integration Project Introduction to Integration Projects 6-1 Oracle Data Integrator Project Components 6-1 Oracle Data Integrator Project Components 6-1 Global Components 6-3 Project Life Cycle 6-3 Creating a New Project 6-3 Managing Knowledge Modules 6-3 Project and Global Knowledge Modules 6-4 Knowledge Module Naming Conventions 6-5 Choosing the Right Knowledge Modules 6-7 Importing and Replacing Knowledge Modules 6-7 Encrypting and Decrypting a Knowledge Module 6-9 Organizing the Project with Folders 6-10 7 Creating and Using Packages Introduction to Packages 7-1 Introduction to Steps 7-1 Introduction to Creating Packages 7-2 v Introduction to the Package editor 7-3 Creating a new Package 7-4 Working with Steps 7-4 Adding a Step 7-4 Adding a Mapping step 7-4 Adding a Procedure step 7-5 Variable Steps 7-5 Adding Oracle Data Integrator Tool Steps 7-6 Adding a Model, Sub-Model or Datastore 7-7 Deleting a Step 7-8 Duplicating a Step 7-8 Running a Step 7-8 Editing a Step's Linked Object 7-9 Arranging the Steps Layout 7-9 Defining the Sequence of Steps 7-9 Running a Package 7-11 8 Creating and Using Mappings Introduction to Mappings 8-1 Parts of a Mapping 8-1 Navigating the Mapping Editor 8-3 Creating a Mapping 8-4 Creating a New Mapping 8-5 Adding and Removing Components 8-6 Connecting and Configuring Components 8-6 Attribute Matching 8-6 Connector Points and Connector Ports 8-7 Defining New Attributes 8-7 Defining Expressions and Conditions 8-8 Defining a Physical Configuration 8-10 Running Mappings 8-10 Using Mapping Components 8-11 The Expression Editor 8-12 Source and Target Datastores 8-13 Creating Multiple Targets 8-13 Specifying Target Order 8-14 Adding a Reusable Mapping 8-15 Creating Aggregates 8-15 Creating Distincts 8-16 Creating Expressions 8-16 vi Creating Filters 8-17 Creating Joins and Lookups 8-18 Creating Pivots 8-22 Example: Pivoting Sales Data 8-22 The Row Locator 8-23 Using the Pivot Component 8-23 Creating Sets 8-24 Creating Sorts 8-25 Creating Splits 8-25 Creating Subquery Filters 8-26 Creating Table Functions 8-27 Creating Unpivots 8-28 Example: Unpivoting Sales Data 8-29 The Row Locator 8-29 Using the Unpivot Component 8-30 Creating Flatten Components 8-31 Using a Flatten Component in a Mapping 8-31 Considerations for using Flatten component with JSON Source 8-32 Creating Jagged Components 8-32 Creating a Mapping Using a Dataset 8-33 Differences Between Flow and Dataset Modeling 8-34 Creating a Dataset in a Mapping 8-34 Converting a Dataset to Flow-Based Mapping 8-34 Physical Design 8-35 About the Physical Mapping Diagram 8-35 Selecting LKMs, IKMs and CKMs 8-37 Configuring Execution Locations 8-38 Moving Physical Nodes 8-39 Moving Expressions 8-39 Defining New Execution Units 8-39 Adding Commands to be Executed Before and After a Mapping 8-39 Configuring In-Session Parallelism 8-40 Configuring Parallel Target Table Load 8-40 Configuring