PROFESSIONAL PETROLEUM DATA MANAGEMENT ASSOCIATION
Introduction to Master Data Management
Master Data Management Data Quality Integration with PPDM 3.8 Data Management with PPDM 3.8
1 Copyright 2013, PPDM Association. All Rights Reserved LEARNING OBJECTIVES
Understand and discuss Effective Data Management Strategies Understand and discuss Data Governance Understand and discuss Master Data Management Understand and discuss Data Quality Understand and discuss Data Organization and Architecture Understand how to use PPDM 3.8 for MDM
2 Copyright 2013, PPDM Association. All Rights Reserved PROFESSIONAL PETROLEUM DATA MANAGEMENT ASSOCIATION
How Does Data Management Fit into Corporate Life?
Mainstream vs “burden”
3 Copyright 2013, PPDM Association. All Rights Reserved EFFECTIVE DATA MANAGEMENT STRATEGIES
• Effective data management is essentially being able to deliver the data that people need, when they need it, and in the format that they need it • Any data management strategy needs to start with the needs of the business and the key decisions that they make • What information is used to make the decisions? • What data is used to derive that information? • This can then be used to assess the value of data: • Who uses it and what do they use it for? • What impact does it have on business decisions? • This, in turn, will drive decisions around governance, architecture, organization, quality, and security.
4 Copyright 2013, PPDM Association. All Rights Reserved EFFECTIVE DATA MANAGEMENT STRATEGIES
5 Copyright 2013, PPDM Association. All Rights Reserved EFFECTIVE DATA MANAGEMENT STRATEGIES
6 Copyright 2013, PPDM Association. All Rights Reserved EFFECTIVE DATA MANAGEMENT STRATEGIES
7 Copyright 2013, PPDM Association. All Rights Reserved EFFECTIVE DATA MANAGEMENT STRATEGIES
Data Value
Data Master Data Governance Data Mgmt Quality
Data Data Data Organization Architecture Security
People Process
Technology
Effective Data Management
8 Copyright 2013, PPDM Association. All Rights Reserved ELEMENTS OF EFFECTIVE DATA MANAGEMENT STRATEGIES
Data Management Capabilities
Data Data Data Master Data Data Governance Organization Architecture & Metadata Quality
. Data Sizing, . Data Policies . Data Taxonomy Storage & . Data Quality Rules . Data Definitions . Data Standards . Logical Data Movement & Policies . Master Data . Business Data Models Architecture . Data Cleansing . Metadata Ownership . Business Process . Data Retention & Standards Process . Reference Data . Data Workflow Flows Deletion Policies . Compliance Rules . Phys Data Models . Data . Data Stewards . Solution Architects Administration . Business Data Enterprise Data . Data Quality . Business Data . . Storage/ Administration Owners Architects Technical Services Team . Data Stewards . Data Czar or Mgt. . Data Modelers Architects . Data Governance People . Data Owners Committee . DBAs . DBAs
. Archiving Tools . Master Data Mgt. . Data Profiling, Data Rules Library Data Modeling Tools . . . Storage Quality & . Automated Tools Management & . Reference Data
Monitoring Tools Notifications . Design/CASE Hardware Architecture . ETL Tools (Workflow) Tools . Technical . Metadata . Audit Reports Technology Architecture Repository
Copyright 2006 Accenture All Rights Reserved
9 Copyright 2013, PPDM Association. All Rights Reserved PROFESSIONAL PETROLEUM DATA MANAGEMENT ASSOCIATION
Data Governance
10 Copyright 2013, PPDM Association. All Rights Reserved DATA GOVERNANCE
Data Governance is how an organization manages its data assets. • The business rules, policies, practices, procedures, roles, and responsibilities to create a consistent enterprise view of an organization’s data • Data Governance is the foundation of effective data management but is too often neglected • It involves a great deal of work to establish and maintain a comprehensive Data Governance program • The PPDM 3.7/3.8 data management module provides a strong framework for Data Governance
11 Copyright 2013, PPDM Association. All Rights Reserved DATA GOVERNANCE
Legislation Shareholders PPP Policy Upper mgmt
Associations Practice Standards Mid mgmt
People Procedure Docs SW
12 Copyright 2013, PPDM Association. All Rights Reserved DATA GOVERNANCE
• POLICY – We need accountability each master data store to establish a focal point for responsibility. • PRACTICE – We must document what job functions are responsible for data creation, updates and deletes. • PRACTICE – We must document what software applications are responsible for data creation, updates and deletes. • PRACTICE – We must document where in the life cycle of a business asset it is created and used by various functional groups in our organization. • PROCEDURE – all replicated (or derived) copies will be derived from the master store, using an appropriate mechanism for replication management.
13 Copyright 2013, PPDM Association. All Rights Reserved DATA GOVERNANCE
• Business Rules describe database or application procedures that are used to enforce Policies, Practices, and Procedures • These rules help ensure that the data within the database: • Conforms to convention in terms of meaning and loading practice • Is the most trusted data available; the source is known, the history is known, and the quality is consistent and understood • Is maintained consistently over its lifecycle
14 Copyright 2013, PPDM Association. All Rights Reserved BUSINESS RULES
Business rules • Create rules and associate with tables,
columns, procedures etc. PPDM_RULE • Rules enforcement methods R_PPDM_RULE_CLASS Kinds of business rules R_PPDM_RULE_PURPOSE • Policies, Practices, Procedures PPDM_RULE_XREF PPDM_RULE_DETAIL • Quality (trustworthiness) PPDM_RULE_COMPONENT
• Entitlements and security R_PPDM_XREF_TYPE
• Relationships PPDM_GROUP • Constraints (database and data) PPDM_RULE_ENFORCEMENT • Conversions or transformations R_PPDM_ENFORCE_METHOD R_PPDM_FAIL_RESULT • Calculations
15 Copyright 2013, PPDM Association. All Rights Reserved PROFESSIONAL PETROLEUM DATA MANAGEMENT ASSOCIATION
Master Data Management
16 Copyright 2013, PPDM Association. All Rights Reserved WHAT DO WE HOPE TO ACHIEVE?
Any kind of data management is a lot of work.
Data management is not always recognized as a good value proposition by management.
- As a company - As an industry
17 Copyright 2013, PPDM Association. All Rights Reserved THE CHALLENGE TIP: SOURCE SYSTEMS LIE
• Source systems may not be able to accurately reflect the business! • Legacy systems • Assume out dated processes • May flatten or denormalize your data • May not account for regional variations • May not account for temporal variations • May not capture all the important information • May replicate information that they don’t “own” • …. • Just because data is in a legacy system does not mean you should load it into an MDM! • Ask lots of questions; ask everybody! • Don’t be afraid to recommend against loading certain data types!
19 Copyright 2013, PPDM Association. All Rights Reserved
COST VS BENEFIT
What is the cost of bad data? • Use the fear factor to engage management • Threat management succeeds in the short term – sometimes. • Does it really help you to avoid costly errors? What is the value of good data management? • Is data management really a “core” function? • Does it really save time and effort? • Can’t I just leave this to my vendors? Is data an asset? • How much is it worth? • Can data be put “on the books” as a tangible asset? - Will the IFRS help us?
20 Copyright 2013, PPDM Association. All Rights Reserved INTEGRATED ARCHITECTURE
Ontologies Meta Data Taxonomies Physical Spatial/GIS
PPDM
SOR MDM SOR
SOR
Business Rules Policies, Practices & Procedures VALUE OF INTEGRATED ARCHITECTURES
The asset value of core data is Life cycle based management recognised • You know who the “real owners” of • By the users data are • By the data managers Data flows can be identified and managed • By IT staff • Beware the “rogue” user! • By managers Maintenance effort should be Business rules can be applied reduced consistently Integration effort is probably higher! Key data is stored once, used many times! • Managed replication • May require translation, granularity shifts, equivalences …
22 Copyright 2013, PPDM Association. All Rights Reserved SOR ORDER FROM CHAOS
MDM / SOR SOR Warehouse
SOR = System of Record Gold Standard
MDM = Master Data Management
23 Copyright 2013, PPDM Association. All Rights Reserved MASTER DATA MANAGEMENT
Financial Data Physical Data
Spatial Data
HSE Data Unstructured / Semi-structured Data
Operational Structured Data Data 24 Copyright 2013, PPDM Association. All Rights Reserved AN OBJECTIVE: THE DIGITAL OILFIELD
001111000111011101100101011011000110110001011111010010010100010000111110001100010011001000110011001101000011010100110110001101 110011100000111001001100000011110000101111011101110110010101101100011011000101111101001001010001000011111000101100001111000101 001101010000010101010100010001011111010001000100000101010100010001010011111000110010001100000011000000110111010111110011000000 110110010111110011001000110111001111000010111101010011010100000101010101000100010111110100010001000001010101000100010100111110
25 Copyright 2013, PPDM Association. All Rights Reserved OBJECTIVES REALITY
Correct
Complete Current
26 Copyright 2013, PPDM Association. All Rights Reserved WORK SMARTER – NOT HARDER!
2005 study at Institute of Psychiatry • University of London IQ test with distraction is 10 points lower than with no distractions.
100
95
90
85
80 IQ test with distraction is No Smoking Pot Distractions Distractions 6 points lower than those who smoked marijuana. 27 Copyright 2013, PPDM Association. All Rights Reserved ABOUT KNOWLEDGE WORKERS
We have less productive time available University of California study of Knowledge Workers (in the US)
11 minutes between 2.1 hours lost per knowledge worker interruptions! per day to interruptions and recovery time
28,000,000,000 Costs in the US hours / year!! $588 Billion / year Gloria Marks, University of California at Irvine 28 Copyright 2013, PPDM Association. All Rights Reserved WHAT IS MASTER DATA MANAGEMENT (MDM)? Technical MDM • In computing, master data User Interface User Interfaces (Many) management (MDM) comprises a set of processes and tools which consistently define and manage the non-transactional data entities of an organization (also called reference Bus Analysis data). Bus Intel Collaborate • MDM has the objective of providing Decisions processes for collecting, aggregating, matching, consolidating, quality- Technical Life Cycles assuring, persisting and distributing Transact such data throughout an organization in such a way as to ensure Interpret consistency and control in the Master Data (One) ongoing maintenance and application use of this information. PPP • At a basic level, MDM seeks to Bus Rules ensure that an organization does not use multiple (potentially inconsistent) Quality versions of the same master data in Metrics different parts of its operations, which Ref data can occur in large organizations. Source Data (Many) Wikipedia 29 Copyright 2013, PPDM Association. All Rights Reserved WHAT IS MASTER DATA MANAGEMENT?
“Master data management (MDM) comprises a set of processes and tools which centrally and persistently define the non-transactional entities of an organization (also called Reference data). The objective of MDM is to collect from, and supply to various processes, unique instances of each entity.”
People
Technology Process
It’s People, Processes and Technology!
http://en.wikipedia.org/wiki/Master_Data_Management
30 Copyright 2013, PPDM Association. All Rights Reserved WHAT IS DATA GOVERNANCE?
“Data governance encompasses the people, processes and technology required to create a consistent, enterprise view of an organization's data in order to: • Increase consistency & confidence in decision making • Decrease the risk of regulatory fines • Improve data security People • Consistent information quality across the organization • Maximize the income generation potential of data • Designate accountability for information quality.”
Technology Process
It’s People, Processes and Technology! http://en.wikipedia.org/wiki/Data_governance
31 Copyright 2013, PPDM Association. All Rights Reserved WHAT’S THE DIFFERENCE?
Master data management keeps the data centralized, organized, unique
Data governance makes the data worth keeping!
Data governance can exist without MDM,
but the value of good MDM depends on good data governance! 32 Copyright 2013, PPDM Association. All Rights Reserved MASTER DATA MANAGEMENT ARCHITECTURES
Central (coexistence) master, where data coexists as managed replication from a source. Better data quality.
Registry, where the location and identity of the master is located, but most data is not replicated (meta data). Data issues not resolved.
Transaction hub, in which the master takes over the Sometimes underlying function of the source. systems are eliminated Highest data quality is possible.
33 Copyright 2013, PPDM Association. All Rights Reserved ALONG THE WAY
User Interfaces (Many)
Bus Analysis Bus Intel Collaborate Decisions Life Cycles
PPDM / MDM “Single Version of the Truth”
Data Governance Data Quality Business Rules
SOURCE SOURCE SOURCE A B C
34 Copyright 2013, PPDM Association. All Rights Reserved MAINTAINING MASTER DATA
Single Copy • Best but can be expensive • Requires a lot of software changes, vendors may not support Multiple Copies, Single Maintenance Master Data (One) • Data is updated in the master and disseminated to where it’s needed • SOA approach Continuous Merge • Data is updated in the source and propagated to the master for reconciliation and then propagation to other systems • Problems - Concurrent updates: What if two sources change the same entity – how do you deal with at the master level? Source Data (Many) - How do you integrate merged data back into the sources, once two entries have been harmonized? - Units of measure conversions – scale errors
35 Copyright 2013, PPDM Association. All Rights Reserved HOW MDM ARE USED
Collaboration Effective use of an MDM depends on • Create information once, use it many times your ability to define Operational controls (Consistency) content standards • Data flows through many life cycle transactional processes (SOA) • Data is born, modified, used, transformed …. Analytical systems • Identity analytics: Understand the objects critical to our business - What wells do I have an interest in • Analytics on master data: Improve operational efficiencies - Which wells have critical deadlines coming up? - Which pumps need to be changed out, so they don’t fail in the well bore? • Analytics integration - With warehouses, technical applications … - Improve the quality of data provided to other systems to improve their analytic capability
36 Copyright 2013, PPDM Association. All Rights Reserved USUAL COMPONENTS OF AN MDM (NOUNS)
People • Who are the people and organizations that we do business with? • Can I be unambiguous? Places • What is a country? (geographic, geopolitical) Things • What are the assets of the organization? • How do they relate to each other? Concepts • What contracts, interest sets, licenses are in place? • Am I managing these properly? http://msdn.microsoft.com/en-us/library/bb190163.aspx 37 Copyright 2013, PPDM Association. All Rights Reserved TO GET SEMANTIC CLARITY YOU MUST DEVELOP OR USE STANDARDS!
http://msdn.microsoft.com/en-us/library/bb190163.aspx 38 Copyright 2013, PPDM Association. All Rights Reserved
WHAT BELONGS IN THE MDM?
How long How itis important? of sharing common data? Value
How often and how much does it change? Where does your information sit?
http://msdn.microsoft.com/en-us/library/bb190163.aspx 39 Copyright 2013, PPDM Association. All Rights Reserved PROFESSIONAL PETROLEUM DATA MANAGEMENT ASSOCIATION What MDM Brings to your Information Architecture
Opportunities for improvement
40 Copyright 2013, PPDM Association. All Rights Reserved INTEGRATED COMMUNICATIONS
41 Copyright 2013, PPDM Association. All Rights Reserved DATA’S ASSET VALUE AND LIFE CYCLE
6 Planning Operations Administration 5
4
3 Value
2
1
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Time
42 Copyright 2013, PPDM Association. All Rights Reserved AGGREGATION HIERARCHIES
These may not exist in a source system!
Reporting Hierarchy Hierarchy Type (Specific) (General) Level 1 = COUNTRY CANADA
Level 2 = PROVINCE ALBERTA SASK MANITOBA
LEVEL 3 = FIELD Organizational RED GREEN BLUE ORANGE FIELD FEILD FIELD FIELD Hierarchies Resent Contributions to a Reporting 50% 85% 60% 50% 25% Hierarchy 50% 15% 40% 25%
RESENT A RESENT C RESENT E RESENT G RESENT I
RESENT B RESENT D RESENT F RESENT H
43 Copyright 2013, PPDM Association. All Rights Reserved THE IMPORTANCE OF VERSIONING
What did this data look like in the source? What rules did I use to harmonize my sources? • Granularity issues • Blending data If the value has changed, who made the change, and why? • Authentication • Provenance
Process standards are critical
44 Copyright 2013, PPDM Association. All Rights Reserved CONTENT – THE HARD BIT
Units of measure Organizational hierarchies Geographic areas identification and definition These are content standards! Well status • Corporate Products and substances • Industry • Global What is a well, anyway?
Company names and identities People and their roles
Create the set that works for the business and map the sources to that set Don’t force users to a single layer hierarchy – reality is a multidimensional business
45 Copyright 2013, PPDM Association. All Rights Reserved Phase 1 may be a small part of the LIFE CYCLE MANAGEMENT business
Acquire Land Rights Identify Opportunity Well Planning
Form Partnerships Interpretation
Drilling and Completion
Facility Exploration Data Management Management
Well Management
Seismic Exploration
Production Tracking
Decommission / Relinquish 46 Copyright 2013, PPDM Association. All Rights Reserved PROFESSIONAL PETROLEUM DATA MANAGEMENT ASSOCIATION Components of an MDM Strategy
47 Copyright 2013, PPDM Association. All Rights Reserved CHALLENGES AND COMPONENTS
People
Technology Process
48 Copyright 2013, PPDM Association. All Rights Reserved MDM CHALLENGES
Data ownership (who owns the data) Strategic vision (silo view master view)
Change management (how are changes to bad Performance management data propagated?) (fear, as MDM exposes People problems in process)
Data quality management (what is Transition planning (keep it, what are the rules?) working while you move forward) Where does bad data get fixed and how is the fix propagated?
Performance (speed of Technology Process queries, latency for data Data standards (semantics, what corrections to be happens when data that has been propagated) integrated has to be distributed back to the source? Meaning may be lost) What is the architecture going to be? Integration (how does the data move around?)
49 Copyright 2013, PPDM Association. All Rights Reserved PEOPLE COMPONENTS
In house users Project leaders Integrators Regulators
Data managers Software engineers Partners MDM team Governance leaders Public DBAs
Business users Computer Evangelists engineers IT Specialists People Network engineers
50 Copyright 2013, PPDM Association. All Rights Reserved PROCESS COMPONENTS
Business Design (the objectives)
Technical Design (the vision / plan) Define the process improvement goals Situation analysis Shewart Cycle / Deming Wheel Gaps Measure the current process Opportunities Process
Set Objectives and Priorities Analyze needs
Six Sigma Data collection (identify and harvest) Improve or optimize the Normalization and process reconciliation (reference codes)
Control to Data Transformation correct (mapping / business rules) variances
Data governance (process controls)
51 Copyright 2013, PPDM Association. All Rights Reserved TECHNOLOGY COMPONENTS
Flat files
Corporate Databases Data and Information Data warehouses and Storage Datamarts
Operational data store
Data storage (hardware)
Technology Data mining
Business Data visualization Intelligence
Data analysis
GIS
Service Oriented Architecture
Networks Communication Real time feedback (Digital Oilfield)
52 Copyright 2013, PPDM Association. All Rights Reserved PROFESSIONAL PETROLEUM DATA MANAGEMENT ASSOCIATION
MDM in PPDM 3.8
53 Copyright 2013, PPDM Association. All Rights Reserved MASTER DATA MANAGEMENT
• The Well tables are being widely adopted within MDM
solutions. WELL
• This can be of tremendous benefit to industry as WELL_XREF WELL_ALIAS WELL_VERSION operators demand that companies that they work with adhere to the same standard
54 Copyright 2013, PPDM Association. All Rights Reserved MAPPINGS AND MAPPING RULES
Persist mappings in PPDM • Not in a spreadsheet Mappings PPDM_SYSTEM_MAP • Database to database or schema PPDM_MAP_DETAIL • Schema to database or schema
• Rule driven PPDM_MAP_RULE • PPDM mappings will be released in the sample data
PPDM PPDM 3.7.1 3.8
55 Copyright 2013, PPDM Association. All Rights Reserved MAPPING LEVELS
PPDM_SYSTEM_MAP PPDM_MAP_DETAIL
Which systems are you mapping?
• system to system • table to table • column to column • schema to column • information to be created (not mapped) • order to handle
56 Copyright 2013, PPDM Association. All Rights Reserved MAPPING RULES
PPDM_MAP_RULE • How is this mapping connected with mappings to other columns? • If a value is created, how is it created? • What are the min and max values that are acceptable? • If the condition is expressed procedurally, where is the code that validates? • If the condition is dependent on the value of another column, which one. • How are dates formatted? • What order to I process the rules in? • What version of the rule is this? • Is this the preferred rule? • What rule did I used last time I did a conversion? 57 Copyright 2013, PPDM Association. All Rights Reserved REFERENCE VALUES
Support for sophisticated reference PPDM_SYSTEM behaviour • Multiple sources • Hierarchy and granularity CLASS_LEVEL_COMPONENT PPDM_TABLE • Equivalences
• Cross Referencing PPDM_CODE_VERSION PPDM_COLUMN Sandbox to prepare reference values for use PPDM_CODE_VERSION_COLUMN
PPDM_CODE_VERSION_USE
PPDM_CODE_VERSION_XREF
PPDM_GROUP
R_CODE_VERSION_XREF
58 Copyright 2013, PPDM Association. All Rights Reserved PROFESSIONAL PETROLEUM DATA MANAGEMENT ASSOCIATION
Meta Data in PPDM 3.8
59 Copyright 2013, PPDM Association. All Rights Reserved META MODEL – TABLE GROUPINGS
• PPDM_SYSTEM, PPDM_TABLE, PPDM_COLUMN
• PPDM_GROUP, PPDM_GROUP_OBJECT
• PPDM_CONSTRAINT, PPDM_CONS_COLUMN
• PPDM_SYSTEM_MAP, PPDM_MAP_DETAIL, PPDM_MAP_RULE
• PPDM_RULE, PPDM_RULE_DETAIL
• PPDM_EXCEPTION
• This will be detailed in Meta Model course
60 Copyright 2013, PPDM Association. All Rights Reserved
PROFESSIONAL PETROLEUM DATA MANAGEMENT ASSOCIATION
Data Quality You cannot inspect quality into your data!
You can’t prove data is right. You can only prove it is not wrong.
61 Copyright 2013, PPDM Association. All Rights Reserved EFFECTIVE DATA MANAGEMENT STRATEGIES
Data Value
Data Master Data Governance Data Mgmt Quality
Data Data Data Organization Architecture Security
People Process
Technology
Effective Data Management
62 Copyright 2013, PPDM Association. All Rights Reserved DATA QUALITY
Data Quality • Data Quality is fundamental to improving business workflows and decision-making processes • Billions of $ are lost annually in terms of inefficiency, poor decisions, and missed opportunities • Data quality initiatives are attracting a great deal of attention across all industries and within oil and gas in particular • However, quality programs can be overwhelming in scope, very expensive, and hampered by a scarcity of qualified resources • It is vital to focus available resources on addressing those data problems that have the most potential impact on the business through an analysis of data value
63 Copyright 2013, PPDM Association. All Rights Reserved DATA QUALITY • Using our Data Value approach each data element is provided with a general value ranking on a scale of 1 to 4 • Level 1: Critical • Level 2: Important • Level 3: Useful • Level 4: Supportive • These value rankings can then be represented within the PPDM data model. • One approach to this would be to use the PPDM_GROUP tables to store value • As an example, the GROUP_NAME (in PPDM_GROUP) could be ‘Critical’ • Then each attribute falling in this category would be defined by TABLE_NAME and COLUMN_NAME in PPDM_GROUP_OBJECT
PPDM_GROUP
PPDM_GROUP_OWNER PPDM_GROUP_OBJECT
PPDM_GROUP_XREF PPDM_GROUP_REMARK 64 Copyright 2013, PPDM Association. All Rights Reserved GROUPS
Link objects in PPDM together to PPDM CODE support business processes VERSION In PPDM 3.8 Groups are used to associate tables into modules (52 PPDM TABLE subject areas) PPDM COLUMN • Core tables • Referenced tables PPDM GROUP • Supports the on-line documentation, which is divided by module PPDM PPDM RULE PROCEDURE • Supports the modular DDL Generator (on hold) PPDM PROPERTY
65 Copyright 2013, PPDM Association. All Rights Reserved DATA QUALITY
• Using the same scale, value rankings could then be established for the different dimensions of Data Quality: • Accuracy • Currency • Timeliness • Consistency • Completeness • Standards • Again, the PPDM_GROUP tables could be used to store the value for each quality dimension for the different attributes • In this case, GROUP_NAME could be ‘Critical’ and GROUP_TYPE could be set to ‘Accuracy’ as an example. • Then each attribute falling in this category would be defined by TABLE_NAME and COLUMN_NAME in the PPDM_GROUP_OBJECT_TABLE
66 Copyright 2013, PPDM Association. All Rights Reserved DATA QUALITY
67 Copyright 2013, PPDM Association. All Rights Reserved DATA QUALITY
• Data Quality Rules for capturing and maintaining the attributes are then developed from the value rankings • Value I (Critical) data: Accuracy: Well location data should be validated from at least 2 independent sources Completeness: 100% of attributes should be populated Timeliness: New bid rounds should be entered within 5 days of announcement • Value II (Important) data: Completeness: 95% of attributes should be populated • …….
68 Copyright 2013, PPDM Association. All Rights Reserved DATA QUALITY
PPDM_RULE PPDM_PROCEDURE
SYSTEM_ID SYSTEM_ID RULE_ID PROCEDURE_ID ACTIVE_IND ACTIVE_IND COLUMN_NAME EFFECTIVE_DATE EFFECTIVE_DATE EXPIRY_DATE PPDM_RULE EXPIRY_DATE PPDM_GUID PPDM_GUID PROCEDURE_DESC REMARK PROCEDURE_NAME RULE_CLASS PROCEDURE_TYPE RULE_DESC REMARK RULE_QUERY SOURCE P R_P P _FK RULE_PURPOSE TABLE_NAME SOURCE ROW_CHANGED_BY PPDM RULE allows you to keep track of TABLE_NAME ROW_CHANGED_DATE USE_CONDITION_DESC ROW_CREATED_BY USE_CONDITION_TYPE ROW_CREATED_DATE the data management rules that you apply ROW_CHANGED_BY ROW_QUALITY ROW_CHANGED_DATE ROW_CREATED_BY ROW_CREATED_DATE to creating, managing and using your ROW_QUALITY data. A good example of a business rule PROCEDURE_ID would be “If you populate a well location’s
latitude, you should also populate a
PRX_PR_FK
PRX_PR_FK2
PR_R_PRC_FK PRD_PR_FK
longitude value and the coordinate PR_R_PRP_FK PPDM_RULE_XREF
R_PPDM_RULE_CLASS PPDM PROCEDURE can PRG_PR_FK system reference”. SYSTEM_ID RULE_ID PPDM_RULE_DETAIL be used to store the R_PPDM_RULE_PURPOSE SYSTEM_ID2 RULE_ID2 XREF_OBS_NO SYSTEM_ID procedure or a pointer to a ACTIVE_IND RULE_ID PPDM_RULE_GROUP DESCRIPTION These rules can be expressed in plain DETAIL_SEQ_NO EFFECTIVE_DATE ACTIVE_IND procedure (or trigger, or EXPIRY_DATE AVERAGE_VALUE GROUP_ID PPDM_GUID narrative language or in structured form, AVERAGE_VALUE_OUOM SYSTEM_ID REMARK AVERAGE_VALUE_UOM RULE_XREF_TYPE script, or function, or RULE_ID BOOLEAN_RULE ACTIVE_IND SOURCE as procedures or SQL. PRE_PR_FK BUSINESS_RULE EFFECTIVE_DATE ROW_CHANGED_BY EFFECTIVE_DATE EXPIRY_DATE ROW_CHANGED_DATE application etc) that is used EXPIRY_DATE PPDM_GUID ROW_CREATED_BY MAX_DATE REMARK ROW_CREATED_DATE MAX_VALUE SOURCE ROW_QUALITY to enforce the rule. MAX_VALUE_OUOM ROW_CHANGED_BY MAX_VALUE_UOM ROW_CHANGED_DATE MIN_DATE ROW_CREATED_BY MIN_VALUE ROW_CREATED_DATE MIN_VALUE_OUOM ROW_QUALITY MIN_VALUE_UOM PPDM_GUID REFERENCE_SYSTEM_ID
REFERENCE_TABLE_NAME PRX_R_PRXT_FK REFERENCE_TABLE_NAME2 PPDM_RULE_ENFORCEMENT R_PPDM_RULE_XREF_TYPE REFERENCE_COLUMN_NAME REFERENCE_COLUMN_NAME2 REFERENCE_VALUE PPDM_PROCEDURE REFERENCE_VALUE_OUOM
PRG_PG_FK REFERENCE_VALUE_TYPE REFERENCE_VALUE_UOM PPDM_RULE_ENFORCEMENT REMARK PPDM_GROUP RULE_DETAIL_TYPE SOURCE PPDM RULE ENFORCEMENT SYSTEM_ID RULE_DESC RULE_ID GROUP_ID ROW_CHANGED_BY ENFORCEMENT_ID ACTIVE_IND ROW_CHANGED_DATE describes how the rule is ABBREVIATION ROW_CREATED_BY DEFAULT_UNIT_SYSTEM_ID ACTIVE_IND EFFECTIVE_DATE ROW_CREATED_DATE APPLICATION_ID EXPIRY_DATE ROW_QUALITY EFFECTIVE_DATE enforced, such as manually GROUP_NAME ENFORCE_METHOD GROUP_TYPE EXPIRY_DATE PPDM_GUID LONG_NAME during a load, or by insert trigger REMARK OWNER_BA_ID SOURCE PPDM_GUID ROW_CHANGED_BY PROCEDURE_ID or by monthly QC run etc. ROW_CHANGED_DATE REMARK R_PPDM_ENFORCE_METHOD ROW_CREATED_BY PRE_R_PR EM_FK SHORT_NAME ROW_CREATED_DATE SOURCE You can also define the owner of ROW_QUALITY ROW_CHANGED_BY ROW_CHANGED_DATE ROW_CREATED_BY this enforcement method, or the ROW_CREATED_DATE application or procedure used to ROW_QUALITY enforce the process.
69 Copyright 2013, PPDM Association. All Rights Reserved DATA QUALITY: METRICS
70 Copyright 2013, PPDM Association. All Rights Reserved METRICS
PPDM_METRIC Data Loading Progress • Measured benchmarks PPDM_METRIC_COMPONENT
• Database loading and QC PPDM_METRIC_VALUE • Link to objects in PPDM
3000 Jan Feb 2500 Mar Apr 2000
1500
1000
500
0 Not Done Prepared Loaded QC Initial QC Verify
71 Copyright 2013, PPDM Association. All Rights Reserved QUALITY CONTROL AND AUDIT HISTORIES
How good is my data, and what have I done to validate it? Quality control by table, column, row • Status history, process… Audit history • Column and row level (full audit-ability) • Table level (usually deletes)
PPDM_TABLE_HISTORY
PPDM_AUDIT_HISTORY PPDM_AUDIT_HISTORY_REM
PPDM_QUALITY_CONTROL
72 Copyright 2013, PPDM Association. All Rights Reserved QUALITY CONTROL
Who checked this value? When was it checked? How many checks have been completed? If the value has changed – what is the current QC status for the current value? What is the quality of this value? • How trustworthy is it? If the value is NULL, why is it NULL and is that OK? Who authorized the change? Who actually did it? How long should I keep this information?
73 Copyright 2013, PPDM Association. All Rights Reserved AUDIT HISTORIES
What was the original value? • Specific column for each data type What is the new value? • Specific column for each data type Where did the new information come from? Why was the information changed? How long should I keep this information’
74 Copyright 2013, PPDM Association. All Rights Reserved DATA QUALITY: WORKFLOW SUMMARY
Business Workflows
Decision Points Business Data Requirements Analysis
Data Value
Data Quality
Data Business Rules Analysis (PPDM) Metrics
Fix/Audit
75 Copyright 2013, PPDM Association. All Rights Reserved PROFESSIONAL PETROLEUM DATA MANAGEMENT ASSOCIATION Data Organization and Architecture
76 Copyright 2013, PPDM Association. All Rights Reserved EFFECTIVE DATA MANAGEMENT STRATEGIES
Data Value
Data Master Data Governance Data Mgmt Quality
Data Data Data Organization Architecture Security
People Process
Technology
Effective Data Management
77 Copyright 2013, PPDM Association. All Rights Reserved DATA ORGANIZATION
• Data organization refers to how data is structured within a company and the meaning assigned to that data • What something means to a geologist may be different to what it means to a reservoir engineer or, more so, to the financial people • PPDM has performed a lot of work to establish standard meanings to key data types and to define standard taxonomies • The ‘What is a Well?’ project has established a baseline for the definition of well components
78 Copyright 2013, PPDM Association. All Rights Reserved DATA ORGANIZATION
Ontologies Meta Data Taxonomies Physical
KID
Work Data Management Rules Policies, Practices & Procedures
79 Copyright 2013, PPDM Association. All Rights Reserved DATA ARCHITECTURE
Data Architecture • Data Architecture refers to the people, processes, and technology required to gather, store, maintain, and distribute data
Accum/ • Data must be Acquisition managed effectively Archival/ Deletion/ Storage across its full Renewal lifecycle to meet the
requirements Transform- Usage of the business ation
80 Copyright 2013, PPDM Association. All Rights Reserved DATA ARCHITECTURE
Data is Integrated Often PPDM Warehouse standards
based GIS GIS Systems Data is Managed As an Asset Combination System of PPDM standards SystemSystem of Record based and RecordRecord proprietary Data is Master Data Updated or Deleted
Often Source Source proprietary SystemSource SystemSource systems, with SystemSource Data is SourceSystem vendor System Created System developed data stores
81 Copyright 2013, PPDM Association. All Rights Reserved PROFESSIONAL PETROLEUM DATA MANAGEMENT ASSOCIATION
Summary
82 Copyright 2013, PPDM Association. All Rights Reserved WHAT WE TALKED ABOUT
• Data management is a fundamental part of business • It’s not about IT • Understand the business (what it really is) • Understand how processes and SW damage data • Master data management adds value • Takes thought to plan and execute • It’s very hard to do, but it’s worth it • Master data management strategies and components • MDM without Data Governance may not add value • You can’t prove your data is right • But you can make a decision about whether to trust it
83 Copyright 2013, PPDM Association. All Rights Reserved PROFESSIONAL PETROLEUM DATA MANAGEMENT ASSOCIATION
Questions
84 Copyright 2013, PPDM Association. All Rights Reserved