The Latest in the Database – Metadata Relationship Donna Burbank Global Data Strategy Ltd
Total Page:16
File Type:pdf, Size:1020Kb
The Latest in the Database – Metadata Relationship Donna Burbank Global Data Strategy Ltd. Database Now! Online Conference July 19, 2017 Donna Burbank of business drivers with data-centric latest BI and Analytics software in the technology. In past roles, she has served in market. key brand strategy and product management roles at CA Technologies and She has worked with dozens of Fortune Embarcadero Technologies for several of 500 companies worldwide in the Americas, the leading data management products in Europe, Asia, and Africa and speaks the market. regularly at industry conferences. She has co-authored two books: Data Modeling for As an active contributor to the data the Business and Data Modeling Made Donna is a recognised industry expert in management community, she is a long Simple with ERwin Data Modeler and is a information management with over 20 time DAMA International member, Past regular contributor to industry years of experience in data strategy, President and Advisor to the DAMA Rocky publications. She can be reached at information management, data modeling, Mountain chapter, and was recently [email protected] metadata management, and enterprise awarded the Excellence in Data Donna is based in Boulder, Colorado, USA. architecture. Her background is multi- Management Award from DAMA faceted across consulting, product International in 2016. She was on the development, product management, brand review committee for the Object strategy, marketing, and business Management Group’s (OMG) Information leadership. Management Metamodel (IMM) and the Business Process Modeling Notation She is currently the Managing Director at (BPMN). Donna is also an analyst at the Global Data Strategy, Ltd., an international Boulder BI Train Trust (BBBT) where she information management consulting provides advices and gains insight on the company that specializes in the alignment Follow on Twitter @donnaburbank Global Data Strategy, Ltd. 2017 Today’s hashtag: #DBNow 2 Agenda What we’ll cover today • Emerging Trends in Metadata Management • The Business Value of Metadata Management • Metadata as Part of Wider Enterprise Data Management • Metadata Isn’t Just for Relational Databases Anymore • Technical Innovation & Best Practices in Managing Metadata Global Data Strategy, Ltd. 2017 3 Metadata is Hotter than ever A Growing Trend In a recent DATAVERSITY survey, over 80% of respondents stated that: Metadata is as important, if not more important, than in the past. Global Data Strategy, Ltd. 2017 4 What is Metadata? Metadata is Data In Context Global Data Strategy, Ltd. 2017 5 Metadata is the “Who, What, Where, Why, When & How” of Data Who What Where Why When How Who created this What is the business Where is this data Why are we storing When was this data How is this data data? definition of this data stored? this data? created? formatted? element? (character, numeric, etc.) Who is the Steward of What are the business Where did this data What is its usage & When was this data How many databases this data? rules for this data? come from? purpose? last updated? or data sources store this data? Who is using this What is the security Where is this data What are the business How long should it be data? level or privacy level used & shared? drivers for using this stored? of this data? data? Who “owns” this What is the Where is the backup When does it need to data? abbreviation or for this data? be purged/deleted? acronym for this data element? Who is regulating or What are the technical Are there regional auditing this data? naming standards for privacy or security database policies that regulate implementation? this data? Global Data Strategy, Ltd. 2017 6 Metadata is Needed by Business Stakeholders Making business decisions on accurate and well-understood data 80% of users of metadata are from the business, according to the recent DATAVERSITY survey. Business users often “get” metadata more than IT does! Global Data Strategy, Ltd. 2017 7 Poor Metadata Management Can be Expensive On average organizations waste 56% of UK marketing organizations 15-18% of their budgets dealing say managing data quality is a with data problems. “significant challenge” . Source: Experian Source: UK Marketing Today The US economy loses $3.1 trillion a year due to poor data quality . Correcting poor data quality is a Source: Artemis Ventures In the US, 6.9 billion pieces of mail Data Scientist’s least favorite task, are undeliverable annually because consuming on average 80% of their of address issues . working day Source: Forbes 2016 Source: US Postal Service Global Data Strategy, Ltd. 2017 8 A Very Expensive Example - NASA • On September 23, 1999 NASA lost the $125 million Mars Climate Orbiter spacecraft after a 286-day journey to Mars. • Missing Metadata was the culprit • Thruster data was sent in English units of pound-seconds (lbf s) instead of Metric units of newton- seconds (N s) • This metadata inconsistency caused thrusters to fire incorrectly, sending the craft off course – 60 miles in all (96.56 km). • In addition to the cost of the orbiter were: • Brand and Reputational Damage • Lost Opportunities for research on the Martian atmosphere & climate Global Data Strategy, Ltd. 2017 9 Human Metadata Avoid the dreaded “I just know” • Much business metadata and the history of the business exists in employee’s heads. • It is important to capture this metadata in an electronic format for sharing with others. • Avoid the dreaded “I just know” Part Number is what used to be called Component Number before the Business Glossary acquisition. Metadata Repository Data Models Etc. Global Data Strategy, Ltd. 2017 10 Metadata is Part of a Larger Enterprise Landscape A Successful Data Strategy Requires Many Inter-related Disciplines “Top-Down” alignment with business priorities Managing the people, process, policies & culture around data Leveraging & managing data for strategic advantage Coordinating & integrating disparate data sources “Bottom-Up” management & inventory of data sources Copyright Global Data Strategy, Ltd. 2017 Global Data Strategy, Ltd. 2017 11 Metadata Use Cases – Now & In the Future • Business Use Cases for Metadata area Evolving, according to the DATAVERSITY Emerging Trends survey. • The “Top 5” are changing – Less BI/DW and Software Dev & More Big Data & Data Science • Data Governance, Master Data Management, and Data Quality remain constants Now Future Global Data Strategy, Ltd. 2017 12 Types of Metadata – Now & In the Future • The types of metadata sources being managed are also evolving. • Business Glossaries & Data Warehouses remain constants • Data Quality & Big Data Platform sources are growing Now Future Global Data Strategy, Ltd. 2017 13 Metadata isn’t just for Relational Databases anymore… • There are many Sources and Types of Metadata Relational databases Application Code Data Models Data Transformation / ETL Tools Text Documents Spreadsheets XML Data Quality Tools Open Data Business Process Models Internet of Things (IoT) Business Intelligence (BI) Tools Photos / Images ERP, CRM, and Packed Applications Social Media Big Data platforms COBOL Copybooks Etc.… there are many more Graph Databases Global Data Strategy, Ltd. 2017 14 Relational Database Metadata • The technical structure of a relational database is defined by DDL (data definition language). It describes the structure / schema for how data is stored in a database. • A Glossary or Data Dictionary generally stores the business metadata. Data Technical Metadata Business Metadata CREATE TABLE EMPLOYEE ( employee_id INTEGER NOT NULL, department_id INTEGER NOT NULL, employee_fname VARCHAR(50) NULL, Term Definition employee_lname VARCHAR(50) NULL, An employee is an individual who currently employee_ssn CHAR(9) NULL); Employee works for the organization or who has been recently employed within the past 6 months. CREATE TABLE CUSTOMER ( customer_id INTEGER NOT NULL, A customer is a person or organization who customer_name VARCHAR(50) NULL, has purchased from the organization within Customer customer_address VARCHAR(150) NULL, the past 2 years and has an active loyalty card customer_city VARCHAR(50) NULL, customer_state CHAR(2) NULL, or maintenance contract. customer_zip CHAR(9) NULL); Glossary or Data Dictionary DDL John Smith Global Data Strategy, Ltd. 2017 15 Data Models are a Good Source of Metadata • Data Models are another good source of both business & technical metadata for relational databases. • They store structural metadata as well as business rules & definitions. Technical Metadata Business Metadata Customer Customer_ID CHAR(18) NOT NULL First Name CHAR(18) NOT NULL Last Name CHAR(18) NOT NULL City CHAR(18) NULL Date Purchased CHAR(18) NULL Global Data Strategy, Ltd. 2017 16 ERP, CRM and Packaged Application Metadata • Packaged applications such as CRM and ERP systems (e.g. Salesforce, PeopleSoft, etc.) are typically based on a relational database system. • Therefore, there is important metadata about both the physical table structures as well as the business names & definitions. Technical Metadata Business Metadata Global Data Strategy, Ltd. 2017 17 NoSQL – Key Value Databases • NoSQL Databases are often optimal solutions for flexibility & performance in certain scenarios. • One common NoSQL database is a key-value pair database (e.g. Redis, Oracle NoSQL, etc.) • They can support extremely high volumes of records & state changes per second through distributed processing and distributed storage. • Use cases include: Managing user sessions