<<

Track and improve Quality using SAP Agile Data Preparation

Ginny Segbers, Manager Conversion and Data Readiness, Duke Energy Michael Eacrett, VP Product Management, SAP Session ID 84169

May 7 – 9, 2019 About the Speakers

Ginny Segbers Michael Eacrett • Manager Conversion and Data • VP Product Management, SAP Data Readiness, Duke Energy Hub and EIM • Experienced technical developer • Previously lead for SAP HANA PM and and business manager, 28 years was a member of the SAP NetWeaver with Duke Energy, 10 years in PM. 27 years of SAP product, the web space, 4th year on implementation, and customer Customer Connect project experience • Math major with art background • Lack of art and math talent = a computer science degree Key Outcomes/Objectives

1. What is SAP Agile Data Preparation 2. How can it be used in my company 3. Accelerate your data trustworthiness 4. From business to action Agenda

• SAP Agile Data Preparation Overview • Duke Energy Overview • Project Presentation • Next Steps • Q&A SAP Agile Data Preparation Overview

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 5 Multi-user-oriented, visual, interactive data prep

Drive User Collaboration Discover Data in a New, Enable for and Sharing More Agile Way Better Analysis From IT generated data to From data gathering to From data nightmares to multiple-user generated data data exploration beautiful storytelling

Driven by digital business transformation

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 6 Business Users Increasing Need

10% of organizations have adopted some form of self-service data preparation, which is estimated to grow to 30% of organizations by 20201

Through 2020, spending on self-service visual discovery and data preparation market will grow 2.5x faster than traditional IT-controlled tools for similar functionality2

20% of global data and analytics decision-makers said the lack of end user self-service capabilities is the biggest challenge with their BI strategy1

1 Gartner: Forecast Snapshot: Self-Service Data Preparation, Worldwide, 2016; 2. IDC: Self-Service Data Preparation Market Supply” 2016; 3. Forrester: “Vendor Landscape: Data Preparation Tools” 2016

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 7 Simplifying the Way you Discover and Shape Data

SAP Agile Data Preparation is a self-service data preparation application providing data discovery, integration, and transformation capabilities. Quickly transform your data into actionable, easily consumable information. Built for all types of users – the software can help you drive more successful analytics, , and master (MDM) initiatives. Simplify how you access and discover the shape of data and become far more productive and agile than you ever dreamed.

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 8 SAP Agile Data Preparation ?

SAP Agile Data Preparation is a self-service data preparation application that helps multiple user roles explore, integrate and • Perform data discovery transform raw data into actionable information. It provides users and profiling with greater agility to respond to new data sources, business • Enrich, blend and requirements and market opportunities. augment data • Harmonize disparate data sources Combine data Transform data Share data • Share and collaborate sources and with one-click discoveries prepared data sets find hidden fixes to ensure and take patterns data trust action

Export prepared data sets to any analytical tool, including SAP Analytics & BI solutions

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 9 SAP Agile Data Preparation Powered by SAP HANA Platform

SAP HANA , access, and quality services: Simplified landscape, lower TCO, in-memory speed, common metadata SAP Agile Data Preparation repository, integrated modeling environment, simple UI, for cloud or on premise.

◼Applications that leverage the SAP HANA Information SAP HANA Management Option reap all the benefits of a simplified, unified smart development platform with in-memory performance DQ Assessment Cleansing ◼SAP HANA smart data quality: information management Matching Best Record requirements (cleanse, match, best record, metadata and Metadata & 3rd Party semantics, enrichment, etc.) with in-built HANA services Semantics Enrichments ◼SAP HANA enterprise semantic services supports semantic enterprise semantic services Packaged together as Dataset search of datasets by business users Dataset discovery relationships SAP HANA Information Management Option smart data integration SAP HANA smart data integration: Supports real-time replication, Real time Data physical bulk/batch data movement, and federation in a unified provisioning Transformations framework. Support for both on-premise and cloud sources, with built-in adapters for common sources and an open and easily On Premise and Cloud sources of data extensible SDK for ecosystem to offer custom adapters. Relational, semi-structured, and unstructured

Out of the box Custom adapters adapters

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 10 SAP Agile Data Preparation Multiple User Roles

& Scientist

◼IT

◼Business Analyst

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 11 SAP Agile Data Preparation Data Preparation Key Capabilities

1 SAP Agile Data Preparation Ingest data from variety of SAP BW sources Analyst, Scientist, Combine Steward 2 Explore Discover and profile the SAP Business data Suite Shape 3 Enrich Combine, shape, enrich, SAP HANA or cleanse data ingest Clean distribute 4 De-dup. Output data for SAP HANA downstream uses

5 IT Data Steward assess and Govern Delimited Files improve the data quality Delimited 6 Files IT govern, analyze and optimize user processes

Sources Prepare Targets

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 12 SAP Agile Data Preparation Data Acquisition

1 SAP Agile Data Preparation can acquire data from any remote Ingest data from variety of sources sources that HANA Smart Data Integration can access including NoSQL databases. 2 It is also possible to acquire data from flat files such as csv, Discover, profile data excel or XML files.

3 Combine, shape, enrich, or cleanse data

4 Output data for downstream uses

5 IT Governance team - analyze and optimize user processes

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 13 SAP Agile Data Preparation Data Profiling

1 SAP Agile Data Preparation automatically detects the content Ingest data from variety of sources type of your data (such as Addresses, Firms, Peoples, SSN...) as well as profile your data to easily understand them and 2 highlight data quality issues you need to focus on (minimum and Discover, profile data maximum value, % of null, blank, distinct values or patterns.

3 Combine, shape, enrich, or cleanse data

4 Output data for downstream uses

5 IT Governance team - analyze and optimize user processes

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 14 SAP Agile Data Preparation &

1 SAP Agile Data Preparation allows to easily transform your data Ingest data from variety of sources in a single click such as trim, replace, combine, split, data type conversions. But also allows to compute aggregates or formulas 2 without writing a single line of code. Finally you can merge or Discover, profile data append multiple sources in a single action.

3 Combine, shape, enrich, cleanse data

4 Output data for downstream uses

5 IT Governance team - analyze and optimize user processes

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 15 SAP Agile Data Preparation Data Quality

1 Ingest data from variety of sources SAP Agile Data Preparation gives the ability to cleanse and remove duplicates in your data based on content types in few 2 seconds. All cleansing and de-duplication actions comes with Discover, profile data debriefing results that guides you on the next step that needs to be performed. 3 Combine, shape, enrich, cleanse data

4 Output data for downstream uses

5 IT Governance team - analyze and optimize user processes

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 16 SAP Agile Data Preparation Data Distribution

1 SAP Agile Data Preparation provide the ability to export your Ingest data from variety of sources results in HANA (HANA tables or HANA Calculation Views) as well as flat files (csv files or Excel files). Many functionalities also 2 provide the capability to export the results for further analysis or Discover, profile data action such as de-duplicate records, rule based failing records…

3 Combine, shape, enrich, cleanse data

4 Output data for downstream uses

5 IT Governance team - analyze and optimize user processes

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 17 SAP Agile Data Preparation IT Governance & Monitoring

1 SAP Agile Data Preparation allows ITs to govern the users Ingest data from variety of sources accesses and monitor their activities to optimize the HANA platform usage by allowing to control users, roles and data 2 sources access and by allowing to monitor dataset usages, the Discover, profile data memory consumption and scheduling activity.

3 Combine, shape, enrich, cleanse data

4 Output data for downstream uses

5 IT Governance team - analyze and optimize user processes

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 18 Duke Energy Overview

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 19 About Duke Energy

▪ 150+ years of service ▪ 7.6 million electric customers in six states: North Carolina, South Carolina, Florida, Indiana, Ohio, Kentucky ▪ Electric service area’s estimated population is 24 million people ▪ 1.6 million natural gas customers ▪ Fortune 125 company ▪ Stock has paid cash dividends for 92 consecutive years ▪ Listed on:

▪ Fortune’s 2018 “World’s Most Admired Companies” list

▪ Forbes’ 2018 “America’s Best Employers” list 20

▪ Dow Jones Sustainability Index

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 20 Customer Connect

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 21 Customer Connect will Transform the Customer Experience Customer Connect is foundational to transforming the customer experience: ▪ Customer engagement platform that ensures we deliver universal, CUSTOMER METER-TO-CASH ENGAGEMENT simple and consistent experiences across channels based on retail industry models ▪ Core meter-to-cash platform that evolves with the market Solution DATA & ANALYTICS

Components ▪ Integrated operational/analytics platform that enables us to personalize experiences and serve our customers as individuals

Customer Connect will help enable our evolution into a more customer-centric organization.

DATA & ANALYTICS

CUSTOMER ENGAGEMENT

METER-TO-CASH

2018 2019 2020 2021 2022

Release 1 Release 2 & 3 Release 4 Release 5-8 Early Analytics Customer Engagement Universal Bill Core Solution © 2019 SAP SE or an SAP affiliate company. All rights reserved. 22 Leveraging the power and relevancy of SAP

Customer • Omni-channel B2B / B2C customer engagement platform Engagement • Capabilities – Marketing, Sales, Service, Commerce, Billing

Marketing

Core • 785 utilities worldwide, nearly 4x SAP’s nearest competitor SALES SELF MKT COMM Meter to Cash • 12 utilities with customers > Duke Energy, largest = 30 Solution million CUSTOMER PROFILE • #1 Customer Information System (“CIS”) ranking by Gartner / Utilipoint over last 10 years M2C NON REG

DATA Integrated • State-of-the-art in-memory transactional / analytics Analytics • Dramatically improved processing speed of large data sets • Real-time, predictive analytics tightly integrated with solution

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 23 Conversion and Data Readiness Team

1. Conversion Objective Artifacts and Deliverables At each Release’s Go-Live Weekend, run all the conversion programs to bring the required • Conversion Strategy • Development Standards data from legacy systems into SAP Solution. We must be very confident that the data • Conversion RICEFW • Designs & Data Maps brought over is complete, accurate, usable, and that our programs finish on time. Inventory • Conversion Workplan • Mock Conversion Plan • Conversion Resource • Key Conversion Scope Plan 2. Controls Objective Decisions • Data Protection Demonstrate the success of conversion by automating a set of completeness and • Control Specifications Approach for Conversion accuracy controls to measure if every object is converted with the right record count and • Go-Live Conversion Process every critical field is converted with the right value. Controls

3. Objective • Cleansing Strategy • Data Cleansing dev Manage the cleansing lifecycle of identifying, profiling, cleansing, and monitoring any data • Data Issue Inventory standards quality issues that impact conversion, solution functionality, or business outcomes. • Data Cleansing One- • Cleansing Workplan Pagers • Cleansing Resource Plan • Data Cleansing Specs 4. Manual Conversion Objective • Manual Conversion Strategy Ensure conversion goes smoothly by tracking and correcting any data exceptions post- • Manual Conversion Inventory conversion. Manage the inventory of required data entry or correction tasks and coordinate (with controls and timings) the execution of every task in that inventory • Manual Conversion Specs/Guides • Manual Conversion Workplan 5. Support • Manual Conversion Resource Plan Provide other teams with what they need to succeed, including: • Convert data into QA so Test Teams can run their test scripts • Convert data into TRN so OCM can run their training exercises • Generate cross reference reports so interfacing apps know how to trace accounts

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 24 Data Cleansing Roadmap

De-duplication First 10 Data Issues 3 Waves of Issues Plan B and Freeze Apr - Jun 2018 Jul 2018 - Mar 2019 Apr 2019 - Sep 2020 Oct 2020 - Jan 2021 Set up data cleansing team & Expand process Expand process Expand process processes

Cleanse critical Cleanse Customer Cleanse Top Data Cleanse Wave ORT discovered De-Duplication Issues Issues issues

Compile data issue Gather new issues Gather new issues Define and sign off inventory from what’s from sprints from project Plan B’s for known today outstanding issues

Pick Wave 1 Issues Pick Top Data Issues Define Plan B’s for for next iteration. for next iteration. outstanding issues Prepare. Prepare. Freeze cleansing

Set up initial people / process / Iterate to tackle inventory of Cleanse what’s critical for ORT technology to tackle Customer De- Expand to tackle First Data project issues. Aim to optimize Define and sign off Plan B’s for Duplication. Issues. Start production cleansing. lifecycle processes. other outstanding issues. Freeze cleansing.

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 25 Data Cleansing Lifecycle

• Identify a data issue • Evaluate data in legacy • Define business rule systems • Consult with Program • Conduct RCA Functional Team Leads • Develop Remediation Plan

Identify Profile

Monitor Cleanse

• Develop scorecards and monitor • Develop and run scripts cleansing progress against DB2 tables (IT) or • Monitor on-going state of the data • Perform manual updates • Work with business owners to (Business) or address data cleansing gaps • Update using robotics

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 26 Architecture

Data Provisioning Meter Export Results (Oracle) SAP Agile Data Preparation Excel Data Provisioning CSV Preference s (SQL) SAP HANA smart data quality DQ Assessment Cleansing

Matching Best Record

rd 90s Data Replication Metadata & 3 Party - Semantics Enrichments enterprise semantic services Dataset Dataset discovery

relationships c.1980s smart data integration Real time Data provisioning Transformations

4 separate, On Premise and Cloud sources of data Relational, semi-structured, and unstructured

homegrown systems Out of the box Custom adapters developed with over adapters 1,100 interfaces

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 27 Transforming Customer Data with SAP Agile Data Preparation ~634,000

SAP Agile Data Preparation is giving Duke Energy the ability to scale on data readiness by ‘Business Partner records targeted for empowering the business to address the right data issues at the right time consolidation Challenges and Opportunities ▪ Four separate CIS databases and other disparate systems (i.e. Meter Tracking, Customer Choice) ▪ Millions of records to be analyzed and evaluated ~4M

▪ Variety of platforms (i.e. DB2, Oracle, Hana) Items identified for cleansing in first 6 months

Advantages to Agile Data Preparation ▪ Full integration with SAP HANA 210% ▪ User-friendly, self-service data management, reducing reliance on IT and streamlining Cleansing analysis to plan ▪ Scalable self-service data preparation, and discovery ▪ Quickly identify patterns and outliers ▪ Create and re-use rules for analysis and scorecard development 1.5 months ▪ Easily export results in user-friendly, consumable formats To get SAP Agile Data Preparation up and running – from development to “SAP Agile Data Preparation is improving our data governance challenges. Our data production, deployment, and training is cleaner and analysts can collaborate more easily, which improves business outcomes” Ginny Segbers, Manager, Conversion & Data Readiness, Duke Energy

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 28 Next Steps

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 29 Embedded Self Service Data Preparation within SAP Data Hub

Connected Systems

SAP Data Hub SAP S/4HANA Self-Service Metadata Data Pipeline Data Access API Management Preparation Development Workflows Governance Access SAP BW/4HANA

Distributed Runtime Pipelines & Workflows Metadata & Applications SAP Data Services Relational Streaming Scripting Built-In Scheduling Profiling & Tables (JS, Python) Connectors & Discovery SAP LT Replication Server SQL App server Templates Custom Operators Metadata Catalog Connectivity SAP Vora Engines Flow-Based Applications Application Services SAP HANA

SAP Data Hub System Management Databases User & Access Content Lifecycle Cluster Multitenancy Diagnostics Management Management Management SAP cloud applications (API-driven) Open connectivity Data Storages Cloud Stores Hadoop SAP Data Hub SAP Vora for third-party & open source Cloud / On-Premise AWS S3, GCP GCS, Azure ADL & WASB HDFS (optional) Adapter Spark Extensions

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 30 SAP Data Intelligence One tool to set up, manage and automate a continuous lifecycle – using SAP Data Hub Data Preparation capabilities

SAP Data Intelligence

Identify Automation & Deployment & Operations Maintenance Integration into Application

Machine Learning Scenario Model Deployment Data Model Processing Validation

Data Model Discovery Training

Connection Model /Storage Management Creation

Data Preprocessing Model Management

© 2019 SAP SE or an SAP affiliate company. All rights reserved. 31 Take the Session Survey.

We want to hear from you! Be sure to complete the session evaluation on the SAPPHIRE NOW and ASUG Annual Conference mobile app. Presentation Materials Access the slides from 2019 ASUG Annual Conference here: http://info.asug.com/2019-ac-slides Q&A For questions after this session, contact us at [email protected] [email protected] Let’s Be Social. Stay connected. Share your SAP experiences anytime, anywhere. Join the ASUG conversation on social media: @ASUG365 #ASUG