Performance Dashboards: Measuring, Monitoring, and Managing Your Business Wayne W. Eckerson
• WAYNE W. ECKERSON is the Director of Research and Services for The Data Warehousing Institute. • Eckerson has 17 years of industry experience and has covered data warehousing and business intelligence since 1995. • Eckerson is the author of many in- depths reports, a columnist for several business and technology magazines, and a noted speaker and consultant. • Eckerson has recently written a book titled, Performance Dashboards: Measuring, Monitoring, and Managing Your Business (Wiley & Sons, 2005). • He can be reached at weckerson@ tdwi.org
© Copyright TDWI, 2006 Slide 2 Course Agenda 1. Evolution of Performance Dashboards 2. Why Performance Dashboards? 3. What are Performance Dashboards? 4. Architecting Performance Dashboards 5. Case Studies 7. Costs of Deployment 8. How to Build Effective Metrics 9. How to Design Effective Dashboard Screens 10. Criteria for Evaluating Dashboard Products
© Copyright TDWI, 2006 Slide 3 Appendices • A - Performance Dashboard Trends • B- Readiness Assessment • C - How to Ensure Adoption • D - Performance Dashboard Market Segmentation • E – Sample Metrics Report
© Copyright TDWI, 2006 Slide 4 Evolution of Performance Dashboards
© Copyright TDWI, 2006 Slide 5 The Business Challenge Decision makers suffer from…
• too much data…. • too little information… • delivered too late… to make effective decisions.
© Copyright TDWI, 2006 Slide 6 Evolution of a Solution The search for the perfect “business insight system”: • 1980s – Executive information systems (EIS) – Decision support systems (DSS) • 1990s – Data warehousing (DW) – Business intelligence (BI) • 2000s – Dashboards and scorecards – Performance management • 2010+??
© Copyright TDWI, 2006 Slide 7 Two Metaphors
Dashboard Performance Chart
Performance Dashboard
© Copyright TDWI, 2006 Slide 8 Two Disciplines
Business Corporate Intelligence Performance Management STRATEGY
e 2 z ls . i a B P Wisdom g o u s F d la te G e o g n , v r e a s ti I e ts tr e n n c , lu e it a P S a c s ia s ents . p t ts la Ev Review, Measure, Refine V In iv , n Act 1 , , a e M s n s M s o , io e y , d s iv g T e is t e a l c t rg s M je a e b tr t Data Plans O S s Rules and Models Integrated Data A ct
Knowledge io W e n 3
s I/D c n .
Analytical Tools R , B a s
e D d M
t v e rm r
a o
D s is c fo
a i r ni
u io s e bo
t j i h
a n o P s t o
d s n a
s D r
Information A , /
/ A
Data Warehouses t n c
a
A
l
y .
z 4
DATA REFINERY e EXECUTION Performance Dashboards
© Copyright TDWI, 2006 Slide 9 Corporate Performance Management STRATEGY
e ls 2 iz a o B . g s u P e G e F d l t , v o g a a s ti r e n tr e n e t lu e c s S a c a , . n s P 1 V I I ts la , , s n , n n s re it T s io e u ia a , s iv s t r s t a iv g i c e e e M je s ts b M O Integrated A c Data g, R tio in W n rt o e s o ls
r vi , p o
k s D e o 3
fl io t .
o e )
w n c , R I e Mo st s y B s
, , is r ( u
u D i e o n j A o s
i l n u i h
s i
d s e Q e
c s y r t r l o
u ts , a a
/A
s n r
t ,
W / s A A
c io ta
n a n
s D a
l
y 4. A 4.
z e
EXECUTION
© Copyright TDWI, 2006 Slide 10 Waves of Software Automation
Æ Effectiveness 0 0 0 2 Mgmt M (Strategy P C Execution) , M M R C C S Æ
5 e , , 9 Cross-Functional c n r r 9 o o i e 1 t t n Value Chains F a n g i s e (Customers, Supplies, Products)) m a e C l o p t l t a l u m s S a m a e A g C C g M Æ a k s 0 r Front-Office c 9 e 9 (Sales, Service, Marketing) a l 1 P b
a P n
R E
E T I 0 / e 0 Back Office r 0 2 (Manufacturing, Finance, Human Resources, a - w 5 t 8 Procurement, Logistics) f 9 o 1 S Efficiency Business Activities Operations
© Copyright TDWI, 2006 Slide 11 Why Performance Dashboards?
© Copyright TDWI, 2006 Slide 12 Status of Performance Dashboards
No plans, 17%
Deployed, 51% Under development 33%
From Wayne Eckerson, “Development Techniques for Creating Analytic Applications,” TDWI, 2005.
© Copyright TDWI, 2006 Slide 13 Business Week Cover Story - February 13, 2006
© Copyright TDWI, 2006 Slide 14 Tactical Drivers Resonates with users • Monitors status of several areas on one screen • Graphical view of key metrics • Alerts users to exception conditions • Click to analyze and drill to detail • Customized views based on role • Personalized views based on interest • No training required!
© Copyright TDWI, 2006 Slide 15 Tactical Drivers (cont.) Rich data • Blends data from multiple sources • Both detailed and aggregated • Both historical and real-time Empowers workers • Focuses users on what’s really important • Shows them how their contributions count • Motivates with goals, competition, & incentives • Drives proactive intervention
© Copyright TDWI, 2006 Slide 16 Strategic Drivers Aligns the business • Everyone uses the same data • Everyone uses the same metrics • Everyone works toward the same strategy Improves communication • Tool for communicating strategy • Managers & staff - collaboration • Among departments - coordination Improves visibility and compliance • Fewer surprises
© Copyright TDWI, 2006 Slide 17 Strategic Drivers - The “Five Cs” • Communicate • Compare • Collaborate • Coordinate • Congratulate
© Copyright TDWI, 2006 Slide 18 Agent of Organizational Change
© Copyright TDWI, 2006 Slide 19 Charting a Course
Goal
Direction without a Performance Dashboard
Direction with a Performance Dashboard
© Copyright TDWI, 2006 Slide 20 What are Performance Dashboards?
© Copyright TDWI, 2006 Slide 21 The “Three Threes” • Three Applications • Three Layers • Three Types
© Copyright TDWI, 2006 Slide 22 Three Applications
Monitoring Analysis Collaboration
Purpose Convey performance Analyze exceptions and Collaborate, plan, status & trends at a find root cause and ACT glance Elements - Multi-paneled screens - Drill down/up -Telephone - Graphical metrics (i.e. hierarchies -Meetings dials, gauges, symbols) - Pivot and swap out - Email (notification) - Charts and tables dimensions - Annotations - Status, trend, and - Drill through to - Threaded threshold indicators operational data discussions - Color-coded, - Time series, - Recommended conditional formatting segmentation, predictive, analysis, actions - Alerts: Web-based, and other analyses - Publish to server email, other - Reporting -Workflow - Triggers, Updates
© Copyright TDWI, 2006 Slide 23 Three Layers of Information
Graphical Abstracted Data Monitoring Graphs, Symbols, Charts
Summarized Dimensional Data Analysis Dimensions, hierarchies, “slice/dice” Planning Detailed, Operational Data Reporting DW queries, Operational reports
Performance Dashboard Plans, models, forecasts, updates
© Copyright TDWI, 2006 Slide 24 Dashboard Usage “Our executives will drill one or two levels down before they call someone who can fix the problem, while our managers will often drill three or four layers down before they make a call.”
– Thomas Tomlinson, director of BI for Bull Moose Tube, a steel manufacturer in Chesterfield, MO.
© Copyright TDWI, 2006 Slide 25 Dashboards vs Scorecards • Distinct? • Synonymous? •Both?
Rule of thumb: Use whatever term business users prefer!
© Copyright TDWI, 2006 Slide 26 Dashboards vs Scorecards Dashboard Scorecard
Purpose Measures current activity Charts progress Users Executives, managers, staff Executives, managers, staff Updates “Right time” feeds Periodic snapshots Data Events Summaries Queries Run against remote systems Run against local data mart Display Charts Symbols
Dashboards and scorecards are visual interfaces for monitoring business performance
© Copyright TDWI, 2006 Slide 27 Three Types
Operational Tactical Strategic
Focus Monitor operations Optimize process Execute strategy Emphasis Monitoring Analysis Collaboration Users Supervisors+ Managers+ Executives+ Scope Operational Departmental Enterprise Information Detailed Detailed/Summary Summary Updates Intra-day Daily/Weekly Monthly/Quarterly “Looks like a…” “Dashboard” “BI Portal” “Scorecard”
© Copyright TDWI, 2006 Slide 28 Pretenders to the throne
– Too Flat “A prettified spreadsheet”
– Too Isolated “Another spreadmart”
– Too Manual “Not scalable or sustainable”
– Too Cheap “You get what you pay for!”
© Copyright TDWI, 2006 Slide 29 How Do You Architect a Performance Dashboard?
© Copyright TDWI, 2006 Slide 30 Three Architectures
Business Architecture BI Architecture Data Architecture
© Copyright TDWI, 2006 Slide 31 Business architecture
Stakeholders Investors Board Workforce Customers Suppliers Regulators Business Architecture
Strategy Mission Vision Values Goals Objectives Strategy Map
Tactics Assets People Knowledge Plans Process Projects
Semantics Terms Definitions Rules Metadata Education Governance
Metrics Leading Lagging Diagnostic
Displays Dashboard BI Portal Scorecard Technical Architecture
Applications Monitoring Analysis Management
Data Stores ODS Memory cache Data warehouse Data marts Reports Documents
Integration Custom API EAI EII Direct queries ETL Manual
Data Sources Legacy Systems Packaged Apps Web pages Files Surveys Text Performance Dashboard
© Copyright TDWI, 2006 Slide 32 BI Architecture
Business Architecture
Monitoring Layer Dashboards, scorecards, KPIs, Alerts r Analysis Layer Dimensions, hierarchies, “slice/dice”
Reporting Layer Planning Laye
Management and operational reports Plans, models, forecasts Integrated BI Capabilities
BI Platform (Analytic Server) - Common services, object model, API, file formats, etc. Data Delivery Architecture
© Copyright TDWI, 2006 Slide 33 Data Architecture – Quicken Loans
Packaged Phone Mainframe AS/400 Web Applications System
Messaging Backbone (Enterprise Application Integration software) e c i b v r We Se
ETL Operational ETL Data Warehouse Real-time Store Data Store (2 days of data) (7 years of data) (2 months of data)
250 OLAP Cubes 100 OLAP Cubes Cubes (7 years of data) Cubes (2 weeks of data)
Reporting & Operational and Analysis Tools Tactical Dashboards
© Copyright TDWI, 2006 Slide 34 Direct Query Architecture
Data Packaged Phone Data Marts Mainframe Reports Spreadsheets Warehouse Applications System
Query engine Pros: Screen elements - Deploy quickly linked directly to - Low cost individual queries Cons - No depth, limited drill down - No dimensions - Hard-wired queries
© Copyright TDWI, 2006 Slide 35 Query and Cache Architecture
Data Packaged Phone Data Marts Mainframe Reports Spreadsheets Warehouse Applications System
Query Engine Pros: In-memory or disk cache - Deploy quickly - Fast response Queries populate a - Rapid navigation queryable cache Cons - Static data sets
© Copyright TDWI, 2006 Slide 36 BI Semantic Layer
Data Packaged Phone Data Marts Mainframe Reports Spreadsheets Warehouse Applications System
ODBC ODBC ODBC
Query Engine Semantic SemanticSemantic Semantic Semantic Layer LayerLayerLayer Layer BI tools provide Pros: - Abstract query objects query objects that - Dimensionalized views represent a database Cons in business terms - Generic ODBC connections for users. -Primarily historical data in DW
© Copyright TDWI, 2006 Slide 37 Federated Query Architecture
Data Packaged Phone Data Marts Mainframe Reports Spreadsheets Warehouse Applications System
An EII tool Pros: dynamically Distributed Query Engine/EII - Multiple sources Semantic Layer - Semantic layer abstraction queries data from - Quick to deploy multiple sources - Prototype before you persist Cons to populate screen - No history elements. - Data quality issues - Complexity
© Copyright TDWI, 2006 Slide 38 Data Mart Architecture
Data Packaged Phone Data Marts Mainframe Reports Spreadsheets Warehouse Applications System
ETL Dashboard Dimensionalized Data Mart queries its own (OLAP or star schema) Pros: - Multiple sources persistent data Query engine and semantic layer - Dimensional model mart loaded in - Historical context batch. - Fast complex queries Cons - No right time data - Non-integrated?
© Copyright TDWI, 2006 Slide 39 Event-driven Architecture
Workflow Operational Outputs Engine Systems
Triggers SQL/Stored Procedures Alerts
Operational Dashboard Data Capture, Data Aggregation, Metrics Management, Event Detection, Rule Processing, Agents/Triggers
Enterprise Service Bus Historical Context
Operational Operational Operational Data Inputs System System System Warehouse
© Copyright TDWI, 2006 Slide 40 “Manual” Architecture
Use when…. • Data doesn’t exist • Strategy is short-term • Want to prototype the KPIs • Executives can’t wait But don’t be fooled… • Permanent prototypes • No scale, depth, value • Reputations on the line!
© Copyright TDWI, 2006 Slide 41 Performance Dashboard Case Studies
© Copyright TDWI, 2006 Slide 42 Operational Dashboard Case - Quicken Loans
• The largest U.S.-based online lender – $12 billion in loans in 2004, 2,500 employees – Sells mortgages via call center and Web • Web Call Center in Livonia, Michigan – 500+ “mortgage lenders” on one giant floor – Disruptions costs millions of dollars an hour
© Copyright TDWI, 2006 Slide 43 Situation • Company philosophy/culture – What gets measured, gets approved – Leverage “velocity as a competitive problem” • Information systems – pre-2002 – Reports run off operational systems • Run slowly, Deliver obsolete data – Disjointed data for historical analysis • Took three weeks to do 18-24 month analysis – Executives and users very frustrated! • Negative view of data warehousing/OLAP
© Copyright TDWI, 2006 Slide 44 Solution • Right-time data warehousing architecture – 1 year at $1 million – Trickle fed OLAP cubes – Existing ESB • Different dashboards for different users – Dashboard ticker – mortgage specialists – Kanban reports – Sales managers, TV monitors – Managerial dashboards – Call center managers – Analytical dashboards/BI tools – Analysts • Metrics – Phone statistics, Number and quality of leads, Sales pipeline, Web traffic, Commissions, products mix
© Copyright TDWI, 2006 Slide 45 Quicken Loans Architecture
Packaged Phone Mainframe AS/400 Web Applications System
Messaging Backbone (Enterprise Application Integration software) e c i b v r We Se
ETL Operational ETL Data Warehouse Real-time Store Data Store (2 days of data) (7 years of data) (2 months of data)
250 OLAP Cubes 100 OLAP Cubes Cubes (7 years of data) Cubes (2 weeks of data)
Reporting & Operational and Analysis Tools Tactical Dashboards
© Copyright TDWI, 2006 Slide 46 Dashboard Ticker
Personal and group metrics updated daily
Personal and group metrics updated instantly
Personal forecasts updated every 15 minutes
© Copyright TDWI, 2006 Slide 47 Kanban Report
Metric 1 Metric 2 Metric 3 Metric 4 Metric 5 Metric 6 Metric 7 Metric 8 Metric 9 Employee 1 Employee 2 Employee 3 Employee 4 Employee 5 Employee 6 Employee 7 Employee 8 Employee 9 Employee 10 Employee 11 Employee 12 Employee 13 Employee 14 Employee 15 Employee 16 Employee 17
© Copyright TDWI, 2006 Slide 48 Managerial Dashboard
© Copyright TDWI, 2006 Slide 49 Tactical Dashboard Case International Truck and Engine • $9.7 billion manufacturer of trucks, buses, diesel engines, and parts based in Illinois • Key business issues: – Market reality: Global competition, new regulations, emerging markets – Goals: 1) $15b in revenues 2) reduced costs, 3) improved quality, 4) reduced risk
© Copyright TDWI, 2006 Slide 50 Situation
• Finance department goals in 2001: – Provide access to financial information any time – Focus on analysis rather than data collection – Deliver proactive rather than reactive analysis – Use financial data as a predictive tool for decisions • Programs – Accelerate closing of books – Standardize company’s information infrastructure – Replace legacy systems with packaged applications – Implement a Web-based “reporting portal.”
© Copyright TDWI, 2006 Slide 51 KBI Portal • Purpose – Deliver actionable information to financial analysts • Scope – Spans 32 source systems across five divisions – 130 key business indicators, updated daily – Supports 500 financial executives, managers, and analysts • Upshot – Bridges gulf between finance and operations – Replaces hodge-podge of paper reports – Saves analysts time creating custom reports – Shuts down dozens of reporting systems
© Copyright TDWI, 2006 Slide 52 Architecture
Purpose Data
Run the business Source Systems Transactional
ETLTools
Gather all data in one place. Transactional Keep a copy for future reuse. Staging Area Database Data Warehouse Transactional & Integrate data for easy loading into Star Schema Database lightly summarized OLAP cubes. ETLTools
Moderately Store data dimensionally to support OLAP Cubes fast queries and easy navigation. summarized
Web Server
Display key metrics so they can be Highly viewed at a glance. Reporting Portal summarized
© Copyright TDWI, 2006 Slide 53 Monitoring Layer
© Copyright TDWI, 2006 Slide 54 Analysis Layer
© Copyright TDWI, 2006 Slide 55 Detail Transaction Layer
‘Click’‘Click’anyany VIN VIN to to expand expand fullfull page page order/build order/build report report
© Copyright TDWI, 2006 Slide 56 Personalization
© Copyright TDWI, 2006 Slide 57 Metadata
© Copyright TDWI, 2006 Slide 58 Strategic Dashboard Case Hewlett-Packard TSG
• HP Technology Services Group – Provides consulting, support services, and software globally for HP – $12 billion division of Hewlett Packard • Situation – Dozens of overlapping reporting systems with inconsistent metrics – No consistent means of measuring regional and business unit performance against company objectives and holding individuals accountable
© Copyright TDWI, 2006 Slide 59 Solution • HP Performance Measurement and Management System (PMMS) – Executive scorecard (LIBRA) deployed to EMEA region in 2001, then globally thereafter – Cascaded down multiple levels in each region – Implemented unified reporting system underneath (MUSE) • Upshot: $26 million cost-savings in 3 years on $1m expenditure – $8.6 million – Shut down dozens of report systems – $10.6 million – Reduced time spent looking for reports – $1.3 million - Training users on BI tools, etc. – Raised customer satisfaction scores, lowered missed service-level commitments, correlate to revenue
© Copyright TDWI, 2006 Slide 60 PMMS Architecture
Value Content Data
“Balanced” Scorecards 12 OLAP cubes updated monthly with 100MB of data Knowledge “LIBRA” “Unbalanced” scorecards (OLAP)
2,500 OLAP cubes updated daily to monthly with 200 GB Information “MUSE” Scorecard Reports of data, spread across four (OLAP) data centers
Operational Systems Data Warehouses 40 data sources, only Data SOURCE DATA exception conditions captured Budgets, Surveys (Databases, Spreadsheets, Files) Data Marts Volume
© Copyright TDWI, 2006 Slide 61 Monitoring Layer
© Copyright TDWI, 2006 Slide 62 Analysis Layer
© Copyright TDWI, 2006 Slide 63 Analysis Layer (cont.)
© Copyright TDWI, 2006 Slide 64 Reporting Layer
© Copyright TDWI, 2006 Slide 65 Rohm & Haas • Company – $8 billion global chemicals manufacturer • Impetus – CFO restructures finance to improve efficiency – Eliminate spreadmarts – offer consistent metrics • Time and cost – First iteration: 12 months, $500k – Subsequent dashboards (12): $100k • Tools – Custom built – SAP Portal, SAP Web Application Developer – Runs against SAP BW with Hyperion data imported
© Copyright TDWI, 2006 Slide 66 © Copyright TDWI, 2006 Slide 67 © Copyright TDWI, 2006 Slide 68 Railroad Company • Impetus – Automate daily performance report (paper) • 120 page report, 45 measures, 4 levels, all locations • Reduce time spent analyzing data • Time and Cost – First iteration: 7 months and $500k – Current view: 1.5 years and $1M • Tools – Existing: Teradata, Essbase, Alphablox, ESRI – New: Treemap software (~$2k)
© Copyright TDWI, 2006 Slide 69 © Copyright TDWI, 2006 Slide 70 © Copyright TDWI, 2006 Slide 71 © Copyright TDWI, 2006 Slide 72 IBM • Impetus - 1998 – Dueling spreadsheets in weekly sales meetings • Time and cost – First iteration: 6 months, $200k – Became basis of BI and EDW initiative • 25,000 users, thousands of reports, 40 dashboards • Tools – Lotus Notes (email, disconnected, unstructured) – Web portal for mid-level managers
© Copyright TDWI, 2006 Slide 73 End to End Business Process
Dashboard designed to allow management by exception. Requires agreement across the enterprise on thresholds.
© Copyright TDWI, 2006 Slide 74 Simple text based inputs. Updated as required.
© Copyright TDWI, 2006 Slide 75 Contact Information Wayne Eckerson
Director of Research and Services The Data Warehousing Institute 70 Martins Lane, Hingham, MA USA 781 740-9504 [email protected]
© Copyright TDWI, 2006 Slide 76