How Do We Know that a Change is an Improvement?
Presenters: Farashta Zainal, MBA, PMP Senior Improvement Advisor Joy Dionisio, MPH Improvement Advisor
June 30, 2020 Webinar Instructions
Figure 1
Figure 2
***There are two options to Switch Connection to telephone audio. 2 Webinar Instructions
• All participants have been muted Figure 1 to eliminate any possible noise/ interference/distraction. • Please take a moment and open your chat box by clicking the chat icon found at the bottom of your screen and as shown in Figure 1. • Be sure to select “All Participants” when sending a message. • If you have any questions, please type your questions into the chat box, and they will be answered throughout the presentation. Conflict of Interest
All presenters have signed a conflict of interest form and have declared that there is no conflict of interest and nothing to disclose for this presentation.
4 Learning Objectives
1 2 3
Understand the Learn how to Understand the importance of construct run role of data context to charts using measurement in interpret current special cause improvement and be able to develop data variation a data collection plan and strategies for implementing it
5 Review Session 1
Model for Improvement What are we trying to accomplish? How will we know that a change is an improvement? What changes can we make that will result in improvement?
Act Plan
Study Do
From Associates in Process Improvement. 6 Review Session I – Aim Statement
• Aim statements should meet the SMART criteria: • Specific • Measureable • Achievable Ambitious • Relevant • Time-bound • Aim statements should be developed with a team and should consider what factors might influence the scope
7 Review Session 1 – Project Charter
Background/ Reason for Effort
Team Aim Roles / Statement Logistics
Charter
Elements Project Project Scope and Milestones Approach
Project Expected Deliverable/ Risks Outcomes 8 Questions?
9 Data for Quality Improvement
10 What is Data?
• Merriam-Webster: – Facts or information used as a basis for reasoning, discussion, or calculation
• FreeDictionary.com: – A series of observations, measurements, or facts; information
11 Data Drives Personal Performance
12 Data for Quality Improvement
Understand • How does the current system perform?
• What interventions might improve the Predict performance of the current system?
• Did our interventions result in Evaluate improvement?
• Are our improvements sustained over Monitor time?
Engage • What do stakeholders need to know?
13 Important Questions about Data
• Consider the context of the data How do the data compare to… • Data in previous months, quarters or years? • My organization’s performance goals? • Performance of similar organizations (benchmarking)? • Industry standards? – Is this what I expected to see? Does it make sense given what I know about my organization? – Does performance differ by subgroup?
14 Calculating Percent
• Numerator/Denominator * 100 = percent • “North Dakota” • Examples:
o 3/4 = .75 .75*100 = 75%
o 4/16 = .25 .25*100 = 25%
Online tutorials on percent available at: https://www.khanacademy.org/math/cc-sixth- grade-math/cc-6th-ratios-prop-topic#cc-6th-percentages 15 Examples of Data Without Context
• The Dow Jones Industrial Average plunged 1,033 points. – It was a 4.2% drop from the previous day. – The Dow has been >20,000 points since March 2020.
• The unemployment rate in June 2020 is at 14.7% - It was 3.7% in June 2019 - It was 9.2% in June 2011
• 90% of patients are satisfied with Partnership Clinic – What if the average satisfaction among peer clinics is 95%?
16 Looking at Variation in Data
• How do the data vary over time? – All data demonstrate variation – How we react to variation depends on how we interpret it
• Two types of variation in data – Common Cause – Special Cause
17 Two Types of Variation
• Random Common • “Natural” or expected Cause variation • Inherent to the system
• Non-random Special • Attributable to a cause Cause • Not inherent to the system
18 Example 1
19 Tools to Understand Variation in Data
Track data over time Distribution of February arrival times 12:00 PM 8 7 10:48 AM 6 9:36 AM 5
8:24 AM 4 3 7:12 AM Number of Days of Number 2 6:00 AM 1 0 4:48 AM < 7 am 7 - 7:30 7:30 - 8:00 - 8:30 - 9:00 - >9:30 1 2 3 4 5 6 7 8 9 10 11 12 13 8:00 8:30 9:00 9:30 am
Stratification: February on time The relationship between variables: Wake-up and Arrival time arrivals by day of the week 12:00 PM 5 10:48 AM 4 9:36 AM 3 8:24 AM 7:12 AM
2 Arrival Time 6:00 AM
Number of of Number Days 1 4:48 AM 0 AM 4:48 AM 5:02 AM 5:16 AM 5:31 AM 5:45 AM 6:00 AM 6:14 AM 6:28 AM 6:43 AM 6:57 AM 7:12
Wake-up Time 20 20 Why Track Data Over Time?
• Make performance of the process visible • Determine if change is an improvement by comparing data before and after test • Determine if holding the gain
21 Run Charts Can Help Prevent Misinterpretation Before and after implementation of a new protocol to reduce readmissions. 30% WOW!
28% A “significant drop” from 28% to 7%
Readmission Rate Readmission 7%
0% Time 1 Time 2 Conclusion -The protocol was a success! We saw a 75% reduction in readmission rate
22 Run Charts Help Tell the Whole Story
Before and after implementation of a new protocol
30% Protocol implemented here Time 1 (28%)
Readmission Rate Readmission Time 2 (7%)
0% 24 Months
23 Colorectal Cancer Screening Example
How to increase colorectal cancer screening rate?
24 Data Help Evaluate the Interventions & Impacts
Colorectal Cancer Screening, Pts Age 50-75 50%
45% EHR Upgrade
40%
35%
30% Start PDSA
25%
20% EHR Upgrade 15%
10%
5%
0%
• Median: arrange a set of observations from lowest to highest and find the value in the middle • Mean (average): the sum of the values divided by the number of values • Examples: 2, 3, 5, 6, 7 2, 3, 5, 6, 7, 100 – Median = 5 – Mean = (2+3+5+6+7)/5 = 4.6 – Median #2: (5+6)/2 = 5.5 – Mean #2: (2+3+5+6+7+100)/6 = 20.5
26 Anatomy of a Run Chart
Goal Title Colorectal Cancer Screening, Pts Age 50-75 yrs 90% 80% 70% Center line 60% (median) 50% Start PDSA EHR Upgrade EHR Upgrade 40% 30% Performance 20% 10%
Well Child Visits, Pts Age 3-6 yrs Median Goal
Time 27 Run Chart Rules
Interpreting Run Charts Using Special Cause Variation Rules
Rule Definition Astronomical - One value that is clearly different from the rest point Shift - An indication of movement, where 6 consecutive points have ‘shifted’ to the other side of the median - If 1 point is on the median, skip it and keep counting
Trend - 5 or more points in a row, each one consecutively higher or lower in value than the previous data point - If 2 or more consecutive points have the same value, skip all but one of the matching points when counting
28 Astronomical Point
Weekly No-Show Rate 40%
35%
30%
25%
20%
15%
10% Median
5% Goal
0% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
29 Shift?
Days Until Third Next Available Appointment 30
25
20
15
10
5
0 July July May May April April June June March March August August October October January January January October February February November December November December November December September September TNAA Median
30 Trend?
A. NO
J F M A M J J A S O N D
B. YES
J F M A M J J A S O N D 31 Which Run Chart Rule Applies?
Diabetic Retinopathy Screening Rate
90% 80% 70% 60% 50%
40% Clinic began 30% using digital camera 20% Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov
Diabetic retinopathy screening rate Median
32 Example 2
TNAA for Established Pts, Dr. Z 30
Dr. Z on Start robust 25 vacation confirmation calls, tetrising 20 the schedule
15 TNAA
10
5
0 1 2 3 4 5 6 7 8 9 3 10 11 12 13 14 15 16 17 Weeks
Days to Third Next Available Appointment, Established Pt Median Goal
33 Run Chart Rules Recap
• 3 Decision rules: – Astronomical point – Shift – Trend • Only 1 rule needs to be fulfilled to suggest non-random (special cause) variation • These rules help address gut reactions to the data
34 Exercise - Run Chart 1
Breast Cancer Screening Rates 100 90 80 70 60 50 40 30 20 10
within within the recommended timeframe 0 % of patientsrecieving eligible mammogram a
Breast Cancer Screening Rate Median
35 Exercise - Run Chart 2
Office Visit Cycle Time 100
90
80
70
60 Average Minutes Average
50
40
Office Visit Cycle Time Median
36 Exercise - Run Chart 3
Days to Third Next Available Appt- Practice A 60
50
40
30
Average Days Average 20
10
0
TNAA Median 37 Scale Matters
From the Institute for Healthcare Improvement
38 Exercise
39 In Summary: Using Data for QI
• Understanding the context of the data helps with interpreting the data. • All data exhibit variation, either common cause or special cause. • A run chart is one of the easiest and most widely-used QI tools to track data over time and to help analyze the data.
40 Questions?
41 How Do We Know That a Change is an Improvement?
42 “Measurement” within Model for Improvement Model for Improvement What are we trying to accomplish? How will we know that a Measure! change is an improvement? What changes can we make that will result in improvement?
Act Plan
Study Do
From Associates in Process Improvement. 43 Measurement for Improvement
• Purpose: To track progress over time and to engage practice staff and leaders. ‒ Not for scientific research or provider feedback. • Character: – Simple (small samples) – Rapid (frequent sampling) – Motivating (immediate response) • Audience: QI Team, front-line staff and providers, senior sponsor, and leadership
44 Data for Quality Improvement
Understand • How does the current system perform?
• What interventions might improve the Predict performance of the current system?
• Did our interventions result in Evaluate improvement?
• Are our improvements sustained over Monitor time?
Engage • What do stakeholders need to know?
45 Set of Measures
• Relates directly to aim Outcome • Answers question – did we achieve our aim? • Measures whether a change has been accomplished Process • Helps us understand why we did or did not achieve our aim • “Unintended” impact Balance • Can be + or - impact
46 Selecting an Outcome Measure
• Directly relates to aim –> what you want to change • Answers the questions: – Did we achieve our aim? – Is anybody better off?
47 Process Measures
• Measures whether a change has been accomplished • Helps us understand why we did or did not achieve our aim • Should be sensitive to changes
48 Driver Diagram to Guide Measure Development
Primary Drivers Secondary Drivers • Wake-up time • Getting dressed Kids • Eating breakfast • Distractions • Lunches
I will improve Vehicle • Flat tire my arrival time • Need gas • Vehicle won’t start to work from 70% to 95% by December • Bad weather (need to drive Environment 31, 20XX more slowly) • Accident blocking traffic • Road work
My • Wake-up time Preparation • Wardrobe • My lunch • Workout clothes 49 Arrival Time Measures Set
Aim: I will improve my arrival time to work from 70% to 95% by December 31, 20XX Measures:
• % of days with arrival to work at or Outcome before 8:30 am
• % of days my kids are ready by 7:45 am Process • % of days that I am ready by 7:15 am
50 CRC Measures Set
Aim: Center #1 will improve early detection of colorectal cancer for patients 50-75 years of age by increasing colon cancer screening from 23% to 56.6% by June 30, 20XX Measures:
• % of patients 50-75 years old with Outcome colorectal cancer screening
• % of patients 50-75 for whom Process colorectal cancer screening is ordered
51 Well-Child Visit Measure Set
Aim: We will increase the percentage of well-child visits in the 3rd, 4th, 5th, and 6th years of life, from 56% to 75% by December 31, 20XX Measures:
Outcome • Well-child visit rate
Process • No-show rate for well-child visits
52 Considerations When You Are Selecting Measures
• Improvement Topic • Key Players: data – Manageable number selection/data of measures collection – Define patient – QI team population – Involve “front line” in the • Methodology selection of measures – Sample size – Involve the data – Frequency collectors in planning collection – Sources – Get senior leader support for your measures
53 Exercise
Here is an aim statement and a list of measures for each. At your table, categorize each measure as an outcome (O), process (P), or balance (B) measure.
Aim statement: We will improve the cervical cancer screening rate for women ages 24-64 from 45% to 65% by September 30, 20XX
Measures: • Third next available appointment • % of women with cervical cancer screening ordered • % of women with cervical cancer screening completed • % of women due who received a CCS reminder
54 Once You’ve Selected Your Measures
Create a data Define them collection plan
Numerator and Where will the denominator data come criteria from?
Who will collect “Lingo” and analyze definitions data?
How often will data be collected and reported?
55 How Big is this Elephant?
56 Defining Your Measures
• Conceptual definition or the measure “name” – Tells what will be measured – For example: Patient wait time
• Operational definition – “Specify and Quantify” – Tells how it will be measured – Ex: Time elapsed from the patient appointment time until time patient enters exam room, in minutes
Adopted from the Dartmouth Institute for Health Policy & Clinical Practice
57 Defining Your Measures
• Be specific. The more specific your measure definitions, the better! • For proportions and percentages: Define numerator and denominator criteria • Create operational definitions (common language definitions) for all “lingo”: – No-show rate – Wait time, cycle time, etc. – Patient satisfaction
58 How to Measure a Banana?
Measure the banana from top to bottom.
59 How to Measure a Banana?
60 How to Measure a Banana?
Measure the banana curve side out from top of the stem to the black tip.
61 Defining Measures – Example
Measure: Percentage of patients 50-75 with colorectal cancer screening. o Numerator: Include any of the following • Fecal occult blood test during the measurement year. • Flexible sigmoidoscopy during the measurement year or the four years prior to the measurement year. • Colonoscopy during the measurement year or the nine years prior to the measurement year. o Denominator: Patients 51–75 years of age at end of measurement year. o Exclusions: Patients with a diagnosis of colorectal cancer or total colectomy
From National Quality Forum: www.qualityforum.org 62 Data Collection Plan
• Who is responsible for getting the data (measurement)? • Sample size – Will data be collected on the entire eligible population? Or will you sample? • Sources – EHR or registry report – Chart review • Frequency – How often will the data be collected?
63 Example Data Collection Tool
64 Measures Definition Worksheet
66 Recap of Measures
• Relates directly to aim Outcome • Answers question – did we achieve our aim? • Measures whether a change has been accomplished Process • Helps us understand why we did or did not achieve our aim • “Unintended” impact Balance • Can be + or - impact
67 Exercise 1
Aim: We will increase the percentage of Dr. Seuss’s diabetic patients’ A1C control compliance (A1C Value < 9.0) from 62% to 70% by December 31, 20XX.
Measure: Percent of diabetic patients who are due for A1C screening completes their screening during their next visit to the clinic. a. Outcome b. Process c. Balance 68 Example 2
Aim: ABC Clinic will increase asthma medication ratio compliance from 74% to 85% by November 1, 20XX.
Measure: # of asthma-specific appointments scheduled a. Outcome b. Process c. Balance
69 Exercise 3
AIM: Clinic Hope will increase Well Child Visits rates from 14.5% to 75% by April 30, 20XX
Measure: rate of CIS-10 a. Outcome b. Process c. Balance
70 Recap – Measurement for Improvement
Purpose: To track progress over time and to engage practice staff and leaders. Set of Measure • Outcome – relates directly to the aim • Process – measures whether a change has been accomplished • Balance – “unintended” impact When Defining your Measure • Be Specific • Create operational definitions (common language) 71 Questions?
72 ABC’s of QI – Upcoming Sessions
Tips for Developing Change Ideas for Improvement Webinar Date: Tuesday, July 7 Time: Noon - 1:15 p.m. Registration: https://partnershiphp.webex.com/partnershiphp/onstage/g.php? MTID=e065b81e47ebb0d34c410ecd11ce06af6
Testing and Implementing Changes via the Plan-Do-Study-Act Cycle Webinar Date: Tuesday, July 14 Time: Noon - 1 p.m. Registration: https://partnershiphp.webex.com/partnershiphp/onstage/g.php? MTID=e357b36d652e9f0c3b26f38c6b435141d
73 Evaluations
Please complete your evaluation. Your feedback is important to us!
74 Continuing Education Credits
Approved for 1.25 AAFP Elective credits.**CME is for physicians and physician assistants and other healthcare professionals whose continuing educational requirements can be met with AAFP CME.
Provider approved by the California Board of Registered Nursing, Provider #CEP16728 for 1.25 hours.
75 Thank You!
Quality Improvement Advisors: Farashta Zainal ([email protected]) Flora Maiki ([email protected]) Joy Dionisio ([email protected])
QI/Performance Team: [email protected]
76 77 Example Run Chart
Outcome Measure
Cervical Cancer Screening
Outreach Gap List: 85% phone calls/letters Saturday Saturday Clinic Clinic #2 #1 75% 72% 69% 69% 70% 69% 69% 70% 66% 67% 66% 67%
65% Ashley on PTO
55%
45% Jan-19 Feb-19 Mar-19 Apr-19 May-19 Jun-19 Jul-19 Aug-19 Sep-19 Oct-19 Nov-19 YTD 66% 67% 66% 67% 69% 69% 70% 69% 69% 70% 72% Goal 70% 70% 70% 70% 70% 70% 70% 70% 70% 70% 70% Example Run Chart
% of Missed Opportunities for Cervical Cancer Screening 40%
35% 33% 34% 30% 31% 29% 27% 28% 25% 25% 22% 20%
Percentage 15%
10%
5%
0% Jan-19 Feb-19 Mar-19 Apr-19 May-19 Jun-19 Jul-19 Aug-19 Month Example Run Chart
Process Measure – Cervical Cancer Screening Appointments Kept 90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
% of Cervical Cancer Screening Appointments Kept Goal Median
80 Example Measures
Aim: XYZ Medical Clinic will improve our asthma medication ratio by increasing the number of patients who are prescribed both a rescue inhaler as well as a controller inhaler from 74% to 85% by November 1, 2019.
Outcome: Percentage of patients age 5-64 years who are identified as having persistent asthma and had a ratio of controller medications to total asthma medications of 50% or greater within the measurement period.
Process: Number of asthma-specific appointments occurring
81 Example Measures
Aim: We will improve the health of our female patients by increasing cervical cancer screening rates at XXX Health Center from 59% to 65% by January 2020
Outcome Measure: Percentage of patients 24-64 years age who are up to date with their CCS
Process Measure: Percent of women scheduled for a Well Visit
82 Example Data Collection Tool
83 Run Chart Exercise
84