How Do We Know That a Change Is an Improvement?
Total Page:16
File Type:pdf, Size:1020Kb
How Do We Know that a Change is an Improvement? Presenters: Farashta Zainal, MBA, PMP Senior Improvement Advisor Joy Dionisio, MPH Improvement Advisor June 30, 2020 Webinar Instructions Figure 1 Figure 2 ***There are two options to Switch Connection to telephone audio. 2 Webinar Instructions • All participants have been muted Figure 1 to eliminate any possible noise/ interference/distraction. • Please take a moment and open your chat box by clicking the chat icon found at the bottom of your screen and as shown in Figure 1. • Be sure to select “All Participants” when sending a message. • If you have any questions, please type your questions into the chat box, and they will be answered throughout the presentation. Conflict of Interest All presenters have signed a conflict of interest form and have declared that there is no conflict of interest and nothing to disclose for this presentation. 4 Learning Objectives 1 2 3 Understand the Learn how to Understand the importance of construct run role of data context to charts using measurement in interpret current special cause improvement and be able to develop data variation a data collection plan and strategies for implementing it 5 Review Session 1 Model for Improvement What are we trying to accomplish? How will we know that a change is an improvement? What changes can we make that will result in improvement? Act Plan Study Do From Associates in Process Improvement. 6 Review Session I – Aim Statement • Aim statements should meet the SMART criteria: • Specific • Measureable • Achievable Ambitious • Relevant • Time-bound • Aim statements should be developed with a team and should consider what factors might influence the scope 7 Review Session 1 – Project Charter Background/ Reason for Effort Team Aim Roles / Statement Logistics Charter Elements Project Project Scope and Milestones Approach Project Expected Deliverable/ Risks Outcomes 8 Questions? 9 Data for Quality Improvement 10 What is Data? • Merriam-Webster: – Facts or information used as a basis for reasoning, discussion, or calculation • FreeDictionary.com: – A series of observations, measurements, or facts; information 11 Data Drives Personal Performance 12 Data for Quality Improvement Understand • How does the current system perform? • What interventions might improve the Predict performance of the current system? • Did our interventions result in Evaluate improvement? • Are our improvements sustained over Monitor time? Engage • What do stakeholders need to know? 13 Important Questions about Data • Consider the context of the data How do the data compare to… • Data in previous months, quarters or years? • My organization’s performance goals? • Performance of similar organizations (benchmarking)? • Industry standards? – Is this what I expected to see? Does it make sense given what I know about my organization? – Does performance differ by subgroup? 14 Calculating Percent • Numerator/Denominator * 100 = percent • “North Dakota” • Examples: o 3/4 = .75 .75*100 = 75% o 4/16 = .25 .25*100 = 25% Online tutorials on percent available at: https://www.khanacademy.org/math/cc-sixth- grade-math/cc-6th-ratios-prop-topic#cc-6th-percentages 15 Examples of Data Without Context • The Dow Jones Industrial Average plunged 1,033 points. – It was a 4.2% drop from the previous day. – The Dow has been >20,000 points since March 2020. • The unemployment rate in June 2020 is at 14.7% - It was 3.7% in June 2019 - It was 9.2% in June 2011 • 90% of patients are satisfied with Partnership Clinic – What if the average satisfaction among peer clinics is 95%? 16 Looking at Variation in Data • How do the data vary over time? – All data demonstrate variation – How we react to variation depends on how we interpret it • Two types of variation in data – Common Cause – Special Cause 17 Two Types of Variation • Random Common • “Natural” or expected Cause variation • Inherent to the system • Non-random Special • Attributable to a cause Cause • Not inherent to the system 18 Example 1 19 Tools to Understand Variation in Data Track data over time Distribution of February arrival times 12:00 PM 8 7 10:48 AM 6 9:36 AM 5 8:24 AM 4 3 7:12 AM Number of Days of Number 2 6:00 AM 1 0 4:48 AM < 7 am 7 - 7:30 7:30 - 8:00 - 8:30 - 9:00 - >9:30 1 2 3 4 5 6 7 8 9 10 11 12 13 8:00 8:30 9:00 9:30 am Stratification: February on time The relationship between variables: Wake-up and Arrival time arrivals by day of the week 12:00 PM 5 10:48 AM 4 9:36 AM 3 8:24 AM 7:12 AM 2 Arrival Time 6:00 AM Number of of Number Days 1 4:48 AM 0 4:48 AM 5:02 AM 5:16 AM 5:31 AM 5:45 AM 6:00 AM 6:14 AM 6:28 AM 6:43 AM 6:57 AM 7:12 AM Wake-up Time 20 20 Why Track Data Over Time? • Make performance of the process visible • Determine if change is an improvement by comparing data before and after test • Determine if holding the gain 21 Run Charts Can Help Prevent Misinterpretation Before and after implementation of a new protocol to reduce readmissions. 30% WOW! 28% A “significant drop” from 28% to 7% Readmission Rate Readmission 7% 0% Time 1 Time 2 Conclusion -The protocol was a success! We saw a 75% reduction in readmission rate 22 Run Charts Help Tell the Whole Story Before and after implementation of a new protocol 30% Protocol implemented here Time 1 (28%) Readmission Rate Readmission Time 2 (7%) 0% 24 Months 23 Colorectal Cancer Screening Example How to increase colorectal cancer screening rate? 24 Data Help Evaluate the Interventions & Impacts Colorectal Cancer Screening, Pts Age 50-75 50% 45% EHR Upgrade 40% 35% 30% Start PDSA 25% 20% EHR Upgrade 15% 10% 5% 0% 25 Median and Mean • Median: arrange a set of observations from lowest to highest and find the value in the middle • Mean (average): the sum of the values divided by the number of values • Examples: 2, 3, 5, 6, 7 2, 3, 5, 6, 7, 100 – Median = 5 – Mean = (2+3+5+6+7)/5 = 4.6 – Median #2: (5+6)/2 = 5.5 – Mean #2: (2+3+5+6+7+100)/6 = 20.5 26 Anatomy of a Run Chart Goal Title Colorectal Cancer Screening, Pts Age 50-75 yrs 90% 80% 70% Center line 60% (median) 50% Start PDSA EHR Upgrade EHR Upgrade 40% 30% Performance 20% 10% Well Child Visits, Pts Age 3-6 yrs Median Goal Time 27 Run Chart Rules Interpreting Run Charts Using Special Cause Variation Rules Rule Definition Astronomical - One value that is clearly different from the rest point Shift - An indication of movement, where 6 consecutive points have ‘shifted’ to the other side of the median - If 1 point is on the median, skip it and keep counting Trend - 5 or more points in a row, each one consecutively higher or lower in value than the previous data point - If 2 or more consecutive points have the same value, skip all but one of the matching points when counting 28 25 29 24 23 22 21 20 19 18 17 16 15 14 Show Rate 13 - 12 11 10 9 8 Weekly No Weekly 7 6 Astronomical Point Astronomical 5 4 3 2 Median Goal 1 0% 5% 10% 15% 20% 25% 30% 35% 40% 10 15 20 25 30 0 5 October November December Days Until Third Appointment Available Next January February March April May June July August TNAA September October Shift? November Median December January February March April May June July August September October November December January 30 Trend? A. NO J F M A M J J A S O N D B. YES J F M A M J J A S O N D 31 Which Run Chart Rule Applies? Diabetic Retinopathy Screening Rate 90% 80% 70% 60% 50% 40% Clinic began 30% using digital camera 20% Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Diabetic retinopathy screening rate Median 32 Example 2 TNAA for Established Pts, Dr. Z 30 Dr. Z on Start robust 25 vacation confirmation calls, tetrising 20 the schedule 15 TNAA 10 5 0 1 2 3 4 5 6 7 8 9 3 10 11 12 13 14 15 16 17 Weeks Days to Third Next Available Appointment, Established Pt Median Goal 33 Run Chart Rules Recap • 3 Decision rules: – Astronomical point – Shift – Trend • Only 1 rule needs to be fulfilled to suggest non-random (special cause) variation • These rules help address gut reactions to the data 34 Exercise - Run Chart 1 Breast Cancer Screening Rates 100 90 80 70 60 50 40 30 20 10 within within the recommended timeframe 0 % of patientsrecieving eligiblea mammogram Breast Cancer Screening Rate Median 35 Exercise - Run Chart 2 Office Visit Cycle Time 100 90 80 70 60 Average Minutes Average 50 40 Office Visit Cycle Time Median 36 Exercise - Run Chart 3 Days to Third Next Available Appt- Practice A 60 50 40 30 Average Days Average 20 10 0 TNAA Median 37 Scale Matters From the Institute for Healthcare Improvement 38 Exercise 39 In Summary: Using Data for QI • Understanding the context of the data helps with interpreting the data. • All data exhibit variation, either common cause or special cause. • A run chart is one of the easiest and most widely-used QI tools to track data over time and to help analyze the data. 40 Questions? 41 How Do We Know That a Change is an Improvement? 42 “Measurement” within Model for Improvement Model for Improvement What are we trying to accomplish? How will we know that a Measure! change is an improvement? What changes can we make that will result in improvement? Act Plan Study Do From Associates in Process Improvement.