Run Charts Robert Lloyd, Ph.D

Home , Run chart

Better Quality Through Better Measurement: Run Charts Robert Lloyd, Ph.D. Executive Director Performance Improvement Institute for Healthcare Improvement Discussion Topics

• The Quality Measurement Journey

• Understanding variation conceptually

• Understanding variation with Run Charts

• Linking measurement to improvement The Quality Measurement Journey

AIM (Why are you measuring?) Concept Measures Operational Definitions Data Collection Plan Data Collection Analysis ACTION

Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.

AIM (Why are you measuring?) Concept Measures Operational Definitions Data Collection Plan Data Collection Analysis ACTION

Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.

Now, what the heck do you do with it?

Aggregated data presented in tabular formats or with summary statistics, will not help you measure the impact of process improvement/redesign efforts. Aggregated data can only lead to judgment, not to improvement.

time

Every process displays variation: LC • Controlled variation L stable, consistent pattern of variation “chance”, constant causes

• Special cause variation Static View “assignable” pattern changes over time

Common Cause Variation Special Cause Variation • Is inherent in the design of the • Is due to irregular or unnatural process causes that are not inherent in the design of the process • Is due to regular, natural or ordinary causes • Affect some, but not necessarily all aspects of the process • Affects all the outcomes of a process • Results in an “unstable” process that is not predictable • Results in a “stable” process that is predictable • Also known as non-random or • Also known as random or assignable variation unassignable variation

Is the process stable? YES NO Special + Type of variation Only Common Common Change the Investigate the origin of the Right Choice process special cause Treat normal variation as a Change the Wrong Choice special cause (tampering) process

Consequences of Increased Wasted making the wrong choice variation! resources!

Unplanned Returns to Ed w/in 72 Hours Month M A M J J A S O N D J F M A M J J A S ED/100 41.78 43.89 39.86 40.03 38.01 43.43 39.21 41.90 41.78 43.00 39.66 40.03 48.21 43.89 39.86 36.21 41.78 43.89 31.45 Returns 17 26 13 16 24 27 19 14 33 20 17 22 29 17 36 19 22 24 22 u chart 1.2

1.0

UCL = 0.88

0.8

0.6 Mean = 0.54 Rate per 100 ED per Patients Rate

0.4

0.2 LCL = 0.19

0.0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

STATIC VIEW DYNAMIC VIEW

Run Chart Descriptive Statistics Mean, Median & Mode Control Chart Minimum/Maximum/Range (plot data over time) Standard Deviation Bar graphs/Pie charts Statistical Process Control (SPC)

Run and Control Charts are the best tools to determine if our improvement strategies have had the desired effect.

12 ©Copyright 2012 IHI/R. Lloyd How many data points? Typically you should have between 15 – 20 data points before constructing a chart 15 – 20 patients 15 – 20 days 15 – 20 weeks 15 – 20 months 15 - 20 quarters…?

320 The centerline (CL) on 300 a Run Chart is the Median 280

260 ~X (CL) 240

220 LOS (minutes) LOS Measure 200

180

160 2/16/11 3/16 4/13 5/11 6/8 Time Week Four simple run rules are used to determine if special cause variation is present ©Copyright 2012 IHI/R. Lloyd Why Median Rather Than Mean? Mean = arithmetic average of data Median = middle value of ordered data (50th percentile)

(n + 1)/2 = Median Position which leads you to the Median Value

• 8,10,11,14,16,18,20 Mean = 13.8 Median Position = 4 Median = 14

• 8,10,11,14,16,18,95 Mean = 24.5 Median Position = 4 Median = 14

• 1,10,11,14,16,18,20 Mean = 12.8 Median Position = 4 Median = 14

The Median with an even number of data points?

(n + 1)/2 = Median Position which leads you to the Median Value

• 8,10,11,14,16,18,20,35 Mean = 16.5 Median Position = 4.5 Median = 15

• 8,10,11,14,16,18,30,95 Mean = 25.3 Median Position = 4.5 Median = 15

• 1,10,11,14,14,18,19,20 Mean = 13.4 Median Position = 4.5 Median = 14

Run Chart: Medical Waste (n + 1)/2 How do you find the media? (29 + 1)/2 = 30/2 = 15 6.00

5.75 When you slide a piece of paper down, you reveal the dots in descending order. 5.50 When you have revealed the 15th data point you have found where the median lives. 5.25

5.00 The Median 4.75 Lives Median=4.610 4.50 here at But, the the 15th 4.25 Median Value= 4.6 data Pounds of Red Bag Waste Red BagPounds of Waste 4.00 point

3.75

3.50

3.25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 Point Number

“How will I know what the Run Chart is trying to tell me?”

There are 4 simple run chart rules that help you decide if your data reflect a random or non-random pattern.

©Copyright 2012 IHI/R. Lloyd First, you need to determine the number of Runs What is a Run? • One or more consecutive data points on the same side of the Median • Do not include data points that fall on the Median

How do we count the number of runs? • Draw a circle around each run and count the number of circles you have drawn • Count the number of times the sequence of data points crosses the Median and add “1”

320

300

280

260

240

220 LOS (minutes) LOS Measure 200 Points on the Median (don’t count these when 180 counting the number of runs)

160 2/16/11 3/16 4/13 5/11 6/8 Time Week

320 300 4 runs 280

260

240

220 LOS (minutes) LOS Measure 200 Points on the Median (don’t count these when 180 counting the number of runs)

160 2/16/11 3/16 4/13 5/11 6/8 Time Week

• Rule #1: A shift in the process, or too many data points in a run (6 or more consecutive points above or below the median)

• Rule #2: A trend (5 or more consecutive points all increasing or decreasing)

• Rule #3: Too many or too few runs (use a table to determine this one)

• Rule #4: An “astronomical” data point

A Shift: A Trend 6 or more 5 or more

Too many or too few runs An astronomical data point

Source: The Data Guide by L. Provost and S. Murray, Austin, Texas, February, 2007: p3-10.

A Shift: A Trend 6 or more 5 or more

Too many or too few runs An astronomical data point

Source: The Data Guide by L. Provost and S. Murray, Austin, Texas, February, 2007: p3-10.

©Copyright 2012 IHI/R. Lloyd Rule #3: Too few or too many runs Use this table by first calculating the number of "useful observations" in your data set. This is done by subtracting the number of data points on the median from the total number of data points. Then, find this number in the first column. The lower number of runs is found in the second column. The upper number of runs can be found in the third column. If the number of runs in your data falls below the lower limit or above the upper limit then this is a signal of a special cause.

# of Useful Lower Number Upper Number Observations of Runs of Runs 15 5 12 16 5 13 17 5 13 18 6 14 19 6 15 20 6 16 21 7 16 22 7 17 23 7 17 24 8 18 25 8 18 26 9 19 27 10 19 28 10 20 29 10 20 30 11 21

©Copyright 20122011 IHI/R. Lloyd Source: Swed, F. and Eisenhart, C. (1943) “Tables for Testing Randomness of Grouping in a Sequence of Alternatives.” Annals of Mathematical Statistics. Vol. XIV, pp. 66-87, Tables II and III.

A Shift: A Trend 6 or more 5 or more

Too many or too few runs An astronomical data point

Source: The Data Guide by L. Provost and S. Murray, Austin, Texas, February, 2007: p3-10.

25 Men and a Test

25 What do you think about this data point? 20 Is it astronomical? 15

Score 10

0 1 3 5 7 9 11 13 15 17 19 21 23 25 Individuals

©Copyright 2012 IHI/R. Lloyd ©Copyright 2011 IHI/R. Lloyd Total data points = 21 Are there any non- Data points on the Median = 1 random patterns Number of “useful observations” = 20 present? (should be between 6 &16 runs) 320 The number of runs = 4 300 Number of times the data line crosses the Median = 3 + 1 = 4 280

260

240

220 LOS (minutes) LOS Measure 200 Points on the Median (don’t count these when 180 counting the number of runs)

160 2/16/11 3/16 4/13 5/11 6/8 Time Week

# of Useful Lower Number Upper Number Observations of Runs of Runs 15 5 12 16 5 13 17 5 13 18 6 14 19 6 15 20 Total useful observations 6 16 21 Total data points 7 16 22 7 17 23 7 17 24 8 18 25 8 18 26 9 19 27 10 19 28 10 20 29 10 20 30 11 21

©Copyright 2012 IHI/R. Lloyd Total data points = 21 Are there any non- Data points on the Median = 1 random patterns Number of “useful observations” = 20 present? (should be between 6 &16 runs) 320 The number of runs = 4 300 Number of times the data line crosses the Median = 3 + 1 = 4 280

260

240

220 LOS (minutes) LOS Measure 200 Points on the Median (don’t count these when 180 counting the number of runs)

160 2/16/11 3/16 4/13 5/11 6/8 Time Week

1. % of patients with Length of Stay shorter than six days

2. Average Length of Stay DRG373

3. Number of Acute Surgical Procedures

Source: Peter Kammerlind, ([email protected]), Project Leader Jönköping County Council, Jonkoping, Sweden.

Grundläggande statistik och analys

Antal patienter med vårdtid < 6dygn i % vid primär elektiv knäplastik (operationsdag= dag1)

40 Antal patienter i % 30

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Månad Source: Peter Kammerlind, ([email protected]), Project Leader Jönköping County Council, Jonkoping, Sweden. Mäta för att leda ©Copyright 2012 IHI/R. Lloyd Antal patienter med vårdtid < 6dygn i % vid primär elektiv knäplastik % of patients with Length(operationsdag= of Stay dag1) shorter than six days

90 Source: Peter Kammerlind, 80 ([email protected]), Project Leader, Jönköping Rule 1 & 3 70 County Council, Jonkoping, Sweden. 60

40 Median=52 Antal patienter i % i patienter Antal 30 Rule 1 & 3 18 useful observations

20 Rule 1: 2 runs (6-14 runs), not OK Rule 2: OK 10 Rule 3: OK Rule 4: OK 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Månad Grundläggande statistik och analys

2,5

Vårddygn 1,5

0,5

Maj Juni Juli Maj Juni Juli 2005 2006 Febr Mars April Febr Mars April januari januari Augusti Oktober Augusti September NovemberDecember September Tidsenhet Source: Peter Kammerlind, ([email protected]), Project Leader Jönköping County Council, Jonkoping, Sweden.

3 Rule 3

2,5

Vårddygn 1,5 Median=2.35 22 useful observations

1 Rule 1: 12 runs (7-17 runs), OK Rule 2: not OK 0,5 Rule 3: OK Rule 4: OK 0

Akut kirurgi

160

150

140

130

120

Antal operationer Antal 110

100

80 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Månad

Source: Peter Kammerlind, ([email protected] kirurgi ), Project Leader Jönköping County Council, Jonkoping, Sweden.

160 20 useful observations 150 Rule 1: 13 runs (6-16 runs),OK Rule 2: OK 140 Rule 3: OK Rule 4: OK 130

120

Antal operationer Antal 110

100 Median=115.5

80 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Månad

…to gain more knowledge about Shewhart Charts (a.k.a. control charts) ©Copyright 2012 IHI/R. Lloyd Why are Shewhart Charts preferred over Run Charts? Because Shewhart Charts… 1. Are more sensitive than run charts  A run chart cannot detect special causes that are due to point-to- point variation (median versus the mean)  Tests for detecting special causes can be used with Shewhart charts

2. Have the added feature of control limits, which allow us to determine if the process is stable (common cause variation) or not stable (special cause variation).

3. Can be used to define process capability.

4. Allow us to more accurately predict process behavior and future performance.

50.0

45.0 UCL=44.855 A 40.0 An indication of a (Upper

special cause B Control Limit) 35.0

C 30.0 CL=29.250 C 25.0 X (Mean) B 20.0

A 15.0

Number Complaints of LCL=13.645 Measure 10.0 (Lower Control Limit) 5.0 Jan01 Mar01 May01 July01 Sept01 Nov01 Jan02 Mar02 May02 July02 Sept02 Nov02 Time Month

• The reasons(s) for a Special Cause

• Whether or not a Common Cause process should be improved (Is the performance of the process acceptable?)

• How the process should actually be improved or redesigned

©Copyright 2012 IHI/R. Lloyd A Simple Improvement Plan 1. Which process do you want to improve or redesign? 2. Does the process contain non-random patterns or special causes? 3. How do you plan on actually making improvements? What strategies do you plan to follow to make things better? 4. What effect (if any) did your plan have on the process performance?

Run & Shewhart Charts will help you answer Questions 2 & 4. YOU need to figure out the answers to Questions 1 & 3.

Make part of Sustaining and routine Spreading a change operations to other locations

Test under a Implementing a variety of change conditions Theory Testing a Act Plan and change Prediction Study Do Developing a change

W. E. Deming, Out of the Crisis, p.5

• The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. G. Langley, K. Nolan, T. Nolan, C. Norman, L. Provost. Jossey-Bass Publishers., San Francisco, 1996.

• Quality Improvement Through Planned Experimentation. 2nd edition. R. Moen, T. Nolan, L. Provost, McGraw-Hill, NY, 1998.

• The Improvement Handbook. Associates in Process Improvement. Austin, TX, January, 2005.

• A Primer on Leading the Improvement of Systems,” Don M. Berwick, BMJ, 312: pp 619-622, 1996.

• “Accelerating the Pace of Improvement - An Interview with Thomas Nolan,” Journal of Quality Improvement, Volume 23, No. 4, The Joint Commission, April, 1997.

• Brook, R. et. al. “Health System Reform and Quality.” Journal of the American Medical Association 276, no. 6 (1996): 476-480.

• Carey, R. and Lloyd, R. Measuring Quality Improvement in healthcare: A Guide to Statistical Process Control Applications. ASQ Press, Milwaukee, WI, 2001.

• Lloyd, R. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, Sudbury, MA, 2004.

• Nelson, E. et al, “Report Cards or Instrument Panels: Who Needs What? Journal of Quality Improvement, Volume 21, Number 4, April, 1995.

• Solberg. L. et. al. “The Three Faces of Performance Improvement: Improvement, Accountability and Research.” Journal of Quality Improvement 23, no.3 (1997): 135-147.

©Copyright 2012 IHI/R. Lloyd References on Measurement (continued) Benneyan, J., Lloyd, R. and Plsek, P. “Statistical process control as a tool for research and healthcare improvement” Quality and Safety in Health Care 2003;12:458–464

Berwick, Donald M., 1991, Controlling Variation in Health Care: A Consultation from Walter Shewhart, Medical Care, Vol. 29, No. 12, December 1991.

Carey, Raymond G. 2003. Improving Healthcare with Control Charts. ASQ Press, Milwaukee.

Nolan, Tom W. and Provost, Lloyd P., 1990. "Understanding Variation." Quality Progress. May, 1990.

Norman, Clifford and Provost, Lloyd, 1990, "Variation Through the Ages," Quality Progress, Special Variation Issue, ASQC, Milwaukee, December, 1990. p 39-44.

Shewhart, W. A. (1931). Economic Control of Quality of Manufactured Product. New York: D. Van Nostrand Company (reprinted by the American Society for Quality Control, 1980).

Shewhart, W. A. 1939. Statistical Method from the Viewpoint of Quality Control. Washington, D.C.: The Graduate School of the Department of Agriculture.

©Copyright 2012 IHI/R. Lloyd Robert Lloyd, Ph.D. Executive Director Performance Improvement can be reached at: The Institute for Healthcare Improvement (630) 717-5383 Chicago office (630) 717-8564 fax [email protected]