Healthcare Quality and Safety Commission Scientific Symposium October 2016 Designing Experiments Professor Nigel Grigg “My life is an experiment I never had a chance to properly design." Diana Ballard Presentation overview • Experience with experiments • Process and system variation recap • Investigating processes and systems • Principles of experimentation and some experimental designs • Case study Experience with Experimentation • Industry (doing) – Aircraft industry: 2+ year parallel improvement projects aimed at optimising the integrity of superalloy turbines and other forged products • Industry (training / advising) – Nanotechnology startup company : 2+ year providing advice on DOE method and analysis (assays, process improvement) – Food Industry: 2+ year M.Phil supervision (filling processes); six sigma training for process managers – Medical device manufacture; banking (6 sigma training) • Research (supervision and advisory roles) – PhDs: Rice growing trials (Cambodia); Rice chewing trials (Thailand) Modelling the tongue, and mastication process (NZ) Optimising textile production processes (Sri Lanka) – Various Masters and Post Graduate Diploma projects Inputs Process Outputs Suppliers Materials Customers / Information Stakeholders Customers Facilities Products staff and Services ‘Voice’ of the process feedback / quality data ‘Voice’ of the customer Operating System External External operation operation suppliers customers Customer Customer Customer Customer requirements requirements requirements ...Supply Chain / Value Stream… Inside the ‘black box’: the processing steps Process variation recap Process data Inputs Process Outputs Performance measurement Process measure exhibits variability BUT How much of it is actually the process?? Graphical prediction methods in health care – humble beginnings A ‘run chart’ used by Dr. Carl Wunderlich in 1870, to study nonrandom oral temperature patterns of a pneumonia patients Ring, E. F. J. (2007), The historical development of temperature measurement in medicine, Infrared Physics & Technology, 49(3), 297-301 Sources of process variation include... 1. Process – Common cause – Special cause 2. Measurement system – People – Devices – Methods – Metrics – Analysis – (even) the interpretation of analysis! Processes can be... • Stable (in control) – Common cause variation only present – Variation is therefore stable – Future variation can be therefore predicted (within certain limits) • Unstable (out of control) – Both common and special cause variation present – Variation is therefore unpredictable Q. Which type of variation is easier to eliminate? So how do we fix the thorny problem of common cause variation? Study the process Test theories Develop knowledge Make changes Classification of research studies Enumerative studies Analytic studies Population-based Process-based • System conditions are essentially • System conditions are varying static over time over time • Purpose is parameter estimation • Purpose is process • Analytical methods are statistical, improvement (hypothesis Tests, CIs, • Analytical Methods are probabilities) graphical (time-based) Overarching cycles Enumerative studies Analytic studies (traditional science) (‘quality science’) Theory Plan Empirical generalisation Hypotheses Act Do Experimental Study observations Both are ‘probes for knowledge’, involving testing interventions by manipulating variables and observing effect upon other variables. Improvement Cycle • Define problem • Carry out the study or experimental design • Define Objectives • Observe the stability of • Evaluate current conditions and other knowledge sources of variation • Develop hypotheses during study • Make predictions Plan Do • Design study Act Study • Decide whether or not • Analyse the data to make the system change • Compare the results to the predictions • Compare to current knowledge Nothing is quite as practical as a good theory Other theories… The Interplay between theory and data “Experience teaches us nothing without theory, but theory without experience is mere intellectual play” Immanuel Kant “without theory it is impossible to make sense of empirically generated data”. Voss et al (2002) Campbell & Stanley classification of research designs • Pre-experimental – Clinical case study – Pretest-posttest design (before-after study) – Static-group comparison • Experimental – Randomised controlled trials • Quasi-experimental – Experiments lacking the random allocation Campbell DT, Stanley JC & Gage NL (1966), Experimental and Quasi-Experimental Designs for Research, Chicago: Rand McNally College Pub Co Classifications of Quality Improvement study designs • Observational • Retrospective • Pre-experimental • Quasi-experimental • Experimental (e.g. factorial designs) • Time-series Speroff T. (2004), Study Designs for PDSA Quality Improvement Research Quality Management in Health Care, 13(1), pp.17-32 Example: Patient falls (observational) Problem • The objective of the study was to provide a methodology to analyse the incidence of inpatient falls in elder health wards of the hospital. • Prior to the study, the incidence of patient falls was difficult to determine due to the complex interaction of patient factors and environmental factors • For this reason there was no proactive interventional procedure in place to reduce the incidence of patient falls. Jayamaha, N. P. & Grigg, N. P. (2009). Monitoring and improving the operational performance of a New Zealand healthcare organisation: a study on patient falls. Proceedings of the 7th ANZAM Operations, Supply Chain and Services Management Symposium, pp.238-250 Falls rate at the Elder Health Unit (Financial Year: 2007-2008) 25 UCL=23.49 20 s y a d - d e 15 b 0 _ 0 0 U=11.62 1 r 10 e p s l l a F 5 0 LCL=0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2WK Period Tests performed with unequal sample sizes (bed-days change from period to period) Example: Cause and Effect Diagram for High Incidence of Inpatient Falls Outcomes of the study • Development of control chart methodology for monitoring patient falls • Searching the root causes for high incidence of falls • Appreciation that there is a wide gap between existing state of the process and the desired state of the process • Experimentation with high-tech infrared patient movement monitors fitted to the beds of high risk patients • Monitored the drop in average incidence rates to develop new control limits What is Design of Experiments (DoE)? ‘A collection of methods and a strategy to make a change to a product or process and observe the effect of that change on one of more quality characteristics with the purpose of helping experimenters gain the most information with the resources available’. (Moen, Nolan and Provost, ‘Quality improvement through planned experimentation’ (2nd ed). p405 A few prominent industrial statisticians / experimenters 1900 1920 1940 1960 A few key contemporary (and not-so- contemporary) authors Davis Balestracci Ronald Moen 1999 Thomas Nolan Lloyd Provost George Box 1991 Stuart Hunter William Hunter 1978 Douglas Montgomery 1976 R.A. Fisher 1935 Experimental objectives Controllable factors x1 x2 xp . Output inputs y . z1 z2 zq Uncontrollable factors Experimental objectives x1 x2 xp . y . 1. Where to set the input factors (x’s) so that output (y) is nearer to its designed target value Experimental objectives x1 x2 xp . y . 2. Where to set the input factors (x’s) so that variability in y is reduced Experimental objectives 3. Where to set the input factors (x’s) so that the influence of the uncontrollable variables (z’s) is minimised (robustisation) . y . z1 z2 zq Experimental objectives . y . 4. To simply determine which variables are significantly influential on the response, y KISS (Keep it simple and sequential) Assessing current knowledge 38 Some common designs ...and necessary key terms… Response variable • The measured outcome(s) of each experimental trial (yi) • Key questions – Do you know the measurement error? – Do you know the measurement variability – Can you measure it? – What kind of value results? – How can / should those data be analysed? Q. What happens if the response variable’s measurement error is excessive? measurement system Does it exhibit necessary... • Stability (of mean) • Lack of Bias • Consistency (of variation) • Linearity • Discrimination And how do you know?? (Also, beware of Likert scales) 2 Likert score distributions, each with n = 1100 observations 350 450 400 300 350 250 300 200 250 150 200 150 100 100 50 50 0 0 1 2 3 4 5 1 2 3 4 5 Value Freq Value Freq 1 300 1 100 2 200 2 250 3 100 Equivalent? 3 400 4 200 4 250 5 300 5 100 Count 1100 Count 1100 Mean 3 Mean 3 Example: Recent measurement project (cold chain consistency for HIV testing kits) • Complexity of global supply chains can result in undesirable consequences • Common for NZ healthcare providers to procure pharmaceutical products and laboratory products from as far as the USA and UK. • Cold chain products are highly impacted by supply chain inefficiency or ineffectiveness. • Problem: The lab manager (customer) wanted the kits to be delivered at the specified temperature: 20C – 80C • Proposition: Laboratory cold chain products do not receive the same treatment as pharmaceutical cold chain products, even though the quality of the former (e.g. HIV AIDS test kits, reagents) can have as significant an effect on the patient as the quality of the latter. Dixon,J.; Jayamaha, N.P.; Grigg, N. (2014) Laboratory Cold Chain Quality Performance - An