Screening Designs and Design Evaluation Bradley Jones Science of Test April 2017 Copyright © 2008, SAS Institute Inc. All rights reserved. Statistical Facts of Life 1. All real processes experience variability 2. Statistical models are approximations What is the purpose of designing experiments? Model -> Design Goals 1. Make the variance of unknowns in a model as small as possible 2. Avoid bias in estimates of unknowns. Visualizing Bias and Variance What makes a design good? 1. Low variance of the coefficients. 2. Low variance of predicted responses. 3. Minimal aliasing of terms in the model from likely effects that are not in the model (0.5 or less). 4. Likely effects that are not in the model are not confounded (smaller correlations are better). The first two deal with variance – the last two with bias. Reducing variance and bias are fundamental goals. How to lower variance using design 1. Replicate or otherwise add runs to a design 2. For continuous factors, increase the range between high and low settings 3. Use blocking, i.e. run your experiment in homogeneous groups of runs 4. Reduce or eliminate correlation between factors How to lower bias using design 1. Randomize the run order 2. Avoid confounding JMP Variance/Bias Demonstration Summary 1. Variance and bias are fundamental criteria for evaluating designs. 2. Designed experiments aim to reduce variance or both variance and bias compared to trial and error testing. Screening or finding the “vital few” important factors Screening Experiments Goals 1. Define Screening 2. Introduce three types of screening design – pros and cons 3. Compare a textbook screening design to a modern design. What is a screening experiment? A major use of fractional factorials is in screening experiments – experiments in which many factors are considered and the objective is to identify those factors (if any) that have large effects. Montgomery, D.C. (2005) Design and Analysis of Experiments, Wiley, New York, page 283. More about screening experiments Screening experiments generally involve many factors (>5) 1. Traditionally, each factor has two levels. 2. The a priori model is either main effects only or main effects plus two-factor interactions. 3. The number of experimental runs is kept small to make them inexpensive and fast. 4. Screening experiments are the most commonly run industrial applications. In factor screening experiments… We start with little prior knowledge and a large initial set of potential factors influencing the response Our purpose is to identify the smaller set of active factors. Primary goal : identify active main effects (MEs) Secondary goal: identify a few active 2nd order effects What makes a factor important? 1. Having a large main effect 2. Being involved in a two-factor interaction effect 3. Having a large nonlinear effect Typical Screening Assumptions Sparsity of effects – fewer than half of the factors will be active. Hierarchy of effects – main effects larger and more likely than 2nd order effect Effect heredity – two-factor interactions are much more likely if both main effects are active Supporting article in Complexity (2006) Complexity article method and results Full-factorial designs from three to seven factors 113 sets of responses re-analyzed to check screening assumptions Findings: 1) Percentage of Main Effects was 41% 2) Main effects averaged twice as large and three times as many as 2FIs 3) Percentage of 2FI was 11% overall 4) Conditional percentages 1) Both main effects active 33% 2) One main effect active 4.5% 3) No main effect active 0.5% Number of active 2FIs about 1/3 of the number of active MEs So, if there are 6 factors, for example, you would expect 2 or 3 active main effects and one or two 2FIs Traditional Screening Designs Resolution III Fractional Factorial Resolution IV Fractional Factorial Plackett-Burman Industrial experience indicates that most designs are not replicated. Budget constraints limit the allowable number of runs. A Graph for Quick Design Comparison The correlation cell plot… 1. Cells off the diagonal are white for orthogonal effects 2. Confounded effects are colored black 3. The greater the correlation, the darker the cell Cell plots of 4 screening designs for 6 factors Desirable design features orthogonality of the MEs, orthogonality of MEs and 2FIs, orthogonality of 2FIs with each other, small run size Problems with Standard Screening Designs Resolution III fractional factorial designs confound main effects with two-factor interactions and two-factor interactions with each other. Orthogonal Main Effects A = CE BC = EF Possible Consequences 1. An active two-factor interaction may cause its confounded main effect to look insignificant when it is actually active. 2. Conversely, an active two-factor interaction can cause its confounded main effect to look significant when it is not. 3. Active two-factor interactions are generally not uniquely identifiable. Problems with Standard Screening Designs Nonregular orthogonal designs such as Plackett-Burman designs have “complex aliasing” of the main effects by two-factor interactions. Two-factor interactions are also correlated. Main Effects Main Effects correlated with orthogonal Two-Factor Interactions Two-Factor Interactions correlated with each other Possible Consequences 1. An active two-factor interaction may cause its correlated main effect to look insignificant when it is actually active. 2. Conversely, an active two-factor interaction can cause its correlated main effect to look significant when it is not. Problems with Standard Screening Designs Resolution IV fractional factorial designs confound two-factor interactions with each other. Main Effects Orthogonal orthogonal to Main Effects Two-Factor Interactions Two-Factor Interactions are confounded with each other. Possible Consequence An active two-factor interaction may not be uniquely identifiable, so further runs may be necessary to resolve the active effects. Further Problems with Standard Screening Designs 1. Adding center points to detect curvature works but does not identify which factors are responsible. 2. Blocking is restrictive – the number of blocks may only be a power of two. Screening Conundrum – too many potential effects Number of Factors Number of Two-Factor Interactions 6 15 7 21 8 28 9 36 10 45 Two-factor interactions matter but for standard screening scenarios there are so many of them that estimating them all is too expensive. Screening Design Example Scenario 1. There are 6 factors. 2. The model consists of all the main effects. 3. Budget for 12 experimental runs. Engineering Data with Responses Run Methanol Ethanol Propanol Butanol pH Time Yield (mg) 1 0 0 0 10 6 1 10.94 2 0 10 0 0 9 1 15.79 3 0 10 0 10 9 2 25.96 4 10 10 10 0 6 1 35.92 5 0 0 10 0 6 2 22.92 6 0 10 10 10 6 1 23.54 7 10 10 0 0 6 2 47.44 8 10 0 0 0 9 1 19.80 9 10 0 10 10 9 1 29.48 10 0 0 10 0 9 2 17.13 11 10 10 10 10 9 2 43.75 12 10 0 0 10 6 2 40.86 Bie, Xiaomei, et. al. “Screening the main factors affecting extraction of the antimicrobial substance from Bacillus sp. fmbJ using the Plackett–Burman method “ World Journal of Microbiology & Biotechnology (2005) 21: 925–928 Springer 2005 Main – Effects Models All Main Effects Model in Published Paper Researchers removed two main effects that were not statistically significant. Copyright © 2008, SAS Institute Inc. All rights reserved. JMP Screening Design Demonstration All Main Effects Model Definitive Screening Design Tutorial 1. What is a definitive screening design (DSD)? 2. Conference matrix based DSDs 3. Analyzing DSDs 4. Adding two-level categorical factors 5. Blocking schemes for DSDs A six-factor screening example Experimenter has six factors to investigate and is concerned about two-factor interactions Can only afford 12 runs Standard advice: Plackett-Burman 12-run design. But the PB design has aliasing between main effects and 2FIs It is possible to create an alias-optimal design where main effects are completely independent of 2FIs 36 Alias optimal design, keeping D-efficiency > 91% D-efficiency: 92% vs. orthogonal design Cost: 10% longer confidence intervals for MEs 37 Alias matrix: Can we get rid of the nonzero guys in row 1? 38 Answer: Yes, lower the D-efficiency constraint to 85% New Design 1 New Design 2 D-efficiency: 92% D-efficiency: 85% (Not Orthogonal) (Orthogonal) “Size” of Alias Matrix = 6/9 “Size” of Alias Matrix = 0 39 Some Interesting Facts About Design 2 Each pair of rows mirror each other (foldover pairs). Exactly one center point in each run. D-efficiency is 85%. Size of A is zero, so the alias matrix is all zeros! Hey, pretty cool structure! 40 OK, Nice Result for Six Factors Does This Work for Any Number of Factors? Foldover structure guarantees A = 0 (I will show this later) So all main effects will be unbiased by active interactions Center points in each row guarantees your ability to estimate quadratic effects for all factors If we impose this structure, we’re guaranteed a nice design if we pick the plus-or-minus one entries carefully May not be orthogonal, however 41 Design structure for definitive screening designs 2 42 JOURNAL OF QUALITY TECHNOLOGY , VOL. 43, NO. 1, QICID: 33051, January 2011, pp. 1-15 43 Design Properties 1. The number of required runs is only one more than twice the number of factors. 44 Design Properties 1. The number of required runs is only one more than twice the number of factors. 2. Unlike resolution III designs, main effects are completely independent of two-factor interactions. 45 Design Properties 1. The number of required runs is only one more than twice the number of factors.