Effect Size and Statistical Power
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
Effect Size (ES)
Lee A. Becker <http://web.uccs.edu/lbecker/Psy590/es.htm> Effect Size (ES) © 2000 Lee A. Becker I. Overview II. Effect Size Measures for Two Independent Groups 1. Standardized difference between two groups. 2. Correlation measures of effect size. 3. Computational examples III. Effect Size Measures for Two Dependent Groups. IV. Meta Analysis V. Effect Size Measures in Analysis of Variance VI. References Effect Size Calculators Answers to the Effect Size Computation Questions I. Overview Effect size (ES) is a name given to a family of indices that measure the magnitude of a treatment effect. Unlike significance tests, these indices are independent of sample size. ES measures are the common currency of meta-analysis studies that summarize the findings from a specific area of research. See, for example, the influential meta- analysis of psychological, educational, and behavioral treatments by Lipsey and Wilson (1993). There is a wide array of formulas used to measure ES. For the occasional reader of meta-analysis studies, like myself, this diversity can be confusing. One of my objectives in putting together this set of lecture notes was to organize and summarize the various measures of ES. In general, ES can be measured in two ways: a) as the standardized difference between two means, or b) as the correlation between the independent variable classification and the individual scores on the dependent variable. This correlation is called the "effect size correlation" (Rosnow & Rosenthal, 1996). These notes begin with the presentation of the basic ES measures for studies with two independent groups. The issues involved when assessing ES for two dependent groups are then described. -
Harmonizing Fully Optimal Designs with Classic Randomization in Fixed Trial Experiments Arxiv:1810.08389V1 [Stat.ME] 19 Oct 20
Harmonizing Fully Optimal Designs with Classic Randomization in Fixed Trial Experiments Adam Kapelner, Department of Mathematics, Queens College, CUNY, Abba M. Krieger, Department of Statistics, The Wharton School of the University of Pennsylvania, Uri Shalit and David Azriel Faculty of Industrial Engineering and Management, The Technion October 22, 2018 Abstract There is a movement in design of experiments away from the classic randomization put forward by Fisher, Cochran and others to one based on optimization. In fixed-sample trials comparing two groups, measurements of subjects are known in advance and subjects can be divided optimally into two groups based on a criterion of homogeneity or \imbalance" between the two groups. These designs are far from random. This paper seeks to understand the benefits and the costs over classic randomization in the context of different performance criterions such as Efron's worst-case analysis. In the criterion that we motivate, randomization beats optimization. However, the optimal arXiv:1810.08389v1 [stat.ME] 19 Oct 2018 design is shown to lie between these two extremes. Much-needed further work will provide a procedure to find this optimal designs in different scenarios in practice. Until then, it is best to randomize. Keywords: randomization, experimental design, optimization, restricted randomization 1 1 Introduction In this short survey, we wish to investigate performance differences between completely random experimental designs and non-random designs that optimize for observed covariate imbalance. We demonstrate that depending on how we wish to evaluate our estimator, the optimal strategy will change. We motivate a performance criterion that when applied, does not crown either as the better choice, but a design that is a harmony between the two of them. -
Optimal Design of Experiments Session 4: Some Theory
Optimal design of experiments Session 4: Some theory Peter Goos 1 / 40 Optimal design theory Ï continuous or approximate optimal designs Ï implicitly assume an infinitely large number of observations are available Ï is mathematically convenient Ï exact or discrete designs Ï finite number of observations Ï fewer theoretical results 2 / 40 Continuous versus exact designs continuous Ï ½ ¾ x1 x2 ... xh Ï » Æ w1 w2 ... wh Ï x1,x2,...,xh: design points or support points w ,w ,...,w : weights (w 0, P w 1) Ï 1 2 h i ¸ i i Æ Ï h: number of different points exact Ï ½ ¾ x1 x2 ... xh Ï » Æ n1 n2 ... nh Ï n1,n2,...,nh: (integer) numbers of observations at x1,...,xn P n n Ï i i Æ Ï h: number of different points 3 / 40 Information matrix Ï all criteria to select a design are based on information matrix Ï model matrix 2 3 2 T 3 1 1 1 1 1 1 f (x1) ¡ ¡ Å Å Å T 61 1 1 1 1 17 6f (x2)7 6 Å ¡ ¡ Å Å 7 6 T 7 X 61 1 1 1 1 17 6f (x3)7 6 ¡ Å ¡ Å Å 7 6 7 Æ 6 . 7 Æ 6 . 7 4 . 5 4 . 5 T 100000 f (xn) " " " " "2 "2 I x1 x2 x1x2 x1 x2 4 / 40 Information matrix Ï (total) information matrix 1 1 Xn M XT X f(x )fT (x ) 2 2 i i Æ σ Æ σ i 1 Æ Ï per observation information matrix 1 T f(xi)f (xi) σ2 5 / 40 Information matrix industrial example 2 3 11 0 0 0 6 6 6 0 6 0 0 0 07 6 7 1 6 0 0 6 0 0 07 ¡XT X¢ 6 7 2 6 7 σ Æ 6 0 0 0 4 0 07 6 7 4 6 0 0 0 6 45 6 0 0 0 4 6 6 / 40 Information matrix Ï exact designs h X T M ni f(xi)f (xi) Æ i 1 Æ where h = number of different points ni = number of replications of point i Ï continuous designs h X T M wi f(xi)f (xi) Æ i 1 Æ 7 / 40 D-optimality criterion Ï seeks designs that minimize variance-covariance matrix of ¯ˆ ¯ 2 T 1¯ Ï . -
Statistical Power, Sample Size, and Their Reporting in Randomized Controlled Trials
Statistical Power, Sample Size, and Their Reporting in Randomized Controlled Trials David Moher, MSc; Corinne S. Dulberg, PhD, MPH; George A. Wells, PhD Objective.\p=m-\Todescribe the pattern over time in the level of statistical power and least 80% power to detect a 25% relative the reporting of sample size calculations in published randomized controlled trials change between treatment groups and (RCTs) with negative results. that 31% (22/71) had a 50% relative Design.\p=m-\Ourstudy was a descriptive survey. Power to detect 25% and 50% change, as statistically significant (a=.05, relative differences was calculated for the subset of trials with results in one tailed). negative Since its the of which a was used. Criteria were both publication, report simple two-group parallel design developed Freiman and has been cited to trial results as or and to the outcomes. colleagues2 classify positive negative identify primary more than 700 times, possibly indicating Power calculations were based on results from the primary outcomes reported in the seriousness with which investi¬ the trials. gators have taken the findings. Given Population.\p=m-\Wereviewed all 383 RCTs published in JAMA, Lancet, and the this citation record, one might expect an New England Journal of Medicine in 1975, 1980, 1985, and 1990. increase over time in the awareness of Results.\p=m-\Twenty-sevenpercent of the 383 RCTs (n=102) were classified as the consequences of low power in pub¬ having negative results. The number of published RCTs more than doubled from lished RCTs and, hence, an increase in 1975 to 1990, with the proportion of trials with negative results remaining fairly the reporting of sample size calcula¬ stable. -
Statistical Inference Bibliography 1920-Present 1. Pearson, K
StatisticalInferenceBiblio.doc © 2006, Timothy G. Gregoire, Yale University http://www.yale.edu/forestry/gregoire/downloads/stats/StatisticalInferenceBiblio.pdf Last revised: July 2006 Statistical Inference Bibliography 1920-Present 1. Pearson, K. (1920) “The Fundamental Problem in Practical Statistics.” Biometrika, 13(1): 1- 16. 2. Edgeworth, F.Y. (1921) “Molecular Statistics.” Journal of the Royal Statistical Society, 84(1): 71-89. 3. Fisher, R. A. (1922) “On the Mathematical Foundations of Theoretical Statistics.” Philosophical Transactions of the Royal Society of London, Series A, Containing Papers of a Mathematical or Physical Character, 222: 309-268. 4. Neyman, J. and E. S. Pearson. (1928) “On the Use and Interpretation of Certain Test Criteria for Purposes of Statistical Inference: Part I.” Biometrika, 20A(1/2): 175-240. 5. Fisher, R. A. (1933) “The Concepts of Inverse Probability and Fiducial Probability Referring to Unknown Parameters.” Proceedings of the Royal Society of London, Series A, Containing Papers of Mathematical and Physical Character, 139(838): 343-348. 6. Fisher, R. A. (1935) “The Logic of Inductive Inference.” Journal of the Royal Statistical Society, 98(1): 39-82. 7. Fisher, R. A. (1936) “Uncertain inference.” Proceedings of the American Academy of Arts and Sciences, 71: 245-258. 8. Berkson, J. (1942) “Tests of Significance Considered as Evidence.” Journal of the American Statistical Association, 37(219): 325-335. 9. Barnard, G. A. (1949) “Statistical Inference.” Journal of the Royal Statistical Society, Series B (Methodological), 11(2): 115-149. 10. Fisher, R. (1955) “Statistical Methods and Scientific Induction.” Journal of the Royal Statistical Society, Series B (Methodological), 17(1): 69-78. -
Advanced Power Analysis Workshop
Advanced Power Analysis Workshop www.nicebread.de PD Dr. Felix Schönbrodt & Dr. Stella Bollmann www.researchtransparency.org Ludwig-Maximilians-Universität München Twitter: @nicebread303 •Part I: General concepts of power analysis •Part II: Hands-on: Repeated measures ANOVA and multiple regression •Part III: Power analysis in multilevel models •Part IV: Tailored design analyses by simulations in 2 Part I: General concepts of power analysis •What is “statistical power”? •Why power is important •From power analysis to design analysis: Planning for precision (and other stuff) •How to determine the expected/minimally interesting effect size 3 What is statistical power? A 2x2 classification matrix Reality: Reality: Effect present No effect present Test indicates: True Positive False Positive Effect present Test indicates: False Negative True Negative No effect present 5 https://effectsizefaq.files.wordpress.com/2010/05/type-i-and-type-ii-errors.jpg 6 https://effectsizefaq.files.wordpress.com/2010/05/type-i-and-type-ii-errors.jpg 7 A priori power analysis: We assume that the effect exists in reality Reality: Reality: Effect present No effect present Power = α=5% Test indicates: 1- β p < .05 True Positive False Positive Test indicates: β=20% p > .05 False Negative True Negative 8 Calibrate your power feeling total n Two-sample t test (between design), d = 0.5 128 (64 each group) One-sample t test (within design), d = 0.5 34 Correlation: r = .21 173 Difference between two correlations, 428 r₁ = .15, r₂ = .40 ➙ q = 0.273 ANOVA, 2x2 Design: Interaction effect, f = 0.21 180 (45 each group) All a priori power analyses with α = 5%, β = 20% and two-tailed 9 total n Two-sample t test (between design), d = 0.5 128 (64 each group) One-sample t test (within design), d = 0.5 34 Correlation: r = .21 173 Difference between two correlations, 428 r₁ = .15, r₂ = .40 ➙ q = 0.273 ANOVA, 2x2 Design: Interaction effect, f = 0.21 180 (45 each group) All a priori power analyses with α = 5%, β = 20% and two-tailed 10 The power ofwithin-SS designs Thepower 11 May, K., & Hittner, J. -
Approximation Algorithms for D-Optimal Design
Approximation Algorithms for D-optimal Design Mohit Singh School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA 30332, [email protected]. Weijun Xie Department of Industrial and Systems Engineering, Virginia Tech, Blacksburg, VA 24061, [email protected]. Experimental design is a classical statistics problem and its aim is to estimate an unknown m-dimensional vector β from linear measurements where a Gaussian noise is introduced in each measurement. For the com- binatorial experimental design problem, the goal is to pick k out of the given n experiments so as to make the most accurate estimate of the unknown parameters, denoted as βb. In this paper, we will study one of the most robust measures of error estimation - D-optimality criterion, which corresponds to minimizing the volume of the confidence ellipsoid for the estimation error β − βb. The problem gives rise to two natural vari- ants depending on whether repetitions of experiments are allowed or not. We first propose an approximation 1 algorithm with a e -approximation for the D-optimal design problem with and without repetitions, giving the first constant factor approximation for the problem. We then analyze another sampling approximation algo- 4m 12 1 rithm and prove that it is (1 − )-approximation if k ≥ + 2 log( ) for any 2 (0; 1). Finally, for D-optimal design with repetitions, we study a different algorithm proposed by literature and show that it can improve this asymptotic approximation ratio. Key words: D-optimal Design; approximation algorithm; determinant; derandomization. History: 1. Introduction Experimental design is a classical problem in statistics [5, 17, 18, 23, 31] and recently has also been applied to machine learning [2, 39]. -
A Review on Optimal Experimental Design
A Review On Optimal Experimental Design Noha A. Youssef London School Of Economics 1 Introduction Finding an optimal experimental design is considered one of the most impor- tant topics in the context of the experimental design. Optimal design is the design that achieves some targets of our interest. The Bayesian and the Non- Bayesian approaches have introduced some criteria that coincide with the target of the experiment based on some speci¯c utility or loss functions. The choice between the Bayesian and Non-Bayesian approaches depends on the availability of the prior information and also the computational di±culties that might be encountered when using any of them. This report aims mainly to provide a short summary on optimal experi- mental design focusing more on Bayesian optimal one. Therefore a background about the Bayesian analysis was given in Section 2. Optimality criteria for non-Bayesian design of experiments are reviewed in Section 3 . Section 4 il- lustrates how the Bayesian analysis is employed in the design of experiments. The remaining sections of this report give a brief view of the paper written by (Chalenor & Verdinelli 1995). An illustration for the Bayesian optimality criteria for normal linear model associated with di®erent objectives is given in Section 5. Also, some problems related to Bayesian optimal designs for nor- mal linear model are discussed in Section 5. Section 6 presents some ideas for Bayesian optimal design for one way and two way analysis of variance. Section 7 discusses the problems associated with nonlinear models and presents some ideas for solving these problems. -
Minimax Efficient Random Designs with Application to Model-Robust Design
Minimax efficient random designs with application to model-robust design for prediction Tim Waite [email protected] School of Mathematics University of Manchester, UK Joint work with Dave Woods S3RI, University of Southampton, UK Supported by the UK Engineering and Physical Sciences Research Council 8 Aug 2017, Banff, AB, Canada Tim Waite (U. Manchester) Minimax efficient random designs Banff - 8 Aug 2017 1 / 36 Outline Randomized decisions and experimental design Random designs for prediction - correct model Extension of G-optimality Model-robust random designs for prediction Theoretical results - tractable classes Algorithms for optimization Examples: illustration of bias-variance tradeoff Tim Waite (U. Manchester) Minimax efficient random designs Banff - 8 Aug 2017 2 / 36 Randomized decisions A well known fact in statistical decision theory and game theory: Under minimax expected loss, random decisions beat deterministic ones. Experimental design can be viewed as a game played by the Statistician against nature (Wu, 1981; Berger, 1985). Therefore a random design strategy should often be beneficial. Despite this, consideration of minimax efficient random design strategies is relatively unusual. Tim Waite (U. Manchester) Minimax efficient random designs Banff - 8 Aug 2017 3 / 36 Game theory Consider a two-person zero-sum game. Player I takes action θ 2 Θ and Player II takes action ξ 2 Ξ. Player II experiences a loss L(θ; ξ), to be minimized. A random strategy for Player II is a probability measure π on Ξ. Deterministic actions are a special case (point mass distribution). Strategy π1 is preferred to π2 (π1 π2) iff Eπ1 L(θ; ξ) < Eπ2 L(θ; ξ) : Tim Waite (U. -
Effect Sizes (ES) for Meta-Analyses the Standardized Mean Difference
Kinds of Effect Sizes The effect size (ES) is the DV in the meta analysis. d - standardized mean difference Effect Sizes (ES) for Meta-Analyses – quantitative DV – between groups designs standardized gain score – pre-post differences • ES – d, r/eta & OR – quantitative DV • computing ESs – within-groups design • estimating ESs r – correlation/eta • ESs to beware! – converted from sig test (e.g., F, t, X2)or set of means/stds • interpreting ES – between or within-groups designs or tests of association • ES transformations odds ratio A useful ES: • ES adustments – binary DVs • is standardized – between groups designs • outlier identification Univariate (proportion or mean) • a standard error can be – prevalence rates calculated The Standardized Mean Difference (d) • A Z-like summary statistic that tells the size of the difference between the means of the two groups • Expresses the mean difference in Standard Deviation units – d = 1.00 Tx mean is 1 std larger than Cx mean – d = .50 Tx mean is 1/2 std larger than Cx mean – d = -.33 Tx mean is 1/3 std smaller than Cx mean • Null effect = 0.00 • Range from -∞ to ∞ • Cohen’s effect size categories – small = 0.20 medium = 0.50 large = 0.80 The Standardized Mean Difference (d) Equivalent formulas to calculate The Standardized Mean Difference (d) •Calculate Spooled using MSerror from a 2BG ANOVA √MSerror = Spooled • Represents a standardized group mean difference on an inherently continuous (quantitative) DV. • Uses the pooled standard deviation •Calculate Spooled from F, condition means & ns • There is a wide variety of d-like ESs – not all are equivalent – Some intended as sample descriptions while some intended as population estimates – define and use “n,” “nk” or “N” in different ways – compute the variability of mean difference differently – correct for various potential biases Equivalent formulas to calculate The Standardized Mean Difference (d) • Calculate d directly from significance tests – t or F • Calculate t or F from exact p-value & df. -
One-Way Analysis of Variance F-Tests Using Effect Size
PASS Sample Size Software NCSS.com Chapter 597 One-Way Analysis of Variance F-Tests using Effect Size Introduction A common task in research is to compare the averages of two or more populations (groups). We might want to compare the income level of two regions, the nitrogen content of three lakes, or the effectiveness of four drugs. The one-way analysis of variance compares the means of two or more groups to determine if at least one mean is different from the others. The F test is used to determine statistical significance. F tests are non-directional in that the null hypothesis specifies that all means are equal, and the alternative hypothesis simply states that at least one mean is different from the rest. The methods described here are usually applied to the one-way experimental design. This design is an extension of the design used for the two-sample t test. Instead of two groups, there are three or more groups. In our more advanced one-way ANOVA procedure, you are required to enter hypothesized means and variances. This simplified procedure only requires the input of an effect size, usually f, as proposed by Cohen (1988). Assumptions Using the F test requires certain assumptions. One reason for the popularity of the F test is its robustness in the face of assumption violation. However, if an assumption is not even approximately met, the significance levels and the power of the F test are invalidated. Unfortunately, in practice it often happens that several assumptions are not met. This makes matters even worse. -
NBER TECHNICAL WORKING PAPER SERIES USING RANDOMIZATION in DEVELOPMENT ECONOMICS RESEARCH: a TOOLKIT Esther Duflo Rachel Glenner
NBER TECHNICAL WORKING PAPER SERIES USING RANDOMIZATION IN DEVELOPMENT ECONOMICS RESEARCH: A TOOLKIT Esther Duflo Rachel Glennerster Michael Kremer Technical Working Paper 333 http://www.nber.org/papers/t0333 NATIONAL BUREAU OF ECONOMIC RESEARCH 1050 Massachusetts Avenue Cambridge, MA 02138 December 2006 We thank the editor T.Paul Schultz, as well Abhijit Banerjee, Guido Imbens and Jeffrey Kling for extensive discussions, David Clingingsmith, Greg Fischer, Trang Nguyen and Heidi Williams for outstanding research assistance, and Paul Glewwe and Emmanuel Saez, whose previous collaboration with us inspired parts of this chapter. The views expressed herein are those of the author(s) and do not necessarily reflect the views of the National Bureau of Economic Research. © 2006 by Esther Duflo, Rachel Glennerster, and Michael Kremer. All rights reserved. Short sections of text, not to exceed two paragraphs, may be quoted without explicit permission provided that full credit, including © notice, is given to the source. Using Randomization in Development Economics Research: A Toolkit Esther Duflo, Rachel Glennerster, and Michael Kremer NBER Technical Working Paper No. 333 December 2006 JEL No. C93,I0,J0,O0 ABSTRACT This paper is a practical guide (a toolkit) for researchers, students and practitioners wishing to introduce randomization as part of a research design in the field. It first covers the rationale for the use of randomization, as a solution to selection bias and a partial solution to publication biases. Second, it discusses various ways in which randomization can be practically introduced in a field settings. Third, it discusses designs issues such as sample size requirements, stratification, level of randomization and data collection methods.