University of South Carolina Scholar Commons Theses and Dissertations 12-14-2015 Statistical Power in Meta-Analysis Jin Liu University of South Carolina - Columbia Follow this and additional works at: https://scholarcommons.sc.edu/etd Part of the Educational Psychology Commons Recommended Citation Liu, J.(2015). Statistical Power in Meta-Analysis. (Doctoral dissertation). Retrieved from https://scholarcommons.sc.edu/etd/3221 This Open Access Dissertation is brought to you by Scholar Commons. It has been accepted for inclusion in Theses and Dissertations by an authorized administrator of Scholar Commons. For more information, please contact [email protected]. STATISTICAL POWER IN META-ANALYSIS by Jin Liu Bachelor of Arts Chongqing University of Arts and Sciences, 2009 Master of Education University of South Carolina, 2012 Submitted in Partial Fulfillment of the Requirements For the Degree of Doctor of Philosophy in Educational Psychology and Research College of Education University of South Carolina 2015 Accepted by: Xiaofeng Liu, Major Professor Christine DiStefano, Committee Member Robert Johnson, Committee Member Brian Habing, Committee Member Lacy Ford, Senior Vice Provost and Dean of Graduate Studies © Copyright by Jin Liu, 2015 All Rights Reserved. ii ACKNOWLEDGEMENTS I would like to express my thanks to all committee members who continuously supported me in the dissertation work. My dissertation chair, Xiaofeng Liu, always inspired and trusted me during this study. I could not finish my dissertation at this time without his support. Brian Habing helped me with R coding and research design. I could not finish the coding process so quickly without his instruction. I appreciate the encouragement from Christine DiStefano and Robert Johnson. When I feel depressed, they always make me believe in myself. They also offered valuable suggestions for my dissertation. I am very grateful to my husband, who always supports and loves me. I cannot imagine how I can go through the whole Ph.D. process without him. I also want to thank my friends at the USC who give me encouragement during the process. I also would like to express my gratitude to my parents and sister for everything they have given to me. iii ABSTRACT Statistical power is important in a meta-analysis study, although few studies have examined the performance of simulated power in meta-analysis. The purpose of this study is to inform researchers about statistical power estimation on two sample mean difference test under different situations: (1) the discrepancy between the analytical power and the actual power and (2) the influence of unequal sample size and unbalanced design on the power. Results indicated that there are noticeable discrepancies between the estimated power and actual power under certain conditions. In general, unbalanced design decreases the statistical power in the meta-analysis. Recommendations are provided for researchers who are interested in power of meta-analysis. iv TABLE OF CONTENTS Acknowledgements ............................................................................................................ iii Abstract .............................................................................................................................. iv List of Tables .................................................................................................................... vii List of Figures .................................................................................................................... ix CHAPTER 1 Statistical Power ........................................................................................... 1 Determinants of Statistical Power ................................................................................... 5 Prospective and Retrospective Power ............................................................................. 8 History of Statistical Power............................................................................................. 9 Computation of Statistical Power .................................................................................. 10 Simulation of Statistical Power ..................................................................................... 16 CHAPTER 2 Meta-analysis .............................................................................................. 19 Limitations of Primary Studies and Narrative Review ................................................. 19 Advantages of Meta-analysis ........................................................................................ 21 Effect Size in Meta-analysis .......................................................................................... 23 Analytical Procedures ................................................................................................... 24 Challenges in Meta-analysis ......................................................................................... 34 Meta-analysis Application............................................................................................. 35 Research Questions ....................................................................................................... 37 v CHAPTER 3 Analysis and Simulation of Statistical Power in Meta-analysis ................. 42 Meta-analysis Practice................................................................................................... 42 Computation of Power in Meta-analysis ....................................................................... 43 Simulation of Statistical Power in Meta-analysis ......................................................... 45 CHAPTER 4 Results......................................................................................................... 54 Type I Error Control ...................................................................................................... 54 Discrepancy in Power Estimation ................................................................................. 57 Influence of Unequal Sample Size and Unbalanced Design on Statistical Power ........ 60 CHAPTER 5 Conclusion ................................................................................................ 101 References ....................................................................................................................... 109 Appendix A R Code ........................................................................................................ 113 Basic Power Simulation (Chapter 2) ........................................................................... 113 Meta-analysis Application (Chapter 3) ....................................................................... 113 Simulation and Analytical Power – Equal Sample Size and Balanced Design .......... 115 Simulation and Analytical Power – Unequal Sample Size and Balanced Design ...... 121 Simulated and Analytical Power – Equal Sample Size and Unbalanced Design ........ 128 Simulated and Analytical Power – Unequal Sample Size and Unbalanced Design ... 134 Graph Functions .......................................................................................................... 141 vi LIST OF TABLES Table 1.1 Decision Making in a Hypothesis Test ............................................................. 18 Table 2.1 Effect Sizes and Sample Sizes of Studies for Mental Rotation Tasks .............. 40 Table 2.2 Summary Results of Meta-analysis across Methods ........................................ 40 Table 2.3 Effect Sizes and Sample Sizes of Studies for Pride .......................................... 40 Table 3.1 Results of Power for the Fixed-effects Model and Random-effects Model ..... 52 Table 4.1 Type I Error Rates of Three Models – Equal Sample size and Balanced Design................................................................................................. 64 Table 4.2 Type I Error Rates of Fixed-Effects model with Varied Population Effect Sizes ............................................................................................. 65 Table 4.3 Statistical Power of the Fixed-effects Model (Equal Sample Size and Balanced Design) ................................................................................................ 66 Table 4.4 Statistical Power of the Random-effects Model (Balanced Design and Equal Sample Size across Studies) ...................................................................... 68 Table 4.5 Statistical Power of the Fixed-effects Model (Maximum sample size: Average sample size * 3) ........................................................................................... 70 Table 4.6 Statistical Power of the Random-effects Model (Maximum sample size: Average sample size * 3) ........................................................................................... 72 Table 4.7 Statistical Power of the Fixed-effects Model (Average sample size ratio: 1:2) ............................................................................... 74 Table 4.8 Statistical Power of the Random-effects Model (Average sample size ratio: 1:2) ................................................................................ 76 Table 4.9 Statistical Power of the Fixed-effects Model (Average sample size ratio – 1:2; Maximum sample size: Average sample size * 3) .............................................. 78 vii Table 4.10 Statistical Power of the Random-effects Model (Average sample size ratio – 1:2; Maximum sample size: Average sample size * 3) ..............................................