APPROPRIATE STATISTICAL ANALYSIS FOR TWO INDEPENDENT GROUPS OF LIKERT-TYPE DATA By Boonyasit Warachan Submitted to the Faculty of the College of Arts and Sciences of American University in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy In Mathematics Education Chair: Mary Gray, Ph.D. Monica Jackson, Ph.D. Jun Lu, Ph.D. Dean of the College of Arts and Sciences Date 2011 American University Washington, D.C. 20016 © COPYRIGHT by Boonyasit Warachan 2011 ALL RIGHTS RESERVED DEDICATION This dissertation is dedicated to my beloved grandfather, Sighto Chankomkhai. APPROPRIATE STATISTICAL ANALYSIS FOR TWO INDEPENDENT GROUPS OF LIKERT-TYPE DATA BY Boonyasit Warachan ABSTRACT The objective of this research was to determine the robustness and statistical power of three different methods for testing the hypothesis that ordinal samples of five and seven Likert categories come from equal populations. The three methods are the two sample t-test with equal variances, the Mann-Whitney test, and the Kolmogorov-Smirnov test. In additional, these methods were employed over a wide range of scenarios with respect to sample size, significance level, effect size, population distribution, and the number of categories of response scale. The data simulations and statistical analyses were performed by using R programming language version 2.13.2. To assess the robustness and power, samples were generated from known distributions and compared. According to returned p-values at different nominal significance levels, empirical error rates and power were computed from the rejection of null hypotheses. Results indicate that the two sample t-test and the Mann-Whitney test were robust for Likert-type data. Also the t-test performed the best to control of Type I error for both 5-point and 7-point Likert scale. For the Kolmogorov-Smirnov test, the Type I error rate ii was not controlled for all circumstances. This means the testing procedure computed from R was not robust for the ordinal Likert-type data because the Type I error rate of this test was lower than the minimum of the robustness criteria. Therefore, the Kolmogorov-Smirnov test was quite conservative. For the statistical power of the test, the Mann-Whitney test was the most powerful for most of the distributions, especially under highly skewed and bimodal distributions. The t-test obtained high statistical power or close to the power from the Man-Whitney test under the uniform, moderate skewed or symmetric distribution with large location shift. iii ACKNOWLEDGMENTS Its accomplishment is owed to my loving family: my parents Jomnian and Somchai, my wife Nisagorn, and my children Chayanit and Chayanon. I would like to acknowledge and extend my heartfelt gratitude to Dr. Mary Gray for her support, patience, understanding, and encouragement which has made the completion of this dissertation possible. I am thankful for the contributions of dissertation committee members Dr. Monica Jackson, and Dr. Jun Lu. Lastly, I offer my regards and blessings to all of those who supported me in any respect during the completion of the dissertation. iv TABLE OF CONTENTS ABSTRACT ........................................................................................................................ ii ACKNOWLEDGMENTS ................................................................................................. iv LIST OF TABLES ............................................................................................................ vii LIST OF FIGURES ......................................................................................................... viii Chapter 1. INTRODUCTION .............................................................................................. 1 Statement of the problem ............................................................................ 1 The purpose of the study ............................................................................. 2 Likert-type data ........................................................................................... 2 Level of measurement ................................................................................. 3 Testing for two independent groups ........................................................... 5 Research questions ...................................................................................... 6 Limitation of the study ................................................................................ 7 2. LITERATURE REVIEW ................................................................................... 8 Treating ordinal as interval data ................................................................. 8 Parametric and nonparametric procedures .................................................. 9 Problem in statistical procedure’s assumtions .......................................... 10 Studies of the robustness and power ......................................................... 11 v Likert-type data in data in education ........................................................ 12 3. RESEARCH METHODOLOGY...................................................................... 14 Simulation of population distributions...................................................... 14 Sample sizes .............................................................................................. 18 Robustness and significance levels ........................................................... 19 Power and levels of effect size .................................................................. 19 The number of Likert categories and the tests .......................................... 19 4. RESULTS ......................................................................................................... 21 Robustness of the statistical tests .............................................................. 21 Relative power of the statistical tests ........................................................ 21 5. CONCLUSION AND RECOMMENDATIONS ............................................. 87 Conclusion ................................................................................................ 87 Recommendations ..................................................................................... 88 Directions for the future research .............................................................. 88 APPENDIX A: R code for the Type I error rate. .............................................................. 89 APPENDIX B: R code for the statistical power ............................................................... 91 REFERENCES ................................................................................................................. 93 vi LIST OF TABLES Table 1. Five population distribution shapes of the 5-point response scale ....................... 17 2. Five population distribution shapes of the 7-point response scale ....................... 17 3. Type I error rate estimates of the uniform distribution . ...................................... 22 4. Statistical power estimates of the uniform distribution at α = 0.01 ...................... 29 5. Statistical power estimates of the uniform distribution at α = 0.05 ...................... 31 6. Statistical power estimates of the uniform distribution at α = 0.10 ...................... 33 7. Type I error rate estimates of the moderate skewed distribution .......................... 35 8. Statistical power estimates of the moderate skewed distribution at α = 0.01. ...... 42 9. Statistical power estimates of the moderate skewed distribution at α = 0.05 ....... 44 10. Statistical power estimates of the moderate skewed distribution at α = 0.10…....46 11. Type I error rate estimates of the highly skewed distribution . ........................... 48 12. Statistical power estimates of the highly skewed distribution at α = 0.01 ............ 55 13. Statistical power estimates of the highly skewed distribution at α = 0.05 ............ 57 14. Statistical power estimates of the highly skewed distribution at α = 0.10 ............ 59 15. Type I error rate estimates of the unimodal symmetric distribution ..................... 61 16. Statistical power estimates of the unimodal symmetric distribution at α = 0.01 .. 68 17. Statistical power estimates of the unimodal symmetric distribution at α = 0.05... 70 18. Statistical power estimates of the unimodal symmetric distribution at α = 010....72 19. Type I error rate estimates of the bimodal symmetric distribution………………74 vii 20. Statistical power estimates of the bimodal symmetric distribution at α = 0.01 .... 81 21. Statistical power estimates of the bimodal symmetric distribution at α = 0.05 .... 83 22. Statistical power estimates of the bimodal symmetric distribution at α = 0.10 .... 85 LIST OF FIGURES Figure 1. Five marginal distributions of 5-point response scale. ......................................... 16 2. Five marginal distributions of 7-point response scale. ......................................... 16 3. Type I error rate estimates of the uniform distribution at α = 0.01 for the 5-point Likert scale……………………………………………………….23 4. Type I error rate estimates of the uniform distribution at α = 0.01 for the 7-point Likert cale……………………………………………………... .. 24 5. Type I error rate estimates of the uniform distribution at α = 0.05 for the 5-point Likert scale……………………………………………………….25 6. Type I error rate estimates of the uniform distribution at α = 0.05 for the 7-point Likert scale……………………………………………………….26 7. Type I error rate