The Role of Between-Case Effect Size in Conducting, Interpreting, and Summarizing Single-Case Research Authors William R. Shadish, University of California, Merced Larry V. Hedges, Northwestern University Robert H. Horner, University of Oregon Samuel L. Odom, University of North Carolina, Chapel Hill National Center for Education Research (NCER) Meredith Larson (Project Officer) Phill Gagné National Center for Special Education Research (NCSER) Kimberley Sprague December 2015 NCER 2015-002 U.S. DEPARTMENT OF EDUCATION This paper was prepared for the National Center for Education Research, Institute of Education Sciences under Contract ED-IES-12-D-0015. Meredith Larson was the project officer. Disclaimer The Institute of Education Sciences at the U.S. Department of Education contracted with Westat to develop a paper on the use of effect sizes in single-case design research. The views expressed in this report are those of the authors, and they do not necessarily represent the opinions and positions of the Institute of Education Sciences or the U.S. Department of Education. U.S. Department of Education Arne Duncan, Secretary Institute of Education Sciences Ruth Neild, Deputy Director for Policy and Research, Delegated Duties of the Director National Center for Education Research Thomas W. Brock, Commissioner National Center for Special Education Research Joan McLaughlin, Commissioner December 2015 This report is in the public domain. Authorization to reproduce it in whole or in part is granted. Although permission to reprint this publication is not necessary, the citation should be: Shadish, W.R., Hedges, L.V., Horner, R.H., and Odom, S.L. (2015). The Role of Between-Case Effect Size in Conducting, Interpreting, and Summarizing Single-Case Research (NCER 2015-002) Washington, DC: National Center for Education Research, Institute of Education Sciences, U.S. Department of Education. This report is available on the Institute website at http://ies.ed.gov/. Alternate Formats Upon request, this report is available in alternate formats such as Braille, large print, audiotape, or computer diskette. For more information, please contact the Department’s Alternate Format Center at 202-260-0852 or 202-260-0818. Disclosure of Potential Conflict of Interest Westat Inc. is the prime contractor for the NCER Analysis and Research Management Support project, under which this paper was developed. Tamara Nimkoff was the project manager. The authors of this paper are not aware of any conflicts of interest. Acknowledgments The authors thank the members of the Technical Advisory Group (Ann Kaiser, Vanderbilt University; Thomas Kratochwill, University of Wisconsin Madison; Kathleen Lane, University of Kansas; Daniel Maggin, University of Illinois, Chicago) for their helpful comments on a previous draft of this paper. Of course, remaining problems and errors are the responsibility of the authors. Contents List of Tables .......................................................................................................................... iii List of Figures ........................................................................................................................ iv 1. Overview and Purposes .................................................................................................... 1 1.1 Who Are the Audiences for This Paper? ............................................... 4 1.2 An Overview of the Paper ....................................................................... 4 2. Background: Single-Case Designs .................................................................................... 7 2.1 Basic Forms of Single-Case Designs ...................................................... 9 2.1.1 Multiple Baseline Design ......................................................... 9 2.1.2 Reversal Design ...................................................................... 11 2.1.3 Alternating Treatments Designs .......................................... 13 2.1.4 Changing Criterion Designs .................................................. 14 2.2 Causal Inference in Single-Case Designs ............................................ 16 2.2.1 Ruling Out Threats to Validity ............................................. 17 2.2.2 Make Your Causal Predictions Elaborate ........................... 18 2.2.3 Causal Inference and Professional Standards..................... 19 2.3 Single-Case Designs: Some Salient Evaluative Issues ....................... 21 2.3.1 Two Reasonable Concerns ................................................... 21 2.3.2 Concerns That May Be Less Compelling ........................... 22 3. Effect Sizes and Single-Case Designs ........................................................................... 26 3.1 What Are Standardized Effect Sizes? ................................................... 27 3.1.1 How Do Standardized Effect Sizes Help Evidence- Based-Practice Reviewers? .............................................. 27 3.1.2 Within-Case Versus Between-Case Effect Sizes ................ 28 3.1.3 Serial Dependence in SCDs and Its Effect on Effect Sizes ........................................................................ 32 3.2 Worked-Through Examples and Related Application Issues .......... 32 3.2.1 Software for Computing Effect Sizes and for Doing Meta-Analysis .................................................................... 33 3.2.2 Computing Effect Sizes Proactively and Retroactively ...................................................................... 33 3.3 Between-Case Effect Sizes for SCDs ................................................... 34 3.3.1 Evaluating Between-Case Effect Sizes for SCDs ............... 37 3.4 Making No Choice Is a Bad Choice: The Perfect as the Enemy of the Good .......................................................................... 41 4. How to Report Between-Case Effect Sizes in Single-Case Designs ........................ 43 4.1 Report a Between-Case Effect Size, Standard Error, and Inferential Test .................................................................................. 43 4.2 Report Citations to the Between-Case Method Used and to Associated Software or Syntax ........................................................ 45 i 4.3 Report Assumptions of the Effect Size Used and Results of Any Tests of Those Assumptions .................................................. 46 4.4 Make Raw Numerical Outcome Data Available ................................ 47 5. How to Use Between-Case Effect Sizes With Individual Single-Case Design Studies .............................................................................................................. 51 5.1 Improving Design Sensitivity ................................................................ 51 5.2 Comparing Results From Visual and Statistical Analysis .................. 54 5.3 Accumulating Descriptive Data About Effect Sizes .......................... 55 6. How to Use Between-Case Effect Sizes to Identify Evidence-Based Practices With Single-Case Designs ............................................................. 58 6.1 An Example of Including SCD Results in an Evidence-Based Practice Review ................................................................................. 58 6.2 Using Modern Meta-Analytic Methods to Review SCDs ................. 60 6.2.1 Issues in Computing the Average Effect Size .................... 61 6.2.2 Heterogeneity Testing ............................................................. 64 6.2.3 Forest Plots and Cumulative Meta-Analyses ...................... 66 6.2.4 Meta-Analytic Diagnostics .................................................... 69 6.2.5 Moderator Analyses ............................................................... 69 6.2.6 Publication Bias Analyses ...................................................... 70 6.3 Issues That Arise When Combining Results From SCDs With Results From Between-Groups Experiments ..................... 73 6.4 Design Standards as Inclusion Criteria in Reviews With SCDs ................................................................................................... 77 7. Future Directions and Development for Between-Case Single-Case Design Effect Sizes ...................................................................................................... 79 7.1. Research to Improve Between-Case Effect Sizes ............................. 79 7.2 Conclusions .............................................................................................. 83 References.. ........................................................................................................................... 85 Glossary ..... ........................................................................................................................... 96 ii List of Tables Table Page 1. An evaluative summary of three between-case effect sizes ....................................... 38 2. Between-case effect sizes from meta-analyses on three different single-case design topics .................................................................................................... 56 3. Effects of the Spelling Mastery curriculum .................................................................. 59 iii List of Figures Figure Page 1. Sample forest plot ............................................................................................................ 66 2. Sample cumulative meta-analysis ..................................................................................