A Comparative Study of Exact Versus Propensity Matching Techniques Using Monte Carlo Simulation

Western Michigan University ScholarWorks at WMU

Dissertations Graduate College

4-2013

Mukaria J. J. Itang'ata Western Michigan University, [email protected]

Follow this and additional works at: https://scholarworks.wmich.edu/dissertations

Part of the Design of Experiments and Sample Surveys Commons, and the Educational Assessment, Evaluation, and Research Commons

Recommended Citation Itang'ata, Mukaria J. J., "A Comparative Study of Exact Versus Propensity Matching Techniques Using Monte Carlo Simulation" (2013). Dissertations. 148. https://scholarworks.wmich.edu/dissertations/148

This Dissertation-Open Access is brought to you for free and open access by the Graduate College at ScholarWorks at WMU. It has been accepted for inclusion in Dissertations by an authorized administrator of ScholarWorks at WMU. For more information, please contact [email protected]. A COMPARATIVE STUDY OF EXACT VERSUS PROPENSITY MATCHING TECHNIQUES USING MONTE CARLO SIMULATION

Mukaria J. J. Itang’ata

A Dissertation Submitted to the Graduate College in partial fulfillment of the requirements for the Degree of Doctor of Philosophy Educational Leadership, Research and Technology Western Michigan University April 2013

Doctoral Committee:

Brooks E. Applegate, Ph.D., Chair Joseph Kretovics, Ph.D. Warren E. Lacefield, Ph.D.

A COMPARATIVE STUDY OF EXACT VERSUS PROPENSITY MATCHING TECHNIQUES USING MONTE CARLO SIMULATION

Mukaria J. J. Itang’ata, Ph.D.

Western Michigan University, 2013

Often researchers face situations where comparative studies between two or more programs are necessary to make causal inferences for informed policy decision-making.

Experimental designs employing randomization provide the strongest evidence for causal inferences. However, many pragmatic and ethical challenges may preclude the use of randomized designs. In such situations, subject matching provides an alternative design approach for conducting causal inference studies. This study examined various design conditions hypothesized to affect matching procedures’ bias recovery ability.

The study examined three common social science research scenarios for case matching where discrete, continuous, or both types of covariates are employed. For each scenario, the following hypothesized factors were arranged experimentally in a factorial design: (a) bias amount, (b) effect size (ES), (c) covariance among the covariates (CV), and (d) correlation (CR) between the bias amount and the covariate group. Within these between group factors, six matching methods (MM) (random sampling, exact matching, propensity score matching, nearest neighbor matching, radius matching, and Mahalanobis metric matching) were examined in terms of their ability to minimize the difference

between "true" and "estimated" effects (YD ) and hence match groups in terms of bias. Study conditions were investigated using Monte Carlo techniques. One thousand replicates were drawn from a theoretically defined population. Each replicate included a treatment sample of N=200 and a control population of N=10,000 subjects. From the control population, random samples of N=200 were drawn via each matching method to form comparison group samples with known bias and effect size amounts added.

Results revealed that in the discrete covariate scenario there was a significant

MM*Bias*ES 3-way interaction. In the continuous covariate scenarios there were significant 4-way interactions involving MM*ES*CV*CR, MM*Bias*ES*CV, and

MM*Bias*ES*CR, while in the mixed scenario there were significant 4-way interactions among MM*Bias*ES*CV, MM*Bias*ES*CR and a significant 3-way interaction for

MM*CV*CR.

Overall, this study indicates that the study design conditions do impact various

outcomes relative testing the general hypothesis: Yt - YC = YD . Study implications suggest that social science researcher’s need to carefully consider multiple factors when employing a propensity-based or exact matching procedure in quasi-experimental designs. Recommendations for further research are offered.

ACKNOWLEDGMENTS

I would like to thank my main professor and doctoral committee chair, Dr. Brooks

Applegate for his support, guidance, expertise and critical review. The completion of this dissertation’s work would not have been possible without him. Thank you very much

Brooks. To my doctoral committee members, Dr. Warren Lacefield, and Dr. Joseph

Kretovics, thank you very much for your inspiration, support and encouragements throughout the dissertation process.

Thank you to my GEAR UP work and team mates: Ms. Shelly Carpenter, Dr.

Nancy Van Kannal-Ray, Dr. Pamela Zeller, Dr. Maxine Gilling, Dr. Jianping Shen, Ms.

Marie Cannel, Mr. Prabhat Vivekander, and Ms. Dawn Wesaw for their support, encouragements and friendship for many years.

Thank you to Dr. Esther Newlin-Haus and Dr. George Haus, their sons Samuel and Joseph for their friendship, kindness, support and goodwill throughout my scholarship endeavors at Western.

To my many friends and family: Akinyi, Toto Nkatha, Kiogora, Ruth, Kaigera,

Nkirote, Mbiriti, dad and mom, thank you all for your support and for always being there for me all the times. My education would not been a success without you all.

Mukaria J. J. Itang’ata

ii TABLE OF CONTENTS

ACKNOWLEDGEMENTS ...... ii

LIST OF TABLES ...... viii

LIST OF FIGURES ...... xii

CHAPTER

I. INTRODUCTION ...... 1

Problem Statement ...... 1

Goal of Matching ...... 3

Design Effects ...... 6

Background ...... 8

Exact Matching ...... 9

Propensity Score Matching ...... 10

Limitations of Matching Systems ...... 14

Influences on Matching Techniques ...... 16

Influences of Colinearity and Correlation on Independent Continuous Covariates ...... 17

Influences on Sample Size/Population Size Ratio ...... 18

Study Objective ...... 18

Research Questions ...... 20

Study Justification ...... 21

Definitions...... 22

Organization of Dissertation ...... 24

iii

Table of Contents—Continued

CHAPTER

II. LITERATURE OF REVIEW ...... 25

Causal Inference...... 25

Theory of Matching ...... 29

Matching as a Study Design ...... 36

Unconfoundedness/Identification Assumption ...... 37

Data Matching and Integration in Exact and Propensity Score Matching ...... 39

Exact Matching ...... 40

Propensity Score Matching ...... 41

Applications of Matching Methods ...... 43

Applications of Exact Matching…………………………...... 44

Applications of Propensity Scores…………...... 45

Matching Estimators for Treatment Effect ...... 50

Computer Algorithms to Stimulate or Match Datasets ...... 52

Relevance and Significance of this Study ...... 55

Summary ...... 58

III. METHODOLOGY ...... 59

Study Design ...... 59

Type and Number of Covariates ...... 61

Table of Contents—Continued

CHAPTER

Effect Size ...... 63

Amount of Selection Bias ...... 64

Matching Methods ...... 65

Random Sampling (No Matching) ...... 65

Exacting Matching ...... 65

Propensity Score Matching ...... 66

Nearest Neighbor Matching ...... 67

Radius Matching ...... 68

Mahalanobis Metric Matching including Propensity Scores ...... 69

Monte Carlo Data Generation ...... 69

Data Analysis and Presentation of Results ...... 72

Pilot Study ...... 78

Research Questions Analytics...... 90

Summary ...... 92

IV. RESULTS ...... 95

RQ1: What design conditions affect the matching procedures in bias recovery? ...... 95

Case 1: Discrete Covariates ...... 94

Case 2: Continuous Covariates ...... 99

MM*Bias*ES*Correlation Interaction Post Hoc ...... 101

Table of Contents—Continued

CHAPTER

MM*Bias*ES*Covariance Interaction Post Hoc ...... 106

MM*ES* Covariance*Correlation Interaction Post Hoc ...... 110

Case 3: Mixed Covariates ...... 113

MM*Covariance*Correlation Interaction Post Hoc ...... 115

MM*Bias*ES*Covariance Interaction Post Hoc ...... 119

MM*Bias*ES*Correlation Interaction Post Hoc ...... 123

Cases 1, 2 and 3 Compared ...... 128

RQ2: Do the matching procedures recover the selection/sampling bias?...... 129

Case 1: Discrete Covariates ...... 130

Case 2: Continuous Covariates ...... 131

Case 3: Mixed Covariates ...... 132

RQ3: Given the parameters of the experimental design conditions, what is the prevalence of non-matches? ...... 134

Case 1: Discrete Covariates ...... 135

Case 2: Continuous Covariates ...... 136

Case 3: Mixed Covariates ...... 136

Summary ...... 137

IV. SUMMARY, CONCLUSIONS, DISCUSSION, LIMITATIONS AND RECOMMENDATIONS……………………………………………. 139

Summary ...... 139

Conclusions ...... 141

Table of Contents—Continued

CHAPTER

RQ1: What design conditions affect the matching procedures in bias recovery? ...... 141

RQ2: Do the matching procedures recover the selection/sampling bias? ...... 143

RQ3: Given the parameters of the experimental design conditions, what is the prevalence of non-matches? ...... 144

Discussion ...... 144

Limitations and Recommendations...... 146

REFERENCES ...... 149

APPENDICES

A. Case 1: Discrete Covariates Descriptive Statistics ...... 163

B. Case 2: Continuous Covariates Descriptive Statistics...... 168

C. Case 3: Mixed Covariates Descriptive Statistics ...... 181

D. Case 1: Discrete Covariates YD t-test, 95% Confidence Interval and Probability of Coverage Statistics ...... 196

E. Case 2: Continuous Covariates t-test, 95% Confidence Interval and Probability of Coverage Statistics ...... 206

F. Case 3: Mixed Covariates t-test, 95% Confidence Interval and Probability of Coverage Statistics ...... 243

vii

LIST OF TABLES

1. Study Design Summary ...... 60

2. Arrangement and Design of Covariates by Type and Number ...... 63

3. Average Effect Sizes from Previous Research Studies ...... 63

4. The Null Case Descriptive Statistics for Continuous Covariates, N = 200 and N = 10,000...... 79

5. Treatment Sample (N = 200): The Null Case Bivariate Correlations among the Continuous Covariates ...... 79

6. Control Population: The Null Case Bivariate Correlations for Continuous Covariates ...... 80

7. Treatment Sample and Control Population: The Null Case Frequency Counts for Discrete Covariates N = 200 and N = 10,000 ...... 80

8. Control Population: Multiple Regression Analysis for Continuous Covariates: Colinearity = 0.00, Correlation with Bias = 0.00, Bias = 0.00 and ES = 0.00 ...... 81

9. The Null Case: t-tests Statistics for Discrete Covariates by Matching Method when there is Collinearity among Covariates = 0.00, Correlation with Bias = 0.00, Bias = 0.00 and Effect Size = 0.00 ...... 81

10. The Null Case: t-tests Statistics for the average of 1000 Replications for Continuous Covariates by Matching Method when Collinearity among Covariates = 0.00, Correlation with Bias = 0.00, Bias = 0.00 and Effect Size = 0.00 ...... 82

11. The Null Case: t-tests Statistics for Mixed Covariates by Matching Method when Collinearity among Covariates = 0.00, Correlation with Bias = 0.00 and Effect Size = 0.00 ...... 83

12. The Null Case: Comparison Statistics for Discrete Covariates by Matching Method when Collinearity among Discrete Covariates = 0.00, Correlation with Bias = 0.00, Bias = 0.00 and Effect Size = 0.00 ...... 83

viii

List of Tables—Continued

13. The Null Case: Comparison Statistics for Continuous Covariates by Matching Method when Collinearity among Continuous Covariates = 0.00, Correlation with Bias = 0.00, Bias = 0.00 and Effect Size = 0.00………….. 84

14. The Null Case: Comparison Statistics for Mixed Covariates by Matching Method when Collinearity among Mixed Covariates = 0.00, Correlation with Bias = 0.00, Bias = 0.00 and Effect Size = 0.00 ...... 85

15. The Null Case: Nominal Type I Error Rate as Percentage of Statistically Significant t-tests (p < 0.05) by Covariate Type and Matching Method ...... 86

16. Case l: Descriptive Statistics for Matching Method Averaged Over 27000 Replications ...... 96

17. Case 1: MANOVA Results ...... 97

18. Case 1: Simple-simple Effect Least Square Means Among Matching Methods When Bias = 0.10 and ES = 0.00 ...... 98

19. Case 1: Matching Method Main effect Least Square Means When Bias = 0.15 ...... 99

20. Case 2: Descriptive Statistics for Average of 108000 Replications by Matching Method ...... 99

21. Case 2: Primary MANOVA Results ...... 101

22. LS Means for Bias, ES and Correlation by Matching Methods ...... 102

23. Case 2: Simple-simple Effect LS Means for Matching Method When Correlation = 0.75 and Bias = 0.15 ...... 105

24. Case 2: Simple-simple Effect LS Means for the Matching Method When Correlation = 0.90 and ES = 0.00 ...... 105

25. Case 2: Simple-simple Effect LS Means for Matching Method When Correlation= 0.99 and ES = 0.10 ...... 106

26. LS Means for Bias, ES and Covariance by Matching Method ...... 107

List of Tables—Continued

27. Case 2: Simple-simple effect LS means for the Matching Method When Covariance = 0.90 and ES = 0.00……………………….………………….. 110

28. Case 2: LS Means for ES, Covariance and Correlation by Matching Method ...... 111

29. Case 2: Simple-simple Effect LS Means for the Matching Method When ES = 0.00, Correlation = 0.90 and Covariance = 0.90 ...... 112

30. Case 2: Simple-simple Effect LS Means for the Matching Method When ES = 0.00, Correlation = 0.99 and Covariance = 0.90 ...... 113

31. Case 3: Descriptive Statistics for Average of 108000 Replications by Matching Method ...... 114

32. Case 3: Primary MANOVA Results ...... 115

33. Case 3: LS Means for Covariance and Correlation by Matching Method ...... 116

34. Case 3: Simple-simple Effect LS Means for the Matching Method When Correlation = 0.75 ...... 119

35. Case 3: LS Means for Bias*ES*Covariance by Matching Method ...... 120

36. Case 3: Simple-simple Effect LS Means for the Matching Method When Covariance = 0.90, ES = 0.00 and Bias = 0.00 ...... 123

37. Case 3: LS Means for Bias*ES*Correlation by Matching Method ...... 125

38. Case 3: Simple-simple Effect LS Means for Matching Method When Correlation = 0.75, ES = 0.45 and Bias = 0.15 ...... 128

39. Case 1: URB Descriptive Statistics for Average of 27000 Replications by Matching Method ...... 131

40. Case 2: UBR Descriptive Statistics for Average of 108000 Replications by Matching Method ...... 132

41. Case 3: UBR Statistics for Average of 108000 Replications by Matching Method ...... 134

List of Tables—Continued

42. Case 1: Matching Methods Matching Prevalence and Performance ...... 135

43. Case 2: Matching Methods Matching Prevalence and Performance ...... 136

44. Case 3: Matching Methods Matching Prevalence and Performance ...... 137

LIST OF FIGURES

1. Case 1 Estimated Mean Bias for Average of 1000 Replications by Matching Method ...... 87

2. Case 1 Estimated Mean ES for Average of 1000 Replications by Matching Method ...... 87

3. Case 2 Estimated Mean Bias for Average of 1000 Replications by Matching Method ...... 88

4. Case 2 Estimated Mean ES for Average of 1000 Replications by Matching Method ...... 88

5. Case 3 Estimated Mean Bias for Average of 1000 Replications by Matching Method ...... 89

6. Case 3 Estimated Mean ES for Average of 1000 Replications by Matching Method ...... 89

7. Case 1: Pooled After Centering ...... 96

8. Case 2: Pooled After Centering ...... 100

9. Case 3: Pooled After Centering ...... 114

10. Case 1, 2, and 3 Comparison: Pooled After Centering ...... 129

11. Case 1: Mean URB pooled over all conditions and Replications by Matching Method ...... 131

12. Case 2: Mean Unrecovered Bias (UBR) for Average of 10800 Replications by Match Method ...... 132

13. Case 3: Mean Unrecovered (UBR) for Average of 10800 Replications by Match Method ...... 133

xii

CHAPTER I

INTRODUCTION

This chapter presents the problem statement of this study including background information on the concept, types and limitations of matching techniques. It also discusses variables that can influence the matching techniques and procedures. The objectives of the study and research questions are highlighted. The justification of the study is presented, and the key concepts of the study are defined. The chapter ends with a summary highlighting the key points discussed in the chapter, and an explanation of how this dissertation study is organized.

Problem Statement

Often policy analysts, program evaluators and researchers using non/quasi- experimental or observational study datasets face situations where they must conduct comparative studies between two or more programs or outcomes to determine program or outcome effect(s) (e.g., What is the outcome or effect associated with a student attending a charter school compared to a student attending a traditional public school in

Michigan?). In many of these situations there may be one large population compared to a much smaller foci subpopulation. In such situations, a logical way to study the subpopulations is to create one or more comparison samples drawn from the larger population (i.e., traditional public school students in Michigan). However, a question

arises: How best to match the subpopulation subjects (e.g. the treatment group

(charter school students)) to the whole population (e.g. the control group (traditional public school students))?

The strongest research design to study the above question is a fully randomized design (Guo & Fraser, 2010; Stuart & Rubin, 2007; Diamond & Sekhon, 2005;

Rosenbaum, 2002; Rubin, 1977). Random assignment is an experimental design technique for assigning subjects to different groups (treatment or control). It achieves equivalence between treatment and control groups by allocating sampling subjects (units) of a target population to two groups in such a way that chance is allowed to decide whether a subject (e.g. student), receives the treatment or control condition. In a randomized design, the allocation of eligible target sampling subjects or units across the treatment and control groups is randomly determined, thus the probabilities of ending up in the treatment or control groups are the same for all subjects in the study. In practice, however, with small samples randomization often fails, and one cannot always randomly assign students to treatment and control groups due to ethnical and logistical limitations, or lack of cooperation from stakeholders (Shadish, Cook & Campbell, 2002). Thus, an unknown sampling and or selection bias is created that will always present alternative explanations for any differences observed between treatment and control groups

(Kleinbaum et el, 2007; Luellen et al, 2005; Chan, Macaskill, Irwig & Walter, 2004;

Rosenbaum, 2002; Dehejia & Wahba, 1999, 2000; Rubin, 1977, 1991).

As a result, researchers have developed several different design alternatives to random assignment in an attempt to address the primary research question: Is there a difference between the groups when randomization cannot be accomplished in non/quasi

experimental designs? Three broad classes of research designs can be identified: (i) ordinary least squares (OLS) regression; (ii) stratification; and (iii) subject matching.

OLS regression involves including covariates in the study analytics to control for the unknown sampling/selection bias. In stratification, groups of subjects or participants are grouped into homogeneous strata(s) on the basis of group membership or covariate(s) categorical values. Subject matching represents a broad constellation of procedures and methods that attempt to draw a comparison group sample that is exactly like the target

(experimental or treatment) sample (in terms of a number of suspected variables or descriptors relevant to possible bias sources), thus eliminating or equating any sampling bias. This dissertation study specifically focuses on the subject matching method to examine the effects of sampling bias.

Goal of Matching

The main goal of a matching procedure is to reduce the selection bias among covariates thus balancing matching datasets (Guo & Fraser, 2010; Stuart & Rubin, 2007;

Joffe & Rosenbaum, 1999; Rosenbaum & Rubin, 1985). Matching is a design option that can be used in both experimental and non/quasi experimental studies to: (i) balance data,

(ii) reduce or eliminate selection bias, (iii) gain precision in the estimation of an effect measure of interest, (iv) control for variables or covariates that are difficult to measure, and (v) for other practical aspects of data collection such as convenience, time-saving, and cost-saving (Guo & Fraser, 2010; Stuart & Rubin, 2007; Kleinbaum, Sullivan, &

Barker, 2007; Rosenbaum & Rubin, 1983; Rubin, 1974). For example, Kleinbaum,

Sullivan, & Barker (2007) illustrate that matching (i) can enable data balancing thus

reducing or removing the confounding effect of a covariate or variable and (ii) can help in gaining precision by allowing an estimation of a narrower confidence interval around the effect measure that could have been obtained without matching. For instance, matching students in a school district by their socio-economic status could provide a way for controlling for poverty or social class.

Since the early work on matching by Cochran and Rubin in the 1970’s, the use of matching methods have increased in both complexity and application. These are now being used in many disciplines to balance data in the absence of randomization: e.g., economics (Lalonde, 1986; Dehejia & Wahba, 2002; Heckman, Ichimura, Smith & Todd,

1997; Smith & Todd, 2003), medicine (Rubin 1997; Christakis & Iswashyna, 2003;

Baser, 2006), epidemiology (Kurth, Glynn, Chan, Gaziano & Berger, 2006; Kleinbaum et el, 2007), sociology (Smith, 1997; Morgan & Harding 2006; Diprete & Engelhardt, 2004;

Winship & Morgan 1999), and psychology (Fillenbaum, Hybels, Pieper, Konrad,

Burchett, & Blazer, 2006; Tebes, Feinn, Vanderploeg, Chinman, Shepard, Brabham,

2007).

The overall objective of a matching procedure is to reduce sampling bias that results from nonrandom assignment between treatment and control group (Rubin, 1973;

D’Agostino, 1998; Joffe & Rosenbaum, 1999; Rosenbaum, 2002; Baser, 2006;

Kleinbaum et el, 2007; Guo & Fraser, 2010). Thus the goal of the matching exercise is to draw a comparison group that is as similar as possible to the treatment group except for exposure to the experimental condition(s) (Rubin, 1980; Dehejia & Wahba, 1999, 2000;

Rosenbaum, 2002; Smith & Todd, 2003). Two matching methodologies can be differentiated in the literature: exact matching (also called covariate matching) and

propensity score matching. Exact matching matches each treatment sample subject to all possible control population subjects with the exact same value or covariate (Joffe &

Rosenbaum, 1999; Rubin, 1980; Lalonde, 1986; Zhao, 2004; King et al, 2007).

Propensity score matching (an alternative to exact matching when exact matches on covariates cannot be found) requires estimation of a propensity score(s) based on the conditional probability of exposure to a treatment given a vector of observed covariates

(Rosenbaum & Rubin, 1983; Rosenbaum, 2002). The estimated propensity score(s) are then used to match the treatment sample subjects to control population subjects with similar values on the propensity score(s) (Rubin, 2001).

For example, suppose a researcher is interested in examining if students attending a charter school have higher achievements scores in mathematics and reading than students attending a traditional public in a given school year. In such a case, the researcher may wish to draw a comparison sample that is similar to the treatment group

(e.g., the charter school student subpopulation) samples by matching the charter school students’ subpopulation with a similar larger control population (e.g. traditional public school students) on the relevant background and demographic characteristics such as ethnicity, gender, age, socioeconomic status, educational level, etc. (Joffe & Rosenbaum,

1999; Rosenbaum, 2002). An alternative approach to exact matching is to draw a comparison sample by propensity score matching based on the observed demographic pretreatment covariates - e.g., ethnicity, gender, age, socioeconomic status, and educational level - because these covariates can be measured prior to the start of treatment and are unaffected by the treatment (Joffe & Rosenbaum, 1999). Whereas, exact matching procedure matches each subject i in the treatment sample with subject j in

the control population on a vector of observed covariates (Xi) such that Xi=Xj=x; in a propensity score matching procedure each subject i in the treatment sample is matched with a subject j in the control population if e(Xi)=e(Xj)=p or is "in the neighborhood" of p. Rosenbaum and Rubin (1983) suggest the use of a balancing score (the propensity score) as an alternative to exact matching. They called the propensity score a balancing score because of its ability to balance the relevant covariates across the matched groups.

If the control population (e.g., the traditional student population) is large, many possible comparison samples could be drawn by either matching method such that each treatment sample student is attached (matched) to all possible control group population members with the same (exact) or similar-enough (propensity) values on a vector of matching covariates.

Design Effects

A "design effect" is a measure of efficiency for comparing sample designs

(Salganik, 2006; Skinner, 1986). Existing research (e.g., Guo & Fraser, 2010; Rivers,

2006; Salganik, 2006; Zhao, 2004; Kane, 2004; Dehejia & Wahba, 1999) suggest there are several possible sources of influence that may affect sample estimates obtained from a group (thereafter referred to as a group mean); for example, the ratio of sample size to population size, method of sampling (random, e.g., simple random sampling, or nonprobability sampling), effect size (theorized or estimated), and assumed distribution.

Design effects influence matching by improving efficiency in the estimation of treatment effects which may result from or are due to confounders/discrepancies in the empirical distribution of covariates in the design factor(s) between treatment sample subjects and

control population subjects. Additionally several factors are known to affect the accuracy of exact matching and propensity score matching procedures (Zhao, 2004; Dehejia &

Wahba, 1999; Joffe & Rosenbaum, 1999; Rosenbaum & Rubin, 1985). In exact matching protocols, factors such as the number of matching covariates, the number of levels within a matching covariate and the treatment of “continuous” matching variables affect the accuracy of the matching procedures/systems. Similarly, in a propensity score protocol, the number of covariates and the original sampling characteristics can affect substantially the accuracy of a propensity score match (Rubin 2006; Rosenbaum, 2002; Smith & Todd,

2003; Imati, 2005; Dejejia & Wahba; Lalonde, 1986).

In the matching literature - e.g., Caliendo & Kopeining, 2008; Todd, 2006;

Sekhon, 2004; Imati, 2005; Rosenbaum, 2002; Dehejia & Wahba, 1999, 2000, 2002;

Rubin, 2006; Lalonde, 1986) - many methods such as exact matching (Cochran & Rubin,

1973), propensity score matching (Rubin & Rosenbaum, 1983), nearest neighbor matching (Rubin, 1973), radius matching (Dehejia & Wahba, 2002), and Mahalanobis metric distance matching (Cochran & Rubin, 1973; Rubin, 1976, 1980) have been proposed as possible methods or solutions for studying the causal effects in observational data. Determining the design conditions under which these methods yield appropriate estimates represents an important contribution to research in the matching literature. The work in this dissertation was motivated by this problem. There is a need to know which matching procedure(s), exact or propensity score is or are superior method(s) of forming a comparison sample under what design conditions and features. Specifically, this dissertation focused on determining if the construction of a comparison group, via exact matching protocol or through various propensity score protocols, results in different

conclusions related to “true” difference (in means) between the groups under different experimental conditions and design features.

Background

The purpose of matching is finding available subject(s) or respondent(s) who are as similar as possible to the selected subjects(s) or member(s) of a sample or population by linking data from different sources. Matching compares individuals from a nonrandomly generated treatment group to similar individuals in a nonrandomly specified control (comparison) group. This is done by assuming that selection of treatment (study) sample(s) is unrelated to the control (comparison) sample(s) and the outcome is conditional on some set of observed covariates drawn from the large population. The process identifies treatment sample individuals who share the same covariate characteristics as control population individuals. Thus, when all the relevant differences between the treatment and comparison subjects or units (e.g. students or schools, etc.) are captured in the observable pretreatment covariates, matching methods can yield an unbiased estimate of the treatment effect or impact (Guo & Fraser, 2010; Zhao, 2004;

Caliendo & Kopeining 2008; Dehejia & Wahba, 2000, 2002; Rosenbaum, 2002; Joffe &

Rosenbaum, 1999, Rosenbaum & Rubin 1985).

Exact matching (also called covariate matching) (Smith, 1997; Dehejia & Wahba,

1999, 2000; Zhao, 2004) compares subjects with exactly the same values on a vector of observed covariates Xi; where each Xi provides additional information about the subjects.

Propensity score matching refers to a class of multivariate statistical methods where multiple covariates are summarized into one score (the estimated propensity score) and

then those score(s) are used to conduct matching. The difference between exact and propensity score matching is that, whereas in exact matching subjects (e.g., persons) can or are matched on one or more covariates or the same value(s) of the observed covariates, in propensity score matching subjects are matched on one composite statistical score (the propensity score).

Exact Matching

A variable such as ethnicity or gender, measured prior to the start of treatment and likely unaffected by the treatment is called a covariate (Joffe & Rosenbaum, 1999). Let,

Yti and Yci respectively be two potential groups with outcomes: Yti = f1(Xi) + ε1i, and Yci = f2(Xi) +ε2i, where, ε1i and ε2i are independent identically distributed error terms. Then Xi =

Xj =>f1(Xi) = f2(Xj)=X, where X is a vector of matching covariates or variables. This suggests exact matching on the predetermined covariates Xi in the treatment sample can be matched with each subject j in the control population on a vector of observed covariates (Xi) such that Xi=Xj=x (Rosenbaum & Rubin, 1985; Dehejia & Wahba, 2000,

2002; Zhao, 2004). An exact match on X is a matching on X in which X is the same for ns subjects or units in each matched set (S), i.e. Xsi = Xsj for i, j =1, 2…ns for each S.

In exact matching, two approaches (quality class and micro data integration) are commonly used. In the quality class approach, records of subjects assigned to the treatment condition are matched to records of the people assigned to control condition based on the extent to which they match, given their specified matching covariates agreement or disagreement with the matching compliance criteria (Denk & Hack, 2003).

In the micro data integration approach, data records are assigned and then matched to the class of matches or the class of non-matches in accordance to defined matching criteria

(Denk & Hack, 2003). In this study, the quality class approach was followed because it was the most suitable method for guiding the construction of matching sample datasets for which subject i in the treatment sample was matched with subject j in the control population if Xi=Xj=X.

The Federal Subcommittee on Matching Techniques (Radner, Allen, Jabine &

Muller, 1980) summarized the following applications for exact matching:

. To gain precision in estimating an effect measure of interest.

. To adjust for confounding variables, thus reducing selection bias.

. To construct matched samples from treatment and control groups so as to

analyze causal effects from observational or categorical data.

Propensity Score Matching

A propensity score is the conditional probability of subject or unit being assigned to a condition (treatment condition) given a set of observed covariates such as age, race, gender, or ethnicity that are hypothesized to eliminate or reduce the selection bias

(Rosenbaum & Rubin, 1983). These authors defined the propensity score for subject i,

(i=1…., N) as the conditional probability of assignment to a particular treatment (Zi =1) versus control (Zi=2), given a vector of observed covariates xi; such that propensity score

(eXi) = prob(Zi =1|Xi=xi). Where Z is an indicator of treatment, Z=1 if treated and Z=2 if

th control. xi is the vector of observed covariates for the i subject. Z and X are conditionally independent given e(X). Subjects in treatment and control groups with equal or nearly equal propensity scores will tend to have the same or nearest the same distributions on their background covariates (Rosenbaum & Rubin, 1984; Joffe &

Rosenbaum, 1999; Rosenbaum, 2002).

In a randomized experiment with two conditions; the probability of assignment to a treatment condition (z), given the covariates of X is (prob(z=│x))=0.5 in designs where each subject has an equal chance of receiving treatment or no treatment. In a nonrandomized experiment the assignment mechanism of prob(z=│x) is unknown and thus has to be estimated by propensity score methods (Shadish, Cook & Campell, 2002).

Adjustments such as propensity score analysis attempt to estimate the treatment effect of subjects in a nonrandomized experiment as if they are randomly assigned to an experimental condition (Rosenbaum & Rubin, 1985; D’Agostino, 1998; Shadish, Clark &

Steiner, 2008). Propensity score matching has the following applications:

. To adjust for confounding variables, thus reducing selection bias (King et al,

2007; Lalonde, 1986);

. to construct matched samples from treatment groups so as to analyze causal

effects from observational/categorical data (Rosenbaum & Rubin, 1983,

1984);

. to equate groups on observed covariates e.g., ensuring two groups of subjects

are matched equally on the observed covariates or factors (Rosenbaum &

Rubin, 1985);

. to gain precision in estimating the effect measure of interest (Kleinbaum et al,

2007); and

. to balance observed covariates (Joffe & Rosenbaum, 1999).

The propensity score matching literature presents several propensity scores matching techniques that are briefly presented.

Stratification/Subclassification/Interval Matching. In this method the estimated propensity score(s) are used to stratify the subjects into homogenous subclasses (divided into treatment and control groups) with similar propensity scores based on the values of their covariates. The subclasses are often defined by quantiles of the distribution of the covariate in the treatment and control groups. An overall estimate of the treatment effect can be obtained by calculating an estimate within each subclass, and then form a weighted average of the subclass estimates (Cochran, 1968).

Caliper Matching. This method selects matches within a specified range defined by the caliper. A caliper is a common support region (e.g. 0.01, 0.002 etc.), from wherein one randomly selects one control unit (comparison) that matches on the propensity score within the treatment unit (Alhauser & Rubin, 1971; Cochran & Rubin,

1973).

Nearest Neighbor Matching. This method matches each individual subject in the treatment group with the best or closest subject in the control population using a specified closest minimum distance measure. Matches are chosen for each treatment sample subject one at a time and at each matching step a control subject is chosen that is not yet matched but is closest to the treatment sample on the minimum distance measure with or without replacement (Rubin, 1973).

Radius Matching. This method matches each treatment subject with the control subject whose propensity score falls within a predefined interval or neighborhood of the propensity score based on the covariates or values used to estimate the propensity score.

Radius matching differs from caliper matching in that in radius matching instead of matching the treatment sample subject(s) with it(s) closest control subject(s) within a

caliper, the control subject(s) that fall within the caliper or radius are selected (Dehejia &

Wahba, 2002).

Mahalanobis Metric Matching. In this method, subjects are first ordered based on the best score distance and then the distance between the treatment sample and control population subject is calculated. The Mahalanobis metric distance (score distance) is computed as: d(i, j) = (u v)T C 1 (u v) . Where, d(i, j) is the distance between treatment sample subject i and a control population subject j defined by the Mahalanobis distance, u and v are the matching covariate vectors/values for each subject i in treatment sample and each subject j in control population including the propensity score, and C-1 is the sample covariance matrix of the matching covariates from sets of treatment sample and control population subjects. There are two forms of Mahalanobis Metric Matching, with and without propensity score replacement. The Mahalanobis Metric without propensity score replacement method randomly orders subjects and then calculates the distance between first treatment sample subjects and all controls. The Mahalanobis

Metric with propensity score replacement method adds the values of the matching covariates or variables for treatment sample subjects and all control subjects. In this study, the Mahalanobis Metric with propensity score replacement method was used

(Cochran & Rubin, 1973; Rubin, 1976; 1980).

Kernel or Local Weight Matching. In this matching method, all treatment group subjects are matched with a weighted average of all control group subjects. Weights are inversely proportional to the distance between the propensity scores of the treatment and control groups, thus lower variance on treatment and group propensity scores is achieved

because all control subjects contribute to the weights (Ichimura & Todd, 1997 & 1998;

Dehjia & Wahba, 2002).

An extensive review of literature conducted for this study relating to propensity score matching techniques revealed that matching on propensity scores, nearest neighbor matching, radius matching, and Mahalanobis metric matching with propensity scores have been used most widely in research studies. However, no study was found in the literature review comparing random sampling, exacting matching and propensity score matching in terms of their ability to simulate unknown bias effects. As a result, these matching techniques were selected for comparison in this study.

Limitations of Matching Systems

When adequate exact matches cannot be found in a matching procedure, as becomes increasingly problematic as the number of covariates is increased (Lalonde,

1986), exact matching as means of obtaining a comparison group is limited; thus the need for alternation approaches such as propensity score matching (Dehejia & Wahba, 2002;

Zhao, 2004). For example, if there are several matching covariates, each with several levels, the number of cells (defined as a set of covariate values) may become large. This increases the likelihood that many cells may be empty and have no comparison (control) population subjects corresponding to each treatment sample subject (Dehejia & Wahba,

2000, 2002; Zhao, 2004; Lalonde, 1986; Rosenbaum & Rubin, 1985). For instance, if there are five covariates or variables each with three levels/values, there will be 35 = 243 cells. This increase in design dimensionality may to lead to empty data cells thus limiting the sample sizes that can be generated via the exact matching procedure (Guo & Fraser,

2010; Dehejia & Wahba, 1999, 2002). The same dimensionality problem applies to

propensity score matching as the number of covariates increases (Guo & Fraser, 2010;

Zhao, 2004; Dehejia & Wahba, 1999, 2002; Rosenbaum, 2002; Rosenbaum & Rubin,

1985). As in the case of exact matching, the likely occurrence of empty data cells due to failed matches may limit the sample sizes generated including the accuracy of propensity score matches and other parameter estimates.

Rubin (1997); Shadish, Cook & Campell (2002); Quiggley, Munoz & Jacknowitz

(2003); Guo, Barth & Gibbons (2006) highlight the following limitations of propensity score matching:

. Propensity scores cannot adjust for unobserved covariates and they work

better with large samples (Rubin, 1997).

. Propensity score studies require large samples, treatment and control group

overlap on covariates must be substantial, and hidden bias may remain

because matching only controls for observed covariates or variables to the

extent that they are perfectly measured (Shadish, Cook & Campell, 2002).

Also Quiggley, Munoz & Jacknowitz (2003) and Guo, Barth & Gibbons

(2006) argue that potentially significant amounts of unobserved bias and the

potential likelihood of not finding a sufficient match for each treated subject

could minimize matching subjects against a known covariate(s) criterion.

Thus, if two groups do not have substantial overlap of covariates (i.e., for

every matched subject there is a positive probability of nonparticipation by

matched subjects), then substantial statistical error may be introduced in the

matched samples. For example, if only the worst cases from the untreated

comparison group are compared to only the best cases from the treatment

group, regression toward the mean may result, thus making the comparison

group look better, and making the treatment group look worse.

. Kleinbaum, Sullivan, & Barker (2007) cautions that in both exact and

propensity score matching procedures;

i. Matching on weak or irrelevant variables or covariates is unlikely to gain

precision/power.

ii. Matching may be costly in terms of time and money required to carry out

the matching process.

iii. Difficulties in finding matches may arise thus losing sample size and the

precision hoped from matching.

Influences on Matching Techniques

Influence of Sample Size and Number of Matching Covariate Variables. Sample size plays an important role in research because as the sample size increases the power of the statistical analysis also increases, all other factors held constant. The dimensionality problem extends or applies to propensity score matching as the number of covariates increase (Guo & Fraser, 2010; Zhao, 2004; Dehejia & Wahba, 1999, 2002; Rosenbaum,

2002; Rosenbaum & Rubin, 1985). As in the case of exact matching, the likely occurrence of empty data cells due to failed matches may limit the sample sizes generated and thus affect the generalizability of propensity score matches and other parameter estimates.

Influence of Effect Size. The effect size is the most appropriate metric for estimating the magnitude of effect of a treatment or an intervention variable because it

makes possible study comparisons using factorial models such as between and within study designs (Rosenthal, 1994). Effect size and the ability to detect it are indirectly related; the smaller the effect, the more difficult it will be to find it. Since the effect size is a measure of the magnitude of the strength of a relationship between a treatment/intervention outcome and a control outcome, this provides a direct mechanism for estimating and evaluating of the effectiveness of a treatment or intervention. It also provides a way to compare matching methods under known biasing conditions when using various known covariate structures.

Influence of Colinearity and Correlation in Independent Continuous Covariates

Colinearity is the presence of linear relationships or near linear relationships between or among covariates. Perfectly linearly correlated independent covariates lead to indeterminable estimation of parameter coefficients, whereas for orthogonal

(uncorrelated independent covariate(s)) collinearity is non-existent and would have no effect in the estimation of parameter coefficients. In reality, there is almost always some degree of correlation or inter-correlation of (0 < rxixj < 1) between or among independent covariate(s), thus leading to collinearity effects that may affect the accuracy and stability of parameter estimates. Low or moderate collinearity may not make independent covariate(s) redundant but high collinearity (> .90) reflects redundant covariate(s).

Correlation is measures of the covariance between two variables indicating how much two variables co-vary together across sample groups or members. Correlation values range from -1 to +1. A negative correlation implies the scores on one variable/covariate go up when the values on the other variable/covariate go down. A positive correlation implies scores on one variable/covariate increase when scores on the other

variable/covariate increase. A value of zero (0) implies no correlation between variables/covariate.

Influence of Sample/Population Size Ratio

Sampling enables a selection of subjects for a study in such a way that the subjects selected for the study represent the larger population group from which they are selected. Thus, a control group sample size drawn from a large population has a higher probability of finding subjects or units to match with or on than a sample size drawn from a smaller population. For example, drawing a control sample of n=200 subjects from a population of N=10000 has a higher probability of getting sufficient matches to match with a treatment sample of n=200 than a control sample drawn from a population of

N=2000. In educational research settings, drawing a control sample of n=200 from a control population of N=10000 to match with a treatment sample of n=200 represents a reality for studying educational intervention casual effects where treatment and control study population samples are compared.

Study Objective

The construction of a comparison group by different methods may result in differential conclusions related to testing for mean differences between two groups under different experimental conditions and design features. Different matching methods may be more or less sensitive to various experimental and contextual conditions when attempting to restore or recreate unknown biasing conditions. Thus, the purposes of this study were:

1. To examine the design conditions that affect matching procedures in bias

recovery/reduction;

2. to investigate the matching procedures ability to recover/reduce

experimentally induced selection bias;

3. to determine if there were prevalence of non-matches given the parameters of

the study experimental design conditions.

The study used the Monte Carlo stimulation approach to stimulate data to study the performance of six different methods of forming group samples under the following experimental conditions:

. Two between groups (treatment and control)

. Type and number of covariates used in matching procedures

. Covariate Collinearity among covariates

. Covariate correlation with bias in the control population

. Treatment effect size

. Amount of selection bias

. Matching methods: Random sampling (no match), Exact matching, propensity

score matching, nearest neighbor matching, radius matching, and Mahalanobis

metric (distance) matching

One thousand replications per cell were simulated. Each sample had the following variables under each matching method:

. group (grp), treatment sample, and control population;

. dependent variable (Y);

. four levels of matching covariates (4 discrete, 4 continuous, 4 mixed);

. three levels of covariate Collinearity;

. four levels of covariate correlation with bias;

. three levels of effect size;

. three levels of bias; and

. six matching methods or methods of forming comparison group samples.

Research Questions

The research questions of the study were:

1. What design conditions affect the matching procedures in bias recovery?

2. Do the matching procedures recover/reduce the selection bias?

3. Given the parameters of the experimental design conditions, what is the

prevalence for non-matches?

All the above research questions were investigated under the following experimental conditions and design features:

. Matching Covariates/Variables Vector:

Type of covariate (measurement scale): (Discrete (D), Continuous(C).

Number of covariates (4 discrete covariates, 4 continuous covariates, 2

discrete & 2 continuous covariates).

Case 1: D, D, D, D

Case 2: C, C, C, C

Case 3: D, D, C, C

Covariate Collinearity among covariates: 0.00, 0.40, 0.90.

Covariate Correlation with Bias in the Control Population: 0.00, 0.40,

0.75, .99.

. Manipulated External Design Environments:

Amount of Selection Bias: 0.00, 0.10, 0.15

Treatment Effect Size: 0.00, 0.10, 0.45

. Matching Methods: Random Sampling/No Matching (NOM), Exact

Matching (EM); Propensity Score Matching (PSM), Nearest Neighbor

Matching (NNM), Radius Matching (RM), and Mahalanobis Metric

(distance) Matching (MM)

Study Justification

A thorough review of literature on matching suggests that most studies conducted on matching focuses on propensity score estimation and matching but not on exact vs. propensity score matching protocols, particularly not on the comparison of the exact and propensity score matching protocols in non/quasi experimental datasets characterized by large population with smaller subpopulations of foci. The work of this dissertation addressed this gap. It is expected to contribute to the literature on matching pertaining to social science and educational research and evaluation in the following ways:

i. identifying the design conditions that affect matching procedures in bias

recovery.

ii. determining if the matching procedures have ability to recover experimentally

induced selection bias.

iii. determining the nature of prevalence of non-matches given the parameters of

the study experimental design conditions.

Exploration of these research issues in Monte Carlo computational data space can provide empirical results that may demonstrate the similarities, differences, and situational applicability of the exact and propensity score matching methods. This work should provide clarity on which exact or propensity score matching protocols, procedures, and techniques are the most appropriate design, and analytic approaches for estimating causal effects and making causal inferences in non/quasi-experimental study designs or studies. Finally, thorough analysis of these techniques via comprehensive simulations should demonstrate how exact and/or propensity score matching methods can be employed in the design of causal inference studies to draw better inferences or conclusions about the effectiveness of educational interventions or programs and thereby better inform educational leaders, researchers, and policymakers.

Definitions

Control group: A group of subjects or targets who do not receive the treatment or intervention and who are compared on outcome measures with one or more groups who do receive the treatment or intervention.

Effect Size: A statistical measure of the difference between the mean of the treatment or experimental group and the mean of the control or comparison group in a quantitative study.

Exact Matching: The process of matching each treatment sample subject/unit to all possible control subjects/units on the same covariates or with exactly the same values on all covariates.

Matching: A method of sampling from a large control population for which the goal is to select a subset as a control sample that has similar covariate values as the treatment sample.

Monte Carlo Stimulation: A computerized mathematical technique that models experiments empirically and approximates the probability of certain outcomes by running multiple trials or runs using random variables and repeated random sampling.

Propensity score matching: The conditional probability of exposure to a treatment condition given a vector of observed covariates.

Quasi-Experimental Design: A research design in which an experimental procedure is applied but the researcher has little or no control over the allocation of treatment subjects or other factors related to those being studied.

Randomization: A control technique that equates group(s) of subject(s) or participant(s) by ensuring every subject or participant in a study has an equal chance of being assigned to a treatment or control group.

Random Sampling: A sampling method in which each subject in the population has an equal chance of being selected to a sample.

Selection Bias: The systematic error that results from the way subjects are selected into the study or because there are selective losses of subjects prior to data analysis.

Treatment group: A group of subjects or targets who receive the treatment or intervention and whose outcome measures are compared with one or more groups who do not receive the treatment or intervention.

Organization of the Dissertation

This chapter has presented the problem statement of this study. It has provided background information on the concept, types, and limitations of matching. It has discussed the variables that influence the matching techniques and procedures. The objectives of the study and the research questions were highlighted. The justification of the study was presented and the key concepts of the study were defined. The organization of this dissertation study is as follows: Chapter one has presented an introduction of the study. Chapter two presents the review of literature. Chapter three discusses the study’s research design and methodology. Chapter four presents the study findings, and lastly

Chapter five presents a discussion of the study findings and their implications for causal inference studies in social science and educational research and evaluation.

CHAPTER II

REVIEW OF LITERATURE

This chapter presents the review of literature for this study. Section one presents the causal inference theory/model. Section two presents the theory of matching. Section three discusses matching as a study design. Section four highlights the assumptions embedded in matching procedures. Section five shows how data matching are conducted in exact matching and propensity score matching techniques.

Sections six and seven, review previous applications of matching methods. Sections eight and nine highlight the matching estimators for treatment effect and the computer algorithms that have been used in past studies to simulate or match datasets. Section nine justifies the relevance and significance of this study as it pertains to previous studies.

Causal Inference

Inference about the impact of an intervention or treatment on an outcome - e.g., student academic achievement after experiencing a charter school curriculum - involves speculation about how a student would have performed on a given achievement test or examination, had the student not received the intervention or treatment (e.g., not experienced a charter school curriculum). The empirical analysis of such an intervention or treatment impact on student academic achievement can be conducted using the causal inference approach. This is also known as the Potential

Outcome Approach or the Neyman-Rubin Causal Model/Counterfactual Framework attributed to Neyman (1935) and Rubin (1974, 1977, 1990).

The Neyman-Rubin causal model/counterfactual framework conceptualizes causal inference in terms of two potential outcomes: one outcome under treatment

(observed) conditions and another outcome under control (unobserved) conditions.

Only one outcome (treatment or control) is observed for each subject i (Rubin 1974,

1977, 1990; Holland, 1986; Winship & Morgan, 1999; Caliendo, 2006; Caliendo &

Kopeinig 2008; Guo & Fraser, 2010; Sekhon, 2011). Thus, subjects selected into the treatment or control conditions have potential outcomes in both conditions but it is only one outcome that is observed. Rubin (1974) and Rosenbaum & Rubin (1995) defined causal effect (or treatment effect) as the difference between two potential outcomes (treatment and control) for which only one of the two outcomes is observed.

To illustrate the Neyman-Rubin causal model (Neyman, 935; Rubin, 1974,

1977, 1990) or counterfactual framework, let Yti denote the potential outcome for subject i if subject i receives the treatment (e.g., a student enrolled in a charter school). Let Yci denote the potential outcome for subject i if subject i does not receive the treatment (e.g., a student in the traditional public school, not enrolled in a charter school). Let Zi = 1 indicate the treatment condition (receipt of treatment), and

Zi = 2 indicate the control condition (non-receipt of treatment). Let i index the population/subjects under consideration. The causal effect or treatment effect (τ) for subject i is defined as the difference between Yti and Yci.. Thus

τi = Yti - Yci (2.1)

The observed outcome for subject i is:

Yi = ZiYti + (1- Zi)Yci (2.2)

Equation (2.2) illustrates the Neyman-Rubin causal counterfactual framework model which infers that a causal relationship between Zi (the cause) and Yi (the outcome) cannot directly link Yti to Zi under the condition Zi=1; instead an evaluation researcher or a data analyst has to check the outcome of Yci under the condition of

Zi= 2, and compare Yci with Yti because Yti and Yci outcomes cannot be observed jointly on the same i. This implies that causal inference is a missing data problem whose counterfactuals (e.g., what could have happened to the charter school students, had they not enrolled in a charter school curriculum (received treatment)) can only be obtained by estimation (Zhao, 2004; Caliendo, 2006; Guo & Fraser, 2010, Sekhon,

2011).

To handle such missing data problems, causal inference researchers such as

Rubin (1974, 1977, 1990, 2004), Rosenbaum & Rubin (1985), Winship & Morgan

(1999), Dehejia & Wahba (2000, 2002), Zhao (2004), Caliendo, 2006, Caliendo &

Kopeinig 2008, Diamond & Sekhon (2005), and Guo & Fraser (2010) suggest that researchers can determine causal/treatment effect by estimating the counterfactual and then examining the mean outcome of the treatment group subjects and the mean

outcome of the control (comparison) population subjects. Let E(Yt |Z = 1) denote the

mean outcome of the subjects in the treatment group. Let E(Yt | Z = 2) denote the

mean outcome of the subjects in the control population. Now that both E(Yc | Z = 2)

and E(Yt | Z = 1) are observable, the causal/treatment effect (τ) of the population can

be defined as the mean difference between E(Y2 | Z = 2) and E( | Z =1). Therefore

τ = E( | Z = 1) - E( | Z = 2) (2.3)

For experimental data where assignment to treatment condition is randomized, causal inference can be estimated easily because two groups are drawn from the same population by construction and treatment assignment is independent of all baseline covariates and group outcomes (Caliendo & Kopeinig 2006; Guo & Fraser, 2010;

Sekhon, 2011). The randomization of subjects to the treatment condition(s) guarantees that, on average, there no systematic difference or bias in observed or unobserved covariates between subjects assigned to treatment condition(s) (Rubin,

1974; Sekhon, 2011). Thus, as the sample size grows, observed and unobserved baseline covariates are balanced across treatment and control groups with arbitrarily high probability, because treatment assignment is independent of Yt and Yc. Hence,

E(Yij|Zi = 1) = E(Yij|Zi = 2) = E(Yi |Zi = j) (2.4)

For non- or quasi-experimental study settings where covariates are not balanced across treatment and control groups because the treatment and control groups are not ordinarily drawn from the same population, matching becomes an important design element to conduct causal inference studies (Rubin 1974, 1977,

2004; Rosenbaum & Rubin, 1985; Winship & Morgan, 1999; Dehejia & Wahba

2000, 2002; Diamond & Sekhon, 2005; Imai, King & Stuart, 2007; Guo & Fraser,

2010). In such cases, the estimation of τi = Yti - Yci cannot be measured directly because Yci is not observed for subjects in the treatment sample (Rubin, 1974;

Diamond & Sekhon, 2005; Caliendo, 2006; Guo & Fraser, 2010). As a result, a problem arises pertaining to the evaluation of Yt and Yc outcomes (Caliendo, 2006;

Guo & Fraser, 2010; Sekhon, 2011). To make progress with regard to balancing covariates across treatment and control groups, subjects have to be matched assuming that selection of subjects to treatment condition depends on the observable matching covariates Xi. Here, Xi is the covariate or vector of covariates providing additional information about the subjects being studied (Dehejia & Wahba 1999; Rubin &

Stuart, 2007; Stuart, 2010; Sekhon, 2011). Similar to randomization in an experimental design, the role of matching is to balance the distributions of all relevant pretreatment characteristics in the treatment and control groups (Rosenbaum, 2002;

Dehejia & Wahba 1999; Caliendo & Kopeinig 2006; Rubin & Stuart, 2007; Guo &

Fraser, 2010; Sekhon, 2011).

Theory of Matching

Matched sampling is a method of data collection and organization designed to reduce bias and increase precision in the studies in which the random assignment of subjects or units to treatment is absent (Rubin, 1973a). Thus, the theory of matching assumes that all relevant differences between two groups, e.g. treatment and control, are captured by the matching covariates (Joffe & Rosenbaum, 1999; Stuart & Rubin,

2007, Stuart, 2010). The idea of matching method is to match each treatment sample subject (Xi|Zi = 1) to n control population subjects Xj|Zj = 2) on X, where X is a vector of matching covariates and Z is an indicator variable for treatment or control

condition and then to compare the average outcome of the treatment group ( Yt )

sample subjects with the average outcome of the control group (Yc ) sample subjects.

Guo & Fraser (2010) show the resultant difference of the means of Yt and Yc is an estimate of causal effect/treatment effect (τ) estimated from:

match= E(Y match,1|Zmatch = 1) – E( match,2|Zmatch = 2) (2.5)

Here, is the estimated sample means of Yt and Yc with the subscript ‘match’ indicating matched samples or subsamples. For Zmatch = 1, the group comprises all treatment sample subjects. When Zmatch = 2, it comprises all control population sample subjects who are successfully matched to treatment sample subjects. Different methods of forming the comparison group sample are available, such as random sampling, exact matching, propensity score matching, nearest neighbor matching, radius matching, and Mahalanobis metric matching.

The matching procedure identifies the treatment sample subjects who share the same characteristics on a vector of covariates as the control (comparison) population subjects. The main goal of a matching procedure is to balance data and reduce the selection bias (Guo & Fraser, 2010; Stuart & Rubin, 2007; Joffe &

Rosenbaum, 1999; Rosenbaum, 1986; Rosenbaum & Rubin, 1985). Matching is a design option that can be used in experimental and non/quasi experimental studies to:

(i) balance data; (ii) reduce or eliminate selection bias; (iii) gain precision in the estimation of an effect measure of interest; (iv) control for variables or covariates that are difficult to measure; and (v) deal with practical aspects of data collection such as convenience, time-saving and cost-saving (Guo & Fraser, 2010; Stuart & Rubin,

2007; Kleinbaum, Sullivan, & Barker, 2007; Rosenbaum & Rubin, 1983; Rubin,

1974). For example, Kleinbaum, Sullivan, & Barker (2007) explain that matching can

(i) enable data balancing thus reducing or removing the confounding effect of a covariate or variable and (ii) help in gaining precision by allowing an estimation of a narrower confidence interval around the effect measure that could have been obtained without matching. For example, matching students in a school district by socioeconomic status would provide a way for controlling for poverty or social class.

Matching typically involves two groups being compared, the treatment group and the control (comparison) group (but can also involve multiple group designs as well). For example, in a comparison study involving public school and charter school students, the treatment group could be the charter school students and the control group could be the traditional public school students. In a matching design study, the control group is restricted to be similar to the treatment group on the matching covariates (Joffe & Rosenbaum, 1999; Ross, Lipsey & Freedman, 2004; Kleinbaum,

Sullivan, & Barker, 2007). It is assumed that the selection of matching covariates can be explained in terms of the matching covariates' observable characteristics. Thus, for every subject in the treatment group sample, a matching subject is found from the control group sample. When matching on discrete covariate variables such as gender, social-economic status, ethnicity or on continuous variables such as age, income, etc., a rule for deciding the matching criteria is needed or has to be specified (Dehejia &

Wahba, 1999; Rosenbaum, 2002; Rosenbaum & Rubin, 1985; Kleinbaum, Sullivan,

& Barker, 2007). For example, if charter high school students are matched with traditional public school high school students on age and gender, and the given age for matching student in the treatment group sample is 17 years old, black or white,

male students, then the public high school students obtained from the traditional public school population (control) should also be 17 years old, black or white, male students.

In non- or quasi-experimental design, matching seeks to identify group samples of treatment and control populations that are “balanced” with respect to observed covariates (i.e., the observed covariate distributions are the same in the treatment and control population groups). A randomized experiment balances the distributions of both observable and unobservable covariates between treatment and control group samples. In nonrandomized study settings characterized by matching, the exact and propensity score matching processes only balance the observable covariates between treatment and control group samples based on each individual covariate or on a composite propensity score value (Zhao, 2004; Guo, 2010). Thus, for exact matching, each subject i in the treatment sample is matched with each subject j in the control population on a vector of observed covariates (Xi) such that Xi

= Xj = x, and for propensity score matching, each subject i in the treatment sample is matched with each subject j in the control population if e(Xi) = e(Xj) = p or in the neighborhood of p.

The matched sampling is a method of data collection and organization designed to reduce bias and increase precision in the studies in which the random assignment of treatment to subjects or units is absent (Rubin, 1973a). Thus, the theory of matching assumes that all relevant differences between two groups, e.g. treatment and control groups are captured by the matching covariates (Joffe & Rosenbaum,

1999; Stuart & Rubin, 2007, Stuart, 2010). The idea of matching method is to match

each treatment sample subject (Xi|Zi = 1) to n control population subjects Xj|Zj = 2) on X, where X is a vector of matching covariates, and Z is an indicator variable for treatment or control condition. And then compare the average outcome of the

treatment group (Yt ) sample subjects with the average outcome of the control group (

Yc ) sample subjects. Guo & Fraser (2010) show the resultant difference of the means of Yt and Yc is an estimate of causal effect/treatment effect (τ) estimated from:

match= E(Y match,1|Zmatch = 1) – E( match,2|Zmatch = 2) (2.6)

Where, is the estimated sample means of Yt and Yc; the subscript ‘match’ indicates matched sample or subsample. For Zmatch = 1, the group comprises of all treatment sample subjects, and Zmatch = 2, comprise of the control population sample subjects who are successfully matched to treatment sample subjects via different methods of forming comparison group sample such as random sampling, exact matching, propensity score matching, nearest neighbor matching, radius matching and

Mahalanobis metric matching.

Rosenbaum, 1999; Rosenbaum, 1986; Rosenbaum & Rubin, 1985). Matching is a design option that can be used in experimental, and non/quasi experimental studies to:

(i) balance data; (ii) reduce or eliminate selection bias; (iii) gain precision in the

estimation of an effect measure of interest; (iv) control for variables or covariates that are difficult to measure; and (v) for practical aspects of data collection such as convenience, time-saving and cost-saving (Guo & Fraser, 2010; Stuart & Rubin,

2007; Kleinbaum, Sullivan, & Barker, 2007; Rosenbaum & Rubin, 1983; Rubin,

1974). For example, Kleinbaum, Sullivan, & Barker (2007) illustrates that matching;

(i) can enable data balancing thus reducing or removing the confounding effect of a covariate or variable; (ii) can help in gaining precision by allowing an estimation of a narrower confidence interval around the effect measure that could have been obtained without matching; (iii) students in a school district by their socio-economic status would provide a way for controlling for poverty or social class.

Matching typically involves two groups being compared (but can also involve), the treatment group and the control (comparison) group. For example, in a comparison study involving public school and charter school students, the treatment group could be the charter school students and the control group could be the traditional public school students. In a matching design study, the control group is restricted to be similar to the treatment group on the matching covariates (Joffe &

Rosenbaum, 1999; Ross, Lipsey & Freedman, 2004; Kleinbaum, Sullivan, & Barker,

2007). It is assumed that the selection of matching covariates can be explained in terms of the matching covariates observable characteristics. Thus, for every subject in the treatment group sample, a matching subject is found from the control group sample. When matching on discrete covariate variables such as gender, social- economic status, ethnicity or continuous variables such as age, income etc., a rule for deciding the matching criteria is needed or has to be specified (Dehejia & Wahba,

1999; Rosenbaum, 2002; Rosenbaum & Rubin, 1985; Kleinbaum, Sullivan, &

Barker, 2007). For example, if charter high school students are matched with traditional public school high school students on age and gender, and the given age for matching student in the treatment group sample is 17 years old, black/white, male students, then the public high school students obtained from the traditional public school population (control) should also be 17 years old, black/white, male students.

In non- or quasi experimental designs, matching seeks to identify group samples of treatment and control populations that are “balanced” with respect to observed covariates (i.e., the observed covariate distributions are the same in the treatment and control population groups). A randomized experiment balances the distributions of both observable and unobservable covariates between treatment and control group samples. In nonrandomized study settings characterized by matching, the exact and propensity score matching processes only balances the observable covariates between treatment and control group samples based on each individual covariate or propensity score value (Zhao, 2004; Guo, 2010). Thus, for exact matching each subject i in the treatment sample is matched with each subject j in the control population on a vector of observed covariates (Xi) such that Xi = Xj = x, and for propensity score matching each subject i in the treatment sample is matched with each subject j in the control population if e(Xi) = e(Xj) = p or in the neighborhood of p.

At the study design stage, subjects to be compared are selected, without use of the values of the outcome covariates. In other words, the matches are chosen without access to any of the outcome data to prevent or reduce bias in selecting a particular

matched sample (Guo, Barth & Gibbons, 2006; Stuart & Rubin, 2007). After matching is completed at the study design stage, an analysis of the matched data sets can then be conducted (Guo, Barth & Gibbons, 2006).

Matching as a Study Design

Matching as a practical study design technique emerged from the statistics literature to aid causal inference studies and demonstrates a close link to the experimental context (Caliendo & Kopeinig, 2006, 2008). Causal inferential studies involving a matching design have been conducted widely in the field of statistics

(e.g., Fisher & Neyman, 1935; Roy, 1951; Quandt, 1972; Rubin, 1973, 1974, 1977,

1979, 1980, 2006; Rosenbaum, 2002; Rosenbaum & Rubin, 1983, 1985; Holland

1986; Lechener, 1998). However, other than in statistics, the use of matching methods have spread increasingly as practical techniques into other fields such as economics (Lalonde, 1986, Dehejia & Wahba, 2000, 2002; Heckman, Ichimura,

Smith & Todd, 1997; Smith & Todd, 2003), medicine (Rubin 1997; Christakis &

Iswashyna, 2003; Baser, 2006), epidemiology (Kurth, Glynn, Chan, Gaziano &

Berger, 2006; Kleinbaum et el, 2007), political science (Bowers & Hansen, 2005;

Arceneaux, Gerber & Green, 2006; Sekhon, 2004; Imati, 2005), sociology (Smith,

1997, Morgan & Harding 2006; Diprete & Engelhardt,2004; Winship &

Morgan,1999), program evaluation (Heckman & Robb, 1985; Dehejia & Wahba,

1999; Bryson, Dosett & Purdon, 2002), educational psychology (Hoy, 2000), and psychology (Fillenbaum, Hybels, Pieper, Konrad, Burchett, & Blazer, 2006; Tebes,

Feinn, Vanderploeg, Chinman, Shepard, Brabham, 2007).

The role of matching in a matched study design involves identifying in the control population, subjects who are similar on a vector of covariates to treatment sample subjects (often a convenience sample) and then using the mean outcome of the control group sample as an estimate or proxy to estimate the counterfactual of the treatment sample group. The conditional identifying assumption states that outcomes

(Yt,,Yc) have independent participation status (Z) conditional on the vector of X covariates. This assumption is attributed to Dawid (1979) and has been popularized by Rosenbaum & Rubin (1983, 1984, 1985, 1986) and by Rubin (1977), Lechner

(1999); and Heckman, Ichimura & Todd (1997, 1998) for use in matching and casual inference studies. The assumption is known by other names e.g., the unconfoundedness assumption or ignorable treatment assignment (Rosenbaum &

Rubin, 1983); the conditional independence assumption (Rubin, 1977; Lechner, 1999) or the selection on variables assumption (Heckman, Ichimura & Todd (1997, 1998).

Unconfoundedness/Identification Assumption

The unconfoundedness/identifying assumption states that treatment assignment of some given individual subjects is independent of potential outcomes

(Yti,Yci) if the relevant predetermined or observable covariates (Xi) are held constant

(Rosenbaum & Rubin, 1983). This assumption is satisfied if Xi includes the covariates (Xi) that affect both participation and outcomes in a treatment or intervention. As a result, the distribution of covariates of Xi in both treatment and control groups is balanced.The treatment effect can be estimated by comparing the outcomes in both treatment and control groups. Specifically, the unconfoundedness

/identification assumption states that, conditional on relevant predetermined or observable covariates X, outcome Y is independent of Z. That is:

Yt,Yc ╨ Z|X (2.7)

Here, ╨ denotes Yt,Yc are independent given X predetermined or observable covariates, e.g., X1, X2, X3…..Xn, that are not influenced by the treatment. Z is a binary indicator variable indicating participation in the treatment or control condition.

Rosenbaum & Rubin (1983); Heckman, Ichimura, & Todd, (1997); Winship

& Morgan (1999); Dehejia & Wahba (2000); Caliendo (2006) demonstrate that if unconfoundedness assumption as outlined in equation (2.6) holds then:

F(Yt | X, Z = 1) = F(Yt | X,Z = 2) (2.8)

and

F(Yc| X, Z = 1) = F(Yc | X,Z = 2) (2.9)

Equation 2.7 suggests that, conditional on X, the treatment sample (Yt) subjects have the same distribution of outcomes that the control population (Yc) subjects would have experienced, had they participated in a treatment or intervention. Equation 2.8 suggests that control population subjects have the same distribution of outcomes that the treatment sample subjects would have experienced if they had not participated in a treatment or intervention. This demonstrates that in a way similar to randomization in an experimental design, matching balances the distributions of all covariate characteristics in the treatment and control population groups. As a result, independence between potential outcomes (Yt,Yc) and the assignment to treatment is achieved.

Also, Rosenbaum & Rubin (1983); Heckman, Ichimura, & Todd, (1997);

Winship & Morgan (1999); Dehejia & Wahba (2000); Caliendo (2006) demonstrate that if the mean exists for each of the (Yt,Yc) outcomes, then:

E(Yt | X, Z = 1) = E(Yt | X,Z = 2) = E(Yt| X) (2.10)

and

E(Yc | X, Z = 1) = E(Yc | X,Z =2) = E(Yc| X) (2.11)

They further demonstrate that an overlap assumption: 0 < prob (Z = 1|X) < 1 has to exist for all X so that both sides of equations 2.9 and 2.10 are well defined at the same time for all X in order to support X equally in both treatment and control population groups (i.e., S = Support (X|Z = 1) = Support (X|Z = 2)). The overlap assumption prevents X from being a perfect predictor so that for each subject in the treatment sample, a counterpart in the control population and vice versa can be found (Dehejia

& Wahba, 2000; Zhao, 2004; Caliendo, 2006, Stuart, 2010).

Data Matching and Integration in Exact and Propensity Score Matching

Exact matching (e.g., one-to-one matching, one-to-many matching) and propensity score matching (e.g., propensity score, nearest neighbor, radius,

Mahalanobis distance matching) approaches and procedures are some of the frequently used matching approaches. Whereas exact matching procedures match each subject i in the treatment sample with each subject j in the control population on a vector of observed covariates (Xi) such that Xi = Xj = x, propensity score matching procedures match each subject i in the treatment sample with each subject j in the control population if e(Xi) = e(Xj) = p or is in the neighborhood of p.

Rosenbaum and Rubin (1983) suggest the use of a balancing score (the propensity score) as an alternative to exact matching. They called the propensity score a balancing score because of its ability to balance the relevant covariates across the matched groups. They defined a balancing score (b(X)) as a function of observed covariates X for which the conditional distribution of X given b(X) is the same for the treatment (Z = 1) and control population (Z = 2) subjects. The propensity score (e(X)) is the chance that a subject with covariates X will be exposed to treatment, given the conditional probability of assignment to treatment or control conditions. The propensity score summarizes the information of the observed covariates of X into a single value or index.

Following the logic of equation 2.6 that Yt,Yc ╨|X and the overlap/common support assumption that 0 < prob(Z = 1|X) < 1 conducting data matching under exact matching implies

(Yt,Yc) ╨|X = Xi = Xj = x (2.12) and propensity score matching implies

(Yt,Yc) ╨| e(Xi) = e(Xj) = p (2.13)

and

0 < prob(Z = 1| e(Xi) = e(Xj) =p < 1 (2.14)

Following Zhao (2004) how exact and propensity score matching methods work can be illustrated as follows:

Exact Matching

In exact matching each subject i in the treatment sample is matched with each subject j in the control population on a vector of observed covariates (Xi) such that

Xi=Xj=x, implying Xi=Xj=x fz(xi)=fz(xj), z=1 if in treatment and z=2 if in control.

Assuming subject i in the treatment sample is matched with subject j in the control sample (i.e., Xi=Xj=x) let:

e i =Yti – Ycj (2.14)

Yti = f1(xi) + 1i (2.15)

Yci = f2(xi) + 2i (2.16)

Applying equations 2.15 and 2.16 to equation 2.14:

= Yti – Yci

= f1(xi) + 1i - f2(xj) - 2j

= f1(xi) – f2(xj) + 1i - 2j

= i + 1i - 2j (2.17)

Where 1i, 2i are error terms conditional means (conditioning on xi); is the estimator for estimating the treatment effect for subpopulation obtained via exact

matching. is the true treatment effect for subject i and 1i - 2 is the bias.

Propensity Score Matching

Propensity score matching is quite different from exact matching (cell or covariate) matching. In propensity score matching, when the matching is exact at the propensity score p, the distribution of X will be the same for the treatment sample and

the control sample. Thus, Prob(Xi|Zi = 1,p(Xi) = p = prob(Xi|Zi = 2), p(Xi) = prob(Xi|p). If matching at p is impossible and instead matching is on some neighborhood of p, the distribution of X is still approximately the same for the treatment sample and the control sample within the neighborhood of p (Zhao, 2004).

Therefore, d(pk, pl) < d’prob(Xi|pk), prob(Xj|pl) < δ. Where d is distance between pk and pl, δ is the smallest absolute propensity score difference between k and l allowed. When using a propensity score to conduct matching, subject i in the treatment sample is matched with subject j in the control sample if p(xi) = p(xj) = p or the neighborhood of p provided p is within d(pk,pl) < δ. Assuming subject i in the treatment sample is matched with subject j in the control sample based on the propensity score p or in the neighborhood of p (p(xi) = p(xj) = p) let:

p i = Yti – Ycj (2.18)

Applying equations 2.15 and 2.16 to equation 2.18:

= Yti – Ycj

= f1(xi) + 1i - f2(xj) - 2j

= f1(xi) – f2(xj) + 1i - 2j

= i + 1i - 2j (2.19)

Where is the estimator for estimating the treatment effect for subpopulation obtained via propensity score matching method(s).

Applications of Matching Methods

When to use or not to use exact matching or propensity score matching methods depends on the availability of sufficient data (Rajeev & Wahba, 2002). For example, Rosenbaum & Rubin (1983) and also Imbens (2000) show the best case scenario for a data analyst to use exact matching occurs when individuals with exact the same set of characteristics X in the treatment and control groups are matched with or on feasible covariates subject to availability of sufficient data. Otherwise, in scenarios where data are insufficient or matching covariates are insufficient to due to matching failure, propensity scores still can be generated and then used to conduct data matching. Dehejia & Wahba (1999, 2002); Rubin (1984); and Rubin & Thomas

(1996) explain how in estimating propensity scores via logistic regression model interactions and higher order terms can be included as covariates. Rosenbaum &

Rubin (1984) illustrated subclassification based on the propensity score using observational data on two treatments for coronary disease. Theyshowed that subclasses formed from the scalar of the propensity score would balance the covariates and remove over 90% of bias due to covariates.

Several methods have been proposed as the best ways to adjust propensity scores statistically once they are estimated. Rosenbaum (2002) suggested paired matching (one-to-one) and full matching (one-to-many) as two of the best methods for pairing or group matching based on the proximity of the propensity score. Rubin

& Thomas (1996) and D’Agostino (1998) show that methods of sampling groups

(such as equal sample sizes for treatment and control) can help adjust statistically

groups of subjects matched using estimated propensity scores. Rubin (1997) suggest stratification (dividing the distribution of propensity scores into intervals, so that each strata has a group of treated and control populations) is also a good way of statistically adjusting samples matched using propensity scores. He postulates that matching on maximum strata of five groups or strata is the most effective and that treatment and control units or samples should be comparable within each stratum.

Kelcey (2011) suggest multilevel propensity scores and outcome proxies or cross validation are better or more effective ways for covariate selection in propensity scores since these approaches parallel the use of observed outcomes. He postulates that matching across schools based on multilevel propensity scores have much more to offer than matching within schools based on a single level of propensity score.

Kelcey also argues that using approaches such as outcome proxies or cross validation to select covariates in propensity score matching substantively augments knowledge to better inform covariate selection and to improve propensity score and causal inference estimation.

Applications of Exact Matching

Other than the works of The Federal Subcommittee on Matching Techniques

(Radner, Allen, Jabine & Muller, 1980); Lalonde (1986); Rajeev & Wahba (2002);

Zhao (2004), and Ho, Imani, King, & Stuart (2006, 2007), the application of exact matching in the matching literature is scarce. The Federal Subcommittee discussed the applications of exact matching in terms of using exact matched datasets to gain precision in the estimation of an effect measure of interest, adjusting for confounding

variables to reduce selection bias and constructing matched samples from treatment and control groups so as to analyze causal effects from observational or categorical data.

Denk & Hack (2003) illustrated the application of exact matching by showing how records of people are matched to other records based on their extent of agreement or disagreement on specified matching variables (the quality cases approach). He also illustrated the application of exact matching by showing how data records are matched, assigned, and integrated (the classification method). Kawataba,

Tran & Hines (2004) illustrated the application of exacting matching in epidemiological studies by showing how subjects in the treatment sample can be matching with subjects in the control population based on some matching criteria.

Lalonde (1986), Rajeev & Wahba (2002), and Zhao (2004) illustrated how exact matching is limiting as a matching procedure when the number of matching covariates increases and there is an insufficient control population from which to draw comparison samples. Ho, Imani, King, & Stuart (2006, 2007) demonstrated how exact matching can be conducted by matching each observation in the treatment sample to possible observations in the control sample with exactly the same values on all the covariates e.g. gender, age, grade level etc.

Applications of Propensity Scores

Propensity scores have been applied to study designs involving group comparisons, nested designs, and estimation of effects. For instance: Rosenbaum &

Rubin (1985a, 1985b) used propensity scores to study the effects of prenatal

exposures to barbiturates. In both studies, they used propensity scores to match a treatment sample of n=221 children to a control sample of 221 children drawn from a control population of N=7,027 children based on four covariate factors: child characteristics, mother characteristics, pregnancy characteristics and drugs. The propensity scores were estimated using a logit model of 20 covariates. The 1985a study results showed covariate matching using propensity scores removed most of the covariates bias. In the study they also illustrated the use of three multivariate matching sampling methods (nearest available matching on the estimated propensity score, nearest available Mahalanobis metric distance matching on the estimated propensity score, and Mahalanobis metric distance matching within caliper defined by the estimated propensity score. Results indicated the nearest available matching on the estimated propensity score and the nearest available Mahalanobis metric distance matching on the estimated propensity score methods were successful in reducing bias.

In the 1985b study, results indicated that bias due to incomplete exact matching can be severe, whereas matching using propensity score removed all the bias. Further, in addition to using propensity score to conduct matching, Rosenbaum & Rubin (1985b) suggested matching using an appropriate multivariate nearest available matching algorithm to reduce or eliminate bias due incomplete or inexact matching.

Rubin (1998) showed how subclassification comparison study groups can be created based on propensity scores to adjust for a single confounding covariate such as age in a study of smoking and mortality. He further demonstrated how propensity scores methods can be used to generalize subclassification on a single confounding covariate to the case with many confounding covariates, e.g., age, country region, and

gender using datasets of nonsmokers, cigarette smokers, and cigar and pipe smokers from the U.S., the U.K., and Canada in 1994. Joffe & Rosenbaum (1999) reviewed the uses and limitations of propensity scores and provided a brief outline of the statistical theory associated with propensity scores. In their review, they presented results demonstrating the application of propensity scores in case-cohort studies, concluding that in such studies, estimated propensity scores perform better than true propensity scores because they (estimated propensity scores) remove some chance imbalances in covariate that true propensity scores leave behind.

Imbens (2000) examined the role of the propensity score in estimating dose- response functions by computing the probability of receiving a particular level of treatment based on multi-valued treatments. He extended the work of Rosenbaum &

Rubin (1983, 1984) on the estimation of binary-valued treatment effects based on propensity score to propose a propensity score methodology that allowed for the estimation of average causal effects with multi-valued treatments. His work suggested an estimation of causal effects methodology where a population is divided into subpopulation and then the average potential outcomes are estimated. Rubin &

Thomas (2000) used propensity score matching, Mahalanobis metric matching, and a combination of propensity score matching and Mahalanobis metric matching using caliper matching to match their treatment sample children with the control population children. In this study, a dataset with a treatment sample of n=96 children and a control population of N=7,847 children whose mothers were exposed to a hormone during pregnancy was used. The results showed combining propensity score matching

with Mahalanobis metric matching can eliminate most of the bias with respect to a linear model relating outcome values to matching variables/covariates.

Bloom, Hill, Lie & Michalopoulos (2002) examined a combination of different methods of forming groups and matching (experimental control groups and multiple non-experimental comparison generated using propensity score and different statistical models) to determine which of the comparison group methods provided the best effects estimates. The methods combined different types of comparison groups

(in-state, out-of-state, and multi-state) with different propensity score balancing approaches (sub-classification, one-to-one matching), and different statistical methods (ordinary least squares, fixed-effect models, and random growth models).

The methods were compared in their ability to estimate program effect on annual earnings in three time periods: short-run, medium-run and long-run. The results suggested in-state comparison methods performed better than out-state or multi-state methods in the estimation of the medium-run program effects and the non- experimental comparison group method estimation error was larger relatively for medium-run program effect estimates than it was for the short-run program effect estimates.

McCaffrey, Ridgeway, & Morral (2004) demonstrated how to use generalized boosted modeling (GBM) to estimate propensity scores impeded by large number of covariates. GBM is a multivariate non-parametric technique to estimate propensity scores whose algorithm can also estimate non-linear relationships between a variable or covariate of interest and a large number of covariates. The estimated propensity scores are used to predict treatment assignment from a large number of pretreatment

covariates while also allowing for flexibility in the relationships between the covariates and the propensity score. Using the Adolescent Outcomes Project (AOP) data, these researchers shown the propensity scores estimated using the boosting technique eliminated most of the pretreatment group differences and altered substantially the treatment effect estimates for the adolescent substance abuse program.

Hong & Raudenbush (2005) used propensity scores to create a multilevel propensity score stratification data consisting of 6 strata in a nested study design to estimate the effects of kindergarten retention policy on childrens’ cognitive growth in reading and mathematics using the US Early Childhood Longitudinal Study

Kindergarten cohort (ECLS-K) data from fall 1988 to spring 2000. Through propensity score stratification, Hong & Raudenbush balanced 141 non-retention schools and 1,080 retention schools on 238 school-level pretreatment characteristics and then used propensity score stratification to balance 471 retained students with promoted students attending schools that allowed retention based on school-level pretreatment characteristics. Their study results did not find any effect evidence to suggest that a policy of grade retention in kindergarten improves average children’s achievement in mathematics or reading or benefits children who would be promoted under the policy. However, it did note that the effects of retention on retainees are considerably large.

Baser (2006) used the MarketScan data, 2005 to estimate the cost of illness for asthma patients using the matching techniques: nearest neighborhood matching,

Mahalanobis matching, Mahalanobis matching with caliper, kernel matching, radius

matching, and stratification. In the study, he matched 1,184 asthma patients

(treatment sample) with 3,169 patients with no asthma (control population). Five quantifiable steps were used to check for balance between the treatment and control groups in the study. These were: (1) two sample t-statistics to compare the means of the treatment and control groups for each explanatory variable; (2) mean difference as a percentage of the average standard deviations; (3) percentage reduction of bias in the means of the explanatory variables before and after matching; (4) treatment and control density estimates for the explanatory variables; and (5) the density estimates of the propensity scores of the treatment and control subjects. The study’s investigation of matching techniques relating their comparative performance to these quantifiable steps revealed the choice of matching technique has effects on the estimated outcomes. After group data matching, the regression analysis results showed that regression-based difference standard errors (deviation in cost of illness estimates) decreased more than three-fold between the matched control and treatment group samples (Mean = 4456 dollars, SD = 996 dollars). The Mahalanobis matching with caliper technique yielded better results according to the five steps criteria (Mean

= 4463 dollars, SD = 3252 dollars). The Mahalanobis matching and Mahalanobis matching with caliper techniques were able to match the treatment and control groups on all the matching variables: age, gender, region, plan type, and point of service.

Matching Estimators for Treatment Effect

The following matching estimators have been suggested for use to estimate treatment effects: Abadie and Imbens (2001) suggested using the exact matching

estimator to predict mean treatment effect from samples formed via the exact matching approach. Millimet & Tchernis (2008) demonstrated that, given knowledge of propensity scores and sufficient overlap between the distributions of the propensity scores across treatment group, Z=1, and control group, Z=2, the average treatment effect can be estimated via the inverse probability weighted estimator that had been suggested by Horvitz and Thompson (1952) using estimated propensity scores.

Ichino, Mealli, & Nannicini (2004) demonstrate a nearest neighbor matching estimator to estimate average treatment effect if subject i is matched to a control group subject j such that the nearest neighbor matching sets: C(pi) = {j|j = min |pi – j pj|}, where C(pi) is the set or number of control group observations matched to treatment sample observations with an estimated value of the propensity score of pi.

Todd (2006) suggests a caliper matching estimator to estimate mean treatment effect via choosing the nearest neighbor inside a caliper of width δ such that the set of matched comparisons can be represented by {j: |pi − pj| < δ}, where p is propensity score. He demonstrates that caliper matching is a variation of nearest neighbor matching that attempts to avoid “bad” matches (those for which pii is far from pj) by imposing a tolerance on the maximum distance δ such that for a pre-specified δ > 0, treatment sample subject i is matched to control sample subject j thus: δ > │pi-pj│= min {│pi-pj│}, j I2 , where I2 is set of control group subjects. Thus a match for j (t 2) subject/person i is selected only if |pi − pj| < δ, j I2. Beckar & Ichino (2002) illustrated how to estimate treatment effects via the radius matching estimator biased based on propensity scores: pj||pi − pj|| < r where r is radius distance. Rubin (1979,

1980) and Xing & Rosenbaum (1993) showed how a Mahalanobis metric match can be conducted after estimation of the Mahalanobis Metric distance to reduce bias in observational studies.

Computer Algorithms to Stimulate or Match Datasets

An extensive review of the matching literature suggests some work has been done pertaining to the generation of computer algorithms to stimulate matched datasets. Several SAS programmers under the auspices of SAS Users Group

International (SUGI), SAS Global Forum (SGF), and SAS Institute Incorporated have written SAS algorithms to simulate datasets for matching. For example, Waller,

Brantley & Podolsk (2002) wrote a SAS macro to conduct one-to-one exact matching on case and control population datasets using categorical variables such as sex, race, and marital status with age and screening score as continuous variables. Kawataba,

Tran & Hines (2004) wrote a SAS algorithm to conduct exact matching for a group study matching treatment group subjects to a control group subjects using race and age as the matching covariates.

Parsons (2000, 2001) wrote SAS macros to create propensity scores using the

LOGISTIC procedure and demonstrated how to use the created propensity scores to match treatment sample cases to control population cases based on the created propensity scores in order to reduce matched-pair bias caused by incomplete matching and inexact matching.

Parsons (2004) presented a SAS macro to perform 1: N matching. She demonstrated how to match treatment sample cases to control population cases by

matching them on the propensity score and allowing the data analyst to specify the number of controls to match to each treatment and control population cases. Her macro algorithm makes "best" matches first and "next-best" matches next based on the propensity score digit(s), continuing in a hierarchical sequence until no more matches can be obtained.

Perraillion (2006, 2007) presented SAS codes demonstrating how to match samples using matching methods such as propensity score, nearest neighbor, caliper, radius and Mahalanobis metric matching. The 2006 presentation showed how to match datasets using these methods after the propensity scores are created thus offering a way to balance groups by matching treatment and control units based on a set of covariates. In the 2007 article and presentation, he describes how to adopt a

SAS code focusing on propensity score matching to implement Mahalanobis metric matching with propensity scores along with other covariates

Feng, Lilly, Jun, & Xu (2006) demonstrated how the nearest available

Mahalanobis metric matching within calipers defined by the propensity score matching algorithm can be used to perform matching. These authors developed a macro to implement a matching algorithm for a growth hormone observational study demonstrating how Mahalanobis metric matching within calipers defined by the propensity score can be used reduce bias.

D’Agostino (1998) wrote a SAS code to estimate propensity scores using the logistic regression procedure in SAS, and then use the estimated propensity scores to conduct data matching and compute t-tests.

Outside of SUG, SGF and SAS Institute Incorporated, other researchers and author have written algorithms or software to conduct data matching particularly to estimate average treatment effects for two or more treatment samples otherwise known as average treatment effect of the treated. For instance, several forms of multiple R software packages have been developed to conduct matching: e.g., Ho,

Imai, King & Stuart (2006) developed the MatchIt software; Ridgeway, McCaffrey &

Morral (2006) developed the twang software; Sekhon (2006) developed the twang software; Becker & Ichino (2002); Leuven & Sianesi (2003); Abadie, Drukker, Herr

& Imbens (2004) developed the multiple stata packages. .

The MatchIt software is designed for causal inference studies with a dichotomous treatment variable and a set of pretreatment control variables. It implements the suggestions of Ho, Imai, King, and Stuart (2006) for improving parametric statistical models by preprocessing data with nonparametric matching methods. The software implements a wide range of matching methods such as exact matching, subclassification, propensity score matching, nearest neighbor matching, optimal matching, full matching, generic matching and coarsened exact matching.

The Toolkit for Weighting and Analysis of Nonequivalent Groups (twang) software developed by Ridgeway, McCaffrey & Morral in 2006 contains a set of functions and procedures to support causal inference modeling of observational data via the estimation and evaluation of propensity scores and associated weights. The matching Sekhon (2006) is an R package that provides functions for implementing algorithms for multivariate matching including propensity score, Mahalanobis, inverse variance and genetic matching.

The Multiple Stata packages (Leuven & Sianesi, 2003); Abadie, Drukker,

Herr & Imbens, 2004) uses the psmatch2 and nnmatch match commands. In Leuven

& Sianesi (2003) the psmatch2 command which implements full Mahalanobis matching and a variety of propensity score matching to adjust for pretreatment observable differences between treatment and control groups is used while in Abadie,

Drukker, Herr & Imbens (2004) the nnmatch command is used. The nnmatch command allows for an estimation of average effect for all unites/subjects or only for the treatment and control group units, to choose the number of matches, to specify the distance metric, to select the bias adjustment and use robust variance estimators.

This dissertation study did not attempt to estimate average treatment effects for two or more treatment samples or average treatment effect of the treated where the multiple R software packages would be most suitable for matching datasets. Instead this study attempted to estimate treatment effects resulting from the difference between single treatment and control samples where the SAS generated algorithms seemed most suitable for simulating and matching datasets. Given this scenario, SAS generated algorithms were chosen over the multiple R software packages for current study purposes.

Relevance and Significance of this Study

This study’s review of literature reveals that most of the previous studies (e.g;

Rubin, 1973, 1974, 1977, 1979, 1980, 2006; Rosenbaum, 2002; Rosenbaum & Rubin,

1983, 1985; Lechener, 1998; Lalonde, 1986, Dehejia & Wahba, 2000, 2002;

Heckman, Ichimura, Smith & Todd, 1997; Kurth, Glynn, Chan, Gaziano & Berger,

2006; Guo, Barth & Gibbons, 2006; Diamond & Sekhon, 2005, Sekhon, 2010 etc.) investigating matching have focused on studying causal inferences between treatment and control groups. Other studies (e.g., Lalonde, 1986; Smith & Todd, 2003; Ichino,

Mealli, and Nannicini, 2004; Morgan & Harding 2006; Millimet & Tchernis, 2008) have focused on the estimation of treatment effects via different matching estimators such as stratification, kernel, nearest neighbor, caliper, and propensity score.

D’Agostino (1998); Calindo (2006); and Calindo & Kopeining (2008) documented the applications of propensity score matching explaining how propensity score matching methods can be used in causal inference studies to reduce or remove selection bias. D’Agostino (1998) provided a tutorial discussing the uses of propensity score methods for bias reduction such as matching, stratification

(subclassification) and regression adjustment. Calindo (2006) and Calindo &

Kopeining (2008) documented a five steps processes for implementing propensity scores in evaluation studies targeting causal inference and bias reduction. These steps were propensity score estimation, choosing matching algorithm, checking overlap/common support, matching quality/effect estimation and sensitivity analysis.

Guo, Barth & Gibbons (2006) estimated casual effects of child welfare and parental substance abuse services on maltreatment re-reports and developmental well- being for children of substance abuse treatment service recipients and non-recipients using the matching methods (nearest neighbor with calipers, Mahalanobis metric matching, Mahalanobis metric matching including the propensity score, local linear matching) propensity scores. The study findings reported substantial variation in the effect sizes estimated by the three matching methods with nearest neighbor with

calipers removing most of the bias among covariates and producing better effect size estimates followed by local linear matching method, and Mahalanobis metric matching methods performing the worst in reducing or removing bias.

These previous studies have not focused specifically on exact vs. propensity score matching protocols, particularly not on the comparison of the exact and propensity score matching protocols in non/quasi experimental datasets characterized by a large population or populations with smaller subpopulations of foci in order to ascertain whether or not design factors such as sample and population sizes, methods of sampling, effects sizes, and assumed distribution affect the accuracy of the matching systems as this pertains the precision of estimating group means, impact of type I error rates, covariates matching conditions, and bias recovery/reduction especially in the fields of education research and evaluation. This study addressed this gap. This study contributes to matching literature pertaining to social science and educational research and evaluation by: (i) identifying the design conditions that affect matching procedures in bias recovery/reduction; (ii) providing empirical results that demonstrate the similarities, differences and applicability of the exact and propensity score matching methods; (iii) providing clarity on which exact or propensity score matching protocols, procedures, and techniques are the most appropriate design and analytic approaches for estimating causal effects and making causal inferences in non/quasi-experimental study designs or studies; and (iv) demonstrating how exact and propensity score matching techniques can be used in the design of causal inference studies to draw inferences or conclusions about the effectiveness of

educational interventions or programs to better inform educational leaders, researchers, and policymakers.

Summary

This chapter has reviewed the theories of causal inferences and matching, discussed matching as a study design, highlighted the assumptions embedded in matching procedures, and demonstrated how data matching and integration are achieved through exact matching and propensity score matching algorithms. The chapter also has reviewed many previous applications of matching methods and has established the relevance and significance of this study as it pertains to past theory and research.

CHAPTER III

METHODOLOGY

This chapter discusses the research design and methodology that was used to compare six different matching methods of forming comparison groups in a quasi- experimental study design under different experimental conditions in terms of each method's ability to recover causal effect size by recovering, reducing or eliminating specific a-priori amounts of deliberately introduced selection bias. Presented in this chapter are: (1) data generation techniques, (2) data analysis modalities, and (3) the results of a prior pilot study conducted to demonstrate the validity of the computer algorithms' ability to simulate the necessary data for the study.

Study Design

The study was designed to determine if the construction of a comparison group via exact matching or propensity score matching methods results in differential conclusions related to testing for mean differences between two groups under different experimental conditions. Datasets were randomly drawn from stimulated populations with known characteristics. After matching, datasets were analyzed to ascertain the six methods ability to recover effect size and recover, reduce or eliminate a-priori amount of selection bias under specific experimental conditions.

The experimental factors that were considered in this study were: type of covariate and number of covariates, covariate collinearity among covariates, covariate

correlation with bias, effect size, amount of selection bias and matching methods. These experimental factors were arranged in a factorial study design. Table 1 summarizes the between group factors of the study design. Type of covariates, number of covariates, covariate collinearity among covariates, covariate correlation with bias among continuous covariates in the control population, effect size and amount of sampling/selection bias were the between subjects independent variables, and the matching methods (random sampling (no match method (NOM), exact matching (EM), propensity score matching

(PSM), nearest neighbor matching (NNM), radius matching (RM), and Mahalanobis metric (distance) matching (MM) were the within subjects variables. The dependent variable was causal effect size, the difference between the true and the estimated effect

size symbolized by YD .

Table 1 Study Design Summary Type of Number of Covariates Correlation Effect Bias Covariate Covariates Collinearity With bias Size Discrete 4 None None 0.00 0.00 0.10 0.10 0.44/45 0.15 Continuous 4 0.00 0.00 0.00 0.00 0.40 0.40 0.10 0.10 0.90 0.75 0.44/45 0.15 0.99 Mixed* 4 (2D, 2C) 0.00 0.00 0.00 0.00 0.40 0.40 0.10 0.10 0.90 0.75 0.44/45 0.15 0.99

Characteristics of the matching covariate variables vector:

1. Type of covariate (measurement scale): Discrete (D), Continuous(C) and

number of covariates: four discrete covariates, four continuous covariates,

and two discrete with two continuous covariates, arranged in three study

cases:

Case 1: D, D, D, D

Case 2: C, C, C, C

Case 3: D, D, C, C

2. Covariate collinearity among covariates: (0.00, 0.40, 0.90).

3. Covariate correlation between the bias and covariate group among

continuous covariates in the control population: (0.00, 0.40, 0.75, .99).

4. Amount of selection bias: (0.00, 0.10, 0.15)

5. True treatment effect size: (0.00, 0.10, 0.44/0.45)

6. Matching Methods: NOM, EM, PSM, NNM, RM, and MM

A treatment sample of n = 200 and a control population of N = 10000 were simulated with 1000 replications conducted. Each design factor is described in the following section.

Type and Number of Covariates

The type and number of matching covariates per cell (dimensionality) affect the quality of matches per cell (Rosenbaum & Rubin, 1985; Winship & Morgan, 1999;

Dehejia &Wahba, 2000, 2002). For example, if there are several categorical or discrete matching covariates (e.g., gender, level of education, ethnicity) each with several levels/values, the number of empty cells may increase due to lack of matching observations or cases (Rosenbaum & Rubin, 1985; Dehejia &Wahba, 2000, 2002;

Rosenbaum 2002; Zhao, 2004; Guo & Fraser, 2010).

In this study, to obtain sufficient matching covariates per cell under both exact and propensity matching methods, an independence model among covariates was assumed in which the experimental condition for the number of matching covariates were restricted to four levels to check how the number of covariates affect the quality of matching subjects, and influence the effectiveness of the matching methods in recovering bias in nonrandom samples. Covariate type; e.g., discrete, continuous, and a mixture of both discrete and continuous, were into three cases to ascertain if type of covariate affected the quality of overall matching. Ds represented discrete covariates, and Cs represented continuous covariates, DCs represented a mixture of both discrete and continuous covariates. The discrete covariates (Case 1: D1, D2, D3, D4) were measured at three levels as follows: (i) covariates D1 and D2 had two levels of measurement; and

(ii) covariates D3 and D4 had three and four levels of measurement respectively. The continuous covariates (Case 2: C1, C2, C3, C4) had no discrete levels of measurement.

Case 3 represented a mix of covariates that were combined in the in the order of four

(2:2) meaning two discrete covariates (D1, D2) and were mixed with two continuous (C1,

C2). Covariates D1 - D4 and C1 - C4 were used in this study to represent the kind of data typically found in the fields of education and social sciences. The main dependent variable was the observed mean difference among matching methods (NOM, EM, PSM,

NNM, RM, and MM) between the treatment sample and the matched control samples.

This was estimated byY which reflects the difference between Y - Y as presented in D t C

Chapter II. Table 2 below shows the arrangement and design of covariates by type and number.

Table 2 Arrangement and Design of Covariates by Type and Number Type of Covariate Covariates Total Number of Covariates Discrete (Model 1) D1, D2, D3, D4 4 Continuous (Model 2) C1, C2, C3, C4 4 Mixed (Model 3) D1, D2, C1, C2 4

Effect Size

Effect size is the quantification of the difference between two groups e.g., treatment and control. In consideration of typically observed effect sizes in previous educational research, six studies were examined and provided a range of possible effect sizes, see Table 3. Summarized in this table are the observed effect sizes and the computed average effect size was 0.25. Using the effect sizes of these studies as guidelines, two levels of effect sizes were used in this study: small = 0.10 and medium =

0.45 together with a null effect size, = 0.00. The medium level effect size of 0.45 was computed as the average of 0.25 and the larger effect size of 0.65 (Slavin & Madden,

1989).

Table 3 Average Effect Sizes from Previous Research Studies Article Study Effect Size Slavin & Madden Targeted interventions for at-risk students to 0.65 (1989) increase student academic achievement Shymansky, Hedges Inquiry-based vs. traditional science curriculum 0.30 & Woodworth (1990) to increase student academic achievement Finn and Achilles Reducing class size from 23 to 15 students so as 0.30 (1990) to increase students’ test performance in reading Lipsey (1992) Treatment programs for juvenile delinquency 0.18 Fletcher-Flinn & Computer assisted instruction to increase 0.01 Gravatt (1995) academic achievement Light, et. al. (1996) Setting students versus mixed ability grouping to 0.08 compare achievement

In educational evaluation studies, Cohen’s (1986) criteria for determining and interpreting effect sizes have been used widely as benchmarks or standards of comparison. Cohen (1988) suggests that an effect size of 0.2 may be considered as small, whereas effect sizes of 0.5 and 0.8 or greater may be considered medium and large respectively. Other research studies e.g., Bloom et al, 2005; Kane (2004); Nye, Hedges

& Konstantopoulus (1999); and Lipsey (1990) suggests that for educational interventions much smaller effect sizes than the minimum effect size of 0.2 suggested by Cohen should be considered substantively important, thus the need for a greater precision than Cohen is suggested (Bloom et al, 2005). This point is consistent with Fletcher-Flinn & Gravatt

(1995) and Light et al (1996)’ see Table 3, indicating a need for designing studies that could detect smaller effect sizes than 0.20.

Amount of Selection Bias

Three amounts of selection bias (0.00, 0.10, 0.15) were simulated into the treatment sample to investigate how different levels of bias could be recovered by the different matching methods under varying experimental conditions. In the computation of estimated bias and reduced bias amounts, Rosenbaum & Rubin (1985); Rubin (1991); and Baser (2006) formulae for estimating bias and calculation of reduced amounts of bias

^ ^ were modified in this study. As result, bias was estimated as: B = YD - ES , and the amounted of unrecovered or unreduced bias was calculated as: UBR = B - , Where is the estimated bias, is the estimated mean treatment effect plus bias, is the

estimated effect size, UBR is the unrecovered/reduced amount of bias, B is the programmed (known) bias.

Matching Method

Six matching methods: NOM, EM, PSM, NNM, RM, and MM were used to form

group comparison samples for the study to test the primary hypothesis of ( T C ) = 0.

1. Random Sampling (No Matching): Using SAS SURVEYSELECT

procedure, a simple random sample (SRS) of n=200 was drawn from the

simulated control population.

2. Exact Matching: Via exact matching, subject i in the treatment sample was

matched with subject j in the control population if Xi = Xj = X, where X is

a vector of observed covariates (D1-D4; C1-C4; D1, D2, C1, C2). Thus

for discrete covariates (D1, D2, D3, D4 - Case 1), each subject i in the

treatment sample was exactly matched by random number and id number

with each subject j in the control population if each covariate (D1 - D4 –

Case 1) in the treatment sample was equal to each covariate (D1 - D4) in

the control population. For continuous covariates (C1, C2, C3, C4 - Case

2), each subject i in the treatment sample was exactly matched by random

number and id number with each subject j in the control population if each

covariate (C1 - C4) in the treatment sample and control population was

within the range of 0.65 standard deviations. The range of 0.65 was

determined from the pilot study because it was able to consistently

generate a control sample of n=200. For the mixed covariates two discrete

covariates (D1, D2), and two continuous covariates (C1, C2) - Case 3,

discrete covariates (D1, D2) were matched using criteria described for

Case 1 and for continuous covariates (C1, C2) matching was conducted

using criteria described for Case 2.

3. Propensity Score Matching: Prior to conducting propensity score

matching, propensity scores (PS) were computed for the treatment sample

and control populations using covariate cases 1-3. The logit model

(logistic regression approach) was used to estimate the propensity scores

p using the formula: e(xi) = log( ), where e(xi) is estimated propensity 1 p

score given the covariates as previously described, p is the predicted

probability of each covariate. Following Rubin (2001) the estimated

propensity scores were rescaled using the logit transformation formula:

PS PSlogit = log( ) to simplify the interpretation of the estimated 1 PS

propensity scores. In the estimation of propensity scores, group

membership Z was the dependent variable, and DI, D2, D3, D4 (Case 1);

C1, C2, C3, C4 (Case 2); and DI, D2, C1, C2 (Case 3) were the covariate

variables. Data were sorted by id number in ascending order; after data

sorting, each subject i in the treatment sample was matched with each

subject j in the control population if p(Xi) = ( Xj) = p, where X is a vector

of observed covariates and p is the propensity score. Thus, each individual

subject i in the treatment sample was matched with each individual subject

j in the control population if each subject i and j had a propensity score 66

within 0.025 standard deviation of the logit of the propensity score j

(Rubin, 2001).

4. Nearest Neighbor Matching: This method enables a minimization of the

absolute difference between estimated propensity scores for the treatment

and control groups (Guo & Fraser, 2010; Caliendo, 2006; Perraillion,

2007, 2007; Caliendo & Kopeinin, 2008). If Pi and Pj respectively are

propensity scores for individual subjects in the treatment and control

groups, I1 is the set of individual subjects in the treatment group sample,

and I2 is the set of subjects in the control population then a neighborhood

C(Pi) contains a control subject j (that is, jϵI2) as a match for treatment

sample subjects in the treatment group sample i (that is, iϵI1), if the

absolute difference of the propensity scores is the smallest among all

possible pairs of propensity scores i and j. Thus, C(Pi) = min || Pi - Pj ||, j

jϵI2. Under nearest neighbor matching, once a subject j was found to

match a subject i, individual subject j was removed from I2 without

replacement given the size of the control population (N=10000). Subjects

from the control population were chosen as matching partners for the

treatment sample that were closest to their propensity scores. There are

two variants of nearest neighbor matching: matching with replacement and

matching without replacement (Guo & Fraser, 2010; Caliendo &

Kopeinin, 2008; Parsons, 2001). In matching with replacement, control

population subjects can be used more than once as matches, whereas in

matching without replacement control population subjects can only be

considered once as matches (Guo & Fraser, 2010; Caliendo & Kopeinin,

2008). The advantage of matching with replacement is that it allows the

average quality of matching to increase and decreases bias (Caliendo &

Kopeinin, 2008; Dehejia &Wahba, 2002). In this study, matching without

replacement was used.

5. Radius Matching: In radius matching, every subject is the treatment

sample is matched with a corresponding subject in the control population

that is within a predefined interval or neighborhood of the treatment

sample subject’s propensity scores (Guo & Fraser, 2010; Perraillion, 2007,

2007; Baser, 2006; Dehejia & Wahba, 2002). In this study, prior to

matching, the treatment and control population subjects were randomly

ordered in ascending order. Then every subject i, in the treatment sample

was matched with a corresponding subject j in the control population if the

absolute difference of the propensity scores between i and j fell within a

radius of ± 0.007 standard deviations. The range of 0.007 was picked

because it was able to generate a control sample of n =200 as per the pilot

study. In the matching process, both i and j were removed from subjects

matching pool once they were successfully matched. The advantage of

radius matching is that it only uses the number of control subjects

available within the predefined radius thus allowing for use of extra

subjects when good matches are available and fewer cases when they are

not (Guo & Fraser, 2010; Baser, 2006; Caliendo & Kopeinin, 2008;

Dehejia & Wahba, 2002).

6. Mahalanobis Metric Matching including Propensity Scores: In this

method, subjects were first ordered and then the distance between the

treatment sample and control population subject was calculated. The

Mahalanobis metric distance (score distance) was computed as:

d(i, j) = (u v)T C 1 (u v) where, d(i, j) is the distance between

treatment sample subject i and a control population subject j defined by

the Mahalanobis distance, u and v are the matching covariate values for

each subject i in treatment sample and each subject j in control population

including the propensity score, C-1 is the sample covariance matrix of the

matching covariates from sets of treatment sample control population

subjects. After the Mahalanobis metric distance was computed, the control

population subject j with minimum or smallest Mahalanobis metric

distance d(i, j) based on the propensity score was chosen as the match for

the treatment sample i, and both subjects were removed from the pool.

This process was repeated until matches were found for all treatment

sample subjects.

Monte Carlo Data Generation

Data were generated to stimulate the kind of data found in the fields of education and social sciences to compare the ability of six different methods of forming comparison group samples in a quasi-experimental study design. SAS 9.12 (SAS Institute, 2008)

software was used to generate computer algorithms to stimulate raw data, match datasets, and draw comparison samples.

A treatment sample of n = 200 and a control population of size N = 10,000 subjects respectively were generated via Monte Carlo stimulation. From the control population, random samples of sizes of n =200 were drawn by matching to the treatment sample via the six methods of forming comparison (control) group samples. The obtained matched control samples were then merged to the original treatment sample to form a dataset of n = 400 obtained by each method of forming comparison group sample. So there were six sets of 400 observations, each which were analyzed as a single experiment.

1000 such replications were stimulated for each of the cells in each of the three different covariate structure case models.

Following Fan, Felsovalyi, Sivo & Keenan (2001), treatment sample and two control populations were constructed from a Σ variance/covariance matrix comprising of the following attributes.

C1 – C4: Normal ~ N(0, 1)

D1, D2: Binomial 2 levels ~ {π=0.5; π =0.5}

D3: Multinomial 3 levels ~ {p1=0.33; p2=0.33; p3=0.34}

D4: Multinomial 4 levels ~ {p1=0.70; p2=0.21; p3=0.05; p4=0.04}

Covariate Collinearity among Covariates: {0.00, 0.40, 0.90}

Covariate Correlation between bias amount and covariate group among

continuous covariates in the Control Population: {0.00, 0.40, 0.75, .99}

Effect Size (ES): Normal ~ N(0, 1) - 3 levels ~ {π=0.00; π=0.10; π=0.45}

Bias (B): Normal ~ N(0, 1) - 3 levels ~ {π=0.00; π=0.10; π= 0.15}

Group variable: GRP (Treatment & Control) ~ Uniform 2 levels {π=0.5}

Dependent variable: Y = Composite of ES + B

Datasets for covariates C1- C4, ES and B in both treatment sample and control populations were generated using SAS RANNOR function from a normal distribution with a mean of 0 and standard deviation of 1. Covariates C1- C4 had three levels of collinearity (0.00, 0.40, 0.90) among them. Four levels (0.00, 0.40, 0.75, 0.90) of covariate correlation with bias were added in the control population to simulate selection/sampling bias common in non/quasi-experimental designs. Discrete covariates

D1 and D2 were generated using SAS RANBIN function from a binomial distribution with mean np and variance of np(1-p). The data simulation proportions (π) of D1 and D2 common in the social sciences. Covariates D1 and D2 were measured at two levels.

Covariates D3 and D4 were generated using SAS RANTBL function from the multinomial distribution defined by probability proportions p1 through p4. No amount of collinearity or covariate correlation with bias was added in the discrete covariates because these covariates can be measured and stimulated with almost no measurement error. Variable Group (grp) was coded as 1 = Treatment, 2 = Control in treatment sample and control populations datasets respectively. Effect sizes (ES) of 0.00 (no effect), 0.10

(small) and 0.45 (medium) were used to evaluate how these effect sizes were affected or influenced data matching processes. The dependent variable (Y) was generated as composite variable of effect size plus bias. In the raw data simulation processes, effect sizes of 0.00, 0.10, & 0.45, and selection bias amounts of 0.00, 0.10, & 0.15 were added to the treatment sample datasets but there were no amounts of effect size or bias were

added in the control population datasets. Thus the treatment sample datasets included bias and effect size and the control population datasets did not.

Data Analysis and Presentation of Results

Let Xi denote a vector of observed matching baseline covariates (D1, D2, D3, D4,

C1, C2, C3, C4) for each subject i, and E represent error. Let Yti denote outcome if subject i is in the treatment group. Let Yci represent outcome if subject i is in the control population. Let group membership Z=1 if subject i is in the treatment group, and Z=2 if subject i in the control population. With (Yti, Yci) following equation 2.1 the treatment effect (τi) for subject i can be defined as the difference between Yti and Yci. Thus,

τi = Yti - Yci (3.1)

Extending the application of equations 2.1 and 2.3 to sample data; let yt denote a

treatment sample, let yc denote a control sample drawn from control population Yc, let z represent a sample variable denoting Z. The average treatment effect ( ) for population sample data can be estimated as:

= E( yt |z = 1) - E( yc |z= 2) (3.2)

Where, yt is the estimated treatment sample mean, and is the estimated control sample mean. Further, let denote the treatment group sample condition on the

matching system baseline covariates such that Yt│matching system covariance of interest is covariance (effect size (ES), bias (B), matching system covariates). Thus,

yt = (X + E) + ES + B (3.3)

Let yc represent a comparison group sample drawn from the control population Yc using different methods of forming a comparison sample (random sampling, exact matching, propensity score matching, nearest neighbor matching, radius matching, and Mahalanobis metric matching) condition on the matching system baseline covariates such that

Yc│matching system covariance of interest is covariance (matching system covariates).

Thus,

= (X + E) (3.4)

Let YD denote mean true treatment effect & bias estimated from the treatment sample (

yt ), and comparison (control) samples ( yc ). Thus,

= - (3.5) substituting and expanding equation 3.5

= (X+E) + ES + B - (X+E)

= (X-X+E-E) + ES + B

= ES+ B (3.6)

^ ES = - B (3.7)

^ B = - (3.8)

^ URES=ES - ES (3.9)

^ UBR=B- B (3.10)

Where, B = programmed (known) bias, ES = programmed (known) effect size,

YD = is the estimated true mean treatment effect plus bias, = the estimated bias, URES

= Unrecovered Effect Size and UBR = Unrecovered Bias. Equation 3.6 is equivalent or the same as equations 2.17 and 2.19 in terms of expressing the decomposition of true

treatment effect and bias estimated from y1 and y2 via exact matching and propensity score matching methods.

The statistics for the dependent variables ( , , URES, , URB) were computed and then used to compare the results of the matching method’s ability to estimate and recover the known effect size and bias. Below is a description of each of the dependent variables.

Estimated Effect Size ( ). Effect size describes the magnitude of the mean

difference between the estimated treatment sample mean ( yt ) and the estimated control

sample mean ( yc ). The estimated effect size was computed as follows: = - B

^ where, ES = estimated effect size; = is the estimated true mean treatment effect + bias estimated from and ; B = programmed (known) bias. For the null hypothesis the unrecovered Effect Size (URES) = programmed effect size (ES) - . The direction of

^ the unrecovered effect sizes was interpreted as follows: If URES = ES - ES = 0, implies

recovery of bias was sufficient so YD was not underestimated or overestimated and if

URES = ES - > 0, implies the bias recovered was insufficient so was overestimated thus, the matching procedure increased type II error. If URES = ES - <

0, implies the bias recovered was overcompensated so was underestimated thus, the

^ matching procedure increased type I error. Bias ( B ) was estimated to describe the magnitude and direction of given and was estimated as = - ES where = the estimated bias and = is the estimated true mean treatment effect + bias estimated from

yt and yc and ES = programmed (known) effect size. For the null hypothesis the

Unrecovered Bias (URB) = programmed bias (B) - . The direction of bias was interpreted as follows: If UBR = B - = 0, implies effect size was sufficient so was not underestimated or overestimated thus, the matching procedure recovered all the bias.

If UBR = B - > 0, implies the effect size recovered was insufficient so was overestimated thus, the matching procedure failed to recover all the bias. If UBR= B - <

0, implies the effect size recovered was excess so was over compensated thus, the matching procedure over compensated for selection bias.

Independent samples t-test. The independent t-test statistics (t-values and their corresponding p-values (p = 0.05) were used to compare the treatment sample vs. the control group samples in the matching procedures. Thus, if p > 0.05, it was concluded that group samples were balanced, and also if p > 0.05, it was concluded there were no significant differences between treatment and control group samples given the experimental design and matching conditions. If p < 0.05, it was concluded that there were significant differences between treatment and control group samples thus the group samples were different given the experimental design and matching conditions.

Nominal Type I error (α). For pilot study, a nominal alpha was used to measure and test type I error rate at α = 0.05. If the count of significant t-values, p=.05 < 50 (there should be about 50 significant t-tests per 1000 runs) the type I error rate was interpreted as conservative, and if the number of p-values are > 50 the type I error rate was interpreted as inflated. Thus, a count was conducted on the significant t-values, p = 0.05:

HO; α = 0.05. If HO: α > 0.05 > 50, type I error was termed as inflated but if HO: α =.05, α

< 0.05 < 50, type I error was termed conservative.

MANOVA/Dunnett t-test/ANOVA. The multivariate analysis of variance

(MANOVA) was used to examine design conditions hypothesized to affect the matching procedures in bias recovery. The MANOVA Wilks’ Lambda ( ), Roy’s greatest characteristic root (GCR), and the univariate ANOVA F and p value statistics were used

to determine if the design factors affected the YD means estimated the six methods of forming comparison group/matching methods.

The Dunnett t-test was used to perform a pairwise comparison of the treatment sample verses the control sample means by each dependent variable to ascertain if treatment sample mean and control sample means were significantly different from each other. A mean of the dependent variable was determined statistically significant if the

Dunnett test LSMEANS p-values were less than 0.05 (p < 0.05), and not statistically significant if the Dunnett test LSMEANS p-values were great than 0.05 (p > 0.05). The

ANOVA F and p values were used to determine if there were differences in the dependent variables means. Thus, if F-values were greater than 1 (F > 1) and p-values were less than 0.05 (p < 0.05), it was concluded there were differences in the means of the dependent variables. If (F > 1) and p > 0.05 it was concluded that there no differences in the means of dependent variables.

95% Confidence Interval on Coverage Probability and Width. To investigate the prevalence of non-matches given the parameters of the experimental

design conditions, YD t-test means and the degrees of freedom (df) statistics were used to determine the prevalence of non-matches. Both the treatment and control samples had equal sample sizes nt = nc = n = 200. Thus, if the average degrees of freedom (df) for the

1000 replications by each matching method is 398 the matching algorithm was deemed

100% complete. When df are less than 398, one or more of the matching replicates was deemed incomplete. Following Collins & et al (2001) coverage probability was computed as the percentage or proportions of times the true values of ( ) were included in the

95% confidence interval per each 1000 replication. A coverage probability below 90%

was considered a problem thus an indication that the statistics:YD estimated via each matching method produced estimates and or standard errors that were excessive. The width of 95% CI was computed as the average length difference between the Lower

Bound CI and the Lower bound CI. If the 95% CI coverage probability was greater than

90% and the lengths of the widths of the 95% CI were short, it was concluded the statistical estimates produced by each matching methods were accurate.

Pilot Study

A pilot study was initiated f to ascertain the validity of the computer algorithms ability to generate the raw data to investigate six different methods of forming comparison groups based on known population characteristics. Descriptive statistics including univariate tests for normality statistics, bivariate correlations, frequency counts, covariate correlations with bias, t-test statistics, and the matching methods comparison statistics were examined. The null case conditions for Case 1 were: Bias = 0.00, effect size = 0.00. The null Case conditions for Cases 2 and 3 were: Bias = 0.00, effect size =

0.00, covariate collinearity = 0.00, covariate correlation with bias = 0.00, treatment sample (N = 200). Descriptive statistics pertaining to continuous covariates for the treatment sample and the control populations are presented in Table 4.

Table 4 The Null Case Descriptive Statistics for Continuous Covariates, N=200 and N=10,000 Treatment Sample Control Population Covariate Mean SD p-value Covariate Mean SD p-value C1 0.0806 0.9986 >.1500 C1 0.0078 1.0033 >.1500 C2 0.0111 1.0523 >.1500 C2 0.0072 0.9972 >.1500 C3 0.0679 0.9894 >.1500 C3 0.0015 0.9964 >.1500 C4 0.0790 0.9975 >.1500 C4 0.0009 1.0075 >.1500 B 0.0549 1.0023 >.1500 B 0.0012 0.9962 >.1500 ES 0.0004 1.0011 >.1500 ES 0.0070 0.9915 >.1500 Y 0.0553 1.3498 >.1500 Y 0.0082 1.4004 >.1500

The univariate tests for normality were met, all p’s>.05. The bivariate correlations for the treatment sample and the control populations were correctly generated as programed, see Tables 5 and 6. The frequency counts for discrete covariates (D1-D4) were successfully generated as per the programmed categories and probability proportions, see Table 7. Covariates D1 and D2 were successfully generated following the binomial distribution in the proportions of: D1, π = 0.50; D2, π = 0.60. Covariates D3 and D4 were successfully generated following the multinomial distribution with covariate

D3 having probability proportions of: p1 = 0.33, p2 = 0.33, & p3 = 0.34; and covariate D4 having probability proportions of: p1 = 0.04, p2 = 0.05, p3 = 0.21, & p4 = .70.

Table 5 Treatment Sample (N=200): The Null Case Bivariate Correlations Among the Continuous Covariates Covariate C1 C2 C3 C4 B ES C1 1.0000 -0.0028 -0.0883 0.0042 0.0691 -0.0492 C2 0.0028 1.0000 0.0569 -0.0920 -0.0740 0.0044 C3 -0.0883 0.0569 1.0000 -0.0131 -0.0127 -0.0354 C4 0.0042 -0.0920 0.0131 1.0000 0.0671 -0.0241 B 0.0691 -0.0740 -0.0127 0.0671 1.0000 0.0429 ES -0.0492 0.0044 -0.0354 -0.0241 0.0429 1.0000

Table 6 Control Population: The Null Case Bivariate Correlations for Continuous Covariates Covariate C1 C2 C3 C4 B ES C1 1.0000 -0.0074 0.0022 0.0012 0.0098 -0.0011 C2 -0.0074 1.0000 0.0071 -0.0244 0.0074 0.0099 C3 0.0022 0.0071 1.0000 -0.0042 0.0027 -0.0012 C4 0.0012 -0.0244 -0.0042 1.0000 0.0098 0.0014 B 0.0098 0.0074 0.0027 0.0098 1.0000 0.0010 ES -0.0011 0.0099 -0.0012 0.0014 0.0010 1.0000

Table 7 Treatment Sample and Control Population: The Null Case Frequency Counts for Discrete Covariates N=200 & N=10,000 Treatment Sample Control Population Covariate Group Percent Frequency Percent Frequency D1 0 50.50% 101 49.70% 4970 1 49.50% 99 50.30% 5030 Total 100.00% 200 100.00% 10,000 D2 0 63.50% 127 59.83% 5983 1 36.50% 73 40.17% 4017 Total 100.00% 200 100.00% 10,000 D3 1 35.50% 71 32.74% 3274 2 33.00% 66 33.05% 3305 3 31.50% 63 34.21% 3421 Total 100.00% 200 100.00% 10,000 D4 1 4.00% 4 4.02% 402 2 12.00% 5 4.99% 499 3 21.50% 50 20.70% 2070 4 70.50% 141 70.29% 7029 Total 100.00% 200 100.00% 10,000

Bias was regressed on covariate variables C1 to C4 in the Control Population to verify the a-priori design correlation between the covariates and bias amount. Multiple regression results indicated that the a-priori R2 was within 0.0000 - 0.0001 of the pre- programmed value of 0.00 after 1000 replications, see Table 8.

Table 8 Control Population: Multiple Regression Analysis for Continuous Covariates: Collinearity =0.00, Correlation with Bias = 0.00, Bias = 0.00 and ES = 0.00 Source df Sum of Squares Mean Square F p Model 4 7.0435 1.7609 .76 0.1330 Error 9995 9976.6114 0.9982 Corrected Total 9999 9983.6549 0.0000 Root MSE 0.9991 R-Square 0.0000 Dependent Mean 0.0138 Adj R-Square 0.0001

An examination of 1000 replications of independent samples t-test analysis for the three models performed to test for the matching methods algorithms replication ability for each model indicated the matching methods algorithms were successfully, see Tables 9 to

11.

Table 9 The Null Case: t-tests Statistics for Discrete Covariates by Matching Method when there is Collinearity among Covariates = 0.00, Correlation with Bias = 0.00, Bias = 0.00 and Effect Size =0.00 Matching Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0058 0.14 98 0.0419 0.4845 -0.2786 0.2774 95.20 0.5560 EM 0.0024 0.26 98 0.0149 0.3139 -0.2684 0.2787 88.60 0.5471 PSM 0.0060 0.14 98 0.0418 0.4949 -0.2714 0.2835 95.20 0.5550 NNM 0.0036 0.14 98 0.0234 0.4943 -0.2721 0.2836 95.50 0.5557 RM 0.0027 0.14 98 0.0160 0.5051 -0.2759 0.2794 95.40 0.5553 MM 0.0037 0.14 95 0.0237 0.4957 -0.2793 0.2869 95.60 0.5579

Note: Y = LCLM = Lower CL Mean, UCLM = Upper CL Mean, Coverage = 95% CI probability D YD , of coverage

Table 10 The Null Case: t-tests Statistics for the average of 1000 Replications for Continuous Covariates by Matching Method when Collinearity among Covariates =0.00, Correlation with Bias =0.00, Bias = 0.00 and Effect Size =0.00 Matching Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0018 0.15 398 0.0814 0.4916 -0.2890 0.2661 93.60 0.5551 EM 0.0012 0.20 398 0.0547 0.4061 -0.2848 0.2691 86.20 0.5539 PSM 0.0083 0.14 398 0.0295 0.4998 -0.2816 0.2731 95.00 0.5546 NNM 0.0023 0.14 398 0.0701 0.5071 -0.2878 0.2679 94.80 0.5558 RM 0.0015 0.15 398 0.0440 0.4922 -0.2842 0.2718 95.20 0.5560 MM 0.0024 0.14 395 0.0667 0.5085 -0.2884 0.2695 94.70 0.5580

Note: Y = LCLM=Lower CL Mean, UCLM=Upper CL Mean, Coverage = 95% CI probability D YD , of coverage

Table 11 The Null Case: t-tests Statistics for Mixed Covariates by Matching Method when Collinearity among Covariates =0.00, Correlation with Bias =0.00, Bias =0.00 and Effect Size =0.00 Matching Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0006 0.14 398 0.0211 0.5235 -0.2809 0.2749 95.70 0.5557 EM 0.0023 0.27 398 0.0887 0.3302 -0.2733 0.2774 85.20 0.5510 PSM 0.0014 0.13 398 0.0994 0.2857 -0.2784 0.2771 94.90 0.5555 NNM 0.0008 0.13 398 0.0999 0.3002 -0.2785 0.2765 95.00 0.5550 RM 0.0011 0.14 398 0.0910 0.2887 -0.2818 0.2740 94.80 0.5558 MM 0.0013 0.13 395 0.0181 0.4920 -0.2797 0.2775 95.40 0.5572

Note: Y = LCLM=Lower CL Mean, UCLM=Upper CL Mean, Coverage = 95% CI probability D , of coverage

For the null cases, t-tests were not expected to be statistically significant and degrees of freedom were expected to be 398.

The 1000 replications computation of the comparison statistics to the dependent

^ ^ variables ( , ES , URES, B and URB) were successfully simulated, see Tables 12 to

14. As expected the ANOVA results of all the dependent variables were not statistically significant, p-values were greater than .05. The Dunnett tests indicated there were no

mean differences between the treatment sample mean and each of the control sample mean obtained via each matching method of forming a comparison sample.

Table 12 The Null Case: Comparison Statistics for Discrete Covariates by Matching Method when Collinearity among Discrete Covariates = 0.00, Correlation with Bias = 0.00, Bias = 0.00 and Effect Size =0.00 Matching Statistic Mean Dunnette LSMEAN Pairwise Comparison p-value Method of Treatment vs. Control Sample NOM Mean(YD*) 0.0058 Mean 0.0024 1.0000 Est_ES 0.0058 Est_ES 0.0024 1.0000 URES 0.0058 Est_Bias 0.0024 1.0000 Est_Bias 0.0058 URES & URB 0.0024 1.0000 URB 0.0058 Treatment 0.0018 EM Mean(YD*) 0.0024 Mean 0.0032 0.9697 Est_ES 0.0024 Est_ES 0.0032 0.9697 URES 0.0024 Est_Bias 0.0032 0.9697 Est_Bias 0.0024 URES & URB 0.0032 0.9697 URB 0.0024 Treatment 0.0018 PSM Mean(YD*) 0.0060 Mean 0.0040 0.9382 Est_ES 0.0060 Est_ES 0.0040 0.9382 URES 0.0060 Est_Bias 0.0040 0.9382 Est_Bias 0.0060 URES & URB 0.0040 0.9382 URB 0.0060 Treatment 0.0018

NNM Mean(YD*) 0.0036 Mean 0.0039 0.9499 Est_ES 0.0036 Est_ES 0.0039 0.9499 URES 0.0036 Est_Bias 0.0039 0.9499 Est_Bias 0.0036 URES & URB 0.0039 0.9499 URB 0.0036 Treatment 0.0018 RM Mean(YD*) 0.0027 Mean 0.0001 0.9999 Est_ES 0.0027 Est_ES 0.0001 0.9999 URES 0.0027 Est_Bias 0.0001 0.9999 Est_Bias 0.0027 URES & URB 0.0001 0.9999 URB 0.0027 Treatment 0.0018 MM Mean(YD*) 0.0037 Mean 0.0046 0.9151 Est_ES 0.0037 Est_ES 0.0046 0.9151 URES 0.0037 Est_Bias 0.0046 0.9151 Est_Bias 0.0037 URES & URB 0.0046 0.9151 URB 0.0037 Treatment 0.0018

Note: Y = D YD

Table 13 The Null Case: Comparison Statistics for Continuous Covariates by Matching Method when Collinearity among Continuous Covariates = 0.00, Correlation with Bias = 0.00, Bias = 0.00 and Effect Size = 0.00 Matching Statistic Mean Dunnette LSMEAN Pairwise Comparison of p-value Method Treatment vs. Control Sample NOM Mean(YD*) 0.0018 Mean 0.0021 0.1047 Est_ES 0.0018 Est_ES 0.0021 0.1047 URES 0.0018 Est_Bias 0.0021 0.1047 Est_Bias 0.0018 URES & URB 0.0021 0.1047 URB 0.0018 Treatment 0.0025 EM Mean(YD*) 0.0012 Mean 0.0016 0.4367 Est_ES 0.0012 Est_ES 0.0016 0.4367 URES 0.0012 Est_Bias 0.0016 0.4367 Est_Bias 0.0012 URES & URB 0.0016 0.4367 URB 0.0012 Treatment 0.0025 PSM Mean(YD*) 0.0083 Mean 0.0072 0.9136 Est_ES 0.0083 Est_ES 0.0072 0.9136 URES 0.0083 Est_Bias 0.0072 0.9136 Est_Bias 0.0083 URES & URB 0.0072 0.9136 URB 0.0083 Treatment 0.0025

NNM Mean(YD*) 0.0023 Mean 0.0028 0.2035 Est_ES 0.0023 Est_ES 0.0028 0.2035 URES 0.0023 Est_Bias 0.0028 0.2035 Est_Bias 0.0023 URES & URB 0.0078 0.2035 URB 0.0023 Treatment 0.0025 RM Mean(YD*) 0.0015 Mean 0.0019 0.6587 Est_ES 0.0015 Est_ES 0.0019 0.6587 URES 0.0015 Est_Bias 0.001 0.6587 Est_Bias 0.0015 URES & URB 0.0019 0.6587 URB 0.0015 Treatment 0.0025 MM Mean(YD*) 0.0024 Mean 0.0027 0.2444 Est_ES 0.0024 Est_ES 0.0027 0.2444 URES 0.0024 Est_Bias 0.0027 0.2444 Est_Bias 0.0024 URES & URB 0.0027 0.2444 URB 0.0024 Treatment 0.0025

Note: Y = D YD

Table 14 The Null Case: Comparison Statistics for Mixed Covariates by Matching Method when Collinearity among Mixed Covariates = 0.00, Correlation with Bias = 0.00, Bias = 0.00 and Effect Size = 0.00 Matching Statistic Mean Dunnette LSMEAN Pairwise Comparison p-value Method of Treatment vs. Control Sample NOM Mean(YD*) 0.0006 Mean 0.0015 0.9904 Est_ES 0.0006 Est_ES 0.0015 0.9904 URES 0.0006 Est_Bias 0.0015 0.9904 Est_Bias 0.0006 URES & URB 0.0015 0.9904 URB 0.0006 Treatment 0.0014 EM Mean(YD*) 0.0023 Mean 0.0023 0.9978 Est_ES 0.0023 Est_ES 0.0023 0.9978 URES 0.0023 Est_Bias 0.0023 0.9978 Est_Bias 0.0023 URES & URB 0.0023 0.9978 URB 0.0023 Treatment 0.0014 PSM Mean(YD*) 0.0014 Mean 0.0014 1.0000 Est_ES 0.0024 Est_ES 0.0014 1.0000 URES 0.0014 Est_Bias 0.0014 1.0000 Est_Bias 0.0014 URES & URB 0.0014 1.0000 URB 0.0014 Treatment 0.0014

NNM Mean(YD*) 0.0008 Mean 0.0008 1.0000 Est_ES 0.0008 Est_ES 0.0008 1.0000 URES 0.0008 Est_Bias 0.0008 1.0000 Est_Bias 0.0008 URES & URB 0.0008 1.0000 URB 0.0008 Treatment 0.0014 RM Mean(YD*) 0.0011 Mean 0.0011 0.9593 Est_ES 0.0011 Est_ES 0.0011 0.9593 URES 0.0011 Est_Bias 0.0011 0.9593 Est_Bias 0.0011 URES & URB 0.0011 0.9593 URB 0.0011 Treatment 0.0014 MM Mean(YD*) 0.0013 Mean 0.0013 1.0000 Est_ES 0.0013 Est_ES 0.0013 1.0000 URES 0.0013 Est_Bias 0.0013 1.0000 Est_Bias 0.0012 URES & URB 0.0013 1.0000 URB 0.0013 Treatment 0.0014

Note: YD=YD

The nominal type I error rate as a percentage of statistically significant t-tests with p- values less than .05 (p<.05) per 1000 replications indicated that the t-tests were correctly simulated within the acceptable range of 5% by chance except for exact matching, see

Table 15.

Table 15

Table 15 The Null Case: Nominal Type I Error Rate as Percentage of Statistically Significant t-tests (p < 0.05) by Covariate Type and Matching Method Covariate Type Matching Method % Discrete NOM 5 EM 15 PSM 5 NNM 5 RM 5 MM 5 Continuous NOM 5 EM 10 PSM 5 NNM 5 RM 5 MM 5 Mixed NOM 5 EM 9 PSM 5 NNM 5 RM 5 MM 5

All the cases models programmed biases and effect sizes per 1000 replications were correctly estimated within the ranges of 0.00, 0.10, 0.15 and 0.00, 0.10, 0.44/0.45 respectively, see figures 1 to 6 below.

It was concluded that the results of the pilot study ascertained the validity of the computer algorithms ability to generate the necessary data to conduct investigative studies on different methods of forming comparison group samples under different experimental conditions and design features.

0.18 0.17 0.16 0.15 0.14 0.13 0.12 0.11 0.10 0.09 0.08

Estimated Bias Estimated 0.07 0.06 0.05 0.04 0.03 0.02 0.01 0.00 NOM EM PSM NNM RM MM NOM EM PSM NNM RM MM NOM EM PSM NNM RM MM

0.00 0.10 0.15 bias

Figure 1. Case 1 Estimated Mean Bias for Average of 1000 Replications by Matching Method

0.48 0.46 0.44 0.42 0.40 0.38 0.36 0.34 0.32 0.30 0.28 0.26 0.24 0.22 0.20

Estimated ES Estimated 0.18 0.16 0.14 0.12 0.10 0.08 0.06 0.04 0.02 0.00 -0.02 NOM EM PSM NNM RM MM NOM EM PSM NNM RM MM NOM EM PSM NNM RM MM NOM EM PSM NNM RM MM

0.00 0.10 0.44 0.45 es

Figure 2. Case 1 Estimated Mean ES for Average of 1000 Replications by Matching Method

0.16

0.15

0.14

0.13

0.12

0.11

0.10

0.09

0.08

0.07

0.06 Estimated Bias Estimated

0.05

0.04

0.03

0.02

0.01

0.00

-0.01 NOM EM PSM NNM RM MM NOM EM PSM NNM RM MM NOM EM PSM NNM RM MM

0.00 0.10 0.15 bias

Figure 3. Case 2 Estimated Mean Bias for Average of 1000 Replications by Matching Method

0.46 0.44 0.42 0.40 0.38 0.36 0.34 0.32 0.30 0.28 0.26 0.24 0.22 0.20 0.18 0.16

EstimatedSizeEffect 0.14 0.12 0.10 0.08 0.06 0.04 0.02 0.00 -0.02 NOM EM PSM NNM RM MM NOM EM PSM NNM RM MM NOM EM PSM NNM RM MM NOM EM PSM NNM RM MM

0.00 0.10 0.44 0.45 es

Figure 4. Case 2 Estimated Mean ES for Average of 1000 Replications by Matching Method

0.46 0.44 0.42 0.40 0.38 0.36 0.34 0.32 0.30 0.28 0.26 0.24 0.22 0.20 0.18

Estimated Effect SizeEstimated 0.16 0.14 0.12 0.10 0.08 0.06 0.04 0.02 0.00 NOM EM PSM NNM RM MM NOM EM PSM NNM RM MM NOM EM PSM NNM RM MM NOM EM PSM NNM RM MM

0.00 0.10 0.44 0.45 es

Figure 5. Case 3 Estimated Mean Bias for Average of 1000 Replications by Matching Method

0.16

0.15

0.14

0.13

0.12

0.11

0.10

0.09

0.08

0.07

0.06 EstimatedBias 0.05

0.04

0.03

0.02

0.01

0.00

-0.01 NOM) EM PSM NNM RM MM NOM) EM PSM NNM RM MM NOM) EM PSM NNM RM MM

0.00 0.10 0.15 bias

Figure 6. Case 3 Estimated Mean ES for Average of 1000 Replications by Matching Method

Research Questions Analytics

The following dependent variables and their statistics were used to answer the research questions.

RQ1: Mean Effect analysis (YD ):

. Yt = (X+E) + ES + B

. Yc = (X+E)

. = (X+E) + ES + B - (X+E) = ES + B

. MANOVA Statistics: F-Values, P-Values, Wilks’s and Roy’s GCR

. ANOVA Statistics: F-Values, and P-Values

RQ2: Recovered Bias (RB), Unrecovered Bias (URB):

^ . Estimated Effect Size ( ES )= - ( - B)

^ . Unrecovered Bias (URB) = B - B

. Percentage (%) of Recovered Bias (RB) = (1- URB)*100

RQ3: t-test, Probability Coverage, Width and Frequency of Matching Failure

. Independent samples t-test:

(Y Y ) (X X ) t T C T C 1 1 S 2 p( ) nT nC

2 2 2 (nT 1)S T (nC 1)S C Where, S p nT nC 2

2 S p = pooled variance

X T = mean of the treatment sample

YT = mean of the sample obtained from the treatment group sample

2 S T = variance of sample obtained from the treatment group

nT = size of the sample taken from treatment sample

X C = mean of the control sample formed by via each method of forming

comparison group sample

YC = mean of the sample obtained from the control sample via each method of

forming a comparison group

2 S C = variance of the sample obtained from control sample

nC = size of the sample taken from the control sample via each method of forming

a comparison group

. (1-α) 95% Confidence Interval about YD

2 1 1 2 1 1 CI = (Y -Y ) - t α/2 S p ( )  (X X )  (Y -Y ) + t α/2 S p ( ) T C n n T C T C n n T C T C

. Coverage probability of = % age/proportion of 95% CI where the

includes true YD

. Width of = 95% CI distance of between Upper CL YT and Lower

CL YC

. f =the number of matching failures determined by average degrees of

freedom (df) 91

. Dunnett t-test:

M M Dt = 1 2 2MSE N h

Where, Dt =Dunnett’s Test

M 1 = mean of treatment sample

M 2 = mean of control group sample obtained via each method of

forming a comparison group sample

MSE = Mean square error as computed from ANOVA

N h =Harmonic Mean of treatment and control group sample

. Dunnett ANOVA: F-Values & P-Values

Summary

This chapter has discussed the research design and methodology that was used to study the ability of six different methods of forming comparison groups in a quasi experimental study design under different experimental and design conditions. It has also discussed the modalities for data generation, data analysis and presentation of results. The pilot study results successfully demonstrated the computer algorithms validity and ability to simulate the necessary data for the study. An example of the SAS algorithm used to undertake the Monte Carlo simulation is available from the author upon request.

CHAPTER IV

RESULTS

This chapter presents the results of the data analysis for this study. The study examined if the construction of a comparison group via six different methods of forming comparison groups’ results in differential conclusions related to testing for mean differences between two groups under different experimental conditions and design features. The research questions (RQs) were: (1) what design conditions affect the matching procedures in bias recovery, (2) do the matching procedures recover the selection/sampling bias and (3) given the parameters of the experimental design conditions, what is the prevalence for non-matches? The study experimental conditions were:

. Matching Covariates/Variables Vector:

Type of covariate (measurement scale): (Discrete (D),

Continuous(C), and Number of covariates {4 discrete covariates,

4 continuous covariates,

2 discrete & 2 continuous covariates}.

Covariate Collinearity/Covariance among the covariates: {0.00,

0.40, 0.90}.

Covariate Correlation between the bias amount and the covariate

group in the Control Population: {0.00, 0.40, 0.75, 0.99}.

. Manipulatable External Design Environments:

Amount of Selection Bias: {0.00, 0.10, 0.15}.

True treatment Effect Size: {0.00, 0.10, 0.44/0.45}

. Matching Methods: {NOM, EM, PSM, NNM, RM, and MM}.

The study involved three types of mixed factorial design scenarios for case matching. Case 1: 4 Discrete Covariates (D1, D2, D3, D4); Case 2: 4 Continuous

Covariates (C1, C2, C3, C4), and Case 3: 4 Mixed Covariates (D1, D2, C1, C2). Case1 was designed as a 3-between, 1-within mixed factorial. Cases 2 and 3 were designed as 4- between, 1-within mixed factorial. The independent variables for each case were: bias amount (Bias), effect size (ES), covariance (collinearity) among covariates (CV), and correlation between the bias amount and the covariates group (CR) with levels previously described. In Case 1, CV was not included since the construction of the D1 – D4 vector was orthogonal.

A multivariate analysis of variance (MANOVA) was conducted to compare the

statistical estimates of YD among each matching method for each design case, univariate

ANOVA was used for post hoc examination and all tests were conducted at a type I error rate = 0.05. The specifics of the Monte Carlo simulation process required defining 27 separate covariance structures for Case 1, and 108 separate covariance structures for each of Cases 2 and 3 one for each between subjects design cell. For each of the three cases scenarios, 1000 replicate samples were simulated. In general each design cell defined by between group factors required one simulated population covariance matrix. Given this population covariance matrix, one treatment sample and six matched comparison samples were drawn per replication.

RQ 1: What design conditions affect the matching procedures in bias recovery? As previously presented in Chapter III, RQ1 focuses on the observed mean difference among matching methods (NOM, EM, PSM, NNM, RM, and MM) between

the treatment sample and the matched control samples, and was estimated byYD which

reflects the difference between Yt - YC . To examine design conditions hypothesized to affect the matching procedures, a factorial multivariate analysis of variance (MANOVA) was conducted. In the overall analysis, two effect size conditions (0.44 and 0.45) were treated as the same as detailed in Chapter III. In the MANOVA, the six matching methods were the dependent variables.

Case 1: Discrete Covariates

Case 1 matching covariates were: D1, D2, D3, D4. To examine the effects of the experimental variables: bias {0, 0.10, 0.15}, effect size {0, 0.10, 0.44/45}, and correlation between the bias amount and covariate group {0, 0.75, 0.90, 0.99}, and among the six matching methods the overall MANOVA analysis was examined. Table 16 presents the pooled over all study conditions for the six matching methods; these means, centered about zero, are depicted in Figure 7. Case 1 descriptive statistics for all design factors by matching methods are presented in Appendix A. Examination of this figure suggests that there is variability among the matching methods in respect to .

Table 16

Case l: Descriptive Statistics for Matching Method YD Averaged Over 27000 Replications

Matching Method SD NOM 0.2661 0.2504 EM 0.2687 0.4023 PSM 0.2671 0.2485 NNM 0.2670 0.3984 RM 0.2660 0.2513 MM 0.2663 0.3313 Comparison of the pooled matching methods means to the horizontal line drawn represents the average effect size, pooled over all conditions and replications suggest that matching method means above the horizontal line overestimated while means under the horizontal line underestimated . From this figure, it appears that PSM and NNM in the discrete-only case might provide optimal matching which is further investigated below and there are some potential problems with EM.

Figure 7. Case 1: Pooled After Centering

MANOVA results for Case 1 are presented in Table 17. Two multivariate hypotheses (Roy’s greatest characteristic root, GCR) revealed statistically significant differences among matching methods as a function of the experimental conditions.

Traditionally in the social sciences, Wilk’s Lambda is commonly reported. However,

Roy’s GCR tests a different multivariate hypothesis from Wilk’s Lambda, notably if the first root is statistically significant, whereas, Wilk’s Lambda is a multiple root test. While often these two multivariate tests agree, they test different aspects of the multivariate space and thus they may diverge in conclusion (Harris, 2001). Specifically the GCR test focuses only on the first characteristic root’s magnitude and when statistically significant, indicates that there is at least one primary linear combination of the multivariate vector.

Table 17 Case 1: MANOVA Results Multivariate Hypothesis Wilk’s λ p-value Roy’s GCR p-value MatchMethod 0.99 0.7027 0.0001 0.7027 MatchMethod*Bias 0.99 0.0991 0.0004 0.0395 MatchMethod*ES 0.99 0.2499 0.0003 0.2029 MatchMethod*Covariance 0.99 0.3943 0.0004 0.0924 MatchMethod*Bias*ES 0.99 0.5197 0.0005 0.0301 MatchMethod*Bias*Covariance 0.99 0.5223 0.0003 0.1053 MatchMethod*ES*Covariance 0.99 0.9026 0.0003 0.1928 MatchMethod*Bias*ES*covariance 0.99 0.9443 0.0005 0.1301

Post hoc break down of the 3-way interaction followed the general plan as outlined in

Winer (1971). Blocking on Bias revealed a statistically significant 2-way interaction,

MM*ES, when Bias = 0.10 (F(10, 44955) = 2.41, p = 0.0073, Huynh-Feldt Lecoutre

(HFL) p = 0.0319) and a Matching Method main effect when Bias = 0.15 (F(5, 44955) =

3.54, p = 0.0033, HFL p = 0.0182); however, no statistically significant results were noted when Bias = 0.00. Further simple-simple effect analysis of the 2-way MM*ES interaction was initiated via pairwise comparisons among the six Matching Methods for

each Bias and ES combination. Statistically significant differences among the six

Matching Methods were only observed in the Bias = 0.10, ES = 0.00 (F(5, 14985) = 2.78, p = 0.0163, HFL p = 0.0478). This suggests that when ES=0.00 but in the presence of

some degree of Bias (e.g., = 0.10) the matching methods differed as measured by YD .

Further examination of the simple-simple effects after blocking on both ES and Bias, revealed that there was a statistically significant difference among the matching methods when ES = 0.0 and Bias = 0.10 such that calculated via the EM was significantly higher than all other methods, see Table 18.

Table 18 Case 1: Simple-simple Effect Least Square Means Among Matching Methods When Bias = 0.10 and ES = 0.00 Matching Method LS Mean Stderr NOMa 0.0993 0.0027 EMb 0.1141 0.0064 PSMa,c 0.1041 0.0027 NNMa,b,c 0.0955 0.0063 RMa,c 0.1020 0.0027 MMa,b,c 0.0981 0.0048 Note: LS Means with the same letter are not statistically different, all p’s < .05

The Matching Method main effect in the Bias = 0.15 condition was also investigated via pairwise comparisons and are presented in Table 19. As can be seen from this table, the

EM again resulted in higher values and a larger standard error than the other matching methods.

Table 19 Case 1: Matching Method Main effect Least Square Means When Bias = 0.15 Matching Method LS Mean Stderr NOMa 0.3330 0.0016 EMb 0.3443 0.0037 PSMa,c 0.3353 0.0015 NNMa,b,c 0.3374 0.0036 RMa,c 0.3325 0.0016 MMa,c 0.3358 0.0028 Note: LS Means with the same letter are not statistically different, all p’s < .05

Case 2: Continuous Covariates

Case 2 only examined continuous covariates. To overall MANOVA analysis explored the effects of the experimental variables: bias {0, 0.10, 0.15}, effect size {0,

0.10, 0.44/45}, covariance among covariates {0, 0.40, 0.90}, correlation between the bias amount and covariate group {0, 0.75, 0.90, 0.99}, among the six matching methods.

Similar to the presentation for Case 1, Table 20 presents the pooled grand means (YD ) for the six matching methods and Figure 8 depicts these same means centered about zero.

Case 2 descriptive statistics for all design factors by matching methods are presented in

Appendix B.

Table 20 Case 2: Descriptive Statistics for Average of 108000 Replications by Matching Method

Matching Method SD NOM 0.2637 0.2500 EM 0.2639 0.3507 PSM 0.2649 0.2474 NNM 0.2637 0.2483 RM 0.2641 0.2486 MM 0.2636 0.2482

Figure 8. Case 2: Pooled YD After Centering

Comparison of among the matching methods to the horizontal line suggests that in the continuous covariate case PSM deviates from the other matching methods. MANOVA of

Case 2 yielded several statistically significant findings as presented in Table 20. Similar to Case 1, here was deviation between the Wilk’s Lambda and the Roy’s GCR statistics and post hoc analysis of the highest order interaction was conducted for statistically significant GCR results. Moreover, post hoc interaction analysis of the three 4-way interactions precluded any breakdown analysis of statistically significant 3-way and lower order interactions.

Post hoc interaction analysis of the three 4-way interactions were ordered first by breaking down the MM*Bias*ES*Correlation and MM*Bias*ES*Covariance interactions since they share the same 3-way interaction MM*Bias*ES. Lastly the

MM*ES*Covariance*Correlation 4-way interaction was examined.

100

Table 21 Case 2: Primary MANOVA Results Multivariate Hypothesis Wilk’s λ p-value Roy’s GCR p-value MM 0.9998 0.0093 0.0001 0.0093 MM*Bias 0.9999 0.6501 0.0001 0.2621 MM*ES 0.9999 0.1087 0.0001 0.0151 MM*Covariance 0.9999 0.7071 0.0001 0.3152 MM*Correlation 0.9998 0.3058 0.0001 0.0518 MM*Bias*ES 0.9999 0.9694 0.0001 0.3287 MM*Bias*Covariance 0.9999 0.7636 0.0001 0.1747 MM*Bias*Correlation 0.9996 0.1112 0.0002 0.0024 MM*ES* Covariance 0.9997 0.1220 0.0002 0.0058 MM*ES*Correlation 0.9996 0.0428 0.0002 0.0051 MM*Covariance*Correlation 0.9997 0.2517 0.0002 0.0024 MM*Bias*ES*Correlation 0.9995 0.6888 0.0002 0.0387 MM*Bias*ES*Covariance 0.9996 0.3027 0.0002 0.0045 MM*Bias*Covariance*Correlation 0.9996 0.8786 0.0002 0.1059 MM*ES*Covariance*Correlation 0.9993 0.0754 <.0001 <0.0001 MM*Bias*ES*Covariance*Correlation 0.9985 0.8899 0.0004 0.1563

MM*Bias*ES*Correlation Interaction Post Hoc

Simple effect analysis focused on the 3-way interaction involving Matching

Method, Bias and ES while holding Correlation constant (see Table 22). No statistically significant 3-way interactions were noted and no effects were observed when Correlation was 0.00. However a statistically significant 2-way interaction was noted for Correlation

= 0.75, 0.90 and 0.99. There was MM*Bias at Correlation = 0.75, (F(10, 134865) = 2.82, p = 0.0017, HFL p = .0187). Breakdown for this interaction revealed a statistically significant Matching Method effect only under the Bias = 0.15 condition.

101

Table 22 LS Means for Bias, ES and Correlation by Matching Methods Bias ES Correlation NOM EM PSM NNM RM MM 0 0 0 -0.0115 -0.0079 -0.0042 -0.0099 -0.0062 -0.0095 0 0 0.75 0.0017 0.0038 0.0047 -0.0012 -0.0071 -0.0009 0 0 0.9 -0.0028 -0.0011 0.0004 0.0046 -0.0035 0.0042 0 0 0.99 0.0004 0.0012 0.0014 -0.0006 -0.0028 -0.0004 0 0 0 -0.0002 0.0003 0.0024 -0.0066 0.0025 -0.0066 0 0 0.75 -0.0091 -0.0038 -0.0029 -0.0079 -0.0089 -0.0073 0 0 0.9 -0.0056 -0.0046 0.0024 0.0000 -0.0013 0.0000 0 0 0.99 -0.0071 -0.0068 -0.0063 -0.0001 0.0035 -0.0001 0 0 0 0.0060 0.0086 0.0093 0.0000 -0.0008 0.0000 0 0 0.75 0.0035 0.0145 0.0050 0.0076 0.0056 0.0070 0 0 0.9 0.0038 0.0076 0.0069 0.0080 -0.0003 0.0077 0 0 0.99 0.0019 -0.0068 -0.0041 -0.0074 0.0022 -0.0074 0 0.1 0 0.1112 0.1017 0.1039 0.1148 0.1083 0.1147 0 0.1 0.75 0.1050 0.0996 0.1041 0.1046 0.1037 0.1050 0 0.1 0.9 0.1019 0.1045 0.1086 0.1081 0.1016 0.1085 0 0.1 0.99 0.0953 0.1040 0.1051 0.1002 0.1061 0.0995 0 0.1 0 0.0941 0.0918 0.1006 0.0966 0.1018 0.0963 0 0.1 0.75 0.1014 0.0977 0.0991 0.0989 0.1002 0.0988 0 0.1 0.9 0.0971 0.1012 0.1059 0.1006 0.1011 0.1007 0 0.1 0.99 0.0993 0.1036 0.1019 0.0987 0.1008 0.0989 0 0.1 0 0.1039 0.0992 0.0997 0.1017 0.1028 0.1021 0 0.1 0.75 0.1001 0.1092 0.1041 0.1035 0.0974 0.1037 0 0.1 0.9 0.0984 0.0957 0.1007 0.0992 0.0993 0.0989 0 0.1 0.99 0.0943 0.0933 0.1004 0.0992 0.0989 0.0986 0 0.45 0 0.4436 0.4435 0.4398 0.4342 0.4411 0.4340 0 0.45 0.75 0.4433 0.4495 0.4434 0.4422 0.4392 0.4425 0 0.45 0.9 0.4435 0.4332 0.4388 0.4406 0.4423 0.4408 0 0.45 0.99 0.4364 0.4362 0.4355 0.4384 0.4435 0.4385 0 0.45 0 0.4347 0.4312 0.4378 0.4351 0.4322 0.4351 0 0.45 0.75 0.4438 0.4297 0.4385 0.4373 0.4375 0.4362 0 0.45 0.9 0.4359 0.4435 0.4442 0.4323 0.4403 0.4322 0 0.45 0.99 0.4423 0.4379 0.4389 0.4328 0.4337 0.4322 0 0.45 0 0.4352 0.4437 0.4442 0.4350 0.4383 0.4354 0 0.45 0.75 0.4329 0.4333 0.4338 0.4368 0.4376 0.4369 0 0.45 0.9 0.4412 0.4159 0.4366 0.4416 0.4362 0.4418 0 0.45 0.99 0.4404 0.4267 0.4360 0.4440 0.4391 0.4440 0.1 0 0 0.1014 0.1011 0.1035 0.0978 0.0969 0.0978 0.1 0 0.75 0.1076 0.1040 0.1037 0.1059 0.1016 0.1060

102

Table 22 – Continued Bias ES Correlation NOM EM PSM NNM RM MM 0.1 0 0.9 0.0996 0.1073 0.1058 0.1026 0.1012 0.1018 0.1 0 0.99 0.0942 0.0991 0.0974 0.0938 0.0991 0.0936 0.1 0 0 0.0919 0.0997 0.0998 0.0982 0.0905 0.0984 0.1 0 0.75 0.0934 0.0988 0.0947 0.0987 0.0988 0.0990 0.1 0 0.9 0.0908 0.0995 0.0995 0.0969 0.1013 0.0969 0.1 0 0.99 0.1021 0.1073 0.1017 0.0994 0.0996 0.0990 0.1 0 0 0.1000 0.0820 0.0959 0.1035 0.0977 0.1033 0.1 0 0.75 0.0963 0.1124 0.1040 0.0988 0.0965 0.0988 0.1 0 0.9 0.0967 0.1227 0.1002 0.0947 0.1054 0.0946 0.1 0 0.99 0.1037 0.1181 0.1029 0.1037 0.1024 0.1034 0.1 0.1 0 0.2018 0.2090 0.2008 0.2008 0.2015 0.2008 0.1 0.1 0.75 0.1978 0.1982 0.1975 0.1978 0.2013 0.1978 0.1 0.1 0.9 0.2080 0.2093 0.2059 0.2003 0.2004 0.1995 0.1 0.1 0.99 0.0942 0.0991 0.0974 0.0938 0.0991 0.0936 0.1 0.1 0 0.2021 0.1966 0.2041 0.2000 0.2026 0.2000 0.1 0.1 0.75 0.1970 0.2048 0.1976 0.1986 0.1943 0.1989 0.1 0.1 0.9 0.1983 0.2023 0.1994 0.1937 0.1980 0.1934 0.1 0.1 0.99 0.1978 0.1960 0.1973 0.2017 0.2022 0.2013 0.1 0.1 0 0.2002 0.1878 0.1986 0.2035 0.1954 0.2031 0.1 0.1 0.75 0.1944 0.2150 0.1980 0.1966 0.1972 0.1973 0.1 0.1 0.9 0.2102 0.2060 0.2047 0.2087 0.2077 0.2086 0.1 0.1 0.99 0.1962 0.1952 0.1998 0.1991 0.2027 0.1996 0.1 0.5 0 0.5495 0.5378 0.5428 0.5489 0.5494 0.5489 0.1 0.5 0.75 0.5465 0.5460 0.5474 0.5451 0.5493 0.5451 0.1 0.45 0.9 0.5503 0.5469 0.5545 0.5549 0.5534 0.5542 0.1 0.45 0.99 0.5417 0.5452 0.5442 0.5467 0.5484 0.5474 0.1 0.45 0 0.5470 0.5504 0.5511 0.5464 0.5477 0.5457 0.1 0.45 0.75 0.5461 0.5366 0.5468 0.5482 0.5454 0.5478 0.1 0.45 0.9 0.5478 0.5647 0.5497 0.5493 0.5574 0.5493 0.1 0.45 0.99 0.5414 0.5412 0.5445 0.5506 0.5443 0.5500 0.1 0.45 0 0.5500 0.5515 0.5474 0.5478 0.5466 0.5477 0.1 0.45 0.75 0.5450 0.5585 0.5432 0.5413 0.5439 0.5412 0.1 0.45 0.9 0.5499 0.5389 0.5535 0.5487 0.5509 0.5489 0.1 0.45 0.99 0.5414 0.5416 0.5458 0.5422 0.5415 0.5426 0.15 0 0 0.1515 0.1588 0.1545 0.1515 0.1471 0.1517 0.15 0 0.75 0.1492 0.1464 0.1485 0.1518 0.1439 0.1520 0.15 0 0.9 0.1501 0.1498 0.1513 0.1452 0.1564 0.1451 0.15 0 0.99 0.1536 0.1459 0.1540 0.1457 0.1527 0.1457 0.15 0 0 0.1471 0.1660 0.1576 0.1476 0.1528 0.1475

103

Table 22 – Continued Bias ES Correlation NOM EM PSM NNM RM MM 0.15 0 0.75 0.1508 0.1457 0.1535 0.1484 0.1505 0.1483 0.15 0 0.9 0.1503 0.1568 0.1481 0.1486 0.1489 0.1482 0.15 0 0.99 0.1582 0.1432 0.1489 0.1513 0.1543 0.1512 0.15 0 0 0.1486 0.1350 0.1471 0.1445 0.1441 0.1445 0.15 0 0.75 0.1488 0.1439 0.1469 0.1497 0.1437 0.1497 0.15 0 0.9 0.1450 0.1768 0.1513 0.1493 0.1547 0.1501 0.15 0 0.99 0.1568 0.1862 0.1566 0.1582 0.1536 0.1583 0.15 0.1 0 0.2549 0.2565 0.2531 0.2495 0.2479 0.2493 0.15 0.1 0.75 0.2522 0.2437 0.2473 0.2501 0.2495 0.2494 0.15 0.1 0.9 0.2547 0.2598 0.2581 0.2521 0.2536 0.2526 0.15 0.1 0.99 0.2505 0.2518 0.2524 0.2482 0.2475 0.2479 0.15 0.1 0 0.2468 0.2491 0.2481 0.2445 0.2425 0.2441 0.15 0.1 0.75 0.2515 0.2541 0.2517 0.2499 0.2456 0.2499 0.15 0.1 0.9 0.2510 0.2627 0.2568 0.2482 0.2472 0.2481 0.15 0.1 0.99 0.2356 0.2462 0.2483 0.2449 0.2474 0.2445 0.15 0.1 0 0.2483 0.2395 0.2473 0.2486 0.2472 0.2489 0.15 0.1 0.75 0.2532 0.2455 0.2587 0.2646 0.2641 0.2645 0.15 0.1 0.9 0.2540 0.2445 0.2537 0.2466 0.2520 0.2471 0.15 0.1 0.99 0.2487 0.2187 0.2488 0.2499 0.2514 0.2494 0.15 0.45 0 0.5958 0.5972 0.5990 0.6008 0.6011 0.5998 0.15 0.45 0.75 0.6033 0.5945 0.6034 0.6009 0.5996 0.6010 0.15 0.45 0.9 0.6000 0.5960 0.5986 0.6017 0.5997 0.6015 0.15 0.45 0.99 0.6011 0.5998 0.6023 0.5922 0.6053 0.5927 0.15 0.45 0 0.5886 0.5923 0.5929 0.5971 0.5941 0.5972 0.15 0.45 0.75 0.6100 0.5890 0.5939 0.6071 0.6021 0.6073 0.15 0.45 0.9 0.6052 0.6063 0.6051 0.6004 0.6028 0.6009 0.15 0.45 0.99 0.5981 0.5992 0.5934 0.5930 0.5945 0.5924 0.15 0.45 0 0.5923 0.5819 0.5987 0.5910 0.5985 0.5905 0.15 0.45 0.75 0.5939 0.5885 0.5976 0.5926 0.5942 0.5924 0.15 0.45 0.9 0.5903 0.5706 0.5901 0.5989 0.5927 0.5990 0.15 0.45 0.99 0.6045 0.5950 0.5991 0.5959 0.5986 0.5963

Table 23 presents the simple-simple effect comparison and the least square means for

these post hoc tests. It appears that EM YD is significantly lower than all matching methods with considerably greater variance. Second there was a MM*ES interaction at

Correlation = 0.90, (F(10, 134865) = 3.96, p < 0.0001, HFL p = .0022). Breaking this

104

down by both Correlation and ES revealed a statistically significant Matching Method effect (F(5,44955) = 5.19, p < 0.0001, HFL p = 0.0038 when Correlation = 0.90 and ES =

0.00. Pairwise comparisons among the

Table 23 Case 2: Simple-simple Effect LS Means for Matching Method When Correlation = 0.75 and Bias = 0.15 Matching Method LS Mean Stderr NOMa 0.3348 0.0016 EMb 0.3279 0.0030 PSMa,c 0.3335 0.0015 NNMa,c 0.3350 0.0015 RMa,b,c 0.3326 0.0015 MMa,c 0.3349 0.0015 Note: LS Means with the same letter are not statistically different, all p’s < .05

Matching Methods when Correlation = 0.90 and ES = 0.00 are presented in Table 24. As

can be seen from this table, the EM revealed a larger YD difference than any of the other matching methods. Furthermore, NOM yielded lower values than PSM and RM.

Lastly, when Correlation = 0.99 there was a MM*ES interaction (F(10, 134865) = 2.00, p

= 0.0292, HFL p = .0004). Breaking this interaction down further by both Correlation and

ES revealed only a statistically significant Matching Method effect (F(5,44955) = 2.68, p

= 0.0197, HFL p = 0.0595 when Correlation = 0.99 and ES = 0.10.

Table 24 Case 2: Simple-simple Effect LS Means for the Matching Method When Correlation = 0.90 and ES = 0.00 Matching Method LS Mean Stderr NOMa 0.0809 0.0012 EMb 0.0905 0.0030 PSMc 0.0851 0.0015 NNMa,c 0.0833 0.0015 RMc 0.0847 0.0015 MMa,c 0.0832 0.0015 Note: LS Means with the same letter are not statistically different, all p’s < .05

105

Table 25 presents least square means and post hoc summary of the simple-simple effect comparisons among the Matching Methods. Here again; EM is diverging away, lower, than the other methods while also evidencing a larger variance.

Table 25 Case 2: Simple-simple Effect LS Means for Matching Method When Correlation= 0.99 and ES = 0.10 Matching Method LS Mean Stderr NOMa 0.1798 0.0016 EMa 0.1784 0.0030 PSMc 0.1843 0.0015 NNMa,b,c 0.1824 0.0015 RMb,c 0.1839 0.0015 MMa,b,d 0.1822 0.0015 Note: LS Means with the same letter are not statistically different, all p’s < .05

MM*Bias*ES*Covariance Interaction Post Hoc

Simple effect analysis of this interaction focused on the 3-way interaction involving Matching Method, Bias and ES while holding Covariance constant (see Table

26). No statistically significant 3-way interactions were noted and no effects were observed when Covariance was 0.00 or 0.40. However, a statistically significant MM*ES

2-way interaction was noted in the Covariance = 0.90 condition (F(10, 179820) = 3.50, p

= 0.0001, HFL p = 0.0138). Breakdown for this 2-way interaction revealed a statistically significant Matching Method effect only in the ES = 0.00 condition (F(5, 59940) = 2.17, p = 0.0049, HFL p = 0.0138.

106

Table 26 LS Means for Bias, ES and Covariance by Matching Method Bias ES Covariance NOM EM PSM NNM RM MM 0 0 0 -0.0115 -0.0079 -0.0042 -0.0099 -0.0062 -0.0095 0 0 0 0.0017 0.0038 0.0047 -0.0012 -0.0071 -0.0009 0 0 0 -0.0028 -0.0011 0.0004 0.0046 -0.0035 0.0042 0 0 0 0.0004 0.0012 0.0014 -0.0006 -0.0028 -0.0004 0 0 0.40 -0.0002 0.0003 0.0024 -0.0066 0.0025 -0.0066 0 0 0.40 -0.0091 -0.0038 -0.0029 -0.0079 -0.0089 -0.0073 0 0 0.40 -0.0056 -0.0046 0.0024 0.0000 -0.0013 0.0000 0 0 0.40 -0.0071 -0.0068 -0.0063 -0.0001 0.0035 -0.0001 0 0 0.90 0.0060 0.0086 0.0093 0.0000 -0.0008 0.0000 0 0 0.90 0.0035 0.0145 0.0050 0.0076 0.0056 0.0070 0 0 0.90 0.0038 0.0076 0.0069 0.0080 -0.0003 0.0077 0 0 0.90 0.0019 -0.0068 -0.0041 -0.0074 0.0022 -0.0074 0 0.1 0 0.1112 0.1017 0.1039 0.1148 0.1083 0.1147 0 0.1 0 0.1050 0.0996 0.1041 0.1046 0.1037 0.1050 0 0.1 0 0.1019 0.1045 0.1086 0.1081 0.1016 0.1085 0 0.1 0 0.0953 0.1040 0.1051 0.1002 0.1061 0.0995 0 0.1 0.40 0.0941 0.0918 0.1006 0.0966 0.1018 0.0963 0 0.1 0.40 0.1014 0.0977 0.0991 0.0989 0.1002 0.0988 0 0.1 0.40 0.0971 0.1012 0.1059 0.1006 0.1011 0.1007 0 0.1 0.40 0.0993 0.1036 0.1019 0.0987 0.1008 0.0989 0 0.1 0.90 0.1039 0.0992 0.0997 0.1017 0.1028 0.1021 0 0.1 0.90 0.1001 0.1092 0.1041 0.1035 0.0974 0.1037 0 0.1 0.90 0.0984 0.0957 0.1007 0.0992 0.0993 0.0989 0 0.1 0.90 0.0943 0.0933 0.1004 0.0992 0.0989 0.0986 0 0.45 0 0.4436 0.4435 0.4398 0.4342 0.4411 0.4340 0 0.45 0 0.4433 0.4495 0.4434 0.4422 0.4392 0.4425 0 0.45 0 0.4435 0.4332 0.4388 0.4406 0.4423 0.4408 0 0.45 0 0.4364 0.4362 0.4355 0.4384 0.4435 0.4385 0 0.45 0.40 0.4347 0.4312 0.4378 0.4351 0.4322 0.4351 0 0.45 0.40 0.4438 0.4297 0.4385 0.4373 0.4375 0.4362 0 0.45 0.40 0.4359 0.4435 0.4442 0.4323 0.4403 0.4322 0 0.45 0.40 0.4423 0.4379 0.4389 0.4328 0.4337 0.4322 0 0.45 0.90 0.4352 0.4437 0.4442 0.4350 0.4383 0.4354 0 0.45 0.90 0.4329 0.4333 0.4338 0.4368 0.4376 0.4369 0 0.45 0.90 0.4412 0.4159 0.4366 0.4416 0.4362 0.4418 0 0.45 0.90 0.4404 0.4267 0.4360 0.4440 0.4391 0.4440 0.1 0 0 0.1014 0.1011 0.1035 0.0978 0.0969 0.0978 0.1 0 0 0.1076 0.1040 0.1037 0.1059 0.1016 0.1060 0.1 0 0 0.0996 0.1073 0.1058 0.1026 0.1012 0.1018

107

Table 26 – Continued Bias ES Covariance NOM EM PSM NNM RM MM 0.1 0 0 0.0942 0.0991 0.0974 0.0938 0.0991 0.0936 0.1 0 0.40 0.0919 0.0997 0.0998 0.0982 0.0905 0.0984 0.1 0 0.40 0.0934 0.0988 0.0947 0.0987 0.0988 0.0990 0.1 0 0.40 0.0908 0.0995 0.0995 0.0969 0.1013 0.0969 0.1 0 0.40 0.1021 0.1073 0.1017 0.0994 0.0996 0.0990 0.1 0 0.90 0.1000 0.0820 0.0959 0.1035 0.0977 0.1033 0.1 0 0.90 0.0963 0.1124 0.1040 0.0988 0.0965 0.0988 0.1 0 0.90 0.0967 0.1227 0.1002 0.0947 0.1054 0.0946 0.1 0 0.90 0.1037 0.1181 0.1029 0.1037 0.1024 0.1034 0.1 0.1 0 0.2018 0.2090 0.2008 0.2008 0.2015 0.2008 0.1 0.1 0 0.1978 0.1982 0.1975 0.1978 0.2013 0.1978 0.1 0.1 0 0.2080 0.2093 0.2059 0.2003 0.2004 0.1995 0.1 0.1 0 0.0942 0.0991 0.0974 0.0938 0.0991 0.0936 0.1 0.1 0.40 0.2021 0.1966 0.2041 0.2000 0.2026 0.2000 0.1 0.1 0.40 0.1970 0.2048 0.1976 0.1986 0.1943 0.1989 0.1 0.1 0.40 0.1983 0.2023 0.1994 0.1937 0.1980 0.1934 0.1 0.1 0.40 0.1978 0.1960 0.1973 0.2017 0.2022 0.2013 0.1 0.1 0.90 0.2002 0.1878 0.1986 0.2035 0.1954 0.2031 0.1 0.1 0.90 0.1944 0.2150 0.1980 0.1966 0.1972 0.1973 0.1 0.1 0.90 0.2102 0.2060 0.2047 0.2087 0.2077 0.2086 0.1 0.1 0.90 0.1962 0.1952 0.1998 0.1991 0.2027 0.1996 0.1 0.5 0 0.5495 0.5378 0.5428 0.5489 0.5494 0.5489 0.1 0.5 0 0.5465 0.5460 0.5474 0.5451 0.5493 0.5451 0.1 0.45 0 0.5503 0.5469 0.5545 0.5549 0.5534 0.5542 0.1 0.45 0 0.5417 0.5452 0.5442 0.5467 0.5484 0.5474 0.1 0.45 0.40 0.5470 0.5504 0.5511 0.5464 0.5477 0.5457 0.1 0.45 0.40 0.5461 0.5366 0.5468 0.5482 0.5454 0.5478 0.1 0.45 0.40 0.5478 0.5647 0.5497 0.5493 0.5574 0.5493 0.1 0.45 0.40 0.5414 0.5412 0.5445 0.5506 0.5443 0.5500 0.1 0.45 0.90 0.5500 0.5515 0.5474 0.5478 0.5466 0.5477 0.1 0.45 0.90 0.5450 0.5585 0.5432 0.5413 0.5439 0.5412 0.1 0.45 0.90 0.5499 0.5389 0.5535 0.5487 0.5509 0.5489 0.1 0.45 0.90 0.5414 0.5416 0.5458 0.5422 0.5415 0.5426 0.15 0 0 0.1515 0.1588 0.1545 0.1515 0.1471 0.1517 0.15 0 0 0.1492 0.1464 0.1485 0.1518 0.1439 0.1520 0.15 0 0 0.1501 0.1498 0.1513 0.1452 0.1564 0.1451 0.15 0 0 0.1536 0.1459 0.1540 0.1457 0.1527 0.1457 0.15 0 0.40 0.1471 0.1660 0.1576 0.1476 0.1528 0.1475 0.15 0 0.40 0.1508 0.1457 0.1535 0.1484 0.1505 0.1483

108

Table 26 – Continued Bias ES Covariance NOM EM PSM NNM RM MM 0.15 0 0.40 0.1503 0.1568 0.1481 0.1486 0.1489 0.1482 0.15 0 0.40 0.1582 0.1432 0.1489 0.1513 0.1543 0.1512 0.15 0 0.90 0.1486 0.1350 0.1471 0.1445 0.1441 0.1445 0.15 0 0.90 0.1488 0.1439 0.1469 0.1497 0.1437 0.1497 0.15 0 0.90 0.1450 0.1768 0.1513 0.1493 0.1547 0.1501 0.15 0 0.90 0.1568 0.1862 0.1566 0.1582 0.1536 0.1583 0.15 0.1 0 0.2549 0.2565 0.2531 0.2495 0.2479 0.2493 0.15 0.1 0 0.2522 0.2437 0.2473 0.2501 0.2495 0.2494 0.15 0.1 0 0.2547 0.2598 0.2581 0.2521 0.2536 0.2526 0.15 0.1 0 0.2505 0.2518 0.2524 0.2482 0.2475 0.2479 0.15 0.1 0.40 0.2468 0.2491 0.2481 0.2445 0.2425 0.2441 0.15 0.1 0.40 0.2515 0.2541 0.2517 0.2499 0.2456 0.2499 0.15 0.1 0.40 0.2510 0.2627 0.2568 0.2482 0.2472 0.2481 0.15 0.1 0.40 0.2356 0.2462 0.2483 0.2449 0.2474 0.2445 0.15 0.1 0.90 0.2483 0.2395 0.2473 0.2486 0.2472 0.2489 0.15 0.1 0.90 0.2532 0.2455 0.2587 0.2646 0.2641 0.2645 0.15 0.1 0.90 0.2540 0.2445 0.2537 0.2466 0.2520 0.2471 0.15 0.1 0.90 0.2487 0.2187 0.2488 0.2499 0.2514 0.2494 0.15 0.45 0 0.5958 0.5972 0.5990 0.6008 0.6011 0.5998 0.15 0.45 0 0.6033 0.5945 0.6034 0.6009 0.5996 0.6010 0.15 0.45 0 0.6000 0.5960 0.5986 0.6017 0.5997 0.6015 0.15 0.45 0 0.6011 0.5998 0.6023 0.5922 0.6053 0.5927 0.15 0.45 0.40 0.5886 0.5923 0.5929 0.5971 0.5941 0.5972 0.15 0.45 0.40 0.6100 0.5890 0.5939 0.6071 0.6021 0.6073 0.15 0.45 0.40 0.6052 0.6063 0.6051 0.6004 0.6028 0.6009 0.15 0.45 0.40 0.5981 0.5992 0.5934 0.5930 0.5945 0.5924 0.15 0.45 0.90 0.5923 0.5819 0.5987 0.5910 0.5985 0.5905 0.15 0.45 0.90 0.5939 0.5885 0.5976 0.5926 0.5942 0.5924 0.15 0.45 0.90 0.5903 0.5706 0.5901 0.5989 0.5927 0.5990 0.15 0.45 0.90 0.6045 0.5950 0.5991 0.5959 0.5986 0.5963

Table 27 presents the simple-simple effect comparison and the least square means for

Matching Method post hoc tests. From this table, EM YD means are significantly larger than all other methods except for PSM.

109

Table 27 Case 2: Simple-simple effect LS means for the Matching Method When Covariance = 0.90 and ES = 0.00 Matching Method LS Mean Stderr NOMa 0.0843 0.0013 EMb 0.0918 0.0026 PSMa,b 0.0852 0.0013 NNMa,c 0.0842 0.0013 RMa,c 0.0837 0.0013 MMa,c 0.0842 0.0013 Note: LS Means with the same letter are not statistically different, all p’s < .05

MM*ES*Covariance*Correlation Interaction Post Hoc

Simple effect analysis (see Table 28) of this interaction focused on the 3-way interaction involving Matching Method, Correlation and Covariance while holding ES constant. Simple effect 3-way interactions for MM*Covariance*Correlation when

ES=0.0 revealed a statistically significant finding (F(30,179820) = 1.86, p = .0029, HFL p = 0.0279) and when ES = 0.45 (F(30,179820) = 2.56, p < .0001, HFL p = 0.0015) but no statistically significant findings for ES = 0.10. Further simple effect analyses of the 3- way interactions when both ES and Correlation held constant. Simple-simple effects testing for a 2-way interaction, MM*Covariance, revealed statistically significant results when ES = 0.00 and Correlation = 0.90 (F(10,44955) = 2.40, p = 0.0076, HFL p = .0399) and when ES = 0.00 and Correlation = 0.99 (F(10,44955) = 2.32, p = 0.0101, HFL p =

.0452) but no statistically significant findings for any effects when ES = 0.00 and

Correlation was either 0.00 or 0.75. Further examination of the MM*Covariance interactions, tests for simple-simple effects among matching methods while holding ES =

0.00; Correlation = 0.90 and varying Covariance. Results revealed Matching Method

differences in YD only when Covariance = 0.90 (F(5,14985) = 5.03, p = 0.0001, HFL p =

110

.0119). Table 29 presents the result of the pairwise comparisons among the Matching

Method means. As can be seen from this table, YD when calculated via EM was significantly larger than all other methods that did not differ among each other.

Table 28

Case 2: LS Means for ES, Covariance and Correlation by Matching Method ES Covariance Correlation NOM EM PSM NNM RM MM 0 0 0 0.0805 0.0840 0.0846 0.0798 0.0793 0.0800 0 0 0.75 0.0861 0.0847 0.0856 0.0855 0.0795 0.0857 0 0 0.9 0.0823 0.0853 0.0858 0.0841 0.0847 0.0837 0 0 0.99 0.0827 0.0821 0.0843 0.0796 0.0830 0.0796 0 0.4 0 0.0796 0.0887 0.0866 0.0797 0.0819 0.0798 0 0.4 0.75 0.0783 0.0802 0.0818 0.0797 0.0801 0.0800 0 0.4 0.9 0.0785 0.0839 0.0834 0.0818 0.0830 0.0817 0 0.4 0.99 0.0844 0.0812 0.0814 0.0835 0.0858 0.0834 0 0.9 0 0.0849 0.0752 0.0841 0.0827 0.0803 0.0826 0 0.9 0.75 0.0829 0.0903 0.0853 0.0854 0.0819 0.0851 0 0.9 0.9 0.0818 0.1024 0.0861 0.0840 0.0866 0.0841 0 0.9 0.99 0.0875 0.0992 0.0851 0.0848 0.0860 0.0848 0.1 0 0 0.1893 0.1891 0.1859 0.1884 0.1859 0.1883 0.1 0 0.75 0.1850 0.1805 0.1830 0.1842 0.1848 0.1841 0.1 0 0.9 0.1882 0.1912 0.1908 0.1868 0.1852 0.1868 0.1 0 0.99 0.1466 0.1516 0.1516 0.1474 0.1509 0.1470 0.1 0.4 0 0.1810 0.1792 0.1842 0.1804 0.1823 0.1801 0.1 0.4 0.75 0.1833 0.1856 0.1828 0.1825 0.1800 0.1825 0.1 0.4 0.9 0.1821 0.1887 0.1874 0.1808 0.1821 0.1807 0.1 0.4 0.99 0.1776 0.1819 0.1825 0.1818 0.1835 0.1816 0.1 0.9 0 0.1841 0.1755 0.1819 0.1846 0.1818 0.1847 0.1 0.9 0.75 0.1826 0.1899 0.1870 0.1883 0.1862 0.1885 0.1 0.9 0.9 0.1875 0.1821 0.1863 0.1848 0.1863 0.1849 0.1 0.9 0.99 0.1797 0.1691 0.1830 0.1827 0.1843 0.1825 0.45 0 0 0.5296 0.5262 0.5272 0.5280 0.5305 0.5276 0.45 0 0.75 0.5310 0.5300 0.5314 0.5294 0.5294 0.5295 0.45 0 0.9 0.5312 0.5254 0.5306 0.5324 0.5318 0.5322 0.45 0 0.99 0.5264 0.5271 0.5273 0.5258 0.5324 0.5262 0.45 0.4 0 0.5234 0.5246 0.5273 0.5262 0.5247 0.5260 0.45 0.4 0.75 0.5333 0.5184 0.5264 0.5309 0.5283 0.5304 0.45 0.4 0.9 0.5296 0.5382 0.5330 0.5273 0.5335 0.5275

111

Table 28 – Continued ES Covariance Correlation NOM EM PSM NNM RM MM 0.45 0.4 0.99 0.5273 0.5261 0.5256 0.5255 0.5242 0.5249 0.45 0.9 0 0.5258 0.5257 0.5301 0.5246 0.5278 0.5245 0.45 0.9 0.75 0.5239 0.5268 0.5249 0.5236 0.5252 0.5235 0.45 0.9 0.9 0.5271 0.5085 0.5267 0.5297 0.5266 0.5299 0.45 0.9 0.99 0.5288 0.5211 0.5270 0.5274 0.5264 0.5276 NOM=Random Sampling, EM=Exact Match; PSM=Propensity Score Match; NNM=Nearest Neighbor Match; RM=Radius Match; MM=Mahalanobis Metric Match

Table 29 Case 2: Simple-simple Effect LS Means for the Matching Method When ES = 0.00, Correlation = 0.90 and Covariance = 0.90 Matching Method LS Mean Stderr NOMa 0.0818 0.0027 EMb 0.1024 0.0053 PSMa 0.0861 0.0026 NNMa 0.0840 0.0027 RMa 0.0866 0.0027 MMa 0.0841 0.0027 Note: LS Means with the same letter are not statistically different, all p’s < .05

Examination of the MM*Covariance interaction when ES = 0.00 and Correlation = 0.99 proceeded with the same breakdown strategy. Results revealed only a statistically significant Matching Method effect when Covariance = 0.99 (F(5,14985) = 2.89, p =

0.0131, HFL p = .0683). Notably, here the protected p-value does not reach the established type I level suggesting caution be applied in the interpretation of this post hoc

analysis. Table 30 summarizes the pairwise YD comparisons among the six Matching

Methods when ES = 0.00, Correlation = 0.99 and Covariance = 0.90. As can be seen from this table, when calculated via EM was significantly larger than all other methods that did not differ among each other.

112

Table 30 Case 2: Simple-simple Effect LS Means for the Matching Method When ES = 0.00, Correlation = 0.99 and Covariance = 0.90 Matching Method LS Mean Stderr NOMa 0.0875 0.0027 EMb 0.0992 0.0053 PSMa 0.0851 0.0026 NNMa 0.0848 0.0027 RMa 0.0860 0.0027 MMa 0.0848 0.0027 Note: LS Means with the same letter are not statistically different, all p’s < .10

Case 3: Mixed Covariates

Case 3 examined mixed covariates. The overall MANOVA analysis explored the effects of the experimental variables: bias {0, 0.10, 0.15}, effect size {0, 0.10, 0.44/45}, covariance/collinearity among covariates {0, 0.40, 0.90}, and correlation between the bias amount and covariate group {0, 0.75, 0.90, 0.99} among the six matching methods.

As presented for Cases 1 and 2, Table 31 presents the pooled grand means (YD ) for the six matching methods and Figure 9 depicts these same means center about zero. Case 3 descriptive statistics for all design factors by matching methods are presented in

Appendix C.

113

Table 31

Case 3: YD Descriptive Statistics for Average of 108000 Replications by Matching Method

Matching Method SD NOM 0.2654 0.2507 EM 0.2653 0.3283 PSM 0.2658 0.2503 NNM 0.2667 0.2509 RM 0.2657 0.2506 MM 0.2666 0.2504

Comparison of among the matching methods to the horizontal line suggests that in the mixed covariate case PSM and RM methods draw closest to the horizontal line suggesting these two methods, averaged over all design conditions may be superior in estimating .

Figure 9. Case 3: Pooled After Centering

114

MANOVA analysis of Case 3 yielded several statistically significant findings as presented in Table 32.

Table 32 Case 3: Primary MANOVA Results Multivariate Hypothesis Wilk’s p-value Roy’s p-value λ GCR MatchMethod 0.9999 0.0267 0.0001 0.0267 MatchMethod*Bias 0.9999 0.3251 0.0001 0.1453 MatchMethod*ES 0.9999 0.9787 <0.0001 0.7197 MatchMethod*Covariance 0.9999 0.4199 0.0001 0.1369 MatchMethod*Correlation 0.9999 0.8191 0.0001 0.2615 MatchMethod*Bias*ES 0.9998 0.4019 0.0001 0.0291 MatchMethod*Bias*Covariance 0.9998 0.2948 0.0001 0.0087 MatchMethod*ES*Covariance 0.9998 0.1799 0.0001 0.0107 MatchMethod*Bias*Correlation 0.9996 0.1070 0.0002 0.0029 MatchMethod*ES*Correlation 0.9998 0.6349 0.0001 0.0257 MatchMethod*Covariance*Correlation 0.9998 0.8642 0.0001 0.0363 MatchMethod*Bias*ES*Covariance 0.9995 0.0912 0.0002 0.0081 MatchMethod*Bias*ES*Correlation 0.9996 0.9015 0.0002 0.0499 MatchMethod*Bias*Covariance*Correlation 0.9997 0.9923 0.0001 0.2392 MatchMethod*ES*Covariance*Correlation 0.9997 0.9907 0.0001 0.2084 MatchMethod*Bias*ES*Covariance*Correlation 0.9990 0.7803 0.0003 0.1008

Parallel to Case 1 and 2, post hoc analysis of the statistically significant interactions followed the general strategy outlined by Winer (1971) and thus subsequently focused on the MM*Bias*SE*Covariance and MM*Bias*SE*Correlation

4-way interactions. Since jointly both interactions include all other statistically significant effect except the statistically significant 3-way interaction

MM*Covariance*Correlation, this 3-way interaction will be examined first, followed by a breakdown of the MM*Bias*SE by Covariance, then by Correlation.

MM*Covariance*Correlation Interaction Post Hoc

Simple effect analysis of this interaction focused on the 2-way interaction involving Matching Method and Covariance while holding Correlation constant. Refer to

Table 33 for descriptive statistics. Surprisingly no statistically significant

115

MM*Covariance 2-way interactions were observed for any level of Correlation.

Although, a Matching Method simple-simple effect was noted when Correlation = 0.75 and 0.99. Pairwise least square means comparisons for effects are presented in Table 34.

Table 33 Case 3: LS Means for Covariance and Correlation by Matching Method Covariance Correlation NOM EM PSM NNM RM MM 0 0 -0.0030 0.0022 -0.0006 -0.0010 -0.0040 -0.0011 0 0.75 0.0068 0.0128 0.0082 0.0090 0.0106 0.0086 0 0.9 -0.0003 0.0094 0.0048 0.0044 0.0043 0.0047 0 0.99 0.0055 0.0016 0.0068 0.0049 0.0042 0.0050 0.4 0 -0.0026 -0.0050 -0.0022 -0.0026 -0.0023 -0.0030 0.4 0.75 0.0068 0.0026 0.0008 0.0038 0.0070 0.0039 0.4 0.9 -0.0070 0.0080 -0.0021 -0.0027 -0.0006 -0.0032 0.4 0.99 -0.0085 -0.0100 -0.0040 -0.0011 -0.0079 -0.0005 0.9 0 -0.0016 -0.0037 -0.0049 0.0010 -0.0008 0.0009 0.9 0.75 0.0024 -0.0122 0.0003 0.0010 0.0010 0.0008 0.9 0.9 0.0020 -0.0053 -0.0011 0.0088 0.0014 0.0080 0.9 0.99 0.0020 -0.0053 -0.0011 0.0088 0.0014 0.0080 0 0 0.1022 0.1005 0.1024 0.1009 0.1008 0.1012 0 0.75 0.1011 0.1112 0.1067 0.1084 0.1071 0.1083 0 0.9 0.0966 0.1009 0.1053 0.1045 0.1062 0.1047 0 0.99 0.0919 0.1010 0.1043 0.1009 0.0980 0.1011 0.4 0 0.1020 0.1030 0.1001 0.1022 0.0996 0.1022 0.4 0.75 0.1000 0.1013 0.0976 0.1047 0.0998 0.1035 0.4 0.9 0.1040 0.1023 0.1055 0.1022 0.1008 0.1018 0.4 0.99 0.0982 0.0979 0.0971 0.1066 0.1047 0.1059 0.9 0 0.1019 0.1081 0.1020 0.1031 0.0964 0.1028 0.9 0.75 0.1046 0.0985 0.1058 0.1043 0.0995 0.1041 0.9 0.9 0.1046 0.1006 0.1022 0.0989 0.1075 0.0989 0.9 0.99 0.0971 0.0970 0.0975 0.1012 0.0967 0.1016 0 0 0.4412 0.4462 0.4392 0.4366 0.4390 0.4365 0 0.75 0.4375 0.4379 0.4360 0.4362 0.4360 0.4369 0 0.9 0.4408 0.4429 0.4444 0.4387 0.4402 0.4385 0 0.99 0.4386 0.4345 0.4406 0.4376 0.4361 0.4370 0.4 0 0.4399 0.4462 0.4475 0.4461 0.4421 0.4457 0.4 0.75 0.4355 0.4312 0.4425 0.4375 0.4417 0.4379 0.4 0.9 0.4418 0.4434 0.4408 0.4406 0.4453 0.4408 0.4 0.99 0.4412 0.4438 0.4416 0.4444 0.4448 0.4440 0.9 0 0.4365 0.4382 0.4376 0.4352 0.4401 0.4352

116

Table 33 – Continued Covariance Correlation NOM EM PSM NNM RM MM 0.9 0.75 0.4388 0.4454 0.4392 0.4360 0.4336 0.4356 0.9 0.9 0.4430 0.4491 0.4431 0.4423 0.4405 0.4423 0.9 0.99 0.4358 0.4542 0.4464 0.4458 0.4450 0.4457 0 0 0.1079 0.1057 0.1015 0.1050 0.1089 0.1047 0 0.75 0.0908 0.0902 0.0905 0.0993 0.0937 0.0992 0 0.9 0.0992 0.0948 0.0934 0.1004 0.0978 0.0999 0 0.99 0.0976 0.0922 0.1003 0.0965 0.1024 0.0958 0.4 0 0.0999 0.1053 0.0993 0.1066 0.1050 0.1060 0.4 0.75 0.0998 0.1027 0.1031 0.1002 0.1005 0.1000 0.4 0.9 0.1041 0.0977 0.1091 0.1096 0.1015 0.1091 0.4 0.99 0.1010 0.1084 0.1003 0.0990 0.1001 0.0989 0.9 0 0.0989 0.1075 0.0993 0.0967 0.0973 0.0972 0.9 0.75 0.1074 0.1105 0.1092 0.1049 0.1063 0.1052 0.9 0.9 0.1091 0.1136 0.1001 0.1028 0.1011 0.1031 0.9 0.99 0.0980 0.1001 0.1007 0.1022 0.1032 0.1024 0 0 0.1965 0.1957 0.1996 0.2022 0.2020 0.2023 0 0.75 0.0908 0.0902 0.0905 0.0993 0.0937 0.0992 0 0.9 0.1980 0.1963 0.2033 0.2068 0.2036 0.2072 0 0.99 0.1914 0.1937 0.2025 0.2063 0.2075 0.2064 0.4 0 0.2102 0.1931 0.1986 0.2013 0.1989 0.2018 0.4 0.75 0.2044 0.1928 0.2079 0.2096 0.2011 0.2097 0.4 0.9 0.2086 0.1998 0.2064 0.2018 0.2036 0.2021 0.4 0.99 0.1978 0.2025 0.2039 0.2034 0.1998 0.2036 0.9 0 0.2001 0.2038 0.1969 0.2018 0.2045 0.2013 0.9 0.75 0.2054 0.2186 0.2107 0.2011 0.2008 0.2011 0.9 0.9 0.1995 0.2053 0.2082 0.2025 0.2055 0.2026 0.9 0.99 0.2017 0.2042 0.2024 0.2012 0.2037 0.2011 0 0 0.5505 0.5444 0.5455 0.5479 0.5469 0.5482 0 0.75 0.5500 0.5428 0.5452 0.5510 0.5442 0.5509 0 0.9 0.5428 0.5459 0.5511 0.5517 0.5523 0.5516 0 0.99 0.5504 0.5533 0.5564 0.5566 0.5552 0.5561 0.4 0 0.5549 0.5553 0.5571 0.5583 0.5599 0.5586 0.4 0.75 0.5467 0.5460 0.5401 0.5508 0.5504 0.5506 0.4 0.9 0.5553 0.5473 0.5528 0.5537 0.5543 0.5532 0.4 0.99 0.5455 0.5427 0.5536 0.5539 0.5514 0.5539 0.9 0 0.5508 0.5503 0.5468 0.5538 0.5543 0.5539 0.9 0.75 0.5468 0.5524 0.5468 0.5507 0.5543 0.5514 0.9 0.9 0.5490 0.5411 0.5496 0.5492 0.5482 0.5485 0.9 0.99 0.5516 0.5421 0.5562 0.5523 0.5524 0.5524

117

Table 33 – Continued Covariance Correlation NOM EM PSM NNM RM MM 0 0 0.1497 0.1461 0.1492 0.1473 0.1458 0.1482 0 0.75 0.1452 0.1475 0.1528 0.1488 0.1584 0.1496 0 0.9 0.1426 0.1479 0.1455 0.1508 0.1457 0.1509 0 0.99 0.1583 0.1564 0.1562 0.1549 0.1539 0.1556 0.4 0 0.1535 0.1598 0.1532 0.1460 0.1445 0.1460 0.4 0.75 0.1522 0.1600 0.1549 0.1532 0.1509 0.1529 0.4 0.9 0.1500 0.1467 0.1506 0.1524 0.1449 0.1527 0.4 0.99 0.1518 0.1539 0.1521 0.1489 0.1491 0.1486 0.9 0 0.1534 0.1641 0.1532 0.1575 0.1571 0.1569 0.9 0.75 0.1491 0.1323 0.1397 0.1393 0.1400 0.1392 0.9 0.9 0.1552 0.1507 0.1540 0.1509 0.1531 0.1508 0.9 0.99 0.1540 0.1485 0.1482 0.1495 0.1540 0.1500 0 0 0.2502 0.2519 0.2538 0.2516 0.2566 0.2515 0 0.75 0.2510 0.2500 0.2504 0.2436 0.2520 0.2431 0 0.9 0.2517 0.2560 0.2498 0.2585 0.2484 0.2581 0 0.99 0.2499 0.2519 0.2476 0.2468 0.2483 0.2468 0.4 0 0.2500 0.2427 0.2521 0.2535 0.2526 0.2530 0.4 0.75 0.2505 0.2554 0.2521 0.2535 0.2531 0.2530 0.4 0.9 0.2502 0.2494 0.2474 0.2524 0.2465 0.2524 0.4 0.99 0.2651 0.2500 0.2531 0.2585 0.2601 0.2589 0.9 0 0.2523 0.2524 0.2533 0.2503 0.2502 0.2500 0.9 0.75 0.2438 0.2357 0.2446 0.2481 0.2506 0.2480 0.9 0.9 0.2567 0.2605 0.2584 0.2544 0.2492 0.2543 0.9 0.99 0.2492 0.2514 0.2454 0.2549 0.2444 0.2548 0 0 0.6031 0.6034 0.6061 0.6027 0.6003 0.6029 0 0.75 0.6004 0.5942 0.6038 0.6046 0.5946 0.6046 0 0.9 0.6031 0.6154 0.6109 0.6073 0.6078 0.6073 0 0.99 0.6026 0.6063 0.6013 0.6056 0.5980 0.6054 0.4 0 0.6008 0.6118 0.6008 0.6037 0.5996 0.6038 0.4 0.75 0.6100 0.5890 0.5939 0.6071 0.6021 0.6073 0.4 0.9 0.6057 0.6108 0.6041 0.6049 0.6023 0.6044 0.4 0.99 0.6064 0.6036 0.6059 0.6077 0.6068 0.6078 0.9 0 0.5995 0.5977 0.5997 0.5987 0.5989 0.5981 0.9 0.75 0.5942 0.5767 0.5968 0.5986 0.6014 0.5994 0.9 0.9 0.5999 0.6050 0.5987 0.5979 0.6021 0.5977 0.9 0.99 0.6047 0.6023 0.6019 0.6037 0.5995 0.6033

118

As can be seen from Table 34, when Correlation = 0.75, YD is lower for the EM relative to NNM and MM but not any other method. Whereas when Correlation = 0.99, NOM and

EM what the same values (rounded) but the larger observed variance seen in EM appears to mask some the pairwise differences.

Table 34 Case 3: Simple-simple Effect LS Means for the Matching Method When Correlation = 0.75 Matching Method LS Mean Stderr NOMa,b 0.2619 0.0009 EMa,b 0.2599 0.0015 PSMa,b 0.2619 0.0009 NNMa 0.2631 0.0009 RMa,b 0.2624 0.0009 MMa 0.2631 0.0009 Case 3: Simple-simple Effect LS Means for the Matching Method When Correlation = 0.99 NOMa 0.2659 0.0009 EMa,b 0.2659 0.0014 PSMa,b 0.2673 0.0009 NNMb 0.2686 0.0009 RMa,b 0.2671 0.0009 MMb 0.2685 0.0009 Note: LS Means with the same letter are not statistically different, all p’s < .05

MM*Bias*ES*Covariance Interaction Post Hoc

Simple effect analysis of this interaction focused on the 3-way interaction involving

MM*Bias*ES while holding Covariance constant. Refer to Table 35 for descriptive statistics. The pattern of findings across the three levels of Covariance {0.00, 0.40, 0.90} revealed the following. When Covariance was 0.00, a 2-way interaction MM*Bias was noted (F(10,179820) = 2.81, p = 0.0018, HFL p = 0.0100) and a simple-simple effect for

Matching Method (F(5,179820) = 2.69, p = 0.0197, HFL p = 0.0450); at Covariance =

0.40 no statistically significant effects were noted and when Covariance = 0.90 a

119

statistically significant 3-way MM*Bias*ES effect was observed (F(20,179820) = 1.82, p

= 0.0140, HFL p = 0.0659). Further break down analysis focused exclusively on the 3- way interaction, MM*Bias*ES, in the Covariance = 0.90 condition by blocking on ES explicitly looking for a statistically significant 2-way interaction: MM*Bias. Results revealed only a statistically significant MM*Bias interaction when Covariance = 0.90 and

ES = 0.00, (F(10,59940) = 2.22, p = 0.0142, HFL p = 0.0622).

Table 35 Case 3: LS Means for Bias*ES*Covariance by Matching Method Bias ES Covariance NOM EM PSM NNM RM MM 0 0 0 -0.0030 0.0022 -0.0006 -0.0010 -0.0040 -0.0011 0 0 0 0.0068 0.0128 0.0082 0.0090 0.0106 0.0086 0 0 0 -0.0003 0.0094 0.0048 0.0044 0.0043 0.0047 0 0 0 0.0055 0.0016 0.0068 0.0049 0.0042 0.0050 0 0 0.4 -0.0026 -0.0050 -0.0022 -0.0026 -0.0023 -0.0030 0 0 0.4 0.0068 0.0026 0.0008 0.0038 0.0070 0.0039 0 0 0.4 -0.0070 0.0080 -0.0021 -0.0027 -0.0006 -0.0032 0 0 0.4 -0.0085 -0.0100 -0.0040 -0.0011 -0.0079 -0.0005 0 0 0.9 -0.0016 -0.0037 -0.0049 0.0010 -0.0008 0.0009 0 0 0.9 0.0024 -0.0122 0.0003 0.0010 0.0010 0.0008 0 0 0.9 0.0020 -0.0053 -0.0011 0.0088 0.0014 0.0080 0 0 0.9 0.0020 -0.0053 -0.0011 0.0088 0.0014 0.0080 0 0.1 0 0.1022 0.1005 0.1024 0.1009 0.1008 0.1012 0 0.1 0 0.1011 0.1112 0.1067 0.1084 0.1071 0.1083 0 0.1 0 0.0966 0.1009 0.1053 0.1045 0.1062 0.1047 0 0.1 0 0.0919 0.1010 0.1043 0.1009 0.0980 0.1011 0 0.1 0.4 0.1020 0.1030 0.1001 0.1022 0.0996 0.1022 0 0.1 0.4 0.1000 0.1013 0.0976 0.1047 0.0998 0.1035 0 0.1 0.4 0.1040 0.1023 0.1055 0.1022 0.1008 0.1018 0 0.1 0.4 0.0982 0.0979 0.0971 0.1066 0.1047 0.1059 0 0.1 0.9 0.1019 0.1081 0.1020 0.1031 0.0964 0.1028 0 0.1 0.9 0.1046 0.0985 0.1058 0.1043 0.0995 0.1041 0 0.1 0.9 0.1046 0.1006 0.1022 0.0989 0.1075 0.0989 0 0.1 0.9 0.0971 0.0970 0.0975 0.1012 0.0967 0.1016 0 0.45 0 0.4412 0.4462 0.4392 0.4366 0.4390 0.4365 0 0.45 0 0.4375 0.4379 0.4360 0.4362 0.4360 0.4369 0 0.45 0 0.4408 0.4429 0.4444 0.4387 0.4402 0.4385

120

Table 35 – Continued Bias ES Covariance NOM EM PSM NNM RM MM 0 0.45 0 0.4386 0.4345 0.4406 0.4376 0.4361 0.4370 0 0.45 0.4 0.4399 0.4462 0.4475 0.4461 0.4421 0.4457 0 0.45 0.4 0.4355 0.4312 0.4425 0.4375 0.4417 0.4379 0 0.45 0.4 0.4418 0.4434 0.4408 0.4406 0.4453 0.4408 0 0.45 0.4 0.4412 0.4438 0.4416 0.4444 0.4448 0.4440 0 0.45 0.9 0.4365 0.4382 0.4376 0.4352 0.4401 0.4352 0 0.45 0.9 0.4388 0.4454 0.4392 0.4360 0.4336 0.4356 0 0.45 0.9 0.4430 0.4491 0.4431 0.4423 0.4405 0.4423 0 0.45 0.9 0.4358 0.4542 0.4464 0.4458 0.4450 0.4457 0.1 0 0 0.1079 0.1057 0.1015 0.1050 0.1089 0.1047 0.1 0 0 0.0908 0.0902 0.0905 0.0993 0.0937 0.0992 0.1 0 0 0.0992 0.0948 0.0934 0.1004 0.0978 0.0999 0.1 0 0 0.0976 0.0922 0.1003 0.0965 0.1024 0.0958 0.1 0 0.4 0.0999 0.1053 0.0993 0.1066 0.1050 0.1060 0.1 0 0.4 0.0998 0.1027 0.1031 0.1002 0.1005 0.1000 0.1 0 0.4 0.1041 0.0977 0.1091 0.1096 0.1015 0.1091 0.1 0 0.4 0.1010 0.1084 0.1003 0.0990 0.1001 0.0989 0.1 0 0.9 0.0989 0.1075 0.0993 0.0967 0.0973 0.0972 0.1 0 0.9 0.1074 0.1105 0.1092 0.1049 0.1063 0.1052 0.1 0 0.9 0.1091 0.1136 0.1001 0.1028 0.1011 0.1031 0.1 0 0.9 0.0980 0.1001 0.1007 0.1022 0.1032 0.1024 0.1 0.1 0 0.1965 0.1957 0.1996 0.2022 0.2020 0.2023 0.1 0.1 0 0.0908 0.0902 0.0905 0.0993 0.0937 0.0992 0.1 0.1 0 0.1980 0.1963 0.2033 0.2068 0.2036 0.2072 0.1 0.1 0 0.1914 0.1937 0.2025 0.2063 0.2075 0.2064 0.1 0.1 0.4 0.2102 0.1931 0.1986 0.2013 0.1989 0.2018 0.1 0.1 0.4 0.2044 0.1928 0.2079 0.2096 0.2011 0.2097 0.1 0.1 0.4 0.2086 0.1998 0.2064 0.2018 0.2036 0.2021 0.1 0.1 0.4 0.1978 0.2025 0.2039 0.2034 0.1998 0.2036 0.1 0.1 0.9 0.2001 0.2038 0.1969 0.2018 0.2045 0.2013 0.1 0.1 0.9 0.2054 0.2186 0.2107 0.2011 0.2008 0.2011 0.1 0.1 0.9 0.1995 0.2053 0.2082 0.2025 0.2055 0.2026 0.1 0.1 0.9 0.2017 0.2042 0.2024 0.2012 0.2037 0.2011 0.1 0.45 0 0.5505 0.5444 0.5455 0.5479 0.5469 0.5482 0.1 0.45 0 0.5500 0.5428 0.5452 0.5510 0.5442 0.5509 0.1 0.45 0 0.5428 0.5459 0.5511 0.5517 0.5523 0.5516 0.1 0.45 0 0.5504 0.5533 0.5564 0.5566 0.5552 0.5561 0.1 0.45 0.4 0.5549 0.5553 0.5571 0.5583 0.5599 0.5586 0.1 0.45 0.4 0.5467 0.5460 0.5401 0.5508 0.5504 0.5506

121

Table 35 – Continued Bias ES Covariance NOM EM PSM NNM RM MM 0.1 0.45 0.4 0.5553 0.5473 0.5528 0.5537 0.5543 0.5532 0.1 0.45 0.4 0.5455 0.5427 0.5536 0.5539 0.5514 0.5539 0.1 0.45 0.9 0.5508 0.5503 0.5468 0.5538 0.5543 0.5539 0.1 0.45 0.9 0.5468 0.5524 0.5468 0.5507 0.5543 0.5514 0.1 0.45 0.9 0.5490 0.5411 0.5496 0.5492 0.5482 0.5485 0.1 0.45 0.9 0.5516 0.5421 0.5562 0.5523 0.5524 0.5524 0.15 0 0 0.1497 0.1461 0.1492 0.1473 0.1458 0.1482 0.15 0 0 0.1452 0.1475 0.1528 0.1488 0.1584 0.1496 0.15 0 0 0.1426 0.1479 0.1455 0.1508 0.1457 0.1509 0.15 0 0 0.1583 0.1564 0.1562 0.1549 0.1539 0.1556 0.15 0 0.4 0.1535 0.1598 0.1532 0.1460 0.1445 0.1460 0.15 0 0.4 0.1522 0.1600 0.1549 0.1532 0.1509 0.1529 0.15 0 0.4 0.1500 0.1467 0.1506 0.1524 0.1449 0.1527 0.15 0 0.4 0.1518 0.1539 0.1521 0.1489 0.1491 0.1486 0.15 0 0.9 0.1534 0.1641 0.1532 0.1575 0.1571 0.1569 0.15 0 0.9 0.1491 0.1323 0.1397 0.1393 0.1400 0.1392 0.15 0 0.9 0.1552 0.1507 0.1540 0.1509 0.1531 0.1508 0.15 0 0.9 0.1540 0.1485 0.1482 0.1495 0.1540 0.1500 0.15 0.1 0 0.2502 0.2519 0.2538 0.2516 0.2566 0.2515 0.15 0.1 0 0.2510 0.2500 0.2504 0.2436 0.2520 0.2431 0.15 0.1 0 0.2517 0.2560 0.2498 0.2585 0.2484 0.2581 0.15 0.1 0 0.2499 0.2519 0.2476 0.2468 0.2483 0.2468 0.15 0.1 0.4 0.2500 0.2427 0.2521 0.2535 0.2526 0.2530 0.15 0.1 0.4 0.2505 0.2554 0.2521 0.2535 0.2531 0.2530 0.15 0.1 0.4 0.2502 0.2494 0.2474 0.2524 0.2465 0.2524 0.15 0.1 0.4 0.2651 0.2500 0.2531 0.2585 0.2601 0.2589 0.15 0.1 0.9 0.2523 0.2524 0.2533 0.2503 0.2502 0.2500 0.15 0.1 0.9 0.2438 0.2357 0.2446 0.2481 0.2506 0.2480 0.15 0.1 0.9 0.2567 0.2605 0.2584 0.2544 0.2492 0.2543 0.15 0.1 0.9 0.2492 0.2514 0.2454 0.2549 0.2444 0.2548 0.15 0.45 0 0.6031 0.6034 0.6061 0.6027 0.6003 0.6029 0.15 0.45 0 0.6004 0.5942 0.6038 0.6046 0.5946 0.6046 0.15 0.45 0 0.6031 0.6154 0.6109 0.6073 0.6078 0.6073 0.15 0.45 0 0.6026 0.6063 0.6013 0.6056 0.5980 0.6054 0.15 0.45 0.4 0.6008 0.6118 0.6008 0.6037 0.5996 0.6038 0.15 0.45 0.4 0.6100 0.5890 0.5939 0.6071 0.6021 0.6073 0.15 0.45 0.4 0.6057 0.6108 0.6041 0.6049 0.6023 0.6044 0.15 0.45 0.4 0.6064 0.6036 0.6059 0.6077 0.6068 0.6078 0.15 0.45 0.9 0.5995 0.5977 0.5997 0.5987 0.5989 0.5981

122

Table 35 – Continued Bias ES Covariance NOM EM PSM NNM RM MM 0.15 0.45 0.9 0.5942 0.5767 0.5968 0.5986 0.6014 0.5994 0.15 0.45 0.9 0.5999 0.6050 0.5987 0.5979 0.6021 0.5977 0.15 0.45 0.9 0.6047 0.6023 0.6019 0.6037 0.5995 0.6033

The final breakdown included Covariance, ES and Bias. From this breakdown, only in the Covariance = 0.90, ES = 0.00 and Bias = 0.00 was a Matching Method effect observed. Pairwise comparisons among the six Matching Methods for this design cell are presented in Table 36. It appears that high collinearity/covariance among the covariate

vector in the absence of either an effect or bias can create some level of variability in YD that is picked-up due to the overall power of the design.

Table 36 Case 3: Simple-simple Effect LS Means for the Matching Method When Covariance = 0.90, ES = 0.00 and Bias = 0.00 Matching Method LS Mean Stderr NOMe 0.0011 0.0023 EMb,d,e -0.0066 0.0048 PSMd,e -0.0017 0.0023 NNMc,e 0.0049 0.0024 RMa,b,d,e 0.0008 0.0023 MMa,e 0.0044 0.0048 Note: LS Means with the same letter are not statistically different, all p’s < .05

MM*Bias*ES*Correlation Interaction Post Hoc

Simple effect analysis of this interaction focused on the 3-way interaction involving MM*Bias*ES while holding Correlation constant. Refer to Table 37 for descriptive statistics. Results revealed no statistically significant 3-way (MM*Bias*ES) interactions but statistically significant Matching Method effects for Correlation = 0.75

(F(5,134895) = 2.50, p = 0.0286, HFL p = 0.0664) and for Correlation = 0.90

123

(F(5,134895) = 2.56, p = 0.0254, HFL p = 0.0593). Although the expected 3-way

MM*Bias*ES interaction was not observed further examination for a MM*Bias 2-way interaction was initiated by varying ES within the Correlation = 0.75 and 0.90 levels. A

MM*Bias interaction was observed in the ES = 0.45 condition (F(10,44955) = 2.93, p =

0.0011, HFL p = 0.0108) as well as a Matching Method effect (F(5,44955) = 3.35, p =

0.0050, HFL p = 0.0237). Whereas, no statistically significant MM*Bias interactions or simple-effects were noted for Correlation = 0.90. Post hoc examination continued to breakdown the 2-way MM*Bias interaction within Correlation = 0.75, ES = 0.45 by systematically examining each level of Bias. Within this breakdown, the only statistically significant Matching Method effect (F(5,14985) = 7.80, p < 0.0001, HFL p < 0.0001) noted was in the design cell combination: Correlation = 0.75, ES = 0.45 and Bias = 0.15.

Table 38 presents the pairwise comparisons among Matching Methods for this design cell.

124

Table 37 Case 3: LS Means for Bias*ES*Correlation by Matching Method Bias ES Correlation NOM EM PSM NNM RM MM 0 0 0 -0.0030 0.0022 -0.0006 -0.0010 -0.0040 -0.0011 0 0 0.75 0.0068 0.0128 0.0082 0.0090 0.0106 0.0086 0 0 0.9 -0.0003 0.0094 0.0048 0.0044 0.0043 0.0047 0 0 0.99 0.0055 0.0016 0.0068 0.0049 0.0042 0.0050 0 0 0 -0.0026 -0.0050 -0.0022 -0.0026 -0.0023 -0.0030 0 0 0.75 0.0068 0.0026 0.0008 0.0038 0.0070 0.0039 0 0 0.9 -0.0070 0.0080 -0.0021 -0.0027 -0.0006 -0.0032 0 0 0.99 -0.0085 -0.0100 -0.0040 -0.0011 -0.0079 -0.0005 0 0 0 -0.0016 -0.0037 -0.0049 0.0010 -0.0008 0.0009 0 0 0.75 0.0024 -0.0122 0.0003 0.0010 0.0010 0.0008 0 0 0.9 0.0020 -0.0053 -0.0011 0.0088 0.0014 0.0080 0 0 0.99 0.0020 -0.0053 -0.0011 0.0088 0.0014 0.0080 0 0.1 0 0.1022 0.1005 0.1024 0.1009 0.1008 0.1012 0 0.1 0.75 0.1011 0.1112 0.1067 0.1084 0.1071 0.1083 0 0.1 0.9 0.0966 0.1009 0.1053 0.1045 0.1062 0.1047 0 0.1 0.99 0.0919 0.1010 0.1043 0.1009 0.0980 0.1011 0 0.1 0 0.1020 0.1030 0.1001 0.1022 0.0996 0.1022 0 0.1 0.75 0.1000 0.1013 0.0976 0.1047 0.0998 0.1035 0 0.1 0.9 0.1040 0.1023 0.1055 0.1022 0.1008 0.1018 0 0.1 0.99 0.0982 0.0979 0.0971 0.1066 0.1047 0.1059 0 0.1 0 0.1019 0.1081 0.1020 0.1031 0.0964 0.1028 0 0.1 0.75 0.1046 0.0985 0.1058 0.1043 0.0995 0.1041 0 0.1 0.9 0.1046 0.1006 0.1022 0.0989 0.1075 0.0989 0 0.1 0.99 0.0971 0.0970 0.0975 0.1012 0.0967 0.1016 0 0.45 0 0.4412 0.4462 0.4392 0.4366 0.4390 0.4365 0 0.45 0.75 0.4375 0.4379 0.4360 0.4362 0.4360 0.4369 0 0.45 0.9 0.4408 0.4429 0.4444 0.4387 0.4402 0.4385 0 0.45 0.99 0.4386 0.4345 0.4406 0.4376 0.4361 0.4370 0 0.45 0 0.4399 0.4462 0.4475 0.4461 0.4421 0.4457 0 0.45 0.75 0.4355 0.4312 0.4425 0.4375 0.4417 0.4379 0 0.45 0.9 0.4418 0.4434 0.4408 0.4406 0.4453 0.4408 0 0.45 0.99 0.4412 0.4438 0.4416 0.4444 0.4448 0.4440 0 0.45 0 0.4365 0.4382 0.4376 0.4352 0.4401 0.4352 0 0.45 0.75 0.4388 0.4454 0.4392 0.4360 0.4336 0.4356 0 0.45 0.9 0.4430 0.4491 0.4431 0.4423 0.4405 0.4423 0 0.45 0.99 0.4358 0.4542 0.4464 0.4458 0.4450 0.4457 0.1 0 0 0.1079 0.1057 0.1015 0.1050 0.1089 0.1047 0.1 0 0.75 0.0908 0.0902 0.0905 0.0993 0.0937 0.0992

125

Table 37 – Continued Bias ES Correlation NOM EM PSM NNM RM MM 0.1 0 0.9 0.0992 0.0948 0.0934 0.1004 0.0978 0.0999 0.1 0 0.99 0.0976 0.0922 0.1003 0.0965 0.1024 0.0958 0.1 0 0 0.0999 0.1053 0.0993 0.1066 0.1050 0.1060 0.1 0 0.75 0.0998 0.1027 0.1031 0.1002 0.1005 0.1000 0.1 0 0.9 0.1041 0.0977 0.1091 0.1096 0.1015 0.1091 0.1 0 0.99 0.1010 0.1084 0.1003 0.0990 0.1001 0.0989 0.1 0 0 0.0989 0.1075 0.0993 0.0967 0.0973 0.0972 0.1 0 0.75 0.1074 0.1105 0.1092 0.1049 0.1063 0.1052 0.1 0 0.9 0.1091 0.1136 0.1001 0.1028 0.1011 0.1031 0.1 0 0.99 0.0980 0.1001 0.1007 0.1022 0.1032 0.1024 0.1 0.1 0 0.1965 0.1957 0.1996 0.2022 0.2020 0.2023 0.1 0.1 0.75 0.0908 0.0902 0.0905 0.0993 0.0937 0.0992 0.1 0.1 0.9 0.1980 0.1963 0.2033 0.2068 0.2036 0.2072 0.1 0.1 0.99 0.1914 0.1937 0.2025 0.2063 0.2075 0.2064 0.1 0.1 0 0.2102 0.1931 0.1986 0.2013 0.1989 0.2018 0.1 0.1 0.75 0.2044 0.1928 0.2079 0.2096 0.2011 0.2097 0.1 0.1 0.9 0.2086 0.1998 0.2064 0.2018 0.2036 0.2021 0.1 0.1 0.99 0.1978 0.2025 0.2039 0.2034 0.1998 0.2036 0.1 0.1 0 0.2001 0.2038 0.1969 0.2018 0.2045 0.2013 0.1 0.1 0.75 0.2054 0.2186 0.2107 0.2011 0.2008 0.2011 0.1 0.1 0.9 0.1995 0.2053 0.2082 0.2025 0.2055 0.2026 0.1 0.1 0.99 0.2017 0.2042 0.2024 0.2012 0.2037 0.2011 0.1 0.45 0 0.5505 0.5444 0.5455 0.5479 0.5469 0.5482 0.1 0.45 0.75 0.5500 0.5428 0.5452 0.5510 0.5442 0.5509 0.1 0.45 0.9 0.5428 0.5459 0.5511 0.5517 0.5523 0.5516 0.1 0.45 0.99 0.5504 0.5533 0.5564 0.5566 0.5552 0.5561 0.1 0.45 0 0.5549 0.5553 0.5571 0.5583 0.5599 0.5586 0.1 0.45 0.75 0.5467 0.5460 0.5401 0.5508 0.5504 0.5506 0.1 0.45 0.9 0.5553 0.5473 0.5528 0.5537 0.5543 0.5532 0.1 0.45 0.99 0.5455 0.5427 0.5536 0.5539 0.5514 0.5539 0.1 0.45 0 0.5508 0.5503 0.5468 0.5538 0.5543 0.5539 0.1 0.45 0.75 0.5468 0.5524 0.5468 0.5507 0.5543 0.5514 0.1 0.45 0.9 0.5490 0.5411 0.5496 0.5492 0.5482 0.5485 0.1 0.45 0.99 0.5516 0.5421 0.5562 0.5523 0.5524 0.5524 0.15 0 0 0.1497 0.1461 0.1492 0.1473 0.1458 0.1482 0.15 0 0.75 0.1452 0.1475 0.1528 0.1488 0.1584 0.1496 0.15 0 0.9 0.1426 0.1479 0.1455 0.1508 0.1457 0.1509 0.15 0 0.99 0.1583 0.1564 0.1562 0.1549 0.1539 0.1556 0.15 0 0 0.1535 0.1598 0.1532 0.1460 0.1445 0.1460

126

Table 37 – Continued Bias ES Correlation NOM EM PSM NNM RM MM 0.15 0 0.75 0.1522 0.1600 0.1549 0.1532 0.1509 0.1529 0.15 0 0.9 0.1500 0.1467 0.1506 0.1524 0.1449 0.1527 0.15 0 0.99 0.1518 0.1539 0.1521 0.1489 0.1491 0.1486 0.15 0 0 0.1534 0.1641 0.1532 0.1575 0.1571 0.1569 0.15 0 0.75 0.1491 0.1323 0.1397 0.1393 0.1400 0.1392 0.15 0 0.9 0.1552 0.1507 0.1540 0.1509 0.1531 0.1508 0.15 0 0.99 0.1540 0.1485 0.1482 0.1495 0.1540 0.1500 0.15 0.1 0 0.2502 0.2519 0.2538 0.2516 0.2566 0.2515 0.15 0.1 0.75 0.2510 0.2500 0.2504 0.2436 0.2520 0.2431 0.15 0.1 0.9 0.2517 0.2560 0.2498 0.2585 0.2484 0.2581 0.15 0.1 0.99 0.2499 0.2519 0.2476 0.2468 0.2483 0.2468 0.15 0.1 0 0.2500 0.2427 0.2521 0.2535 0.2526 0.2530 0.15 0.1 0.75 0.2505 0.2554 0.2521 0.2535 0.2531 0.2530 0.15 0.1 0.9 0.2502 0.2494 0.2474 0.2524 0.2465 0.2524 0.15 0.1 0.99 0.2651 0.2500 0.2531 0.2585 0.2601 0.2589 0.15 0.1 0 0.2523 0.2524 0.2533 0.2503 0.2502 0.2500 0.15 0.1 0.75 0.2438 0.2357 0.2446 0.2481 0.2506 0.2480 0.15 0.1 0.9 0.2567 0.2605 0.2584 0.2544 0.2492 0.2543 0.15 0.1 0.99 0.2492 0.2514 0.2454 0.2549 0.2444 0.2548 0.15 0.45 0 0.6031 0.6034 0.6061 0.6027 0.6003 0.6029 0.15 0.45 0.75 0.6004 0.5942 0.6038 0.6046 0.5946 0.6046 0.15 0.45 0.9 0.6031 0.6154 0.6109 0.6073 0.6078 0.6073 0.15 0.45 0.99 0.6026 0.6063 0.6013 0.6056 0.5980 0.6054 0.15 0.45 0 0.6008 0.6118 0.6008 0.6037 0.5996 0.6038 0.15 0.45 0.75 0.6100 0.5890 0.5939 0.6071 0.6021 0.6073 0.15 0.45 0.9 0.6057 0.6108 0.6041 0.6049 0.6023 0.6044 0.15 0.45 0.99 0.6064 0.6036 0.6059 0.6077 0.6068 0.6078 0.15 0.45 0 0.5995 0.5977 0.5997 0.5987 0.5989 0.5981 0.15 0.45 0.75 0.5942 0.5767 0.5968 0.5986 0.6014 0.5994 0.15 0.45 0.9 0.5999 0.6050 0.5987 0.5979 0.6021 0.5977 0.15 0.45 0.99 0.6047 0.6023 0.6019 0.6037 0.5995 0.6033

127

Table 38 Case 3: Simple-simple Effect LS Means for Matching Method When Correlation = 0.75, ES = 0.45 and Bias = 0.15 Matching Method LS Mean Stderr NOMa 0.6015 0.0027 EMb 0.5866 0.0047 PSMa,c 0.5982 0.0027 NNMa,d 0.6034 0.0027 RMa,c,d 0.5994 0.0027 MMa,d 0.6038 0.0027 Note: LS Means with the same letter are not statistically different, all p’s < .05

Cases 1, 2 and 3 Compared

Figure 10 presents the YD means of all the three cases by matching method after centering. The horizontal line represents the overall grand mean for all the three cases.

Case 3 estimates deviates the least from the grand mean line compared to Cases 1 and

2. Thus, Case 3 produces the best estimates by all matching method but MM matching method.

128

Figure10. Case 1, 2, and 3 Comparison: Pooled YD After Centering

Figure 10 depicts variability among the pooled and careful examination of the y-axis scale suggests that overall there is minimal variation among the six matching methods.

However analysis of the three cases individually do highlight several relevant findings, specifically the posy hoc analysis of the MM*Bias*ES*Correlation Interaction. Within this interaction was evidence that matching methods do deviate in their estimate of as a function of bias and effect size amount when the matching covariate vector is mixed

(C1, C2, D1, D2) and moderately correlated with Bias. Specifically PSM and RM outperform the other methods.

RQ 2: Do the matching procedures recover the selection/sampling bias?

RQ2 focuses on the ability of matching methods to recover the sampling/selection bias.

129

Thus, to investigate the matching procedures ability to recover experimentally induced selection bias. Each case scenario’s recovered bias (RB) statistics were computed using the mathematical formula: RB = (1 – URB)*100. URB represents unrecovered bias as defined in Chapter III. Results for RQ2 presents average effects of overall design cells and replications.

Case 1: Discrete Covariates

RB among the six matching methods ranged from 99.68% to 99.96% (pooled N =

27,000): NOM = 99.95%, EM = 99.68%, PSM = 99.85%, NNM = 99.86%, RM =

99.96% and MM = 99.92%. The radius matching method recovered most of the bias while the exact matching method recovered the least bias. The URB represents the inverse of the RB and is presented in Figure 11 by the match methods pooled overall design cells and replications. All Matching Methods slightly underestimated the average

ES. Table 39 presents URB descriptive statistics including the percentage of the recovered bias by each matching method for Case 1.

130

0.0035 0.0032

0.0030

0.0025

0.0020 URB 0.0015 0.0014 0.0015 0.0008 0.0010 0.0005 0.0004 0.0005

0.0000 NOM EM PSM NNM RM MM Matching Method

Figure 11. Case 1: Mean URB pooled over all conditions and Replications by Matching Method

Table 39 Case 1: URB Descriptive Statistics for Average of 27000 Replications by Matching Method Matching Method UBR Mean SD Percentage of RB NOM 0.0005 0.1474 99.95 % EM 0.0032 0.3483 99.68% PSM 0.0015 0.1459 99.85% NNM 0.0014 0.1442 99.86% RM 0.0004 0.1492 99.96% MM 0.0008 0.2617 99.92%

Case 2: Continuous Covariates

In Case 2, the matching methods recovered bias ranging from 99.91% to 99.97%.

The RB by each method was as follows: NOM = 99.91%, EM = 99.92%, PSM = 99.97%,

NNM = 99.91%, RM = 99.95% and MM = 99.91%. The PSM recovered most of the bias

(99.97%) followed by NOM, NNM and MM which recovered/reduced the least bias

(99.91%). The URB (see Figure 12) by all matching methods was almost zero (0)

131

implying effect size in YD was sufficient so was not underestimated or overestimated.

This suggests the matching procedure recovered 99.91% of the bias. Table 40 presents the unrecovered bias (URB) descriptive statistics including the percentage of the recovered bias by each method for Case 2 (N=108000).

0.0009 0.0009 0.0009 0.001 0.0008 0.0008 0.0005 0.0006 UBR 0.0003 0.0004 0.0002 0 NOM EM PSM NNM RM MM Matching Method

Figure 12. Case 2: Mean Unrecovered Bias (UBR) for Average of 10800 Replications by Match Method

Table 40 Case 2: UBR Descriptive Statistics for Average of 108000 Replications by Matching Method Matching Method UBR Mean SD Percentage of RB NOM 0.0009 0.1476 99.91 % EM 0.0008 0.2887 99.92% PSM 0.0003 0.1445 99.97% NNM 0.0009 0.1457 99.91% RM 0.0005 0.1453 99.95% MM 0.0009 0.1452 99.91%

Case 3: Mixed Covariates

In Case 3, the matching methods recovered bias ranging from 99.79% to 99.93%.

The RB by each method was as follows: NOM = 99.92%, EM = 99.93%, PSM = 99.88%,

NNM = 99.79%, RM = 99.89% and MM = 99.80%,. The Exact matching method

132

recovered most of the bias (99.93%) whereas the nearest neighbor method recovered/reduced the least bias (99.79%). The URB (Figure 13) by all methods was

almost zero (0) implying effect size in YD was sufficient so was not underestimated or overestimated indicating the matching procedure recovered 99.79% of the bias.

Table 41 displays the unrecovered/unreduced bias (UBR) descriptive statistics including the percentage of bias recovered by each method for Case 3 (N=108000).

0.0025 0.0021 0.0020 0.0020

0.0015 0.0012 UBR 0.0011 0.0010 0.0007 0.0007

0.0005

0.0000 NOM EM PSM NNM RM MM Matching Method

Figure 13. Case 3: Mean Unrecovered (UBR) for Average of 10800 Replications by Match Method

133

Table 41 Case 3: UBR Statistics for Average of 108000 Replications by Matching Method Matching Method UBR Mean SD Percentage of RB NOM 0.0008 0.1480 99.92 % EM 0.0007 0.2588 99.93% PSM 0.0012 0.1473 99.88% NNM 0.0021 0.1484 99.79% RM 0.0011 0.1478 99.89% MM 0.0020 0.1476 99.80%

RQ3: Given the parameters of the experimental design conditions, what is the prevalence of non-matches?

As presented in Chapter III, RQ3 focuses on the matching methods ability to match the treatment and control samples cases including evaluating the performance of matching methods. To address this research question three different outcomes were

examined: (a) coverage probability of YD , (b) width of CI, and (c) number of match failures as measured by degrees of freedom (df) were used to evaluate matching prevalence and overall performance of the six matching methods. Coverage probability is

proportion of 95% CI for which includes true YD and if ≥ 90% it was concluded the

statistics estimated via the matching method produced estimates and standard errors that were accurate and consistent. Width of is the 95% CI distance and df is the frequency of the number of matching failures where df < 398. Since both the treatment and control samples were simulated to have equal sample sizes nt = nc = n = 200 the

Ntotal = 400 or 398 df. Thus the average df over the 1000 replications by each matching

134

method is 398 the matching algorithm was deemed 100% complete. When df are less than 398, one or more of the matching replicates was deemed incomplete.

Case 1: Discrete Covariates

Table 42 presents the results of the matching methods prevalence and performance for Case 1. The results are for the complete experimental design (average of

27000 replications). As can be seen from this table, coverage probability was greater than

90% for all matching methods but EM, widths around YD were stable for all the matching methods but the MM and matching successes (mean df) were complete for all methods but PSM. These results suggest some of the statistical estimates estimated by the

EM and MM methods may be less stable given the observed narrow (EM) or wider (MM)

CI width. Individual t-test statistics testing the Ho: = YD is presented in Appendix D.

Table 42 Case 1: Matching Methods Matching Prevalence and Performance % Probability Matching Method Mean df Coverage* CI Width NOM Greater than 90% Stable 398.00 EM Less than 90% Narrower 398.00 PSM Greater than 90% Stable 397.90 NNM Greater than 90% Stable 398.00 RM Greater than 90% Stable 398.00 MM Greater than 90% Wider 398.00

* Percentage of coverage probability at which t-test mean lies in 95% CI

135

Case 2: Continuous Covariates

Table 43 presents the results of the matching methods prevalence and performance for Case 2. The results are for the complete experimental design (average of

108000 replications). As can be seen from this table, the coverage probability was greater

than 90% for all matching methods except EM implying that the YD estimates for EM may be inconsistent. However, CI widths around among the matching methods were deemed stable. Lastly, the mean df for EM, PSM, RM, and MM matching method were <

398 implying one or more of the matching replicates failed. Individual t-test statistics

testing the Ho: = YD is presented in Appendix E.

Table 43 Case 2: Matching Methods Matching Prevalence and Performance % Probability Matching Method df (Average) Coverage* CI Width of NOM Greater than 90% Stable 398.00 EM Less than 90% Stable 396.60 PSM Greater than 90% Stable 397.90 NNM Greater than 90% Stable 398.00 RM Greater than 90% Stable 397.90 MM Greater than 90% Stable 395.00

* Percentage of coverage probability at which t-test mean lies in 95% CI

Case 3: Mixed Covariates

Table 44 presents the results of the matching methods prevalence and performance for Case 3. The results are for the complete experimental design (average of

108000 replications). As can be seen from this table, the coverage probability was greater

136

398 implying one or more of the matching replicates failed. Individual t-test statistics

testing the Ho: = YD is presented in Appendix E.

Table 44 Case 3: Matching Methods Matching Prevalence and Performance Matching % Probability df (Average) Method Coverage* CI Width of NOM Greater than 90% Stable 398.00 EM Less than 90% Stable 397.80 PSM Greater than 90% Stable 397.90 NNM Greater than 90% Stable 398.00 RM Greater than 90% Stable 397.90 MM Greater than 90% Stable 395.00

* Percentage of coverage probability at which t-test mean lies in 95% CI

Summary

This chapter has presented the data analysis results focused on the three research questions as examined across the various design conditions hypothesized to affect the matching procedures in bias recovery; investigated the matching procedures ability to recover experimentally induced selection bias; and has studied the prevalence of non- matches given the parameters of the study experimental design conditions. The study results revealed statistically significant effects involving all of the design condition in one

137

form or another, e.g. as a main effect or in one or more interactions in all three simulated cases. Post hoc analyses focused on breaking down complex interactions into their constituent components. Overall the ability of all six matching methods to provide a valid

test of the central hypothesis: Yt - YC = YD was very good although meaningful differences did emerge relative to and the prevalence of matching failures.

138

CHAPTER V

SUMMARY, CONCLUSIONS, DISCUSSION, LIMITATIONS AND RECOMMENDATIONS

Summary

This study focused on attention on the general problem associated with finding or constructing a suitable comparison group when randomization of study participants to treatment groups cannot be accomplished ethically or experimentally. In research instances such as these, quasi-experimental designs have been utilized to answer the primary research question related to treatment outcome. However, the validity of a strong conclusion, e.g., that the treatment caused a change in the observed outcome, cannot be supported fully in the absence of a true experimental design. Thus, implementation of a quasi-experimental design requires critical and thoughtful consideration as to how best to control for the unknown selection bias inherent in nonrandomly constructed treatment and comparison groups. One popular design approach is to utilize a matched-comparison design (e.g., Joffe & Rosenbaum, 1999; Zhao, 2004;

Guo, Barth & Gibbons, 2006; Stuart & Rubin, 2007; Guo, 2010). In recent decades several different alternatives methods for matching have been discussed in the literature

(e.g., Rosenbaum and Rubin, 1983; Rosenbaum & Rubin, 1985a, 1985b; Rubin, 1998;

Rubin & Thomas, 2000; Rajeev & Wahba, 2002; Ho, Imani, King, & Stuart, 2006, 2007;

Stuart, 2010; Guo & Fraser, 2010). This study continues this line of inquiry through its

139

investigation of the construction of a comparison group via different matching methods to explore further three research questions:

RQ 1: What design conditions affect the matching procedures in bias

recovery?

RQ 2: Do the matching procedures recover the selection/sampling bias?

RQ3: Given the parameters of the experimental design conditions, what is

the prevalence of non-matches?

The study’s experimental conditions were: type and number of covariates {Case1:

D1 – D4; Case 2: C1 – C4; Case 3: D1, D2, C1, C2}; effect size {0.00, 0.10, 0.45}; amount of selection bias {0.00, 0.10, 0.15}; covariate covariance/collinearity {0.00, 0.40,

0.90}; correlation of the covariate vector with bias {0.00, 0.40, 0.90, 0.99} and matching methods {NOM, EM, PSM, NNM, RM, and MM}. One thousand replicates were simulated from theoretical population covariance matrices corresponding to the fore- mentioned experimental conditions. In total, Case 1 simulated 27,000 unique datasets distributed among 27 design cells and Case 2 and 3 each simulated 108,000 unique datasets distributed among 108 design cells. Various analytical methods were utilized, including (a) MANOVA with univariate repeated measures post hoc analyses directed towards a breakdown of statistically significant interactions and simple effects as described by Winer (1971) and (b) descriptive and graphical data summaries to answer the three study questions.

140

Conclusions

This section is organized to follow the study research questions and is thus presented in three parts. Part one discusses the design conditions that affect matching

   procedures bias recovery. It focuses on the test of the central hypothesis: Yt - YC = YD .

Part two discusses the matching procedures ability to recover experimentally induced selection bias. Part three discusses coverage probability, CI width and the prevalence of non-matches given the parameters of the study experimental design conditions.

RQ 1: What design conditions affect the matching procedures in bias recovery?

The results of the MANOVA analysis conducted to examine if the revealed the following. For Case 1, there was a MM*Bias*ES 3-way interaction. Post hoc analysis indicated Matching Method differences in in the presence of Bias (0.15) only when

ES = 0.00 and pooled over Covariance. Second, differences were observed among the

Matching Methods when pooled over Covariance and ES when Bias = 0.15. Specifically in the former, the EM seemed to overestimate relative to the other Matching Methods suggesting that this matching method may be susceptible to influences in sampling bias when no true ES is present. In the latter, again the EM overestimated in the presence of Bias, e.g., 0.15, even when pooling over all ES and Covariance conditions. Thus in the discrete case, caution should be taken in the use of EM in so far as the results of this study suggest this matching method may overestimate .

141

Case 2 examined all continuous covariates as matching variables. MANOVA results were considerably than was the situation in Case 1. Here they involved the breakdown of three separate 4-way interactions (MM*Bias*ES*Covariance,

MM*Bias*ES*Correlation, MM*ES* Covariance*Correlation). Although the 5-way interaction was not statistically significant, the presence of three of the four possible 4- way interactions indicates that all of the design factors in some way do influence the

 estimate of YD . Post hoc analysis across all three of the 4-way interactions consistently indicates problems with the EM. Across several analyses, this method underestimated

when there was no ES but overestimated in the presence of an effect size effect relative to the other methods over varying levels of Bias, ES Covariance and Correlation.

Moreover, this general under-overestimation reversed under the design conditions with high and very high levels of correlation with bias and or covariance among the covariate vector. Thus, in the absence of an anticipated true effect size, EM cannot be recommended as a matching method when the covariate vector is comprised of all continuous variables.

Case 3 examined a mixture of both discrete and continuous covariates as matching variables. Once again, MANOVA results were considerably more complex than in Case 1 and differed slightly from Case 2. Analyzes and findings involved the breakdown of two separate 4-way interactions (MM*Bias*ES*Covariance,

MM*Bias*ES*Correlation) and one 3-way interaction (MM* Covariance*Correlation).

Post hoc findings indicate that under high and very high levels of correlation between the covariate vector and Bias the EM underestimates regardless of the amount of Bias

142

 and ES. Moreover, NOM begins to underestimate YD as well, especially in the extreme condition where Correlation = 0.99. An unanticipated finding from Case 3 was the variability among and within the six matching methods in the null case. This was when

Bias and ES were both 0.00 but as a set the covariate vector was highly collinear

(Covariance = 0.90). The standard errors for observed for the EM and MM were twice the magnitude of the other matching methods. Finally, and perhaps most importantly, Case 3 found that when Bias = 0.15, ES = 0.10 and Correlation = 0.75 three of the matching methods underestimated : EM, PSM and MM. Moreover, and consistent with the findings of the other two cases, EM evidences substantially more variation within with standard error estimates often twice the magnitude of the other matching methods.

RQ 2: Do the matching procedures recover the selection/sampling bias?

Conclusions related to the ability of the six matching methods bias recovery were presented in summary form. Overall the differences among the matching methods were very small and overall quite high, however differences were observed. In Case 1, RM and

NOM were superior, recovering the bias over 99.9% of the time, whereas EM was least effective in bias recovery. In Case 2, two methods recovered over 99.5% of the bias:

PSM and RM, although all of the methods were over 99.90%. Lastly, Case 3 presents somewhat of a combined picture of Case1 and 2. Although EM and NOM were superior to the other methods, RM in particular was considerably more variable with a standard deviation about twice as large. Although PSM and RM were not quite as good as NOM

143

and EM, the former were considerably better relative to NNM and MM. This suggests that PSM or RM be first considered first as a matching method in situations with a mixture of continuous and discrete covariates.

RQ3: Given the parameters of the experimental design conditions, what is the prevalence of non-matches?

Conclusions related to the coverage probability, CI width and matching failure were presented in summary form. Examination among all three outcomes over all three cases suggests that either PSM or NNM provide the best matching relative to coverage probability, CI width and prevalence of non-matches.

Overall, this study indicates that study design conditions do impact various

   outcomes relative testing the general hypothesis: Yt - YC = YD and in the discrete and continuous covariate cases there is compelling evidence not to use EM or NOM. In the mixed case MM can be added to the “do not use” list. However, deciding which of the propensity-based matching procedures is best (PSM, NNM, RM) may simply be left to the researcher’s preference and future research to determine.

Discussion

This study’s findings pertaining to matching methods ability to reduce selection bias, the choice of matching technique and complications of incomplete matches are consistent with other study findings in the matching literature e.g., Rosenbaum & Rubin

(1985a, 1985b); Rajeev & Wabha (1999, 2000, 2002); Rubin & Thomas (2000); Zhao

(2004); Guo, Barth & Gibbions (2005); and Baser (2006). As previously documented, differences exist between the six matching method’s ability to recover experimentally

144

induced bias. Specifically, differences were noted in matching success rates for all but random sampling and nearest neighbor methods. Propensity based matching methods seem to work best with modest covariance /Collinearity (e.g., 0.40) irrespective of correlation between the bias amount and covariate group. Hence, social science researchers need to consider multiple factors when employing a propensity-based or exact matching procedure in quasi-experimental designs.

For example in this study, the manipulation of bias, effect size, covariance and

 correlation influences YD estimates in different ways. In Case 1, estimates were different when Bias was 0.15 and ES was 0.00 and pooled over Covariance. In Case 2, over several analyses, estimates estimated via EM method were underestimated when there was no ES but overestimated when there was an effect size effect relative to the other methods when varying levels of Bias, ES Covariance and Correlation. In Case 3, in the presence of high and very high levels of correlation between the covariate vector and

Bias, the EM method underestimates regardless of the amount of Bias and ES, whereas the NOM method underestimates when Correlation is extremely high (0.99).

Also in Case 3 when Bias = 0.15, ES = 0.10 and Correlation = 0.75 the matching methods

EM, PSM and MM underestimated implying the bias or effect size recovered were overcompensated.

Type of covariates has implications on how the matching methods perform in bias recovery. For instance, in Case 1 (discrete covariates) where the matching covariates were by design categorical variables, the EM method was less effective relative to other

145

matching methods because the covariates had difference levels which essentially affected or limited the distribution of the matching covariates in the matching process due to covariates homogeneity or duplication. In Case 2 (continuous covariates), an imposition of some tolerance limit in order to secure enough matched cases impacts some matching methods such as EM, PSM, RM and MM. In Case 3 (mixed covariates), where covariates matching improves as a result of combining continuous and categorical variables, propensity score matching and radius matching based on the propensity scores performs better in matching cases.

Pertaining to the specific effects of covariate collinearity/covariance, the matching

 methods as measured by YD estimates seem to perform optimally when collinearity/covariance is modest at 0.40, 0.90 and covariate correlation with Bias is moderate at 0.75 suggesting presences of collinearity and or correlation among matching covariates have implications for matching methods.

Limitations and Recommendations

While this study offers a comprehensive examination of four design factor’s influence on six different matching methods ability to create a comparison group that can recover experimentally induced bias, it was nevertheless designed, and therefore limited in its scope. This dissertation examined three common social science research scenarios for case matching. These include situations where discrete, continuous, or both types of covariates are employed which are thought to relate to design conditions hypothesized to affect matching procedures’ ability to recovery or reduce bias. Future research must be conducted that replicates and extends the parameters manipulated in this study. While

146

there are undoubtedly dozens of ways this study could be expanded, this author suggests consideration of the following.

First, the raw data was simulated using three different forms of distributions, normal, binomial and multinomial. While many demographic variables do conform to the binomial or multinomial (Case 1) the normality restriction placed on the continuous covariates in Case 2 and 3 should be relaxed and at least skewed or kurtotic symmetrical distributions considered as well as censored distributions or even mixed distribution, e.g., normal with Poisson or Log linear. Particularly since these distributions are common in social and educational databases where the matched comparison design is a viable design choice.

Second, the number of matching covariates was limited to four covariates. In future research, the number of covariates could be examined in a step-wise manner; from

1 to p, or until there is no measurable change in the primary outcome decision.

Three, sample size to population size ratio should be examined. Often researchers are faced with considerably smaller samples of intact groups and/or population sizes.

Similar to varying the number of covariates incrementally, the sample size: population size ratio could be examined incrementally as well. This is particularly relevant if a matching method may result in matching failures, e.g., PSM, RM, MM.

Fourth, the matching tolerance limits substantially affects both the number of duplicate matches and the number of matching failures, especially in methods based on an initial propensity score or the use of any of the continuous covariate matching methods in exact matching. A study could be conducted that systematically investigates these

147

limits so that either optimal ranges or further conditions that influence match failures and duplicate matches could be identified.

148

REFERENCES

Abadie, A., Drukker, D., Herr, J. L., & Imbens, G.W. (2004). Implementing matching

estimators for average treatment effects in Stata. The Stata Journal, 4(3), 290–

311.

Abadie, A., & Imbens, G.W. (2001). Large sample properties of matching estimators for

average treatment effects. Econometrica, 74(1), 235–267.

Althauser, R., & Rubin, D. (1971). The computerized construction of a matched sample.

American Journal of Sociology, 76, 25-346.

Arceneaux, K., Gerber, A.S., & Green, D.P. (2006). Comparing experimental and

matching methods using a large-scale voter mobilization experiment. Political

Analysis, 4, 37-62.

Baser, O. (2006). Too much ado about propensity score models? Comparing method of

propensity score matching. International Society for Pharmacoeconomics and

Outcomes Research (ISPOR), 9(6), 377-385.

Becker, S. O., & Ichino, A. (2002). Estimation of average treatment effects based on

propensity scores. The Stata Journal, 2(4), 358–377.

Bloom, S.H., Hayes, L.R., & Black, A, R. (2005). Using Covariates to Improve

Precision: Empirical Guidance for Studies That Randomize Schools to Measure

the Impacts of Educational Interventions. Manpower Demonstration Research

Corporation (MDRC) Working Papers on Research Methodology. New York.

Bowers J, & Hansen, B. (2005). Attributing effects to a cluster randomized get out the

vote campaign. Department of Statistics, University of Michigan. 149

Http://wwwpersonal.umich.edu/~jwbowers/PAPERS/bowershansen10TechReoprt

.pdf

Caliendo, M., (2006). The microeconometric evaluation of labor market policies. Verlag

Berlin Heidelberg: Springer.

Caliendo, M., & Kopeinig, S. (2008). Some practical guidance for the implementation

of propensity score matching. Journal of Economic Surveys. 22(1), 31-72.

Chan, S.F., Macaskill, P., Irwig, L., & Walter, S.D (2004). Adjustment for baseline error

in randomized controlled trials induces bias. Controlled Clinical Trials, 25 408-

416.

Chatterjee, A., & Segev, A . (1992). Resolving data heterogeneity in scientific statistical

databases in H. Hinterberger & J.C. French, editors, Scientific and Statistical

Database Management, 145-159.

Chatterjee, A., & Segev, A (1994). Supporting statistics in extensible databases: A case

study in H. Hinterberger & J.C. French, editors, Scientific and Statistical

Database Management, 54-63.

Christakis, N.A., & Iwashyna, T.I (2003). The health impact of healthcare on families: A

matched cohort study of hospice use by decedents and mortality outcomes in

surviving widowed spouses, 7, 465-475.

Cochran, W.G. (1968). The effectiveness of adjustment by subclassification in removing

bias in observational studies. Biometrics, 24, 295-313.

Cochran, W.G., & Rubin (1973). Controlling bias in observational studies: A review.

The Indian Journal of Statistics, Series A, (35),417 – 446.

150

Collins, L.M., Schafer, J.L., & Kam, C.M (2001). A comparison of inclusive and

restrictive missing data strategies in modern missing data procedures.

Psychological Methods, 6, 333-351.

D’Agostino, R. B. (1998). Tutorial in biostatistics: Propensity score methods for bias

reduction in the comparison of a treatment to a non-randomized control group.

Statistics in Medicine, 17, 2265-2281.

Diamond, A. & Sekhon, J.S. (2005). Generic matching for estimating causal Effects: A

General Multivariate Matching Method for Achieving Balance in Observational

Studies. Technical Report, Department of Political Science, UC Berkeley. URL

http://sekhon.berkeley. du/papers/GenMatch.pdf.

Dawid, A.P.(1979). Conditional independence in statistical theory. Journal of the Royal

Statistical Society, Series B, 1-31.

Delin, T.R. & Rubin, D.B (1995). A method for calibrating false-match rates in record

linkage. JASA. 90(430):694–707.

Dehejia, R.H. & Wahba, S. (1999). Estimating and using propensity scores with partial

missing data. Journal American Statistical Association, (94), 749-759.

Dehejia, R.H. & Wahba, S. (2000). Causal effects in non experimental studies: Re-

evaluating the evaluation of training programs. Journal American Statistical

Association, 4 (10), 53-62.

Dehejia, R.H. & Wahba, S. (2002). Propensity score matching for non experimental

causal studies. Review of Economic Statistics, 84(1), 151-161.

Denk., M & Hackl, P. (2003). Data integration and record matching: An Austrian

contribution to research in official statistics. Austrian Journal of Statistics, 22 (4),

151

305-321.

Diprete, T.A., Engelhart, H. (2004). Estimating causal effects with matching methods in

the presence and absence of bias cancellation. Sociological Methods and

Research, 32 (4), 501-528.

Fan X., Felsovalyi A., Sivo S. A., & Keenan S.C., (2001). SAS for Monte Carlo studies. A

guide for quantitative researchers. SAS Institute Inc. Cary, NC.

Feng, W.W., Lilly, E., & Jun, Y. (2006). A Method/macro based on propensity score and

Mahalanobis distance to reduce bias in treatment comparison in observational

study.

http://www.lexjansen.com/pharmasug/2006/publichealthresearch/pr05.pdfIn.

Fillenbaum, G., Hybels, C., Pieper, C., Konrad, T., Burchett, B., & Blazer, D. (2006).

Provider characteristics related to antidepressant use in older people. Journal of

the American Geriatrics Society, 54(6), 942–949.

Frolich, M. (2002). Program Evaluation and Treatment effect Choice. Springer

Publications.

Guo, S., Barth, R.P., & Gibbons, C. (2006). Propensity score matching strategies for

evaluating abuse services for child welfare clients. Children and Youth Services

Review, 28(4), 359-383.

Guo, S., & Fraser, M.W. (2010). Propensity score analysis: Statistical methods and

applications. Sage Publications.

Harris, J. (2001). A primer of multivariate statistics, 3rd edition.Lawrence Erlbaum

Associates, Mahway Publishers, New Jersey.

152

Imati, K. (2004). Do get out vote calls reduce turnout? The importance of statistical

methods for field experiments. American Political Review.

Imbens, G. W. (2000). The role of the propensity score in estimating dose-response

functions. Biometrika, 87(3), 706-710.

Hastie, T., Tibshirani, R., & Friedman, J. (2001). The elements of statistical learning:

Data mining, inference, and prediction. Springer, New York.

Heckman, J.J., Ichimura, H., Smith, J., & Todd, P (1997). Characterizing selection bias

using experimental data. Econometrika, 64 (5), 1017-1098.

Heckman, J.J., Ichimura, H., Smith, J., & Todd, P (1998), Matching as an econometric

evaluation estimator. Review of Economic Studies, 66, 261-294.

Hirano, K., & Imbems, G.W. (2001). Estimation of causal effects using propensity score

weighting: An application to data on right heart catheterization. Health Services

and Outcomes Research Methodology, (2), 259-278.

Ho, D. E., Imai, K., King, G., & Stuart, E. A. (2006). MatchIt: Nonparametric

preprocessing for parametric causal inference. Software for using matching

methods in R. Available athttp://gking.harvard.edu/matchit

Ho, D., Imai, K., King, G., & Stuart, E. (2007), Matching as nonparametric preprocessing

for reducing model dependence in parametric causal inference. Political Analysis

15(3), 199-236.

Holl, P. W. 1986. Statistics and causal inference. Journal of the American Statistical

Association, 81 (396), 945–960.

153

Hong, G., & Raudenbush, S. W. (2005). Effects of kindergarten retention policy on

children’s cognitive growth in reading and mathematics. Educational Evaluation

and Policy Analysis, 27(3), 205-224.

Hoy, A.W. (2000), Educational psychology in teacher education. Educational

Psychologist, 4 (4), 257-270.

Horvitz, D. G., & Thompson, D. J., (1952). A Generation of sampling without

replacement from a finite Universe. Journal of the American Statistical

Association, 47, 663-685.

Ibtesam, I (2006). The Effect of motivation, family environment and student

characteristics on academic achievement. Journal of Industrial Psychology. June

Issue.

Ichino, A., Mealli, F., & Nannicini, T. (2004). Sensivity of matching estimators to

unconfoundedness. An application to the effect of temporary work on future

employment. JEL Classifications, C2, C8, J6.

Joffe, M.M., & Rosenbaum, P.R. (1999). Propensity scores. American Journal of

Epidemiology, 150 (4), 327-333.

Kadane, J.B. (1978). Some statistical problems in merging data files in 1978

compendium of tax research. US Department of the Treasury (Reprinted in

Journal of Official Statistics. 17(3), 423-433, 2001.

Kawabata, H., Tran, M., & Hines, P. (2004). Using SAS to match cases for control

studies, Proceedings of the Twenty-Nineth Annual SAS_Users Group

International Conference, Montreal, Canada: SAS Institute Inc., 173-29.

154

Kelcey, B. (2011). Covariate selection in propensity scores using outcome proxies.

Multivariate Behavioral Research, 46: 453-476.

Kelcey, B. (2011). Multilevel propensity score matching within and cross schools. ERIC

ED 518897

Kleinbaum, D.G., Sullivan, K.M., & Barker, N.D. (2007). A Pocket Guide to

Epidemiology. Springer Science and Business Media, LLC. New York.

Kovar, J.G. & Whitridge, P.J. (1995). Imputation of business survey data in B. Cox et

al. editors, Business Survey Methods. John Wiley, New York.

Kurth, T., Walker, A. M., Glynn, R. J., Chan, K. A., Gaziano, J. M., & Berger, K.,

(2006). Results of multivariable logistic regression, propensity matching,

propensity adjustment, and propensity-based weighting under conditions of non

uniform effect. American Journal of Epidemiology, 163(3), 262-270.

Lalonde, R. (1986). Evaluating the econometric evaluations of training programs with

experimental data. American Economic Review, 76 (4), 604-620.

Little, R.J.A. & Rubin, D.B. (1987). Statistical analysis with missing data. John Wiley &

Sons, New York.

Lechner, M. (1999). Earning and employment effects of continuous off-the-job training

in East Germany after unification. Journal of Business Economic Statistics,1, 74-

90.

Lenzerini, M. (2002). Data Integration: A Theoretical perspective. PODS, 243-246.

Leuven, E., & Sianesi, B. (2003). psmatch2. Stata module to perform full Mahalanobis

and propensity score matching, common support graphing, & covariate

imbalance testing.Available at http://www1.fee.uva.nl/scholar/ mdw/leuven

155

McCaffrey, D. F., Ridgeway, G., & Morral, A. R. (2004). Propensity score estimation

with boosted regression for evaluating causal effects in observational studies.

Psychological Methods, 9(4), 403-425.

Millimet, D., & Tchernis, R. (2008). On the specification of propensity scores: With

applications to the analysis of trade olicies. Center for Applied Economics and

Policy, Working Paper. http://ssrn.com/abstract=918454.

Moriarity, C & Scheuren, F (2001). Statistical matching: A paradigm for assessing the

uncertainty in the procedure. Journal of Official Statistics, 17(3):407-422.

Morgan, S.L., & Harding, D.J. (2006). Matching estimators of causal effects: prospects

and pitfalls in theory and practice. Sociological Methods and Research, 35(1), 3-

60.

Neyman, J. (1935). Statistical problems in agricultural experiments. Journal of the Royal

Statistical Society, 2, 107-180.

Parsons, L.S. (2000). Using SAS software to perform a case-control match on propensity

score in an observational study. proceedings of the twenty-fifth annual SAS users

group International Conference, Indianapolis, Indiana: SAS Institute Inc., 225-25.

Parsons, L.S. (2001). Reducing bias in a propensity score matched-pair sample using

greedy matching techniques. Proceedings of the twenty-sixth annual SAS users

group International Conference, Long Beach, CA: SAS Institute Inc., 214-26.

Parsons, L.S. (2004). Performing a 1: N case-control match on propensity score.

Proceedings of the twenty-ninth annual SAS users group international conference,

Montreal, Canada, SAS Institute Inc., 165-29.

156

Pearl, J. (2009). Causality: models, reasoning and inference. Cambridge University

Press, Second Edition.

Perraillon M. C. (2007). Local and global optimal propensity score matching. SAS

Global Forum. Orlando, FL, 185-2007.

Perraillon M. C. (2006). Matching with propensity scores to reduce bias in observational

studies.19th Annual Conference Proceedings of SAS NESUG. Philadelphia, PA,

da13.

Pimparyon, P., Caleer, S.M. & Roff, S (2000). Educational environment, student

approaches to learning and academic achievement in a nursing school. Medical

Teacher, 22 (4), 327-333.

Quant, R. (1972). Methods for estimating switching regressions. Journal of the America

Statistical Association, 76(338), 306-310.

W.L. Rodgers & E.B. DeVol (1981). An evaluation of statistical matching. Journal

American Statistical Association, 128-132.

Radner, D.B., Allen, R., Jabine, T.B., Muller, H.J. (1980). Statistical Policy Working

Paper 5: Report on Exact and Statistical Matching Techniques. US Federal

Committee on Statistical Methodology, Office of Management and

Budget.Reynolds, W. M (1998). Measurement of academic self-concept in

college students. Journal of Personality Assessment, 22 (2), 233-240.

Ridgeway, G., McCaffrey, D., & Morral, A. (2006). TWANG: Toolkit for weighting &

analysis of nonequivalent groups. Software for using matching methods in R.

Available at http://cran.r-project.org/src/

157

Rivers, D. (2006). Sample matching: Representative sampling from internet panels.

Polimetrix. www.polimetric.com

Rosenbaum, P.R. & Rubin, D.B (1983). The central role of the propensity score in

observational studies for causal effects. Biometrika, 70, 41-45.

Rosenbaum, P.R. & Rubin, D.B (1984). Reducing bias in observational studies using

subclassification on the propensity score. Journal American Statistical

Association, 79, 516-524.

Rosenbaum, P. R., Rubin, D. B. (1985a). Constructing a Control Group Using

Multivariate Matched Sampling Methods That Incorporate the Propensity Scores.

The American Statistician. 39(1), 33-38.

Rosenbaum, P. R., Rubin, D. B. (1985b). The bias due to incomplete matching.

Biometrics, 41(1), 103-116.

Rosenbaum, P. R. (1986). Dropping out of high school in the United States: An

observational study. Journal of Educational Statistics. 11(3), 207-224.

Rosenbaum, P. R. (2002). Observational studies, 2nd editor. Springer, New York.

Rosenthal, R. (1994). Parametric measures of effect size. In H. Cooper & L. V. Hedges

(Eds.), The handbook of research synthesis (pp. 231-244). New York: Russell

Sage Foundation.

Ross, P.H, Lipsey, M.W, & Freeman, H. E., (2004). Evaluation: A systemic approach. 7th

edition. Sage Publications.

Roy, A.D. (1951). Some thoughts on the distribution of earnings. Oxford Economics

Papers 3, 135-146.

158

Rubin, D.B. (1973a). Matching to remove bias in observational studies. observational

studies. Biometrics, 29(1), 159-183.

Rubin, D.B. (1973b). The use of matched sampling and regression adjustment to remove

bias in observational studies. Biometrics, 29(2), 185-203.

Rubin, D.B. (1974). Estimating causal effects to treatments randomized and non-

randomized studies, Journal of Educational Psychology, 66, 688-701.

Rubin, D.B. (1977). Assignment of treatment group on the basis of a covariate. Journal

of Educational and Behavioral Statistics, 2(1), 1-26.

Rubin, D. B. (1979). Using multivariate matched sampling and regression adjustment to

control bias in observational studies. Journal of the America Statistical

Association, 74(366), 318-328.

Rubin, D. B. (1980). Bias reduction using Mahalanobis-metric matching. Biometrics,

36(2), 293-298.

Rubin, D. B. (1990). Formal mode of statistical inference for causal Effects. Journal of

Statistical Planning and Inference, 25(3), 279-292.

Rubin, D. B., & Thomas, N. (2000). Combining propensity score matching with

additional adjustments for prognostic covariates. Journal of the American

Statistical Association. 95(450)

Rubin, D. B., & Thomas, N. (1996). Matching using estimated propensity scores:

Relating theory to practice. Biometrics, 52(1), 249-264.

Rubin, D.B. (1997). Estimating causal effects from large data sets using propensity

scores. Annals of Internal Medicine, 127, 757-763.

159

Rubin, D. B. (1998). Estimation from nonrandomized treatment comparisons using

subclassification on propensity scores. Ann. Internal Medicine 8 757–763.

Rubin, D. (2001). Using propensity scores to help design observational studies:

Application to the tobacco litigation. Health Services and Outcomes Research

Methodology, 2, 169-188.

Rubin, D. B. (2004). Teaching inference for causal effects in experimental observational

studies. Journal of Educational and Behavioral Statistics, 29(3), 343-367.

Rubin, D.B. (2006). Matching sampling for causal effects. Cambridge University Press.

New York.

Salganik, M.J. (2006). Variance estimation design effects, and Sample size calculations

for Respondent-Driven Sampling. Journal of Urban Health: Bulletin of the New

York Academy of Medicine, 83(7).

Sekhon, J.S. (2004). The varying role of voter information across democratic societies.

working paper. http://sekhon.berkely.edu/papers/sekhoninformation.pdf.

Sekhon, J.S. (2011). Multivariate and propensity score matching software with

automated balance optimization: The matching package for R. Journal of

Statistical Software, 42 (7), 1–52

Skinner, C.J (1986). Design Effects of Two-Stage Sampling. Journal of the Royal

Statistical Society. Series B (Methodological), 48(1), 89-99.

Shadish, W.R., Clark, M.H., & Steiner, M.P. (2008). Can nonrandomized experiments

yield accurate answers? A randomized experiment comparing random and

nonrandom assignments. Journal of the American Statistical Association.

103(484).

160

Shadish, W.R., Cook,T.D., & Campbell, D.T. (2002). Experimental and quasi-

experimental designs for generalized causal inference. Houghton Mifflin

Company.

Smith, H, L. 1997. Matching with multiple controls to estimate treatment effects in

observational studies. Sociological Methodology, 305–353.

Stuart, E.A. (2010). Matching methods for causal inference: A review and a look

forward. Statistical Science 25(1): 1-21.

Stuart, E.A. & Rubin, D.B. (2007). Best practices in quasi-experimental designs:

Matching methods for causal inference. Chapter 11 (pp. 155-176) in Best

Practices in Quantitative Social Science. J. Osborne (Ed.). Thousand Oaks, CA:

Sage Publications

Tebes, J. K., Feinn, R., Verploeg, J. J., Chinman, M. J., Shepard, J., Brabham, T. (2007).

Impact of a positive youth development program in urban after-school settings on

the prevention of adolescent substance use. Journal of Adolescent Health, 41(3),

239-247.

Xing, S.G., & Rosenbaum, P. R. (1993). Comparison of multivariate matching methods:

Structures, distances, and algorithms. Journal of Computational and Graphic

Statistics, 2 (4), 405-420.

Wainer, H. (1989). Eelworms, bullet holes, Geraldine Ferraro: Some problems with

statistical adjustment and some solutions. Journal of Educational Statistics

Summer 1989, 14(2), 121-140.

Waller, J.L., Brantley, V.C., & Podolsky, R.H. (2002). Where’s the match. SAS Institute

Inc.

161

Winer, B.J. (1971). Statistical principles in experimental design, (2nd Ed.). New York:

McGraw Hill.

Winship. C., & Morgan, S. (1999). The estimation of causal effects from observational

data. Annual Review of Sociology, 25, 569-707.

Zeegers, P. (2004) Student learning in higher education: A path analysis of academic

achievement in Science. Higher Education Research and Development, 23 (1) 35-

56.

Zhao, Z., (2004). Using matching to estimate treatment effects; data requirements,

matching metrics, and Monte Carlo evidence. The Review of Economics and

Statistics, 86(1), 91-107.

162

Appendix A

Case 1: Discrete Covariates Descriptive Statistics

163

Case 1: Discrete Covariates YD Descriptive Statistics Bias ES Covariance Matching Method Mean SD 0 0 0 Random Sample 0.0010 0.1458 0 0 0 Exact Matching 0.0124 0.3498 0 0 0 Propensity Score Matching 0.0003 0.1544 0 0 0 Nearest Neighbor Matching 0.0013 0.3406 0 0 0 Radius Matching 0.0020 0.1448 0 0 0 Mahalanobis Metric Matching 0.0052 0.2601 0 0 0.40 Random Sample 0.0024 0.1446 0 0 0.40 Exact Matching 0.0023 0.3489 0 0 0.40 Propensity Score Matching 0.0002 0.1482 0 0 0.40 Nearest Neighbor Matching 0.0068 0.3500 0 0 0.40 Radius Matching 0.0024 0.1439 0 0 0.40 Mahalanobis Metric Matching 0.0053 0.2643 0 0 0.90 Random Sample 0.0018 0.1422 0 0 0.90 Exact Matching 0.0054 0.3455 0 0 0.90 Propensity Score Matching 0.0062 0.1442 0 0 0.90 Nearest Neighbor Matching 0.0047 0.3369 0 0 0.90 Radius Matching 0.0008 0.1390 0 0 0.90 Mahalanobis Metric Matching 0.0090 0.2589 0 0.1 0 Random Sample 0.1022 0.1449 0 0.1 0 Exact Matching 0.0972 0.3518 0 0.1 0 Propensity Score Matching 0.0979 0.1563 0 0.1 0 Nearest Neighbor Matching 0.1089 0.3389 0 0.1 0 Radius Matching 0.0954 0.1516 0 0.1 0 Mahalanobis Metric Matching 0.0954 0.1516 0 0.1 0.40 Random Sample 0.0961 0.1419 0 0.1 0.40 Exact Matching 0.0972 0.3568 0 0.1 0.40 Propensity Score Matching 0.0960 0.1503 0 0.1 0.40 Nearest Neighbor Matching 0.0970 0.3425 0 0.1 0.40 Radius Matching 0.0955 0.1531 0 0.1 0.40 Mahalanobis Metric Matching 0.0965 0.2570 0 0.1 0.90 Random Sample 0.0989 0.1448 0 0.1 0.90 Exact Matching 0.0956 0.3435 0 0.1 0.90 Propensity Score Matching 0.1034 0.1496 0 0.1 0.90 Nearest Neighbor Matching 0.0979 0.3484 0 0.1 0.90 Radius Matching 0.1045 0.1446 0 0.1 0.90 Mahalanobis Metric Matching 0.0973 0.2663 0 0.45 0 Random Sample 0.4482 0.1510 0 0.45 0 Exact Matching 0.4630 0.3495 0 0.45 0 Propensity Score Matching 0.4512 0.1388 0 0.45 0 Nearest Neighbor Matching 0.4477 0.3542 0 0.45 0 Radius Matching 0.4488 0.1604 0 0.45 0 Mahalanobis Metric Matching 0.4502 0.2634 0 0.45 0.40 Random Sample 0.4357 0.1540 0 0.45 0.40 Exact Matching 0.4368 0.3568 0 0.45 0.40 Propensity Score Matching 0.4402 0.1569 0 0.45 0.40 Nearest Neighbor Matching 0.4409 0.3377 0 0.45 0.40 Radius Matching 0.4372 0.1528 0 0.45 0.40 Mahalanobis Metric Matching 0.4425 0.2645

164

Bias ES Covariance Matching Method Mean SD 0 0.45 0.90 Random Sample 0.4351 0.1500 0 0.45 0.90 Exact Matching 0.4346 0.3522 0 0.45 0.90 Propensity Score Matching 0.4389 0.1505 0 0.45 0.90 Nearest Neighbor Matching 0.4350 0.3594 0 0.45 0.90 Radius Matching 0.4359 0.1505 0 0.45 0.90 Mahalanobis Metric Matching 0.4371 0.2718 0.1 0 0 Random Sample 0.0964 0.1432 0.1 0 0 Exact Matching 0.1200 0.3505 0.1 0 0 Propensity Score Matching 0.0997 0.1490 0.1 0 0 Nearest Neighbor Matching 0.0956 0.3439 0.1 0 0 Radius Matching 0.1013 0.1418 0.1 0 0 Mahalanobis Metric Matching 0.0983 0.2632 0.1 0 0.40 Random Sample 0.1023 0.1380 0.1 0 0.40 Exact Matching 0.1073 0.1372 0.1 0 0.40 Propensity Score Matching 0.1037 0.1372 0.1 0 0.40 Nearest Neighbor Matching 0.0997 0.2647 0.1 0 0.40 Radius Matching 0.1013 0.1461 0.1 0 0.40 Mahalanobis Metric Matching 0.0996 0.2647 0.1 0 0.90 Random Sample 0.0996 0.1433 0.1 0 0.90 Exact Matching 0.1152 0.3464 0.1 0 0.90 Propensity Score Matching 0.1088 0.1447 0.1 0 0.90 Nearest Neighbor Matching 0.0959 0.3377 0.1 0 0.90 Radius Matching 0.1035 0.1413 0.1 0 0.90 Mahalanobis Metric Matching 0.0965 0.2604 0.1 0.1 0 Random Sample 0.2054 0.1478 0.1 0.1 0 Exact Matching 0.2019 0.3535 0.1 0.1 0 Propensity Score Matching 0.2014 0.1451 0.1 0.1 0 Nearest Neighbor Matching 0.1966 0.3260 0.1 0.1 0 Radius Matching 0.2074 0.1493 0.1 0.1 0 Mahalanobis Metric Matching 0.2016 0.2549 0.1 0.1 0.40 Random Sample 0.1980 0.1447 0.1 0.1 0.40 Exact Matching 0.1962 0.3638 0.1 0.1 0.40 Propensity Score Matching 0.1982 0.1436 0.1 0.1 0.40 Nearest Neighbor Matching 0.2007 0.3432 0.1 0.1 0.40 Radius Matching 0.2004 0.1367 0.1 0.1 0.40 Mahalanobis Metric Matching 0.1977 0.2550 0.1 0.1 0.90 Random Sample 0.2106 0.1421 0.1 0.1 0.90 Exact Matching 0.1976 0.3464 0.1 0.1 0.90 Propensity Score Matching 0.2069 0.1473 0.1 0.1 0.90 Nearest Neighbor Matching 0.1973 0.3458 0.1 0.1 0.90 Radius Matching 0.2098 0.1574 0.1 0.1 0.90 Mahalanobis Metric Matching 0.2015 0.2595 0.1 0.45 0 Random Sample8 0.5507 0.1631 0.1 0.45 0 Exact Matching 0.5523 0.3592 0.1 0.45 0 Propensity Score Matching 0.5457 0.1315 0.1 0.45 0 Nearest Neighbor Matching 0.5537 0.3562 0.1 0.45 0 Radius Matching 0.5490 0.1632 0.1 0.45 0 Mahalanobis Metric Matching 0.5636 0.2679

165

Bias ES Covariance Matching Method Mean SD 0.1 0.45 0.40 Random Sample 0.5565 0.1562 0.1 0.45 0.40 Exact Matching 0.5429 0.3482 0.1 0.45 0.40 Propensity Score Matching 0.5499 0.1595 0.1 0.45 0.40 Nearest Neighbor Matching 0.5491 0.3492 0.1 0.45 0.40 Radius Matching 0.5545 0.1540 0.1 0.45 0.40 Mahalanobis Metric Matching 0.5506 0.2735 0.1 0.45 0.90 Random Sample 0.5512 0.1621 0.1 0.45 0.90 Exact Matching 0.5495 0.3416 0.1 0.45 0.90 Propensity Score Matching 0.5491 0.1496 0.1 0.45 0.90 Nearest Neighbor Matching 0.5543 0.3527 0.1 0.45 0.90 Radius Matching 0.5476 0.1562 0.1 0.45 0.90 Mahalanobis Metric Matching 0.5482 0.2673 0.15 0 0 Random Sample 0.1480 0.1386 0.15 0 0 Exact Matching 0.1686 0.3482 0.15 0 0 Propensity Score Matching 0.1524 0.1385 0.15 0 0 Nearest Neighbor Matching 0.1487 0.3311 0.15 0 0 Radius Matching 0.1479 0.1391 0.15 0 0 Mahalanobis Metric Matching 0.1471 0.2439 0.15 0 0.40 Random Sample 0.1471 0.1386 0.15 0 0.40 Exact Matching 0.1574 0.3482 0.15 0 0.40 Propensity Score Matching 0.1519 0.1385 0.15 0 0.40 Nearest Neighbor Matching 0.1628 0.3311 0.15 0 0.40 Radius Matching 0.1451 0.1391 0.15 0 0.40 Mahalanobis Metric Matching 0.1617 0.2439 0.15 0 0.90 Random Sample 0.1482 0.1367 0.15 0 0.90 Exact Matching 0.1502 0.3484 0.15 0 0.90 Propensity Score Matching 0.1524 0.1407 0.15 0 0.90 Nearest Neighbor Matching 0.1569 0.3468 0.15 0 0.90 Radius Matching 0.1464 0.1453 0.15 0 0.90 Mahalanobis Metric Matching 0.1503 0.2591 0.15 0.1 0 Random Sample 0.2534 0.1422 0.15 0.1 0 Exact Matching 0.2614 0.3417 0.15 0.1 0 Propensity Score Matching 0.2583 0.1449 0.15 0.1 0 Nearest Neighbor Matching 0.2534 0.3373 0.15 0.1 0 Radius Matching 0.2492 0.1429 0.15 0.1 0 Mahalanobis Metric Matching 0.2500 0.2639 0.15 0.1 0.40 Random Sample 0.2487 0.1410 0.15 0.1 0.40 Exact Matching 0.2595 0.3429 0.15 0.1 0.40 Propensity Score Matching 0.2449 0.1382 0.15 0.1 0.40 Nearest Neighbor Matching 0.2445 0.3524 0.15 0.1 0.40 Radius Matching 0.2447 0.1457 0.15 0.1 0.40 Mahalanobis Metric Matching 0.2444 0.2663

166

Bias ES Covariance Matching Method Mean SD 0.1 0.45 0.40 Random Sample 0.5565 0.1562 0.1 0.45 0.40 Exact Matching 0.5429 0.3482 0.1 0.45 0.40 Propensity Score Matching 0.5499 0.1595 0.1 0.45 0.40 Nearest Neighbor Matching 0.5491 0.3492 0.1 0.45 0.40 Radius Matching 0.5545 0.1540 0.1 0.45 0.40 Mahalanobis Metric Matching 0.5506 0.2735 0.15 0.1 0.90 Random Sample 0.2586 0.1465 0.15 0.1 0.90 Exact Matching 0.2635 0.3290 0.15 0.1 0.90 Propensity Score Matching 0.2576 0.1382 0.15 0.1 0.90 Nearest Neighbor Matching 0.2674 0.3530 0.15 0.1 0.90 Radius Matching 0.2539 0.1481 0.15 0.1 0.90 Mahalanobis Metric Matching 0.2688 0.2688 0.15 0.45 0 Random Sample 0.6031 0.1516 0.15 0.45 0 Exact Matching 0.6165 0.3438 0.15 0.45 0 Propensity Score Matching 0.6029 0.1296 0.15 0.45 0 Nearest Neighbor Matching 0.5953 0.3400 0.15 0.45 0 Radius Matching 0.6019 0.1592 0.15 0.45 0 Mahalanobis Metric Matching 0.5968 0.1558 0.15 0.45 0.40 Random Sample 0.5970 0.1565 0.15 0.45 0.40 Exact Matching 0.5948 0.3632 0.15 0.45 0.40 Propensity Score Matching 0.6014 0.1467 0.15 0.45 0.40 Nearest Neighbor Matching 0.6117 0.3317 0.15 0.45 0.40 Radius Matching 0.5978 0.1583 0.15 0.45 0.40 Mahalanobis Metric Matching 0.6083 0.2480 0.15 0.45 0.90 Random Sample 0.6031 0.1616 0.15 0.45 0.90 Exact Matching 0.6064 0.3565 0.15 0.45 0.90 Propensity Score Matching 0.5977 0.1556 0.15 0.45 0.90 Nearest Neighbor Matching 0.5994 0.3419 0.15 0.45 0.90 Radius Matching 0.6058 0.1599 0.15 0.45 0.90 Mahalanobis Metric Matching 0.6018 0.2648

167

Appendix B

Case 2: Continuous Covariates Descriptive Statistics

168

Case 2: Discrete Covariates YD Descriptive Statistics Bias ES Covariance Correlation Matching Method Mean SD 0 0.1 0.40 0 Random Sample 0.0941 0.1435 0 0.1 0.40 0 Exact Matching 0.0918 0.2205 0 0.1 0.40 0 Propensity Score Matching 0.1006 0.1409 0 0.1 0.40 0 Nearest Neighbor Matching 0.0986 0.1431 0 0.1 0.40 0 Radius Matching 0.1018 0.1414 0 0.1 0.40 0 Mahalanobis Metric Matching 0.0983 0.1428 0 0.1 0.40 0.75 Random Sample 0.1014 0.1452 0 0.1 0.40 0.75 Exact Matching 0.0982 0.2023 0 0.1 0.40 0.75 Propensity Score Matching 0.0991 0.1530 0 0.1 0.40 0.75 Nearest Neighbor Matching 0.0989 0.1476 0 0.1 0.40 0.75 Radius Matching 0.1002 0.1534 0 0.1 0.40 0.75 Mahalanobis Metric Matching 0.0988 0.1473 0 0.1 0.40 0.90 Random Sample 0.0971 0.1417 0 0.1 0.40 0.90 Exact Matching 0.1012 0.2016 0 0.1 0.40 0.90 Propensity Score Matching 0.1059 0.1476 0 0.1 0.40 0.90 Nearest Neighbor Matching 0.1006 0.1543 0 0.1 0.40 0.90 Radius Matching 0.1011 0.1483 0 0.1 0.40 0.90 Mahalanobis Metric Matching 0.1007 0.1532 0 0.1 0.40 0.99 Random Sample 0.1039 0.1464 0 0.1 0.40 0.99 Exact Matching 0.1036 0.1979 0 0.1 0.40 0.99 Propensity Score Matching 0.1019 0.1416 0 0.1 0.40 0.99 Nearest Neighbor Matching 0.0987 0.1488 0 0.1 0.40 0.99 Radius Matching 0.1008 0.1468 0 0.1 0.40 0.99 Mahalanobis Metric Matching 0.0989 0.1485 0 0.1 0.90 0 Random Sample 0.1039 0.1442 0 0.1 0.90 0 Exact Matching 0.0992 0.4365 0 0.1 0.90 0 Propensity Score Matching 0.0997 0.1427 0 0.1 0.90 0 Nearest Neighbor Matching 0.1017 0.1457 0 0.1 0.90 0 Radius Matching 0.1028 0.1436 0 0.1 0.90 0 Mahalanobis Metric Matching 0.1021 0.1430 0 0.1 0.90 0.75 Random Sample 0.1001 0.1430 0 0.1 0.90 0.75 Exact Matching 0.1092 0.4059 0 0.1 0.90 0.75 Propensity Score Matching 0.1041 0.1444 0 0.1 0.90 0.75 Nearest Neighbor Matching 0.1035 0.1440 0 0.1 0.90 0.75 Radius Matching 0.0974 0.1450 0 0.1 0.90 0.75 Mahalanobis Metric Matching 0.1037 0.1438 0 0.1 0.90 0.90 Random Sample 0.0984 0.1419 0 0.1 0.90 0.90 Exact Matching 0.0957 0.3942 0 0.1 0.90 0.90 Propensity Score Matching 0.1007 0.1427 0 0.1 0.90 0.90 Nearest Neighbor Matching 0.0992 0.1456 0 0.1 0.90 0.90 Radius Matching 0.0993 0.1453 0 0.1 0.90 0.90 Mahalanobis Metric Matching 0.0989 0.1455 0 0.1 0.90 0.99 Random Sample 0.0945 0.1426 0 0.1 0.90 0.99 Exact Matching 0.0933 0.4059 0 0.1 0.90 0.99 Propensity Score Matching 0.1004 0.1467 0 0.1 0.90 0.99 Nearest Neighbor Matching 0.0992 0.1492 0 0.1 0.90 0.99 Radius Matching 0.0989 0.1473 0 0.1 0.90 0.99 Mahalanobis Metric Matching 0.0986 0.1489

169

Bias ES Covariance Correlation Match Method Mean SD 0 0.45 0 0 Random Sample 0.4436 0.1548 0 0.45 0 0 Exact Matching 0.4435 0.2126 0 0.45 0 0 Propensity Score Matching 0.4398 0.1553 0 0.45 0 0 Nearest Neighbor Matching 0.4342 0.1621 0 0.45 0 0 Radius Matching 0.4411 0.1593 0 0.45 0 0 Mahalanobis Metric Matching 0.4340 0.1618 0 0.45 0 0.75 Random Sample 0.4433 0.1560 0 0.45 0 0.75 Exact Matching 0.4495 0.1719 0 0.45 0 0.75 Propensity Score Matching 0.4434 0.1396 0 0.45 0 0.75 Nearest Neighbor Matching 0.4422 0.1375 0 0.45 0 0.75 Radius Matching 0.4392 0.1425 0 0.45 0 0.75 Mahalanobis Metric Matching 0.4425 0.1375 0 0.45 0 0.90 Random Sample 0.4435 0.1555 0 0.45 0 0.90 Exact Matching 0.4332 0.3942 0 0.45 0 0.90 Propensity Score Matching 0.4388 0.1540 0 0.45 0 0.90 Nearest Neighbor Matching 0.4406 0.1528 0 0.45 0 0.90 Radius Matching 0.4423 0.1505 0 0.45 0 0.90 Mahalanobis Metric Matching 0.4408 0.1526 0 0.45 0 0.99 Random Sample 0.4364 0.1566 0 0.45 0 0.99 Exact Matching 0.4362 0.1542 0 0.45 0 0.99 Propensity Score Matching 0.4355 0.1358 0 0.45 0 0.99 Nearest Neighbor Matching 0.4384 0.1336 0 0.45 0 0.99 Radius Matching 0.4435 0.1351 0 0.45 0 0.99 Mahalanobis Metric Matching 0.4385 0.1331 0 0.45 0.40 0 Random Sample 0.4347 0.1565 0 0.45 0.40 0 Exact Matching 0.4312 0.2260 0 0.45 0.40 0 Propensity Score Matching 0.4378 0.1520 0 0.45 0.40 0 Nearest Neighbor Matching 0.4351 0.1570 0 0.45 0.40 0 Radius Matching 0.4322 0.1532 0 0.45 0.40 0 Mahalanobis Metric Matching 0.4351 0.1566 0 0.45 0.40 0.75 Random Sample 0.4438 0.1579 0 0.45 0.40 0.75 Exact Matching 0.4297 0.1940 0 0.45 0.40 0.75 Propensity Score Matching 0.4385 0.1476 0 0.45 0.40 0.75 Nearest Neighbor Matching 0.4373 0.1474 0 0.45 0.40 0.75 Radius Matching 0.4375 0.1498 0 0.45 0.40 0.75 Mahalanobis Metric Matching 0.4362 0.1477 0 0.45 0.40 0.90 Random Sample 0.4359 0.1570 0 0.45 0.40 0.90 Exact Matching 0.4435 0.2099 0 0.45 0.40 0.90 Propensity Score Matching 0.4442 0.1478 0 0.45 0.40 0.90 Nearest Neighbor Matching 0.4323 0.1518 0 0.45 0.40 0.90 Radius Matching 0.4403 0.1481 0 0.45 0.40 0.90 Mahalanobis Metric Matching 0.4322 0.1518 0 0.45 0.40 0.99 Random Sample 0.4423 0.1460 0 0.45 0.40 0.99 Exact Matching 0.4379 0.2056 0 0.45 0.40 0.99 Propensity Score Matching 0.4389 0.1467 0 0.45 0.40 0.99 Nearest Neighbor Matching 0.4328 0.1534 0 0.45 0.40 0.99 Radius Matching 0.4337 0.1494

170

Bias ES Covariance Correlation Matching Method Mean SD 0 0.45 0.90 0 Random Sample 0.4352 0.1555 0 0.45 0.90 0 Exact Matching 0.4437 0.4399 0 0.45 0.90 0 Propensity Score Matching 0.4442 0.1590 0 0.45 0.90 0 Nearest Neighbor Matching 0.4350 0.1599 0 0.45 0.90 0 Radius Matching 0.4383 0.1568 0 0.45 0.90 0 Mahalanobis Metric Matching 0.4354 0.1596 0 0.45 0.90 0.75 Random Sample 0.4329 0.1572 0 0.45 0.90 0.75 Exact Matching 0.4333 0.4319 0 0.45 0.90 0.75 Propensity Score Matching 0.4368 0.1554 0 0.45 0.90 0.75 Nearest Neighbor Matching 0.4376 0.1483 0 0.45 0.90 0.75 Radius Matching 0.4376 0.1518 0 0.45 0.90 0.75 Mahalanobis Metric Matching 0.4369 0.1481 0 0.45 0.90 0.90 Random Sample 0.4412 0.1428 0 0.45 0.90 0.90 Exact Matching 0.4259 0.4164 0 0.45 0.90 0.90 Propensity Score Matching 0.4366 0.1478 0 0.45 0.90 0.90 Nearest Neighbor Matching 0.4416 0.1456 0 0.45 0.90 0.90 Radius Matching 0.4362 0.1480 0 0.45 0.90 0.90 Mahalanobis Metric Matching 0.4418 0.1453 0 0.45 0.90 0.99 Random Sample 0.4404 0.1559 0 0.45 0.90 0.99 Exact Matching 0.4267 0.3940 0 0.45 0.90 0.99 Propensity Score Matching 0.4360 0.1490 0 0.45 0.90 0.99 Nearest Neighbor Matching 0.4440 0.1520 0 0.45 0.90 0.99 Radius Matching 0.4391 0.1569 0 0.45 0.90 0.99 Mahalanobis Metric Matching 0.4440 0.1507 0.1 0 0 0 Random Sample 0.1014 0.1402 0.1 0 0 0 Exact Matching 0.1011 0.1934 0.1 0 0 0 Propensity Score Matching 0.1035 0.1397 0.1 0 0 0 Nearest Neighbor Matching 0.0978 0.1395 0.1 0 0 0 Radius Matching 0.0969 0.1408 0.1 0 0 0 Mahalanobis Metric Matching 0.0978 0.1388 0.1 0 0 0.75 Random Sample 0.1076 0.1434 0.1 0 0 0.75 Exact Matching 0.1040 0.1678 0.1 0 0 0.75 Propensity Score Matching 0.1037 0.1446 0.1 0 0 0.75 Nearest Neighbor Matching 0.1059 0.1461 0.1 0 0 0.75 Radius Matching 0.1016 0.1432 0.1 0 0 0.75 Mahalanobis Metric Matching 0.1060 0.1451 0.1 0 0 0.90 Random Sample 0.0996 0.1388 0.1 0 0 0.90 Exact Matching 0.1073 0.1683 0.1 0 0 0.90 Propensity Score Matching 0.1058 0.1417 0.1 0 0 0.90 Nearest Neighbor Matching 0.1026 0.1502 0.1 0 0 0.90 Radius Matching 0.1012 0.1416 0.1 0 0 0.90 Mahalanobis Metric Matching 0.1018 0.1494 0.1 0 0 0.99 Random Sample 0.0942 0.1435 0.1 0 0 0.99 Exact Matching 0.0991 0.1587 0.1 0 0 0.99 Propensity Score Matching 0.0974 0.1429 0.1 0 0 0.99 Nearest Neighbor Matching 0.0938 0.1435 0.1 0 0 0.99 Radius Matching 0.0991 0.1404 0.1 0 0 0.99 Mahalanobis Metric Matching 0.0936 0.1431

171

Bias ES Covariance Correlation Matching Method Mean SD 0.1 0 0.40 0 Random Sample 0.0919 0.1370 0.1 0 0.40 0 Exact Matching 0.0997 0.2042 0.1 0 0.40 0 Propensity Score Matching 0.0998 0.1399 0.1 0 0.40 0 Nearest Neighbor Matching 0.0982 0.1350 0.1 0 0.40 0 Radius Matching 0.0935 0.1398 0.1 0 0.40 0 Mahalanobis Metric Matching 0.0984 0.1343 0.1 0 0.40 0.75 Random Sample 0.0934 0.1356 0.1 0 0.40 0.75 Exact Matching 0.0998 0.1973 0.1 0 0.40 0.75 Propensity Score Matching 0.0947 0.1422 0.1 0 0.40 0.75 Nearest Neighbor Matching 0.0987 0.1411 0.1 0 0.40 0.75 Radius Matching 0.0998 0.1450 0.1 0 0.40 0.75 Mahalanobis Metric Matching 0.0990 0.1406 0.1 0 0.40 0.90 Random Sample 0.0908 0.1366 0.1 0 0.40 0.90 Exact Matching 0.0995 0.1928 0.1 0 0.40 0.90 Propensity Score Matching 0.0997 0.1432 0.1 0 0.40 0.90 Nearest Neighbor Matching 0.0969 0.1412 0.1 0 0.40 0.90 Radius Matching 0.1013 0.1433 0.1 0 0.40 0.90 Mahalanobis Metric Matching 0.0969 0.1409 0.1 0 0.40 0.99 Random Sample 0.1021 0.1396 0.1 0 0.40 0.99 Exact Matching 0.1073 0.2027 0.1 0 0.40 0.99 Propensity Score Matching 0.1017 0.1402 0.1 0 0.40 0.99 Nearest Neighbor Matching 0.0994 0.1436 0.1 0 0.40 0.99 Radius Matching 0.0996 0.1429 0.1 0 0.40 0.99 Mahalanobis Metric Matching 0.0990 0.1428 0.1 0 0.90 0 Random Sample 0.1000 0.1382 0.1 0 0.90 0 Exact Matching 0.0920 0.1427 0.1 0 0.90 0 Propensity Score Matching 0.0977 0.1394 0.1 0 0.90 0 Nearest Neighbor Matching 0.1035 0.1448 0.1 0 0.90 0 Radius Matching 0.0987 0.1394 0.1 0 0.90 0 Mahalanobis Metric Matching 0.1033 0.1448 0.1 0 0.90 0.75 Random Sample 0.0963 0.1415 0.1 0 0.90 0.75 Exact Matching 0.1124 0.4097 0.1 0 0.90 0.75 Propensity Score Matching 0.1040 0.1381 0.1 0 0.90 0.75 Nearest Neighbor Matching 0.0988 0.1460 0.1 0 0.90 0.75 Radius Matching 0.0965 0.1446 0.1 0 0.90 0.75 Mahalanobis Metric Matching 0.0989 0.1451 0.1 0 0.90 0.90 Random Sample 0.0967 0.1452 0.1 0 0.90 0.90 Exact Matching 0.1127 0.4044 0.1 0 0.90 0.90 Propensity Score Matching 0.1002 0.1440 0.1 0 0.90 0.90 Nearest Neighbor Matching 0.0957 0.1434 0.1 0 0.90 0.90 Radius Matching 0.1054 0.1393 0.1 0 0.90 0.90 Mahalanobis Metric Matching 0.0956 0.1429 0.1 0 0.90 0.99 Random Sample 0.1037 0.1419 0.1 0 0.90 0.99 Exact Matching 0.1181 0.4176 0.1 0 0.90 0.99 Propensity Score Matching 0.1029 0.1437 0.1 0 0.90 0.99 Nearest Neighbor Matching 0.1037 0.1484 0.1 0 0.90 0.99 Radius Matching 0.1024 0.1447 0.1 0 0.90 0.99 Mahalanobis Metric Matching 0.1034 0.1482

172

Bias ES Covariance Correlation Matching Method Mean SD 0.1 0.1 0 0 Random Sample 0.2018 0.1485 0.1 0.1 0 0 Exact Matching 0.2090 0.2011 0.1 0.1 0 0 Propensity Score Matching 0.2008 0.1495 0.1 0.1 0 0 Nearest Neighbor Matching 0.2008 0.1486 0.1 0.1 0 0 Radius Matching 0.2015 0.1467 0.1 0.1 0 0 Mahalanobis Metric Matching 0.2008 0.1475 0.1 0.1 0 0.75 Random Sample 0.1978 0.1498 0.1 0.1 0 0.75 Exact Matching 0.1982 0.1667 0.1 0.1 0 0.75 Propensity Score Matching 0.1975 0.1405 0.1 0.1 0 0.75 Nearest Neighbor Matching 0.1978 0.1422 0.1 0.1 0 0.75 Radius Matching 0.2013 0.1492 0.1 0.1 0 0.75 Mahalanobis Metric Matching 0.1978 0.1419 0.1 0.1 0 0.90 Random Sample 0.2080 0.1462 0.1 0.1 0 0.90 Exact Matching 0.2093 0.1640 0.1 0.1 0 0.90 Propensity Score Matching 0.2059 0.1410 0.1 0.1 0 0.90 Nearest Neighbor Matching 0.2003 0.1424 0.1 0.1 0 0.90 Radius Matching 0.2004 0.1400 0.1 0.1 0 0.90 Mahalanobis Metric Matching 0.1978 0.1412 0.1 0.1 0 0.99 Random Sample 0.2001 0.1511 0.1 0.1 0 0.99 Exact Matching 0.0998 0.1621 0.1 0.1 0 0.99 Propensity Score Matching 0.2050 0.1453 0.1 0.1 0 0.99 Nearest Neighbor Matching 0.2000 0.1458 0.1 0.1 0 0.99 Radius Matching 0.2000 0.1394 0.1 0.1 0 0.99 Mahalanobis Metric Matching 0.2000 0.1447 0.1 0.1 0.40 0 Random Sample 0.2021 0.1434 0.1 0.1 0.40 0 Exact Matching 0.1966 0.2136 0.1 0.1 0.40 0 Propensity Score Matching 0.2041 0.1452 0.1 0.1 0.40 0 Nearest Neighbor Matching 0.2000 0.1463 0.1 0.1 0.40 0 Radius Matching 0.2026 0.1441 0.1 0.1 0.40 0 Mahalanobis Metric Matching 0.2000 0.1448 0.1 0.1 0.40 0.75 Random Sample 0.1970 0.1475 0.1 0.1 0.40 0.75 Exact Matching 0.2048 0.1981 0.1 0.1 0.40 0.75 Propensity Score Matching 0.1976 0.1458 0.1 0.1 0.40 0.75 Nearest Neighbor Matching 0.1986 0.1468 0.1 0.1 0.40 0.75 Radius Matching 0.1953 0.1423 0.1 0.1 0.40 0.75 Mahalanobis Metric Matching 0.1989 0.1456 0.1 0.1 0.40 0.90 Random Sample 0.1983 0.1399 0.1 0.1 0.40 0.90 Exact Matching 0.2023 0.2065 0.1 0.1 0.40 0.90 Propensity Score Matching 0.1994 0.1497 0.1 0.1 0.40 0.90 Nearest Neighbor Matching 0.1957 0.1454 0.1 0.1 0.40 0.90 Radius Matching 0.1980 0.1467 0.1 0.1 0.40 0.90 Mahalanobis Metric Matching 0.1956 0.1453 0.1 0.1 0.40 0.99 Random Sample 0.1978 0.1445 0.1 0.1 0.40 0.99 Exact Matching 0.1960 0.1932 0.1 0.1 0.40 0.99 Propensity Score Matching 0.1973 0.1395 0.1 0.1 0.40 0.99 Nearest Neighbor Matching 0.2017 0.1425 0.1 0.1 0.40 0.99 Radius Matching 0.2022 0.1440 0.1 0.1 0.40 0.99 Mahalanobis Metric Matching 0.2013 0.1419

173

Bias ES Covariance Correlation Matching Method Mean SD 0.1 0.1 0.90 0 Random Sample 0.2002 0.1411 0.1 0.1 0.90 0 Exact Matching 0.1878 0.4109 0.1 0.1 0.90 0 Propensity Score Matching 0.1986 0.1458 0.1 0.1 0.90 0 Nearest Neighbor Matching 0.2035 0.1405 0.1 0.1 0.90 0 Radius Matching 0.1954 0.1407 0.1 0.1 0.90 0 Mahalanobis Metric Matching 0.2031 0.1389 0.1 0.1 0.90 0.75 Random Sample 0.1944 0.1404 0.1 0.1 0.90 0.75 Exact Matching 0.2150 0.4224 0.1 0.1 0.90 0.75 Propensity Score Matching 0.1980 0.1489 0.1 0.1 0.90 0.75 Nearest Neighbor Matching 0.1966 0.1446 0.1 0.1 0.90 0.75 Radius Matching 0.1972 0.1434 0.1 0.1 0.90 0.75 Mahalanobis Metric Matching 0.1973 0.1438 0.1 0.1 0.90 0.90 Random Sample 0.2102 0.1444 0.1 0.1 0.90 0.90 Exact Matching 0.2060 0.4247 0.1 0.1 0.90 0.90 Propensity Score Matching 0.2047 0.1484 0.1 0.1 0.90 0.90 Nearest Neighbor Matching 0.2087 0.1493 0.1 0.1 0.90 0.90 Radius Matching 0.2077 0.1426 0.1 0.1 0.90 0.90 Mahalanobis Metric Matching 0.2086 0.1485 0.1 0.1 0.90 0.99 Random Sample 0.1962 0.1474 0.1 0.1 0.90 0.99 Exact Matching 0.1952 0.4132 0.1 0.1 0.90 0.99 Propensity Score Matching 0.1998 0.1476 0.1 0.1 0.90 0.99 Nearest Neighbor Matching 0.1991 0.1440 0.1 0.1 0.90 0.99 Radius Matching 0.2027 0.1456 0.1 0.1 0.90 0.99 Mahalanobis Metric Matching 0.1996 0.1442 0.1 0.45 0 0 Random Sample 0.5495 01724 0.1 0.45 0 0 Exact Matching 0.5378 0.1845 0.1 0.45 0 0 Propensity Score Matching 0.5468 0.1433 0.1 0.45 0 0 Nearest Neighbor Matching 0.5489 0.1492 0.1 0.45 0 0 Radius Matching 0.5494 0.1502 0.1 0.45 0 0 Mahalanobis Metric Matching 0.5489 0.1484 0.1 0.45 0 0.75 Random Sample 0.5465 0.1508 0.1 0.45 0 0.75 Exact Matching 0.5460 0.1660 0.1 0.45 0 0.75 Propensity Score Matching 0.5474 0.1349 0.1 0.45 0 0.75 Nearest Neighbor Matching 0.5451 0.1349 0.1 0.45 0 0.75 Radius Matching 0.5493 0.1288 0.1 0.45 0 0.75 Mahalanobis Metric Matching 0.5452 0.1346 0.1 0.45 0 0.90 Random Sample 0.5503 0.1557 0.1 0.45 0 0.90 Exact Matching 0.5469 0.1499 0.1 0.45 0 0.90 Propensity Score Matching 0.5545 0.1297 0.1 0.45 0 0.90 Nearest Neighbor Matching 0.5549 0.1326 0.1 0.45 0 0.90 Radius Matching 0.5534 0.1290 0.1 0.45 0 0.90 Mahalanobis Metric Matching 0.5542 0.1321 0.1 0.45 0 0.99 Random Sample 0.5417 0.1560 0.1 0.45 0 0.99 Exact Matching 0.5452 0.1588 0.1 0.45 0 0.99 Propensity Score Matching 0.5442 0.1300 0.1 0.45 0 0.99 Nearest Neighbor Matching 0.5467 0.1339 0.1 0.45 0 0.99 Radius Matching 0.5484 0.1341 0.1 0.45 0 0.99 Mahalanobis Metric Matching 0.5474 0.1340

174

Bias ES Covariance Correlation Matching Method Mean SD 0.1 0.45 0.40 0 Random Sample 0.5470 0.1522 0.1 0.45 0.40 0 Exact Matching 0.5504 0.2176 0.1 0.45 0.40 0 Propensity Score Matching 0.5511 0.1592 0.1 0.45 0.40 0 Nearest Neighbor Matching 0.5464 0.1584 0.1 0.45 0.40 0 Radius Matching 0.5477 0.1617 0.1 0.45 0.40 0 Mahalanobis Metric Matching 0.5457 0.1577 0.1 0.45 0.40 0.75 Random Sample 0.5461 0.1674 0.1 0.45 0.40 0.75 Exact Matching 0.5366 0.2118 0.1 0.45 0.40 0.75 Propensity Score Matching 0.5468 0.1483 0.1 0.45 0.40 0.75 Nearest Neighbor Matching 0.5482 0.1489 0.1 0.45 0.40 0.75 Radius Matching 0.5454 0.1583 0.1 0.45 0.40 0.75 Mahalanobis Metric Matching 0.5478 0.1485 0.1 0.45 0.40 0.90 Random Sample 0.5478 0.1610 0.1 0.45 0.40 0.90 Exact Matching 0.5647 0.1954 0.1 0.45 0.40 0.90 Propensity Score Matching 0.5497 0.1420 0.1 0.45 0.40 0.90 Nearest Neighbor Matching 0.5493 0.1459 0.1 0.45 0.40 0.90 Radius Matching 0.5574 0.1438 0.1 0.45 0.40 0.90 Mahalanobis Metric Matching 0.5493 0.1459 0.1 0.45 0.40 0.99 Random Sample 0.5414 0.1549 0.1 0.45 0.40 0.99 Exact Matching 0.5412 0.2006 0.1 0.45 0.40 0.99 Propensity Score Matching 0.5465 0.1484 0.1 0.45 0.40 0.99 Nearest Neighbor Matching 0.5506 0.1486 0.1 0.45 0.40 0.99 Radius Matching 0.5453 0.1396 0.1 0.45 0.40 0.99 Mahalanobis Metric Matching 0.5500 0.1481 0.1 0.45 0.90 0 Random Sample 0.5500 0.1602 0.1 0.45 0.90 0 Exact Matching 0.5515 0.4446 0.1 0.45 0.90 0 Propensity Score Matching 0.5474 0.1581 0.1 0.45 0.90 0 Nearest Neighbor Matching 0.5478 0.1597 0.1 0.45 0.90 0 Radius Matching 0.5466 0.1559 0.1 0.45 0.90 0 Mahalanobis Metric Matching 0.5477 0.1588 0.1 0.45 0.90 0.75 Random Sample 0.5450 0.1547 0.1 0.45 0.90 0.75 Exact Matching 0.5385 0.4156 0.1 0.45 0.90 0.75 Propensity Score Matching 0.5432 0.1481 0.1 0.45 0.90 0.75 Nearest Neighbor Matching 0.5412 0.1501 0.1 0.45 0.90 0.75 Radius Matching 0.5439 0.1511 0.1 0.45 0.90 0.75 Mahalanobis Metric Matching 0.5412 0.1495 0.1 0.45 0.90 0.90 Random Sample 0.5499 0.1452 0.1 0.45 0.90 0.90 Exact Matching 0.5389 0.4044 0.1 0.45 0.90 0.90 Propensity Score Matching 0.5535 0.1491 0.1 0.45 0.90 0.90 Nearest Neighbor Matching 0.5487 0.1525 0.1 0.45 0.90 0.90 Radius Matching 0.5509 0.1496 0.1 0.45 0.90 0.90 Mahalanobis Metric Matching 0.5489 0.1519 0.1 0.45 0.90 0.99 Random Sample 0.5414 0.1592 0.1 0.45 0.90 0.99 Exact Matching 0.5416 0.4233 0.1 0.45 0.90 0.99 Propensity Score Matching 0.5458 0.1465 0.1 0.45 0.90 0.99 Nearest Neighbor Matching 0.5422 0.1525 0.1 0.45 0.90 0.99 Radius Matching 0.5415 0.1496 0.1 0.45 0.90 0.99 Mahalanobis Metric Matching 0.5426 0.1529

175

Bias ES Covariance Correlation Matching Method Mean SD 0.15 0 0 0 Random Sample 0.1515 0.1381 0.15 0 0 0 Exact Matching 0.1588 0.1938 0.15 0 0 0 Propensity Score Matching 0.1545 0.1388 0.15 0 0 0 Nearest Neighbor Matching 0.1513 0.1448 0.15 0 0 0 Radius Matching 0.1471 0.1438 0.15 0 0 0 Mahalanobis Metric Matching 0.1517 0.1440 0.15 0 0 0.75 Random Sample 0.1492 0.1452 0.15 0 0 0.75 Exact Matching 0.1464 0.1726 0.15 0 0 0.75 Propensity Score Matching 0.1485 0.1440 0.15 0 0 0.75 Nearest Neighbor Matching 0.1518 0.1462 0.15 0 0 0.75 Radius Matching 0.1449 0.1471 0.15 0 0 0.75 Mahalanobis Metric Matching 0.1520 0.1450 0.15 0 0 0.90 Random Sample 0.1501 0.1401 0.15 0 0 0.90 Exact Matching 0.1498 0.1624 0.15 0 0 0.90 Propensity Score Matching 0.1513 0.1428 0.15 0 0 0.90 Nearest Neighbor Matching 0.1452 0.1420 0.15 0 0 0.90 Radius Matching 0.1564 0.1422 0.15 0 0 0.90 Mahalanobis Metric Matching 0.1451 0.1411 0.15 0 0 0.99 Random Sample 0.1536 0.1390 0.15 0 0 0.99 Exact Matching 0.1459 0.1659 0.15 0 0 0.99 Propensity Score Matching 0.1540 0.1486 0.15 0 0 0.99 Nearest Neighbor Matching 0.1457 0.1466 0.15 0 0 0.99 Radius Matching 0.1527 0.1450 0.15 0 0 0.99 Mahalanobis Metric Matching 0.1456 0.1461 0.15 0 0.40 0 Random Sample 0.1481 0.1370 0.15 0 0.40 0 Exact Matching 0.1559 0.2188 0.15 0 0.40 0 Propensity Score Matching 0.1510 0.1403 0.15 0 0.40 0 Nearest Neighbor Matching 0.1496 0.1353 0.15 0 0.40 0 Radius Matching 0.1508 0.1372 0.15 0 0.40 0 Mahalanobis Metric Matching 0.1495 0.1352 0.15 0 0.40 0.75 Random Sample 0.1508 0.1431 0.15 0 0.40 0.75 Exact Matching 0.1457 0.1989 0.15 0 0.40 0.75 Propensity Score Matching 0.1484 0.1345 0.15 0 0.40 0.75 Nearest Neighbor Matching 0.1505 0.1372 0.15 0 0.40 0.75 Radius Matching 0.1503 0.1394 0.15 0 0.40 0.75 Mahalanobis Metric Matching 0.1483 0.1365 0.15 0 0.40 0.90 Random Sample 0.1503 0.1418 0.15 0 0.40 0.90 Exact Matching 0.1568 0.2005 0.15 0 0.40 0.90 Propensity Score Matching 0.1481 0.1471 0.15 0 0.40 0.90 Nearest Neighbor Matching 0.1486 0.1447 0.15 0 0.40 0.90 Radius Matching 0.1486 0.1458 0.15 0 0.40 0.90 Mahalanobis Metric Matching 0.1482 0.1442 0.15 0 0.40 0.99 Random Sample 0.1582 0.1482 0.15 0 0.40 0.99 Exact Matching 0.1432 0.1940 0.15 0 0.40 0.99 Propensity Score Matching 0.1489 0.1400 0.15 0 0.40 0.99 Nearest Neighbor Matching 0.1513 0.1407 0.15 0 0.40 0.99 Radius Matching 0.1543 0.1446 0.15 0 0.40 0.99 Mahalanobis Metric Matching 0.1512 0.1404

176

Bias ES Covariance Correlation Matching Method Mean SD 0.15 0 0.90 0 Random Sample 0.1486 0.1370 0.15 0 0.90 0 Exact Matching 0.1350 0.2042 0.15 0 0.90 0 Propensity Score Matching 0.1471 0.1399 0.15 0 0.90 0 Nearest Neighbor Matching 0.1444 0.1350 0.15 0 0.90 0 Radius Matching 0.1441 0.1398 0.15 0 0.90 0 Mahalanobis Metric Matching 0.1445 0.1343 0.15 0 0.90 0.75 Random Sample 0.1488 0.1466 0.15 0 0.90 0.75 Exact Matching 0.1439 0.4109 0.15 0 0.90 0.75 Propensity Score Matching 0.1469 0.1418 0.15 0 0.90 0.75 Nearest Neighbor Matching 0.1497 0.1417 0.15 0 0.90 0.75 Radius Matching 0.1437 0.1454 0.15 0 0.90 0.75 Mahalanobis Metric Matching 0.1497 0.1422 0.15 0 0.90 0.90 Random Sample 0.1450 0.1402 0.15 0 0.90 0.90 Exact Matching 0.1768 0.4022 0.15 0 0.90 0.90 Propensity Score Matching 0.1513 0.1343 0.15 0 0.90 0.90 Nearest Neighbor Matching 0.1493 0.1360 0.15 0 0.90 0.90 Radius Matching 0.1547 0.1365 0.15 0 0.90 0.90 Mahalanobis Metric Matching 0.1501 0.1356 0.15 0 0.90 0.99 Random Sample 0.1568 0.1435 0.15 0 0.90 0.99 Exact Matching 0.1682 0.4084 0.15 0 0.90 0.99 Propensity Score Matching 0.1566 0.1467 0.15 0 0.90 0.99 Nearest Neighbor Matching 0.1582 0.1415 0.15 0 0.90 0.99 Radius Matching 0.1536 0.1457 0.15 0 0.90 0.99 Mahalanobis Metric Matching 0.1583 0.1408 0.15 0.1 0 0 Random Sample 0.2549 0.1420 0.15 0.1 0 0 Exact Matching 0.2565 0.2011 0.15 0.1 0 0 Propensity Score Matching 0.2531 0.1460 0.15 0.1 0 0 Nearest Neighbor Matching 0.2495 0.1454 0.15 0.1 0 0 Radius Matching 0.2479 0.1459 0.15 0.1 0 0 Mahalanobis Metric Matching 0.2493 0.1450 0.15 0.1 0 0.75 Random Sample 0.2522 0.1412 0.15 0.1 0 0.75 Exact Matching 0.2437 0.1720 0.15 0.1 0 0.75 Propensity Score Matching 0.2473 0.1368 0.15 0.1 0 0.75 Nearest Neighbor Matching 0.2501 0.1433 0.15 0.1 0 0.75 Radius Matching 0.2495 0.1482 0.15 0.1 0 0.75 Mahalanobis Metric Matching 0.2494 0.1430 0.15 0.1 0 0.90 Random Sample 0.2547 0.1469 0.15 0.1 0 0.90 Exact Matching 0.2598 0.1617 0.15 0.1 0 0.90 Propensity Score Matching 0.2581 0.1392 0.15 0.1 0 0.90 Nearest Neighbor Matching 0.2521 0.1406 0.15 0.1 0 0.90 Radius Matching 0.2536 0.1373 0.15 0.1 0 0.90 Mahalanobis Metric Matching 0.2526 0.1402 0.15 0.1 0 0.99 Random Sample 0.2505 0.1384 0.15 0.1 0 0.99 Exact Matching 0.2518 0.1525 0.15 0.1 0 0.99 Propensity Score Matching 0.2524 0.1337 0.15 0.1 0 0.99 Nearest Neighbor Matching 0.2482 0.1389 0.15 0.1 0 0.99 Radius Matching 0.2475 0.1382 0.15 0.1 0 0.99 Mahalanobis Metric Matching 0.2479 0.1381

177

Bias ES Covariance Correlation Matching Method Mean SD 0.15 0.1 0.40 0 Random Sample 0.2482 0.1428 0.15 0.1 0.40 0 Exact Matching 0.2496 0.2241 0.15 0.1 0.40 0 Propensity Score Matching 0.2500 0.1440 0.15 0.1 0.40 0 Nearest Neighbor Matching 0.2498 0.1433 0.15 0.1 0.40 0 Radius Matching 0.2497 0.1487 0.15 0.1 0.40 0 Mahalanobis Metric Matching 0.2495 0.1434 0.15 0.1 0.40 0.75 Random Sample 0.2515 0.1416 0.15 0.1 0.40 0.75 Exact Matching 0.2541 0.1942 0.15 0.1 0.40 0.75 Propensity Score Matching 0.2517 0.1399 0.15 0.1 0.40 0.75 Nearest Neighbor Matching 0.2499 0.1430 0.15 0.1 0.40 0.75 Radius Matching 0.2456 0.1487 0.15 0.1 0.40 0.75 Mahalanobis Metric Matching 0.2499 0.1424 0.15 0.1 0.40 0.90 Random Sample 0.2510 0.1465 0.15 0.1 0.40 0.90 Exact Matching 0.2627 0.1988 0.15 0.1 0.40 0.90 Propensity Score Matching 0.2568 0.1421 0.15 0.1 0.40 0.90 Nearest Neighbor Matching 0.2482 0.1459 0.15 0.1 0.40 0.90 Radius Matching 0.2481 0.1443 0.15 0.1 0.40 0.90 Mahalanobis Metric Matching 0.2481 0.1454 0.15 0.1 0.40 0.99 Random Sample 0.2356 0.1523 0.15 0.1 0.40 0.99 Exact Matching 0.2462 0.1927 0.15 0.1 0.40 0.99 Propensity Score Matching 0.2483 0.1444 0.15 0.1 0.40 0.99 Nearest Neighbor Matching 0.2449 0.1435 0.15 0.1 0.40 0.99 Radius Matching 0.2474 0.1454 0.15 0.1 0.40 0.99 Mahalanobis Metric Matching 0.2445 0.1426 0.15 0.1 0.90 0 Random Sample 0.2483 0.1450 0.15 0.1 0.90 0 Exact Matching 0.2395 0.4382 0.15 0.1 0.90 0 Propensity Score Matching 0.2473 0.1423 0.15 0.1 0.90 0 Nearest Neighbor Matching 0.2486 0.1458 0.15 0.1 0.90 0 Radius Matching 0.2472 0.1421 0.15 0.1 0.90 0 Mahalanobis Metric Matching 0.2489 0.1458 0.15 0.1 0.90 0.75 Random Sample 0.2532 0.1409 0.15 0.1 0.90 0.75 Exact Matching 0.2455 0.4291 0.15 0.1 0.90 0.75 Propensity Score Matching 0.2587 0.1416 0.15 0.1 0.90 0.75 Nearest Neighbor Matching 0.2646 0.1396 0.15 0.1 0.90 0.75 Radius Matching 0.2641 0.1444 0.15 0.1 0.90 0.75 Mahalanobis Metric Matching 0.2645 0.1386 0.15 0.1 0.90 0.90 Random Sample 0.2540 0.1445 0.15 0.1 0.90 0.90 Exact Matching 0.2445 0.4006 0.15 0.1 0.90 0.90 Propensity Score Matching 0.2537 0.1405 0.15 0.1 0.90 0.90 Nearest Neighbor Matching 0.2466 0.1464 0.15 0.1 0.90 0.90 Radius Matching 0.2520 0.1405 0.15 0.1 0.90 0.90 Mahalanobis Metric Matching 0.2471 0.1462 0.15 0.1 0.90 0.99 Random Sample 0.2487 0.1507 0.15 0.1 0.90 0.99 Exact Matching 0.2187 0.4041 0.15 0.1 0.90 0.99 Propensity Score Matching 0.2488 0.1415 0.15 0.1 0.90 0.99 Nearest Neighbor Matching 0.2499 0.1396 0.15 0.1 0.90 0.99 Radius Matching 0.2514 0.1371 0.15 0.1 0.90 0.99 Mahalanobis Metric Matching 0.2494 0.1393

178

Bias ES Covariance Correlation Matching Method Mean SD 0.15 0.45 0 0 Random Sample 0.5958 0.1607 0.15 0.45 0 0 Exact Matching 0.5972 0.2141 0.15 0.45 0 0 Propensity Score Matching 0.5990 0.1546 0.15 0.45 0 0 Nearest Neighbor Matching 0.6008 0.1599 0.15 0.45 0 0 Radius Matching 0.6011 0.1597 0.15 0.45 0 0 Mahalanobis Metric Matching 0.5998 0.1598 0.15 0.45 0 0.75 Random Sample 0.6033 0.1566 0.15 0.45 0 0.75 Exact Matching 0.5945 0.1615 0.15 0.45 0 0.75 Propensity Score Matching 0.6034 0.1305 0.15 0.45 0 0.75 Nearest Neighbor Matching 0.6009 0.1309 0.15 0.45 0 0.75 Radius Matching 0.5996 0.1349 0.15 0.45 0 0.75 Mahalanobis Metric Matching 0.6010 0.1312 0.15 0.45 0 0.90 Random Sample 0.6000 0.1536 0.15 0.45 0 0.90 Exact Matching 0.5960 0.1522 0.15 0.45 0 0.90 Propensity Score Matching 0.5986 0.1236 0.15 0.45 0 0.90 Nearest Neighbor Matching 0.6017 0.1357 0.15 0.45 0 0.90 Radius Matching 0.5997 0.1315 0.15 0.45 0 0.90 Mahalanobis Metric Matching 0.6017 0.1355 0.15 0.45 0 0.99 Random Sample 0.6011 0.1595 0.15 0.45 0 0.99 Exact Matching 0.5998 0.1559 0.15 0.45 0 0.99 Propensity Score Matching 0.6023 0.1303 0.15 0.45 0 0.99 Nearest Neighbor Matching 0.5922 0.1271 0.15 0.45 0 0.99 Radius Matching 0.6053 0.1333 0.15 0.45 0 0.99 Mahalanobis Metric Matching 0.5927 0.1273 0.15 0.45 0.40 0 Random Sample 0.5978 0.1582 0.15 0.45 0.40 0 Exact Matching 0.5993 0.2283 0.15 0.45 0.40 0 Propensity Score Matching 0.5997 0.1590 0.15 0.45 0.40 0 Nearest Neighbor Matching 0.6000 0.1595 0.15 0.45 0.40 0 Radius Matching 0.5996 0.1560 0.15 0.45 0.40 0 Mahalanobis Metric Matching 0.5998 0.1588 0.15 0.45 0.40 0.75 Random Sample 0.6032 0.1543 0.15 0.45 0.40 0.75 Exact Matching 0.5955 0.2295 0.15 0.45 0.40 0.75 Propensity Score Matching 0.6020 0.1459 0.15 0.45 0.40 0.75 Nearest Neighbor Matching 0.5965 0.1470 0.15 0.45 0.40 0.75 Radius Matching 0.5971 0.1449 0.15 0.45 0.40 0.75 Mahalanobis Metric Matching 0.5964 0.1460 0.15 0.45 0.40 0.90 Random Sample 0.6052 0.1577 0.15 0.45 0.40 0.90 Exact Matching 0.6063 0.1970 0.15 0.45 0.40 0.90 Propensity Score Matching 0.6051 0.1442 0.15 0.45 0.40 0.90 Nearest Neighbor Matching 0.6004 0.1441 0.15 0.45 0.40 0.90 Radius Matching 0.6028 0.1430 0.15 0.45 0.40 0.90 Mahalanobis Metric Matching 0.6009 0.1437 0.15 0.45 0.40 0.99 Random Sample 0.5981 0.1510 0.15 0.45 0.40 0.99 Exact Matching 0.5992 0.1973 0.15 0.45 0.40 0.99 Propensity Score Matching 0.5934 0.1400 0.15 0.45 0.40 0.99 Nearest Neighbor Matching 0.5930 0.1461 0.15 0.45 0.40 0.99 Radius Matching 0.5945 0.1446 0.15 0.45 0.40 0.99 Mahalanobis Metric Matching 0.5924 0.1453

179

Bias ES Covariance Correlation Matching Method Mean SD 0.15 0.45 0.90 0 Random Sample 0.5923 0.1590 0.15 0.45 0.90 0 Exact Matching 0.5819 0.4572 0.15 0.45 0.90 0 Propensity Score Matching 0.5987 0.1546 0.15 0.45 0.90 0 Nearest Neighbor Matching 0.5910 0.1598 0.15 0.45 0.90 0 Radius Matching 0.5985 0.1571 0.15 0.45 0.90 0 Mahalanobis Metric Matching 0.5905 0.1592 0.15 0.45 0.90 0.75 Random Sample 0.5939 0.1552 0.15 0.45 0.90 0.75 Exact Matching 0.5885 0.4125 0.15 0.45 0.90 0.75 Propensity Score Matching 0.5976 0.1467 0.15 0.45 0.90 0.75 Nearest Neighbor Matching 0.5926 0.1503 0.15 0.45 0.90 0.75 Radius Matching 0.5942 0.1484 0.15 0.45 0.90 0.75 Mahalanobis Metric Matching 0.5924 0.1491 0.15 0.45 0.90 0.90 Random Sample 0.5903 0.1520 0.15 0.45 0.90 0.90 Exact Matching 0.5706 0.4035 0.15 0.45 0.90 0.90 Propensity Score Matching 0.5901 0.1464 0.15 0.45 0.90 0.90 Nearest Neighbor Matching 0.5989 0.1482 0.15 0.45 0.90 0.90 Radius Matching 0.5927 0.1477 0.15 0.45 0.90 0.90 Mahalanobis Metric Matching 0.5990 0.1456 0.15 0.45 0.90 0.99 Random Sample 0.6045 0.1575 0.15 0.45 0.90 0.99 Exact Matching 0.5950 0.3955 0.15 0.45 0.90 0.99 Propensity Score Matching 0.5991 0.1435 0.15 0.45 0.90 0.99 Nearest Neighbor Matching 0.5959 0.1440 0.15 0.45 0.90 0.99 Radius Matching 0.5986 0.1432 0.15 0.45 0.90 0.99 Mahalanobis Metric Matching 0.5963 0.1430

180

Appendix C

Case 3: Continuous Covariates Descriptive Statistics

181

Case 3: Mixed Covariates YD Descriptive Statistics for average of 1000 Replications Bias ES Covariance Correlation Matching Method Mean SD 0 0 0 0 Random Sample 0.0030 0.1388 0 0 0 0 Exact Matching 0.0022 0.2591 0 0 0 0 Propensity Score Matching 0.0006 0.1436 0 0 0 0 Nearest Neighbor Matching 0.0010 0.1432 0 0 0 0 Radius Matching 0.0040 0.1417 0 0 0 0 Mahalanobis Metric Matching 0.0011 0.1426 0 0 0 0.75 Random Sample 0.0068 0.1379 0 0 0 0.75 Exact Matching 0.0128 0.2136 0 0 0 0.75 Propensity Score Matching 0.0082 0.1577 0 0 0 0.75 Nearest Neighbor Matching 0.0090 0.1536 0 0 0 0.75 Radius Matching 0.0011 0.1518 0 0 0 0.75 Mahalanobis Metric Matching 0.0086 0.1525 0 0 0 0.90 Random Sample 0.0003 0.1462 0 0 0 0.90 Exact Matching 0.0094 0.2117 0 0 0 0.90 Propensity Score Matching 0.0048 0.1526 0 0 0 0.90 Nearest Neighbor Matching 0.0044 0.1588 0 0 0 0.90 Radius Matching 0.0043 0.1516 0 0 0 0.90 Mahalanobis Metric Matching 0.0047 0.1584 0 0 0 0.99 Random Sample 0.0055 0.1386 0 0 0 0.99 Exact Matching 0.0016 0.2124 0 0 0 0.99 Propensity Score Matching 0.0068 0.1520 0 0 0 0.99 Nearest Neighbor Matching 0.0049 0.1517 0 0 0 0.99 Radius Matching 0.0042 0.1516 0 0 0 0.99 Mahalanobis Metric Matching 0.0050 0.1511 0 0 0.40 0 Random Sample 0.0026 0.1451 0 0 0.40 0 Exact Matching 0.0050 0.2617 0 0 0.40 0 Propensity Score Matching 0.0022 0.1450 0 0 0.40 0 Nearest Neighbor Matching 0.0026 0.1403 0 0 0.40 0 Radius Matching 0.0023 0.1431 0 0 0.40 0 Mahalanobis Metric Matching 0.0030 0.1402 0 0 0.40 0.75 Random Sample 0.0068 0.1449 0 0 0.40 0.75 Exact Matching 0.0026 0.2352 0 0 0.40 0.75 Propensity Score Matching 0.0008 0.1438 0 0 0.40 0.75 Nearest Neighbor Matching 0.0038 0.1473 0 0 0.40 0.75 Radius Matching 0.0039 0.1476 0 0 0.40 0.75 Mahalanobis Metric Matching 0.0039 0.1469 0 0 0.40 0.90 Random Sample 0.0070 0.1449 0 0 0.40 0.90 Exact Matching 0.0080 0.2354 0 0 0.40 0.90 Propensity Score Matching 0.0021 0.1515 0 0 0.40 0.90 Nearest Neighbor Matching 0.0027 0.1568 0 0 0.40 0.90 Radius Matching 0.0006 0.1557 0 0 0.40 0.90 Mahalanobis Metric Matching 0.0032 0.1558 0 0 0.40 0.99 Random Sample 0.0085 0.1411 0 0 0.40 0.99 Exact Matching 0.0100 0.2258 0 0 0.40 0.99 Propensity Score Matching 0.0040 0.1428 0 0 0.40 0.99 Nearest Neighbor Matching 0.0011 0.1491 0 0 0.40 0.99 Radius Matching 0.0079 0.1515 0 0 0.40 0.99 Mahalanobis Metric Matching 0.0005 0.1480

182

Bias ES Covariance Correlation Matching Method Mean SD 0 0 0.90 0 Random Sample 0.0016 0.1430 0 0 0.90 0 Exact Matching 0.0037 0.3189 0 0 0.90 0 Propensity Score Matching 0.0049 0.1410 0 0 0.90 0 Nearest Neighbor Matching 0.0010 0.1414 0 0 0.90 0 Radius Matching 0.0008 0.1410 0 0 0.90 0 Mahalanobis Metric Matching 0.0009 0.1406 0 0 0.90 0.75 Random Sample 0.0024 0.1399 0 0 0.90 0.75 Exact Matching 0.0122 0.2956 0 0 0.90 0.75 Propensity Score Matching 0.0003 0.1391 0 0 0.90 0.75 Nearest Neighbor Matching 0.0010 0.1489 0 0 0.90 0.75 Radius Matching 0.0009 0.1485 0 0 0.90 0.75 Mahalanobis Metric Matching 0.0008 0.1481 0 0 0.90 0.90 Random Sample 0.0020 0.1459 0 0 0.90 0.90 Exact Matching 0.0053 0.3039 0 0 0.90 0.90 Propensity Score Matching 0.0011 0.1538 0 0 0.90 0.90 Nearest Neighbor Matching 0.0088 0.1491 0 0 0.90 0.90 Radius Matching 0.0014 0.1540 0 0 0.90 0.90 Mahalanobis Metric Matching 0.0080 0.1483 0 0 0.90 0.99 Random Sample 0.0054 0.1379 0 0 0.90 0.99 Exact Matching 0.0015 0.2127 0 0 0.90 0.99 Propensity Score Matching 0.0068 0.1519 0 0 0.90 0.99 Nearest Neighbor Matching 0.0049 0.1520 0 0 0.90 0.99 Radius Matching 0.0041 0.1616 0 0 0.90 0.99 Mahalanobis Metric Matching 0.0051 0.1508 0 0.1 0 0 Random Sample 0.1022 0.1501 0 0.1 0 0 Exact Matching 0.1005 0.2582 0 0.1 0 0 Propensity Score Matching 0.1024 0.1484 0 0.1 0 0 Nearest Neighbor Matching 0.1009 0.1477 0 0.1 0 0 Radius Matching 0.1008 0.1452 0 0.1 0 0 Mahalanobis Metric Matching 0.1012 0.1476 0 0.1 0 0.75 Random Sample 0.1011 0.1417 0 0.1 0 0.75 Exact Matching 0.1112 0.2185 0 0.1 0 0.75 Propensity Score Matching 0.1067 0.1540 0 0.1 0 0.75 Nearest Neighbor Matching 0.1084 0.1539 0 0.1 0 0.75 Radius Matching 0.1071 0.1531 0 0.1 0 0.75 Mahalanobis Metric Matching 0.1083 0.1529 0 0.1 0 0.90 Random Sample 0.0967 0.1460 0 0.1 0 0.90 Exact Matching 0.1009 0.2094 0 0.1 0 0.90 Propensity Score Matching 0.1053 0.1517 0 0.1 0 0.90 Nearest Neighbor Matching 0.1045 0.1507 0 0.1 0 0.90 Radius Matching 0.1062 0.1561 0 0.1 0 0.90 Mahalanobis Metric Matching 0.1047 0.1501 0 0.1 0 0.99 Random Sample 0.0949 0.1435 0 0.1 0 0.99 Exact Matching 0.1010 0.1994 0 0.1 0 0.99 Propensity Score Matching 0.1043 0.1503 0 0.1 0 0.99 Nearest Neighbor Matching 0.1009 0.1565 0 0.1 0 0.99 Radius Matching 0.0980 0.1571 0 0.1 0 0.99 Mahalanobis Metric Matching 0.1011 0.1554

183

Bias ES Covariance Correlation Matching Method Mean SD 0 0.1 0.40 0 Random Sample 0.1020 0.1463 0 0.1 0.40 0 Exact Matching 0.1030 0.2653 0 0.1 0.40 0 Propensity Score Matching 0.1001 0.1526 0 0.1 0.40 0 Nearest Neighbor Matching 0.1022 0.1492 0 0.1 0.40 0 Radius Matching 0.0997 0.1442 0 0.1 0.40 0 Mahalanobis Metric Matching 0.1021 0.1487 0 0.1 0.40 0.75 Random Sample 0.1000 0.1455 0 0.1 0.40 0.75 Exact Matching 0.1013 0.2340 0 0.1 0.40 0.75 Propensity Score Matching 0.0986 0.1515 0 0.1 0.40 0.75 Nearest Neighbor Matching 0.1047 0.1759 0 0.1 0.40 0.75 Radius Matching 0.0998 0.1504 0 0.1 0.40 0.75 Mahalanobis Metric Matching 0.1035 0.1504 0 0.1 0.40 0.90 Random Sample 0.1040 0.1444 0 0.1 0.40 0.90 Exact Matching 0.1023 0.2324 0 0.1 0.40 0.90 Propensity Score Matching 0.1055 0.1506 0 0.1 0.40 0.90 Nearest Neighbor Matching 0.1022 0.1556 0 0.1 0.40 0.90 Radius Matching 0.1008 0.1539 0 0.1 0.40 0.90 Mahalanobis Metric Matching 0.1018 0.1549 0 0.1 0.40 0.99 Random Sample 0.0982 0.1457 0 0.1 0.40 0.99 Exact Matching 0.0979 0.2357 0 0.1 0.40 0.99 Propensity Score Matching 0.0971 0.1490 0 0.1 0.40 0.99 Nearest Neighbor Matching 0.1066 0.1482 0 0.1 0.40 0.99 Radius Matching 0.1047 0.1479 0 0.1 0.40 0.99 Mahalanobis Metric Matching 0.1059 0.1483 0 0.1 0.90 0 Random Sample 0.1019 0.1486 0 0.1 0.90 0 Exact Matching 0.1081 0.3372 0 0.1 0.90 0 Propensity Score Matching 0.1020 0.1484 0 0.1 0.90 0 Nearest Neighbor Matching 0.1031 0.1488 0 0.1 0.90 0 Radius Matching 0.0965 0.1450 0 0.1 0.90 0 Mahalanobis Metric Matching 0.1028 0.1480 0 0.1 0.90 0.75 Random Sample 0.1046 0.1418 0 0.1 0.90 0.75 Exact Matching 0.0985 0.3057 0 0.1 0.90 0.75 Propensity Score Matching 0.1058 0.1500 0 0.1 0.90 0.75 Nearest Neighbor Matching 0.1043 0.1487 0 0.1 0.90 0.75 Radius Matching 0.0995 0.1476 0 0.1 0.90 0.75 Mahalanobis Metric Matching 0.1041 0.1483 0 0.1 0.90 0.90 Random Sample 0.1044 0.1446 0 0.1 0.90 0.90 Exact Matching 0.1006 0.2857 0 0.1 0.90 0.90 Propensity Score Matching 0.1022 0.1470 0 0.1 0.90 0.90 Nearest Neighbor Matching 0.0989 0.1457 0 0.1 0.90 0.90 Radius Matching 0.1075 0.1478 0 0.1 0.90 0.90 Mahalanobis Metric Matching 0.0991 0.1450 0 0.1 0.90 0.99 Random Sample 0.0971 0.1424 0 0.1 0.90 0.99 Exact Matching 0.0970 0.2840 0 0.1 0.90 0.99 Propensity Score Matching 0.0975 0.1503 0 0.1 0.90 0.99 Nearest Neighbor Matching 0.1012 0.1468 0 0.1 0.90 0.99 Radius Matching 0.0967 0.1485 0 0.1 0.90 0.99 Mahalanobis Metric Matching 0.1016 0.1456

184

Bias ES Covariance Correlation Matching Method Mean SD 0 0.45 0 0 Random Sample 0.4412 0.1539 0 0.45 0 0 Exact Matching 0.4462 0.2623 0 0.45 0 0 Propensity Score Matching 0.4392 0.1517 0 0.45 0 0 Nearest Neighbor Matching 0.4366 0.1549 0 0.45 0 0 Radius Matching 0.4390 0.1524 0 0.45 0 0 Mahalanobis Metric Matching 0.4365 0.1544 0 0.45 0 0.75 Random Sample 0.4375 0.1606 0 0.45 0 0.75 Exact Matching 0.4379 0.2251 0 0.45 0 0.75 Propensity Score Matching 0.4360 0.1402 0 0.45 0 0.75 Nearest Neighbor Matching 0.4362 0.1504 0 0.45 0 0.75 Radius Matching 0.4363 0.1497 0 0.45 0 0.75 Mahalanobis Metric Matching 0.4369 0.1509 0 0.45 0 0.90 Random Sample 0.4408 0.1544 0 0.45 0 0.90 Exact Matching 0.4429 0.2208 0 0.45 0 0.90 Propensity Score Matching 0.4444 0.1514 0 0.45 0 0.90 Nearest Neighbor Matching 0.4387 0.1473 0 0.45 0 0.90 Radius Matching 0.4402 0.1478 0 0.45 0 0.90 Mahalanobis Metric Matching 0.4385 0.1470 0 0.45 0 0.99 Random Sample 0.4385 0.1475 0 0.45 0 0.99 Exact Matching 0.4345 0.2013 0 0.45 0 0.99 Propensity Score Matching 0.4406 0.1457 0 0.45 0 0.99 Nearest Neighbor Matching 0.4376 0.1421 0 0.45 0 0.99 Radius Matching 0.4361 0.1439 0 0.45 0 0.99 Mahalanobis Metric Matching 0.4370 0.1406 0 0.45 0.40 0 Random Sample 0.4399 0.1540 0 0.45 0.40 0 Exact Matching 0.4462 0.2688 0 0.45 0.40 0 Propensity Score Matching 0.4475 0.1566 0 0.45 0.40 0 Nearest Neighbor Matching 0.4461 0.1567 0 0.45 0.40 0 Radius Matching 0.4457 0.1564 0 0.45 0.40 0 Mahalanobis Metric Matching 0.4455 0.1560 0 0.45 0.40 0.75 Random Sample 0.4355 0.1626 0 0.45 0.40 0.75 Exact Matching 0.4312 0.2421 0 0.45 0.40 0.75 Propensity Score Matching 0.4425 0.1484 0 0.45 0.40 0.75 Nearest Neighbor Matching 0.4375 0.1543 0 0.45 0.40 0.75 Radius Matching 0.4417 0.1514 0 0.45 0.40 0.75 Mahalanobis Metric Matching 0.4379 0.1540 0 0.45 0.40 0.90 Random Sample 0.4418 0.1538 0 0.45 0.40 0.90 Exact Matching 0.4434 0.2232 0 0.45 0.40 0.90 Propensity Score Matching 0.4408 0.1450 0 0.45 0.40 0.90 Nearest Neighbor Matching 0.4406 0.1519 0 0.45 0.40 0.90 Radius Matching 0.4453 0.1476 0 0.45 0.40 0.90 Mahalanobis Metric Matching 0.4408 0.1466 0 0.45 0.40 0.99 Random Sample 0.4412 0.1541 0 0.45 0.40 0.99 Exact Matching 0.4438 0.2303 0 0.45 0.40 0.99 Propensity Score Matching 0.4416 0.1513 0 0.45 0.40 0.99 Nearest Neighbor Matching 0.4445 0.1515 0 0.45 0.40 0.99 Radius Matching 0.4448 0.1533 0 0.45 0.40 0.99 Mahalanobis Metric Matching 0.4440 0.1513

185

Bias ES Covariance Correlation Matching Method Mean SD 0 0.45 0.90 0 Random Sample 0.4365 0.1591 0 0.45 0.90 0 Exact Matching 0.4383 0.3485 0 0.45 0.90 0 Propensity Score Matching 0.4376 0.1550 0 0.45 0.90 0 Nearest Neighbor Matching 0.4352 0.1606 0 0.45 0.90 0 Radius Matching 0.4401 0.1502 0 0.45 0.90 0 Mahalanobis Metric Matching 0.4355 0.1603 0 0.45 0.90 0.75 Random Sample 0.4388 0.1412 0 0.45 0.90 0.75 Exact Matching 0.4454 0.3061 0 0.45 0.90 0.75 Propensity Score Matching 0.4392 0.1381 0 0.45 0.90 0.75 Nearest Neighbor Matching 0.4360 0.1346 0 0.45 0.90 0.75 Radius Matching 0.4346 0.1359 0 0.45 0.90 0.75 Mahalanobis Metric Matching 0.4359 0.1339 0 0.45 0.90 0.90 Random Sample 0.4430 0.1574 0 0.45 0.90 0.90 Exact Matching 0.4491 0.2901 0 0.45 0.90 0.90 Propensity Score Matching 0.4431 0.1520 0 0.45 0.90 0.90 Nearest Neighbor Matching 0.4423 0.1550 0 0.45 0.90 0.90 Radius Matching 0.4405 0.1518 0 0.45 0.90 0.90 Mahalanobis Metric Matching 0.4423 0.1438 0 0.45 0.90 0.99 Random Sample 0.4358 0.1576 0 0.45 0.90 0.99 Exact Matching 0.4542 0.2928 0 0.45 0.90 0.99 Propensity Score Matching 0.4464 0.1484 0 0.45 0.90 0.99 Nearest Neighbor Matching 0.4458 0.1530 0 0.45 0.90 0.99 Radius Matching 0.4450 0.1512 0 0.45 0.90 0.99 Mahalanobis Metric Matching 0.4447 0.1525 0.1 0 0 0 Random Sample 0.1079 0.1453 0.1 0 0 0 Exact Matching 0.1057 0.2567 0.1 0 0 0 Propensity Score Matching 0.1015 0.1416 0.1 0 0 0 Nearest Neighbor Matching 0.1050 0.1403 0.1 0 0 0 Radius Matching 0.1089 0.1407 0.1 0 0 0 Mahalanobis Metric Matching 0.1047 0.1405 0.1 0 0 0.75 Random Sample 0.1047 0.1424 0.1 0 0 0.75 Exact Matching 0.0886 0.2158 0.1 0 0 0.75 Propensity Score Matching 0.0982 0.1472 0.1 0 0 0.75 Nearest Neighbor Matching 0.0993 0.1504 0.1 0 0 0.75 Radius Matching 0.1022 0.1469 0.1 0 0 0.75 Mahalanobis Metric Matching 0.0992 0.1490 0.1 0 0 0.90 Random Sample 0.0992 0.1430 0.1 0 0 0.90 Exact Matching 0.0948 0.2049 0.1 0 0 0.90 Propensity Score Matching 0.0945 0.1479 0.1 0 0 0.90 Nearest Neighbor Matching 0.1004 0.1468 0.1 0 0 0.90 Radius Matching 0.0978 0.1476 0.1 0 0 0.90 Mahalanobis Metric Matching 0.0999 0.1361 0.1 0 0 0.99 Random Sample 0.0977 0.1362 0.1 0 0 0.99 Exact Matching 0.0959 0.2092 0.1 0 0 0.99 Propensity Score Matching 0.1003 0.1480 0.1 0 0 0.99 Nearest Neighbor Matching 0.0966 0.1478 0.1 0 0 0.99 Radius Matching 0.1024 0.1474 0.1 0 0 0.99 Mahalanobis Metric Matching 0.0968 0.1473

186

Bias ES Covariance Correlation Matching Method Mean SD 0.1 0 0.40 0 Random Sample 0.0999 0.1412 0.1 0 0.40 0 Exact Matching 0.1053 0.2618 0.1 0 0.40 0 Propensity Score Matching 0.0993 0.1442 0.1 0 0.40 0 Nearest Neighbor Matching 0.1066 0.1427 0.1 0 0.40 0 Radius Matching 0.1050 0.1425 0.1 0 0.40 0 Mahalanobis Metric Matching 0.1060 0.1426 0.1 0 0.40 0.75 Random Sample 0.0998 0.1428 0.1 0 0.40 0.75 Exact Matching 0.1027 0.2399 0.1 0 0.40 0.75 Propensity Score Matching 0.1009 0.1470 0.1 0 0.40 0.75 Nearest Neighbor Matching 0.1002 0.1483 0.1 0 0.40 0.75 Radius Matching 0.1005 0.1497 0.1 0 0.40 0.75 Mahalanobis Metric Matching 0.1000 0.1477 0.1 0 0.40 0.90 Random Sample 0.1041 0.1417 0.1 0 0.40 0.90 Exact Matching 0.0978 0.2312 0.1 0 0.40 0.90 Propensity Score Matching 0.1091 0.1458 0.1 0 0.40 0.90 Nearest Neighbor Matching 0.1096 0.1480 0.1 0 0.40 0.90 Radius Matching 0.1015 0.1494 0.1 0 0.40 0.90 Mahalanobis Metric Matching 0.1093 0.1477 0.1 0 0.40 0.99 Random Sample 0.1010 0.1458 0.1 0 0.40 0.99 Exact Matching 0.1084 0.2268 0.1 0 0.40 0.99 Propensity Score Matching 0.1003 0.1492 0.1 0 0.40 0.99 Nearest Neighbor Matching 0.0990 0.1539 0.1 0 0.40 0.99 Radius Matching 0.1001 0.1488 0.1 0 0.40 0.99 Mahalanobis Metric Matching 0.0989 0.1537 0.1 0 0.90 0 Random Sample 0.0991 0.1415 0.1 0 0.90 0 Exact Matching 0.1075 0.3322 0.1 0 0.90 0 Propensity Score Matching 0.0995 0.1351 0.1 0 0.90 0 Nearest Neighbor Matching 0.0969 0.1408 0.1 0 0.90 0 Radius Matching 0.0975 0.1430 0.1 0 0.90 0 Mahalanobis Metric Matching 0.0973 0.1407 0.1 0 0.90 0.75 Random Sample 0.1074 0.1364 0.1 0 0.90 0.75 Exact Matching 0.1105 0.3023 0.1 0 0.90 0.75 Propensity Score Matching 0.1092 0.1455 0.1 0 0.90 0.75 Nearest Neighbor Matching 0.1049 0.1388 0.1 0 0.90 0.75 Radius Matching 0.1063 0.1448 0.1 0 0.90 0.75 Mahalanobis Metric Matching 0.1052 0.1380 0.1 0 0.90 0.90 Random Sample 0.1091 0.1438 0.1 0 0.90 0.90 Exact Matching 0.1136 0.2992 0.1 0 0.90 0.90 Propensity Score Matching 0.1001 0.1490 0.1 0 0.90 0.90 Nearest Neighbor Matching 0.1028 0.1540 0.1 0 0.90 0.90 Radius Matching 0.1011 0.1526 0.1 0 0.90 0.90 Mahalanobis Metric Matching 0.1031 0.1531 0.1 0 0.90 0.99 Random Sample 0.0980 0.1402 0.1 0 0.90 0.99 Exact Matching 0.1001 0.3092 0.1 0 0.90 0.99 Propensity Score Matching 0.1007 0.1469 0.1 0 0.90 0.99 Nearest Neighbor Matching 0.1022 0.1488 0.1 0 0.90 0.99 Radius Matching 0.1032 0.1459 0.1 0 0.90 0.99 Mahalanobis Metric Matching 0.1024 0.1479

187

Bias ES Covariance Correlation Matching Method Mean SD 0.1 0.1 0 0 Random Sample 0.1967 0.1495 0.1 0.1 0 0 Exact Matching 0.1958 0.2520 0.1 0.1 0 0 Propensity Score Matching 0.1996 0.1424 0.1 0.1 0 0 Nearest Neighbor Matching 0.2022 0.1437 0.1 0.1 0 0 Radius Matching 0.2020 0.1449 0.1 0.1 0 0 Mahalanobis Metric Matching 0.2023 0.1429 0.1 0.1 0 0.75 Random Sample 0.2000 0.1420 0.1 0.1 0 0.75 Exact Matching 0.2030 0.2195 0.1 0.1 0 0.75 Propensity Score Matching 0.2027 0.1499 0.1 0.1 0 0.75 Nearest Neighbor Matching 0.2024 0.1445 0.1 0.1 0 0.75 Radius Matching 0.2017 0.1475 0.1 0.1 0 0.75 Mahalanobis Metric Matching 0.2021 0.1434 0.1 0.1 0 0.90 Random Sample 0.1980 0.1441 0.1 0.1 0 0.90 Exact Matching 0.1963 0.2014 0.1 0.1 0 0.90 Propensity Score Matching 0.2033 0.1440 0.1 0.1 0 0.90 Nearest Neighbor Matching 0.2068 0.1486 0.1 0.1 0 0.90 Radius Matching 0.2036 0.1444 0.1 0.1 0 0.90 Mahalanobis Metric Matching 0.2072 0.1480 0.1 0.1 0 0.99 Random Sample 0.1914 0.1451 0.1 0.1 0 0.99 Exact Matching 0.1937 0.1999 0.1 0.1 0 0.99 Propensity Score Matching 0.2025 0.1490 0.1 0.1 0 0.99 Nearest Neighbor Matching 0.2063 0.1479 0.1 0.1 0 0.99 Radius Matching 0.2075 0.1470 0.1 0.1 0 0.99 Mahalanobis Metric Matching 0.2064 0.1471 0.1 0.1 0.40 0 Random Sample 0.2102 0.1467 0.1 0.1 0.40 0 Exact Matching 0.1931 0.2575 0.1 0.1 0.40 0 Propensity Score Matching 0.1986 0.1418 0.1 0.1 0.40 0 Nearest Neighbor Matching 0.2013 0.1401 0.1 0.1 0.40 0 Radius Matching 0.1989 0.1420 0.1 0.1 0.40 0 Mahalanobis Metric Matching 0.2018 0.1400 0.1 0.1 0.40 0.75 Random Sample 0.2044 0.1453 0.1 0.1 0.40 0.75 Exact Matching 0.1928 0.2371 0.1 0.1 0.40 0.75 Propensity Score Matching 0.2079 0.1495 0.1 0.1 0.40 0.75 Nearest Neighbor Matching 0.2096 0.1455 0.1 0.1 0.40 0.75 Radius Matching 0.2011 0.1485 0.1 0.1 0.40 0.75 Mahalanobis Metric Matching 0.2097 0.1447 0.1 0.1 0.40 0.90 Random Sample 0.2086 0.1487 0.1 0.1 0.40 0.90 Exact Matching 0.1998 0.2323 0.1 0.1 0.40 0.90 Propensity Score Matching 0.2064 0.1470 0.1 0.1 0.40 0.90 Nearest Neighbor Matching 0.2018 0.1439 0.1 0.1 0.40 0.90 Radius Matching 0.2036 0.1472 0.1 0.1 0.40 0.90 Mahalanobis Metric Matching 0.2021 0.1428 0.1 0.1 0.40 0.99 Random Sample 0.1978 0.1427 0.1 0.1 0.40 0.99 Exact Matching 0.2925 0.2245 0.1 0.1 0.40 0.99 Propensity Score Matching 0.2039 0.1498 0.1 0.1 0.40 0.99 Nearest Neighbor Matching 0.2034 0.1537 0.1 0.1 0.40 0.99 Radius Matching 0.1998 0.1442 0.1 0.1 0.40 0.99 Mahalanobis Metric Matching 0.2036 0.1529

188

Bias ES Covariance Correlation Matching Method Mean SD 0.1 0.1 0.90 0 Random Sample 0.2001 0.1448 0.1 0.1 0.90 0 Exact Matching 0.2038 0.3348 0.1 0.1 0.90 0 Propensity Score Matching 0.1969 0.1434 0.1 0.1 0.90 0 Nearest Neighbor Matching 0.2018 0.1462 0.1 0.1 0.90 0 Radius Matching 0.2045 0.1463 0.1 0.1 0.90 0 Mahalanobis Metric Matching 0.2013 0.1458 0.1 0.1 0.90 0.75 Random Sample 0.2054 0.1419 0.1 0.1 0.90 0.75 Exact Matching 0.2086 0.2976 0.1 0.1 0.90 0.75 Propensity Score Matching 0.2007 0.1456 0.1 0.1 0.90 0.75 Nearest Neighbor Matching 0.2011 0.1463 0.1 0.1 0.90 0.75 Radius Matching 0.2008 0.1434 0.1 0.1 0.90 0.75 Mahalanobis Metric Matching 0.2010 0.1450 0.1 0.1 0.90 0.90 Random Sample 0.1995 0.1427 0.1 0.1 0.90 0.90 Exact Matching 0.2053 0.2889 0.1 0.1 0.90 0.90 Propensity Score Matching 0.2082 0.1432 0.1 0.1 0.90 0.90 Nearest Neighbor Matching 0.2025 0.1435 0.1 0.1 0.90 0.90 Radius Matching 0.2055 0.1456 0.1 0.1 0.90 0.90 Mahalanobis Metric Matching 0.2026 0.1430 0.1 0.1 0.90 0.99 Random Sample 0.2017 0.1460 0.1 0.1 0.90 0.99 Exact Matching 0.2042 0.3002 0.1 0.1 0.90 0.99 Propensity Score Matching 0.2024 0.1479 0.1 0.1 0.90 0.99 Nearest Neighbor Matching 0.2012 0.1465 0.1 0.1 0.90 0.99 Radius Matching 0.2037 0.1433 0.1 0.1 0.90 0.99 Mahalanobis Metric Matching 0.2011 0.1457 0.1 0.45 0 0 Random Sample 0.5505 0.1529 0.1 0.45 0 0 Exact Matching 0.5444 0.2690 0.1 0.45 0 0 Propensity Score Matching 0.5455 0.1628 0.1 0.45 0 0 Nearest Neighbor Matching 0.5479 0.1565 0.1 0.45 0 0 Radius Matching 0.5469 0.1595 0.1 0.45 0 0 Mahalanobis Metric Matching 0.5482 0.1561 0.1 0.45 0 0.75 Random Sample 0.5500 0.1621 0.1 0.45 0 0.75 Exact Matching 0.5438 0.2057 0.1 0.45 0 0.75 Propensity Score Matching 0.5452 0.1411 0.1 0.45 0 0.75 Nearest Neighbor Matching 0.5510 0.1433 0.1 0.45 0 0.75 Radius Matching 0.5450 0.1415 0.1 0.45 0 0.75 Mahalanobis Metric Matching 0.5509 0.1431 0.1 0.45 0 0.90 Random Sample 0.5428 0.1638 0.1 0.45 0 0.90 Exact Matching 0.5459 0.2104 0.1 0.45 0 0.90 Propensity Score Matching 0.5511 0.1389 0.1 0.45 0 0.90 Nearest Neighbor Matching 0.5517 0.1436 0.1 0.45 0 0.90 Radius Matching 0.5523 0.1447 0.1 0.45 0 0.90 Mahalanobis Metric Matching 0.5516 0.1429 0.1 0.45 0 0.99 Random Sample 0.5504 0.1582 0.1 0.45 0 0.99 Exact Matching 0.5533 0.2079 0.1 0.45 0 0.99 Propensity Score Matching 0.5564 0.1434 0.1 0.45 0 0.99 Nearest Neighbor Matching 0.5566 0.1492 0.1 0.45 0 0.99 Radius Matching 0.5552 0.1508 0.1 0.45 0 0.99 Mahalanobis Metric Matching 0.5561 0.1488

189

Bias ES Covariance Correlation Matching Method Mean SD 0.1 0.45 0.40 0 Random Sample 0.5549 0.1565 0.1 0.45 0.40 0 Exact Matching 0.5553 0.2727 0.1 0.45 0.40 0 Propensity Score Matching 0.5571 0.1560 0.1 0.45 0.40 0 Nearest Neighbor Matching 0.5583 0.1561 0.1 0.45 0.40 0 Radius Matching 0.5599 0.1551 0.1 0.45 0.40 0 Mahalanobis Metric Matching 0.5586 0.1557 0.1 0.45 0.40 0.75 Random Sample 0.5467 0.1616 0.1 0.45 0.40 0.75 Exact Matching 0.5460 0.2427 0.1 0.45 0.40 0.75 Propensity Score Matching 0.5501 0.1474 0.1 0.45 0.40 0.75 Nearest Neighbor Matching 0.5508 0.1409 0.1 0.45 0.40 0.75 Radius Matching 0.5504 0.1466 0.1 0.45 0.40 0.75 Mahalanobis Metric Matching 0.5506 0.1405 0.1 0.45 0.40 0.90 Random Sample 0.5552 0.1583 0.1 0.45 0.40 0.90 Exact Matching 0.5473 0.2333 0.1 0.45 0.40 0.90 Propensity Score Matching 0.5528 0.1485 0.1 0.45 0.40 0.90 Nearest Neighbor Matching 0.5537 0.1462 0.1 0.45 0.40 0.90 Radius Matching 0.5543 0.1490 0.1 0.45 0.40 0.90 Mahalanobis Metric Matching 0.5532 0.1461 0.1 0.45 0.40 0.99 Random Sample 0.5455 0.1534 0.1 0.45 0.40 0.99 Exact Matching 0.5427 0.2284 0.1 0.45 0.40 0.99 Propensity Score Matching 0.5536 0.1478 0.1 0.45 0.40 0.99 Nearest Neighbor Matching 0.5539 0.1411 0.1 0.45 0.40 0.99 Radius Matching 0.5514 0.1434 0.1 0.45 0.40 0.99 Mahalanobis Metric Matching 0.5538 0.1407 0.1 0.45 0.90 0 Random Sample 0.5508 0.1577 0.1 0.45 0.90 0 Exact Matching 0.5503 0.3463 0.1 0.45 0.90 0 Propensity Score Matching 0.5468 0.1572 0.1 0.45 0.90 0 Nearest Neighbor Matching 0.5538 0.1549 0.1 0.45 0.90 0 Radius Matching 0.5543 0.1571 0.1 0.45 0.90 0 Mahalanobis Metric Matching 0.5539 0.1544 0.1 0.45 0.90 0.75 Random Sample 0.5468 0.1601 0.1 0.45 0.90 0.75 Exact Matching 0.5524 0.2962 0.1 0.45 0.90 0.75 Propensity Score Matching 0.5478 0.1471 0.1 0.45 0.90 0.75 Nearest Neighbor Matching 0.5507 0.1532 0.1 0.45 0.90 0.75 Radius Matching 0.5543 0.1497 0.1 0.45 0.90 0.75 Mahalanobis Metric Matching 0.5514 0.1528 0.1 0.45 0.90 0.90 Random Sample 0.5490 0.1589 0.1 0.45 0.90 0.90 Exact Matching 0.5411 0.2927 0.1 0.45 0.90 0.90 Propensity Score Matching 0.5496 0.1578 0.1 0.45 0.90 0.90 Nearest Neighbor Matching 0.5492 0.1539 0.1 0.45 0.90 0.90 Radius Matching 0.5482 0.1496 0.1 0.45 0.90 0.90 Mahalanobis Metric Matching 0.5485 0.1535 0.1 0.45 0.90 0.99 Random Sample 0.5516 0.1573 0.1 0.45 0.90 0.99 Exact Matching 0.5421 0.2907 0.1 0.45 0.90 0.99 Propensity Score Matching 0.5562 0.1491 0.1 0.45 0.90 0.99 Nearest Neighbor Matching 0.5523 0.1556 0.1 0.45 0.90 0.99 Radius Matching 0.5520 0.1530 0.1 0.45 0.90 0.99 Mahalanobis Metric Matching 0.5524 0.1554

190

Bias ES Covariance Correlation Matching Method Mean SD 0.15 0 0 0 Random Sample 0.1497 0.1478 0.15 0 0 0 Exact Matching 0.1461 0.2574 0.15 0 0 0 Propensity Score Matching 0.1492 0.1450 0.15 0 0 0 Nearest Neighbor Matching 0.1473 0.1500 0.15 0 0 0 Radius Matching 0.1458 0.1402 0.15 0 0 0 Mahalanobis Metric Matching 0.1482 0.1490 0.15 0 0 0.75 Random Sample 0.1452 0.1449 0.15 0 0 0.75 Exact Matching 0.1475 0.2069 0.15 0 0 0.75 Propensity Score Matching 0.1528 0.1502 0.15 0 0 0.75 Nearest Neighbor Matching 0.1488 0.1457 0.15 0 0 0.75 Radius Matching 0.1584 0.1509 0.15 0 0 0.75 Mahalanobis Metric Matching 0.1496 0.1457 0.15 0 0 0.90 Random Sample 0.1426 0.1408 0.15 0 0 0.90 Exact Matching 0.1479 0.2119 0.15 0 0 0.90 Propensity Score Matching 0.1455 0.1451 0.15 0 0 0.90 Nearest Neighbor Matching 0.1508 0.1532 0.15 0 0 0.90 Radius Matching 0.1457 0.1481 0.15 0 0 0.90 Mahalanobis Metric Matching 0.1509 0.1517 0.15 0 0 0.99 Random Sample 0.1583 0.1438 0.15 0 0 0.99 Exact Matching 0.1564 0.2033 0.15 0 0 0.99 Propensity Score Matching 0.1562 0.1393 0.15 0 0 0.99 Nearest Neighbor Matching 0.1549 0.1431 0.15 0 0 0.99 Radius Matching 0.1539 0.1438 0.15 0 0 0.99 Mahalanobis Metric Matching 0.1556 0.1423 0.15 0 0.40 0 Random Sample 0.1535 0.1378 0.15 0 0.40 0 Exact Matching 0.1598 0.2549 0.15 0 0.40 0 Propensity Score Matching 0.1532 0.1390 0.15 0 0.40 0 Nearest Neighbor Matching 0.1460 0.1413 0.15 0 0.40 0 Radius Matching 0.1445 0.1430 0.15 0 0.40 0 Mahalanobis Metric Matching 0.1460 0.1410 0.15 0 0.40 0.75 Random Sample 0.1522 0.1443 0.15 0 0.40 0.75 Exact Matching 0.1587 0.2339 0.15 0 0.40 0.75 Propensity Score Matching 0.1549 0.1401 0.15 0 0.40 0.75 Nearest Neighbor Matching 0.1532 0.1479 0.15 0 0.40 0.75 Radius Matching 0.1509 0.1401 0.15 0 0.40 0.75 Mahalanobis Metric Matching 0.1529 0.1465 0.15 0 0.40 0.90 Random Sample 0.1500 0.1440 0.15 0 0.40 0.90 Exact Matching 0.1467 0.2309 0.15 0 0.40 0.90 Propensity Score Matching 0.1506 0.1535 0.15 0 0.40 0.90 Nearest Neighbor Matching 0.1524 0.1484 0.15 0 0.40 0.90 Radius Matching 0.1459 0.1453 0.15 0 0.40 0.90 Mahalanobis Metric Matching 0.1527 0.1478 0.15 0 0.40 0.99 Random Sample 0.1518 0.1399 0.15 0 0.40 0.99 Exact Matching 0.1539 0.2258 0.15 0 0.40 0.99 Propensity Score Matching 0.1521 0.1448 0.15 0 0.40 0.99 Nearest Neighbor Matching 0.1489 0.1471 0.15 0 0.40 0.99 Radius Matching 0.1491 0.1409 0.15 0 0.40 0.99 Mahalanobis Metric Matching 0.1486 0.1465

191

Bias ES Covariance Correlation Matching Method Mean SD 0.15 0 0.90 0 Random Sample 0.1484 0.1415 0.15 0 0.90 0 Exact Matching 0.1532 0.2065 0.15 0 0.90 0 Propensity Score Matching 0.1493 0.1361 0.15 0 0.90 0 Nearest Neighbor Matching 0.1581 0.1451 0.15 0 0.90 0 Radius Matching 0.1535 0.1465 0.15 0 0.90 0 Mahalanobis Metric Matching 0.1580 0.1493 0.15 0 0.90 0.75 Random Sample 0.1539 0.1459 0.15 0 0.90 0.75 Exact Matching 0.1566 0.2973 0.15 0 0.90 0.75 Propensity Score Matching 0.1472 0.1441 0.15 0 0.90 0.75 Nearest Neighbor Matching 0.1495 0.1406 0.15 0 0.90 0.75 Radius Matching 0.1471 0.1426 0.15 0 0.90 0.75 Mahalanobis Metric Matching 0.1492 0.1401 0.15 0 0.90 0.90 Random Sample 0.1552 0.1421 0.15 0 0.90 0.90 Exact Matching 0.1507 0.2957 0.15 0 0.90 0.90 Propensity Score Matching 0.1540 0.1440 0.15 0 0.90 0.90 Nearest Neighbor Matching 0.1509 0.1448 0.15 0 0.90 0.90 Radius Matching 0.1531 0.1393 0.15 0 0.90 0.90 Mahalanobis Metric Matching 0.1508 0.1442 0.15 0 0.90 0.99 Random Sample 0.1540 0.1367 0.15 0 0.90 0.99 Exact Matching 0.1485 0.2861 0.15 0 0.90 0.99 Propensity Score Matching 0.1482 0.1407 0.15 0 0.90 0.99 Nearest Neighbor Matching 0.1495 0.1456 0.15 0 0.90 0.99 Radius Matching 0.1539 0.1423 0.15 0 0.90 0.99 Mahalanobis Metric Matching 0.1500 0.1450 0.15 0.1 0 0 Random Sample 0.2502 0.1497 0.15 0.1 0 0 Exact Matching 0.2519 0.2650 0.15 0.1 0 0 Propensity Score Matching 0.2538 0.1476 0.15 0.1 0 0 Nearest Neighbor Matching 0.2516 0.1489 0.15 0.1 0 0 Radius Matching 0.2566 0.1462 0.15 0.1 0 0 Mahalanobis Metric Matching 0.2515 0.1475 0.15 0.1 0 0.75 Random Sample 0.2010 0.1423 0.15 0.1 0 0.75 Exact Matching 0.2500 0.2134 0.15 0.1 0 0.75 Propensity Score Matching 0.2504 0.1434 0.15 0.1 0 0.75 Nearest Neighbor Matching 0.2489 0.1409 0.15 0.1 0 0.75 Radius Matching 0.2520 0.1437 0.15 0.1 0 0.75 Mahalanobis Metric Matching 0.2485 0.1415 0.15 0.1 0 0.90 Random Sample 0.2517 0.1433 0.15 0.1 0 0.90 Exact Matching 0.2560 0.2011 0.15 0.1 0 0.90 Propensity Score Matching 0.2498 0.1482 0.15 0.1 0 0.90 Nearest Neighbor Matching 0.2585 0.1463 0.15 0.1 0 0.90 Radius Matching 0.2484 0.1468 0.15 0.1 0 0.90 Mahalanobis Metric Matching 0.2581 0.1461 0.15 0.1 0 0.99 Random Sample 0.2499 0.1433 0.15 0.1 0 0.99 Exact Matching 0.2519 0.2025 0.15 0.1 0 0.99 Propensity Score Matching 0.2476 0.1419 0.15 0.1 0 0.99 Nearest Neighbor Matching 0.2468 0.1472 0.15 0.1 0 0.99 Radius Matching 0.2483 0.1397 0.15 0.1 0 0.99 Mahalanobis Metric Matching 0.2469 0.1461

192

Bias ES Covariance Correlation Matching Method Mean SD 0.15 0.1 0.40 0 Random Sample 0.2500 0.1485 0.15 0.1 0.40 0 Exact Matching 0.2447 0.2333 0.15 0.1 0.40 0 Propensity Score Matching 0.2521 0.1394 0.15 0.1 0.40 0 Nearest Neighbor Matching 0.2535 0.1460 0.15 0.1 0.40 0 Radius Matching 0.2526 0.1467 0.15 0.1 0.40 0 Mahalanobis Metric Matching 0.2530 0.1451 0.15 0.1 0.40 0.75 Random Sample 0.2505 0.1446 0.15 0.1 0.40 0.75 Exact Matching 0.2554 0.2295 0.15 0.1 0.40 0.75 Propensity Score Matching 0.2516 0.1393 0.15 0.1 0.40 0.75 Nearest Neighbor Matching 0.2529 0.1365 0.15 0.1 0.40 0.75 Radius Matching 0.2531 0.1463 0.15 0.1 0.40 0.75 Mahalanobis Metric Matching 0.2532 0.1359 0.15 0.1 0.40 0.90 Random Sample 0.2502 0.1456 0.15 0.1 0.40 0.90 Exact Matching 0.2494 0.2278 0.15 0.1 0.40 0.90 Propensity Score Matching 0.2474 0.1449 0.15 0.1 0.40 0.90 Nearest Neighbor Matching 0.2524 0.1434 0.15 0.1 0.40 0.90 Radius Matching 0.2465 0.1474 0.15 0.1 0.40 0.90 Mahalanobis Metric Matching 0.2524 0.1420 0.15 0.1 0.40 0.99 Random Sample 0.2651 0.1452 0.15 0.1 0.40 0.99 Exact Matching 0.2500 0.2331 0.15 0.1 0.40 0.99 Propensity Score Matching 0.2531 0.1445 0.15 0.1 0.40 0.99 Nearest Neighbor Matching 0.2585 0.1467 0.15 0.1 0.40 0.99 Radius Matching 0.2601 0.1492 0.15 0.1 0.40 0.99 Mahalanobis Metric Matching 0.2589 0.1461 0.15 0.1 0.90 0 Random Sample 0.2523 0.1451 0.15 0.1 0.90 0 Exact Matching 0.2524 0.3301 0.15 0.1 0.90 0 Propensity Score Matching 0.2533 0.1483 0.15 0.1 0.90 0 Nearest Neighbor Matching 0.2503 0.1444 0.15 0.1 0.90 0 Radius Matching 0.2502 0.1477 0.15 0.1 0.90 0 Mahalanobis Metric Matching 0.2500 0.1440 0.15 0.1 0.90 0.75 Random Sample 0.2437 0.1466 0.15 0.1 0.90 0.75 Exact Matching 0.2357 0.3117 0.15 0.1 0.90 0.75 Propensity Score Matching 0.2446 0.1463 0.15 0.1 0.90 0.75 Nearest Neighbor Matching 0.2481 0.1449 0.15 0.1 0.90 0.75 Radius Matching 0.2506 0.1451 0.15 0.1 0.90 0.75 Mahalanobis Metric Matching 0.2480 0.1443 0.15 0.1 0.90 0.90 Random Sample 0.2567 0.1489 0.15 0.1 0.90 0.90 Exact Matching 0.2605 0.2818 0.15 0.1 0.90 0.90 Propensity Score Matching 0.2584 0.1408 0.15 0.1 0.90 0.90 Nearest Neighbor Matching 0.2544 0.1465 0.15 0.1 0.90 0.90 Radius Matching 0.2492 0.1486 0.15 0.1 0.90 0.90 Mahalanobis Metric Matching 0.2543 0.1456 0.15 0.1 0.90 0.99 Random Sample 0.2492 0.1467 0.15 0.1 0.90 0.99 Exact Matching 0.2514 0.2883 0.15 0.1 0.90 0.99 Propensity Score Matching 0.2454 0.1429 0.15 0.1 0.90 0.99 Nearest Neighbor Matching 0.2449 0.1495 0.15 0.1 0.90 0.99 Radius Matching 0.2445 0.1470 0.15 0.1 0.90 0.99 Mahalanobis Metric Matching 0.2548 0.1487

193

Bias ES Covariance Correlation Matching Method Mean SD 0.15 0.45 0 0 Random Sample 0.5983 0.1654 0.15 0.45 0 0 Exact Matching 0.5896 0.2631 0.15 0.45 0 0 Propensity Score Matching 0.5986 0.1639 0.15 0.45 0 0 Nearest Neighbor Matching 0.6021 0.1633 0.15 0.45 0 0 Radius Matching 0.5967 0.1619 0.15 0.45 0 0 Mahalanobis Metric Matching 0.6018 0.1630 0.15 0.45 0 0.75 Random Sample 0.6004 0.1547 0.15 0.45 0 0.75 Exact Matching 0.5942 0.2111 0.15 0.45 0 0.75 Propensity Score Matching 0.6038 0.1387 0.15 0.45 0 0.75 Nearest Neighbor Matching 0.6046 0.1416 0.15 0.45 0 0.75 Radius Matching 0.5946 0.1412 0.15 0.45 0 0.75 Mahalanobis Metric Matching 0.6046 0.1409 0.15 0.45 0 0.90 Random Sample 0.6031 0.1579 0.15 0.45 0 0.90 Exact Matching 0.6154 0.2086 0.15 0.45 0 0.90 Propensity Score Matching 0.6079 0.1420 0.15 0.45 0 0.90 Nearest Neighbor Matching 0.6073 0.1411 0.15 0.45 0 0.90 Radius Matching 0.6078 0.1397 0.15 0.45 0 0.90 Mahalanobis Metric Matching 0.6073 0.1398 0.15 0.45 0 0.99 Random Sample 0.6026 0.1553 0.15 0.45 0 0.99 Exact Matching 0.6063 0.2025 0.15 0.45 0 0.99 Propensity Score Matching 0.6013 0.1379 0.15 0.45 0 0.99 Nearest Neighbor Matching 0.6056 0.1399 0.15 0.45 0 0.99 Radius Matching 0.5980 0.1414 0.15 0.45 0 0.99 Mahalanobis Metric Matching 0.6054 0.1394 0.15 0.45 0.40 0 Random Sample 0.6008 0.1618 0.15 0.45 0.40 0 Exact Matching 0.6118 0.2643 0.15 0.45 0.40 0 Propensity Score Matching 0.6004 0.1598 0.15 0.45 0.40 0 Nearest Neighbor Matching 0.6037 0.1619 0.15 0.45 0.40 0 Radius Matching 0.5996 0.1598 0.15 0.45 0.40 0 Mahalanobis Metric Matching 0.6038 0.1606 0.15 0.45 0.40 0.75 Random Sample 0.6084 0.1543 0.15 0.45 0.40 0.75 Exact Matching 0.5890 0.2295 0.15 0.45 0.40 0.75 Propensity Score Matching 0.5957 0.1458 0.15 0.45 0.40 0.75 Nearest Neighbor Matching 0.6071 0.1470 0.15 0.45 0.40 0.75 Radius Matching 0.6021 0.1447 0.15 0.45 0.40 0.75 Mahalanobis Metric Matching 0.6073 0.1460 0.15 0.45 0.40 0.90 Random Sample 0.6057 0.1513 0.15 0.45 0.40 0.90 Exact Matching 0.6108 0.2290 0.15 0.45 0.40 0.90 Propensity Score Matching 0.6041 0.1445 0.15 0.45 0.40 0.90 Nearest Neighbor Matching 0.6049 0.1423 0.15 0.45 0.40 0.90 Radius Matching 0.6029 0.1480 0.15 0.45 0.40 0.90 Mahalanobis Metric Matching 0.6044 0.1422 0.15 0.45 0.40 0.99 Random Sample 0.6064 0.1600 0.15 0.45 0.40 0.99 Exact Matching 0.6036 0.2176 0.15 0.45 0.40 0.99 Propensity Score Matching 0.6059 0.1401 0.15 0.45 0.40 0.99 Nearest Neighbor Matching 0.6077 0.1486 0.15 0.45 0.40 0.99 Radius Matching 0.6068 0.1453 0.15 0.45 0.40 0.99 Mahalanobis Metric Matching 0.6078 0.1484

194

Bias ES Covariance Correlation Match Method Mean SD 0.15 0.45 0.90 0 Random Sample 0.5995 0.1545 0.15 0.45 0.90 0 Exact Matching 0.5977 0.3395 0.15 0.45 0.90 0 Propensity Score Matching 0.5997 0.1548 0.15 0.45 0.90 0 Nearest Neighbor Matching 0.5987 0.1588 0.15 0.45 0.90 0 Radius Matching 0.5989 0.1565 0.15 0.45 0.90 0 Mahalanobis Metric Matching 0.5981 0.1585 0.15 0.45 0.90 0.75 Random Sample 0.5942 0.1550 0.15 0.45 0.90 0.75 Exact Matching 0.5767 0.2965 0.15 0.45 0.90 0.75 Propensity Score Matching 0.5968 0.1511 0.15 0.45 0.90 0.75 Nearest Neighbor Matching 0.5986 0.1495 0.15 0.45 0.90 0.75 Radius Matching 0.6014 0.1496 0.15 0.45 0.90 0.75 Mahalanobis Metric Matching 0.5994 0.1493 0.15 0.45 0.90 0.90 Random Sample 0.5999 0.1553 0.15 0.45 0.90 0.90 Exact Matching 0.6050 0.2983 0.15 0.45 0.90 0.90 Propensity Score Matching 0.5987 0.1490 0.15 0.45 0.90 0.90 Nearest Neighbor Matching 0.5979 0.1478 0.15 0.45 0.90 0.90 Radius Matching 0.6021 0.1497 0.15 0.45 0.90 0.90 Mahalanobis Metric Matching 0.5978 0.1477 0.15 0.45 0.90 0.99 Random Sample 0.6047 0.1558 0.15 0.45 0.90 0.99 Exact Matching 0.6023 0.2973 0.15 0.45 0.90 0.99 Propensity Score Matching 0.6019 0.1477 0.15 0.45 0.90 0.99 Nearest Neighbor Matching 0.6036 0.1459 0.15 0.45 0.90 0.99 Radius Matching 0.5995 0.1433 0.15 0.45 0.90 0.99 Mahalanobis Metric Matching 0.6033 0.1459

195

Appendix D

Case 1: Discrete Covariates YD t-test, 95% Confidence Interval & Probability of Coverage Statistics

196

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=0, Collinearity=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0010 0.1458 398.0 0.0028 0.4849 -0.2786 0.2774 95.20 0.5560 EM 0.0124 0.3498 398.0 0.0399 0.2571 -0.2684 0.2787 58.30 0.5471 PSM 0.0003 0.1544 397.9 0.0436 0.4945 -0.2715 0.2835 95.40 0.5550 NNM 0.0013 0.3406 398.0 0.0401 0.5064 -0.2721 0.2836 95.30 0.5557 RM 0.0020 0.1448 398.0 0.0137 0.5026 -0.2757 0.2794 94.20 0.5551 MM 0.0052 0.2601 398.0 0.0319 0.5946 -0.2881 0.2969 96.00 0.5850

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=0, Collinearity=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1022 0.1449 398.0 0.6883 0.4360 -0.1849 0.3843 94.50 0.5692 EM 0.0975 0.3518 398.0 0.6661 0.2294 -0.1875 0.3752 75.20 0.5627 PSM 0.0989 0.1563 397.9 0.6889 0.4432 -0.1847 0.3837 95.90 0.5784 NNM 0.1089 0.3389 398.0 0.7068 0.4328 -0.1825 0.3871 94.60 0.5696 RM 0.0969 0.1516 398.0 0.7037 0.4319 -0.1826 0.3862 95.50 0.5688 MM 0.0997 0.2504 398.0 0.2084 0.5666 -0.1895 0.3951 96.70 0.5846

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.44, Collinearity=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4482 0.1510 398.0 2.8299 0.0422 0.1341 0.7471 95.20 0.6130 EM 0.4630 0.3495 398.0 2.8064 0.1352 0.1276 0.7322 62.10 0.6046 PSM 0.4512 0.1388 397.9 2.8291 0.0471 0.1339 0.7469 93.90 0.6130 NNM 0.4477 0.3542 398.0 2.8153 0.0506 0.1315 0.7436 94.30 0.6121 RM 0.4488 0.1604 398.0 2.8599 0.0415 0.1385 0.7508 95.50 0.6123 MM 0.4502 0.2634 398.0 2.8829 0.8229 0.1585 0.7829 94.20 0.6244

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

197

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=0, Collinearity=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0965 0.1432 398.0 0.6629 0.4538 -0.1846 0.3715 95.10 0.5561 EM 0.1100 0.3505 398.0 0.9331 0.2183 -0.1443 0.4022 58.00 0.5465 PSM 0.0997 0.1490 397.9 0.7387 0.4269 -0.1737 0.3826 96.20 0.5563 NNM 0.0970 0.3439 398.0 0.7161 0.4336 -0.1771 0.3799 95.60 0.5570 RM 0.1013 0.1418 398.0 0.6738 0.4328 -0.1830 0.3733 95.70 0.5563 MM 0.0985 0.2632 398.0 0.8671 0.5584 -0.1522 0.4102 96.70 0.5624

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.10, Collinearity=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2054 0.1478 398.0 1.3906 0.2778 -0.0834 0.4857 94.40 0.5691 EM 0.2019 0.3535 398.0 1.4415 0.1932 -0.0748 0.4848 66.80 0.5596 PSM 0.2014 0.1451 397.9 1.4194 0.2667 -0.0793 0.4900 96.70 0.5693 NNM 0.1976 0.3260 398.0 1.3710 0.2735 -0.0863 0.4829 95.40 0.5692 RM 0.2074 0.1493 398.0 1.3543 0.2876 -0.0886 0.4811 95.10 0.5697 MM 0.2016 0.2549 398.0 1.4576 0.5384 -0.0983 0.5159 98.40 0.6142

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case1: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.45, Collinearity=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5505 0.1631 398.0 3.4880 0.0164 0.2379 0.8536 94.10 0.6157 EM 0.5523 0.3952 398.0 3.6038 0.0876 0.2512 0.8465 65.90 0.6093 PSM 0.5457 0.1315 397.9 3.5252 0.0138 0.2440 0.8802 93.70 0.6162 NNM 0.5537 0.3552 398.0 3.5119 0.0141 0.2414 0.8568 94.70 0.6154 RM 0.5490 0.1632 398.8 3.5069 0.0148 0.2406 0.8561 94.50 0.6155 MM 0.5536 0.2678 398.8 3.5907 0.0342 0.2551 0.8909 93.70 0.6358

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

198

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=0, Collinearity=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1480 0.1410 398.0 1.1120 0.3326 -0.1192 0.4191 93.80 0.5383 EM 0.1686 0.3426 398.0 1.0245 0.3312 -0.1327 0.4191 83.70 0.5518 PSM 0.1524 0.1468 397.9 1.0556 0.3519 -0.1288 0.4266 95.40 0.5554 NNM 0.1487 0.3473 398.0 1.0724 0.3504 -0.1266 0.4291 95.10 0.5557 RM 0.1486 0.1474 398.0 1.0934 0.3463 -0.1236 0.4322 94.00 0.5559 MM 0.1479 0.2623 398.0 1.0678 0.3497 -0.1277 0.4402 95.10 0.5679

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.10, Collinearity=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2534 0.1422 398.0 1.6286 0.2271 -0.0491 0.5303 93.70 0.5694 EM 0.2614 0.3417 398.0 1.7154 0.2139 -0.0368 0.5292 86.10 0.5661 PSM 0.2583 0.1449 397.9 1.7153 0.2028 -0.0363 0.5328 95.00 0.5691 NNM 0.2534 0.3373 398.0 1.6951 0.2018 -0.0396 0.5294 95.40 0.5690 RM 0.2492 0.1429 398.0 1.7143 0.2045 -0.0372 0.5321 95.10 0.5693 MM 0.2500 0.2639 398.0 1.6857 0.2023 -0.0411 0.5301 95.50 0.5712

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.45, Collinearity=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.6031 0.1516 398.0 3.9063 0.0072 0.3019 0.9157 94.80 0.6138 EM 0.6165 0.3438 398.0 3.9716 0.0617 0.3060 0.9135 63.50 0.6075 PSM 0.6029 0.1296 397.9 3.8643 0.0068 0.2954 0.9093 94.10 0.6139 NNM 0.5953 0.3400 398.0 3.8994 0.0071 0.3011 0.9153 93.80 0.6142 RM 0.6019 0.1592 398.0 3.8829 0.0074 0.2984 0.9121 93.70 0.6137 MM 0.5978 0.2629 398.0 3.8727 0.0069 0.2834 0.9141 91.40 0.6307

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

199

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=0, Collinearity=.40 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0024 0.1446 398.0 0.0179 0.4840 -0.2754 0.2802 94.40 0.5556 EM 0.0023 0.3489 398.0 0.0175 0.2498 -0.2710 0.2756 54.90 0.5466 PSM 0.0002 0.1482 397.9 0.0015 0.4774 -0.2779 0.2784 94.20 0.5563 NNM 0.0068 0.3500 398.0 0.0459 0.2563 -0.2680 0.2815 93.30 0.5495 RM 0.0024 0.1439 398.0 0.0161 0.5006 -0.2758 0.2806 94.80 0.5564 MM 0.0053 0.2643 398.0 0.0226 0.4967 -0.3164 0.3057 94.70 0.6221

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 1:-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.10, Collinearity=.40 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0961 0.1419 398.0 0.7194 0.4308 -0.1804 0.3894 95.90 0.5698 EM 0.0972 0.3568 398.0 0.6862 0.2320 -0.1834 0.3766 56.20 0.5600 PSM 0.0960 0.1503 397.9 0.7262 0.4214 -0.1798 0. 3901 94.20 0.5699 NNM 0.0970 0.3425 398.0 0.7510 0.4328 -0.1746 0.3868 92.60 0.5614 RM 0.0955 0.1531 398.0 0.7384 0.4365 -0.1780 0.3914 94.60 0.5694 MM 0.0965 0.2570 398.0 0.3973 0.4881 -0.2101 0.4165 96.30 0.6266

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.44, Collinearity=.40 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4357 0.1540 398.0 2.7842 0.0453 0.1273 0.7420 95.90 0.6147 EM 0.4368 0.3568 398.0 2.8348 0.1379 0.1310 0.7383 59.00 0.6073 PSM 0.4402 0.1569 397.9 2.8189 0.0458 0.1327 0.7476 94.90 0.6149 NNM 0.4409 0.3377 398.0 2.8707 0.0450 0.1368 0.7450 92.30 0.6082 RM 0.4372 0.1528 398.0 2.7703 0.0467 0.1250 0.7397 95.00 0.6147 MM 0.4425 0.2645 398.0 2.4589 0.0424 0.1569 0.7929 97.70 0.6360

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

200

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=0, Collinearity=.40 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1023 0.1380 398.0 0.7229 0.4238 -0.1756 0.3802 96.30 0.5548 EM 0.1073 0.1372 398.0 0.7648 0.2416 -0.1664 0.3809 58.40 0.5473 PSM 0.1037 0.1372 397.9 0.7328 0.4227 -0.1747 0.3820 95.20 0.5567 NNM 0.0997 0.2647 398.0 0.7149 0.4234 -0.1757 0.3711 92.60 0.5468 RM 0.1013 0.1461 398.0 0.7155 0.4155 -0.1766 0.3791 93.70 0.5557 MM 0.0996 0.2647 398.0 0.7387 0.4686 -0.1741 0.4502 95.30 0.6243

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.10, Collinearity=.40 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2054 0.1478 398.0 1.3669 0.2752 -0.0867 0.4827 96.60 0.5695 EM 0.2019 0.3535 398.0 1.3103 0.2002 -0.0946 0.4670 56.80 0.5617 PSM 0.2014 0.1451 397.9 1.3553 0.2902 -0.0889 0.4810 95.30 0.5699 NNM 0.1966 0.3260 398.0 1.4274 0.2615 -0.0786 0.4799 94.40 0.5585 RM 0.2074 0.1493 398.0 1.3868 0.2696 -0.0841 0.4849 95.70 0.5690 MM 0.2016 0.2549 398.0 1.5476 0.3425 -0.0933 0.5783 96.40 0.6716

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.45, Collinearity=.40 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5565 0.1562 398.0 3.5661 0.0133 0.2492 0.8638 94.90 0.6146 EM 0.5429 0.3482 398.0 3.4060 0.0982 0.2194 0.8264 61.70 0.6070 PSM 0.5499 0.1595 397.9 3.5222 0.0171 0.2424 0.8574 94 0.6150 NNM 0.5491 0.3492 398.0 3.5835 0.0231 0.2457 0.8526 93.70 0.6069 RM 0.5545 0.1540 398.8 3.5522 0.0123 0.2470 0.8618 96.00 0.6148 MM 0.5506 0.2735 398.8 3.8139 0.0145 0.2512 0.9150 97.70 0.6638

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

201

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=0, Collinearity=.40 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1471 0.1386 398.0 1.0383 0.3675 -0.1312 0.4255 95.40 0.5566 EM 0.1574 0.3482 398.0 1.1382 0.2229 -0.1165 0.4312 58.80 0.5477 PSM 0.1519 0.1385 397.9 1.0754 0.3469 -0.1264 0.4302 95.30 0.5566 NNM 0.1628 0.3311 398.0 1.0876 0.3221 -0.1108 0.4363 93.60 0.5471 RM 0.1451 0.1391 398.0 1.0264 0.3652 -0.1332 0.4334 95.50 0.5567 MM 0.1617 0.2439 398.0 1.0672 0.4524 -0.1490 0.5624 96.20 0.7114

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.10, Collinearity=.40 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2487 0.1410 398.0 1.6488 0.2121 -0.0462 0.5235 95.10 0.5697 EM 0.2595 0.3429 398.0 1.8319 0.1817 -0.0214 0.5408 59.60 0.5622 PSM 0.2449 0.1382 397.9 1.6787 0.2056 -0.0417 0.5275 96.50 0.5692 NNM 0.2445 0.3524 398.0 1.6974 0.1927 -0.0385 0.5231 94.80 0.5616 RM 0.2447 0.1457 398.0 1.6911 0.2168 -0.0402 0.5286 95.00 0.5688 MM 0.2444 0.2663 398.0 1.6857 0.2023 -0.0878 0.5973 95.70 0.6851

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.45, Collinearity=.40 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5970 0.1565 398.0 3.8161 0.0068 0.2890 0.9050 94.90 0.6160 EM 0.5948 0.3632 398.0 3.8692 0.0780 0.2909 0.8988 58.70 0.6079 PSM 0.6014 0.1467 397. 3.7586 0.0057 0.2865 0.9163 97.20 0.6298 NNM 0.6117 0.3317 398.0 3.9614 0.0059 0.3064 0.9169 94.00 0.6105 RM 0.5978 0.1583 398.0 3.8197 0.0076 0.2897 0.9059 94.60 0.6162 MM 0.6083 0.2480 398.0 3.9125 0.0069 0.2935 0.9645 95.40 0.6810

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

202

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=0, Collinearity=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0018 0.1422 398.0 0.0131 0.4965 -0.2796 0.2759 94.40 0.5555 EM 0.0054 0.3455 398.0 0.0411 0.2483 -0.2700 0.2809 59.90 0.5509 PSM 0.0062 0.1442 397.9 0.0443 0.4901 -0.2718 0.2842 94.40 0.5559 NNM 0.0047 0.3369 398.0 0.0314 0.4333 -0.2780 0.2890 92.40 0.5586 RM 0.0008 0.1390 398.0 0.0275 0.5107 -0.2774 0.2790 95.30 0.5564 MM 0.0090 0.2589 398.0 0.0353 0.5015 -0.2752 0.2953 95.20 0.5705

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.10, Collinearity=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0989 0.1448 398.0 0.6867 0.4326 -0.1851 0.3829 96.20 0.5680 EM 0.0956 0.3435 398.0 0.6821 0.2380 -0.1846 0.3737 57.60 0.5583 PSM 0.1034 0.1496 397.9 0.7261 0.4165 -0.1793 0. 3892 94.10 0.5685 NNM 0.0979 0.3484 398.0 0.7016 0.4328 -0.1812 0.3769 92.40 0.5581 RM 0.1045 0.1446 398.0 0.7410 0.4206 -0.1775 0.3915 94.40 0.5690 MM 0.0973 0.2663 398.0 0.7364 0.4560 -0.1843 0.3970 95.20 0.5913

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.44, Collinearity=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4351 0.1500 398.0 2.7849 0.0432 0.1277 0.7425 96.00 0.6148 EM 0.4346 0.3522 398.0 2.8246 0.0329 0.1310 0.7382 60.40 0.6072 PSM 0.4389 0.1505 397.9 2.8090 0.0426 0.1313 0.7464 95.20 0.6151 NNM 0.4350 0.3594 398.0 2.7662 0.0353 0.1218 0.7283 91.30 0.6065 RM 0.4359 0.1505 398.0 2.7885 0.0439 0.1283 0.7436 96.00 0.6153 MM 0.4371 0.2718 398.0 2.8601 0.0420 0.1267 0.8175 97.80 0.6908

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

203

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=0, Collinearity=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0996 0.1433 398.0 0.7050 0.4220 -0.1780 0.3773 95.10 0.5553 EM 0.1152 0.3464 398.0 0.8403 0.3234 -0.1588 0.3891 57.50 0.5479 PSM 0.1088 0.1447 397.9 0.7697 0.4064 -0.1691 0.3867 95.10 0.5558 NNM 0.0959 0.3377 398.0 0.6912 0.4243 -0.1786 0.3704 93.50 0.5490 RM 0.1035 0.1413 398.0 0.7304 0.4164 -0.1749 0.3818 95.10 0.5567 MM 0.0965 0.2604 398.0 0.7370 0.4367 -0.1796 0.3981 95.00 0.5777

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.10, Collinearity=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2106 0.1421 398.0 1.4560 0.2616 -0.0744 0.4955 95.30 0.5699 EM 0.1976 0.3464 398.0 1.3990 0.2180 -0.0837 0.4765 57.80 0.5602 PSM 0.2069 0.1473 397.9 1.4315 0.2594 -0.0779 0.4917 94.40 0.5696 NNM 0.1973 0.3458 398.0 1.3850 0.2085 -0.0849 0.4757 92.40 0.5606 RM 0.2098 0.1574 398.0 1.4489 0.2694 -0.0751 0.4946 92.90 0.5696 MM 0.2015 0.2595 398.0 1.4735 0.4274 -0.0814 0.5005 96.40 0.5819

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.45, Collinearity=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5512 0.1621 398.0 3.5280 0.0130 0.2435 0.8589 93.60 0.6154 EM 0.5495 0.3416 398.0 3.5816 0.0809 0.2457 0.8532 63.10 0.6075 PSM 0.5491 0.1496 397.9 3.5140 0.0103 0.2414 0.8567 96.10 0.6153 NNM 0.5543 0.3527 398.0 3.5294 0.0145 0.2425 0.8587 92.70 0.6162 RM 0.5476 0.1562 398.8 3.5057 0.0139 0.2401 0.8551 95.50 0.6150 MM 0.5482 0.2673 398.8 3.5389 0.0169 0.2432 0.8663 97.30 0.6431

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

204

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=0, Collinearity=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1482 0.1367 398.0 1.0383 0.3675 -0.1312 0.4255 95.70 0.5565 EM 0.1502 0.3484 398.0 1.1382 0.2229 -0.1165 0.4312 57.60 0.5455 PSM 0.1524 0.1407 397.9 1.0754 0.3469 -0.1264 0.4302 94.70 0.5555 NNM 0.1569 0.3468 398.0 1.0876 0.3221 -0.1108 0.4363 94.90 0.5485 RM 0.1464 0.1453 398.0 1.0264 0.3652 -0.1332 0.4334 95.10 0.5561 MM 0.1503 0.2591 398.0 1.0672 0.4524 -0.1390 0.5324 94.70 0.6714

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 1; t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.10, Collinearity=.90 1000 Replications Average Method % YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2586 0.1465 398.0 1.7890 0.1825 -0.0260 0.5432 95.10 0.5692 EM 0.2635 0.3290 398.0 1.8675 0.2085 -0.0173 0.5442 61.60 0.5695 PSM 0.2576 0.1382 397.9 1.7814 0.1780 -0.0271 0.5425 96.60 0.5696 NNM 0.2674 0.3530 398.0 1.7951 0.1798 -0.0135 0.5478 92.90 0.5613 RM 0.2539 0.1481 398.0 1.7562 0.2023 -0.0310 0.5387 95.30 0.5697 MM 0.2688 0.2688 398.0 1.7877 0.2365 -0.0362 0.5570 95.10 0.5932

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 1: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.45, Collinearity=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width EM 0.6031 0.1616 398.0 3.8573 0.0062 0.2953 0.9109 95.20 0.6156 PSM 0.6064 0.3565 398.0 3.9354 0.0720 0.3018 0.9100 60.80 0.6092 NNM 0.5977 0.1556 397. 3.8197 0.0065 0.2897 0.9058 95.30 0.6161 RM 0.5994 0.3419 398.0 3.8295 0.0075 0.2852 0.8936 91.90 0.6094 MM 0.6058 0.1599 398.0 3.8737 0.0061 0.2880 0.9135 94.90 0.6155 NOM 0.6018 0.2648 398.0 3.8790 0.0069 0.2935 0.9447 97.50 0.6512

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

205

Appendix E

Case 2: Continuous Covariates YD t-test, 95% Confidence Interval & Probability of Coverage Statistics

206

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=0, Collinearity=0 & Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0081 0.1403 395.0 0.0667 0.5085 -0.2884 0.2695 94.70 0.5580 EM 0.0099 0.1477 398.0 0.0814 0.4916 -0.2890 0.2661 93.60 0.5551 PSM 0.0079 0.1888 395.6 0.0547 0.4061 -0.2848 0.2691 86.20 0.5539 NNM 0.0042 0.1423 397.9 0.0295 0.4998 -0.2816 0.2731 95.00 0.5546 RM 0.0086 0.1409 398.0 0.0701 0.5071 -0.2878 0.2679 94.80 0.5558 MM 0.0062 0.1420 397.9 0.0440 0.4922 -0.2842 0.2718 95.20 0.5560

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.10, Collinearity=0 & Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1145 0.1427 395.0 0.7906 0.4011 -0.1710 0.4004 95.70 0.5714 EM 0.1112 0.1460 398.0 0.7688 0.4152 -0.1732 0.1447 95.00 0.5688 PSM 0.1017 0.1982 395.5 0.7022 0.3612 -0.1838 0.3868 84.20 0.5691 NNM 0.1039 0.1387 397.9 0.7185 0.4229 -0.1809 0.1449 96.10 0.5696 RM 0.1148 0.1427 398.0 0.7945 0.3997 -0.1697 0.3993 95.40 0.5690 MM 0.1083 0.1443 397.9 0.7477 0.4105 -0.1765 0.3931 95.20 0.5696

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

: Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.44, Collinearity=0 & Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4340 0.1618 395.0 2.8254 0.0469 0.1341 0.7505 94.60 0.6165 EM 0.4436 0.1548 398.0 2.8602 0.0482 0.1391 0.7536 94.20 0.6144 PSM 0.4435 0.2126 395.5 2.8325 0.0820 0.1342 0.7470 86.20 0.6130 NNM 0.4398 0.1553 397.9 2.8490 0.0468 0.1374 0.7509 94.40 0.6135 RM 0.4342 0.1621 398.0 2.8355 0.0475 0.1350 0.7487 93.90 0.6137 MM 0.4411 0.1593 397.9 2.8130 0.0479 0.1320 0.7461 95.50 0.6141

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

207

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=0, Collinearity=0 & Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0978 0.1388 395.0 0.6902 0.4383 -0.1813 0.3768 95.60 0.5581 EM 0.1014 0.1402 398.0 0.7179 0.4210 -0.1767 0.3795 95.80 0.5563 PSM 0.1011 0.1934 395.6 0.7169 0.3656 -0.1767 0.3788 85.20 0.5555 NNM 0.1035 0.1397 397.9 0.7331 0.4298 -0.1746 0.3816 95.00 0.5562 RM 0.0978 0.1395 398.0 0.6930 0.4359 -0.1802 0.3758 95.40 0.5559 MM 0.0969 0.1408 397.9 0.6854 0.4331 -0.1811 0.3749 94.10 0.5560

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.10, Collinearity=0 & Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2008 0.1475 395.0 1.3848 0.2742 -0.0849 0.4866 95.70 0.5716 EM 0.2018 0.1485 398.0 1.3981 0.2708 -0.0827 0.4864 93.40 0.5682 PSM 0.2090 0.2011 395.6 1.4515 0.2668 -0.0751 0.1445 86.50 0.5692 NNM 0.2008 0.1495 397.9 1.3938 0.2739 -0.0833 0.4850 94.20 0.5684 RM 0.2008 0.1486 398.0 1.3894 0.2720 -0.0838 0.4854 94.80 0.5693 MM 0.2015 0.1467 397.9 1.3970 0.2832 -0.0838 0.4862 95.30 0.5697

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.45, Collinearity=0 & Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5489 0.1484 395.0 3.5553 0.0136 0.2485 0.8655 94.80 0.6179 EM 0.5495 01724 398.0 3.5542 0.0148 0.2478 0.8620 93.30 0.6149 PSM 0.5378 0.1845 395.5 3.5080 0.0382 0.2399 0.8532 85.50 0.6127 NNM 0.5468 0.1433 397.9 3.5474 0.0122 0.2466 0.8611 94.50 0.6144 RM 0.5489 0.1492 398.0 3.5724 0.0135 0.2502 0.8643 94.30 0.6152 MM 0.5494 0.1502 397.9 3.5322 0.0145 0.2447 0.8601 93.80 0.6148

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

208

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=0, Collinearity=0 & Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1517 0.1440 395.0 1.0448 0.3612 -0.1309 0.4276 94.20 0.5564 EM 0.1515 0.1381 398.0 1.0574 0.3488 -0.1296 0.4266 94.00 0.5558 PSM 0.1588 0.1938 395.7 1.1620 0.2889 -0.1144 0.4420 83.90 0.5563 NNM 0.1545 0.1388 397.9 1.0996 0.3438 -0.1229 0.4335 93.40 0.5562 RM 0.1513 0.1448 398.0 1.0489 0.3601 -0.1299 0.4265 93.90 0.5544 MM 0.1471 0.1438 397.9 1.0767 0.3499 -0.1259 0.4303 93.80 0.5559

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.10, Collinearity=0 & Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2489 0.1458 395.0 1.7155 0.1980 -0.0365 0.5350 94.90 0.5715 EM 0.2483 0.1450 398.0 1.7600 0.1913 -0.0299 0.5396 95.60 0.5696 PSM 0.2395 0.4382 395.6 1.7776 0.2188 -0.0277 0.5408 84.90 0.5686 NNM 0.2473 0.1423 397.9 1.7515 0.1949 -0.0312 0.5375 94.20 0.5697 RM 0.2486 0.1458 398.0 1.7236 0.1979 -0.0351 0.5342 94.60 0.5692 MM 0.2472 0.1421 397.9 1.7125 0.2055 -0.0370 0.5327 95.10 0.5695

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.45, Collinearity=0 & Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5998 0.1598 395.0 3.8185 0.0075 0.2907 0.9089 95.10 0.6182 EM 0.5958 0.1607 398.0 3.8067 0.0074 0.2877 0.9039 94.20 0.6162 PSM 0.5972 0.2141 395.5 3.8315 0.0247 0.2899 0.9045 85.40 0.6146 NNM 0.5990 0.1546 397.9 3.8303 0.0058 0.2910 0.9070 95.70 0.6160 RM 0.6008 0.1599 398.0 3.8421 0.0074 0.2931 0.9085 94.70 0.6153 MM 0.6011 0.1597 397.9 3.8420 0.0071 0.2932 0.9091 94.60 0.6159

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

209

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=0, Collinearity=0 & Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0009 0.1507 395.0 0.0057 0.4858 -0.2798 0.2880 93.60 0.5578 EM 0.0017 0.1398 398.0 0.0122 0.5013 -0.2763 0.2796 95.70 0.5559 PSM 0.0038 0.1765 395.5 0.0278 0.4227 -0.2707 0.2783 88.40 0.5490 NNM 0.0047 0.1469 397.9 0.0344 0.4825 -0.2734 0.2828 94.00 0.5562 RM 0.0012 0.1513 398.0 0.0078 0.4822 -0.2791 0.2766 93.70 0.5557 MM 0.0071 0.1485 397.9 0.0503 0.4803 -0.2850 0.2708 93.70 0.5558

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.10, Collinearity=0 & Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1049 0.1494 395.1 0.6676 0.4264 -0.1887 0.3826 93.30 0.5580 EM 0.1050 0.1459 398.0 0.6928 0.4246 -0.1875 0.3826 94.20 0.5551 PSM 0.0996 0.1729 395.5 0.3983 0.3882 -0.1783 0.3845 89.50 0.5539 NNM 0.1041 0.1461 397.9 0.6994 0.4135 -0.1835 0.3854 94.80 0.5546 RM 0.1046 0.1509 398.0 0.6720 0.4247 -0.1873 0.3817 93.20 0.5558 MM 0.1037 0.1489 397.9 0.7075 0.4226 -0.1823 0.3874 94.40 0.5560

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.44, Collinearity=0 & Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4362 0.1477 395.0 2.8390 0.0375 0.1362 0.7517 96.80 0.6155 EM 0.4438 0.1579 398.0 2.8122 0.0462 0.1317 0.7452 94.570 0.6135 PSM 0.4297 0.1940 395.6 2.8852 0.0560 0.1413 0.7491 90.40 0.6078 NNM 0.4385 0.1476 397.9 2.8728 0.0326 0.1409 0.7543 96.80 0.6134 RM 0.4373 0.1474 398.0 2.8518 0.0374 0.1376 0.7502 96.60 0.6127 MM 0.4375 0.1498 397.9 2.8317 0.0354 0.1346 0.7477 98.80 0.6132

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

210

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=0, Collinearity=0 & Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1060 0.1451 395.0 0.7463 0.4258 -0.1732 0.3853 94.70 0.5586 EM 0.1076 0.1434 398.0 0.7615 0.4079 -0.1705 0.3857 95.30 0.5562 PSM 0.1040 0.1678 395.6 0.7456 0.3898 -0.1711 0.3791 88.10 0.5503 NNM 0.1037 0.1446 397.9 0.7323 0.4189 -0.1743 0.3817 94.50 0.5560 RM 0.1059 0.1461 398.0 0.7481 0.4226 -0.1723 0.3840 94.60 0.5564 MM 0.1016 0.1432 397.9 0.7190 0.4279 -0.1763 0.3795 94.50 0.5558

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.10, Collinearity=0 & Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1978 0.1419 395.0 1.3655 0.2843 -0.0878 0.4835 95.60 0.5713 EM 0.1978 0.1498 398.0 1.3662 0.2801 -0.0869 0.4824 94.60 0.5693 PSM 0.1982 0.1667 395.5 1.3848 0.2701 -0.0837 0.4802 91.50 0.5640 NNM 0.1975 0.1405 397.9 1.3663 0.2714 -0.0872 0.4822 94.40 0.5694 RM 0.1978 0.1422 398.0 1.3705 0.2835 -0.0868 0.4823 95.00 0.5691 MM 0.2013 0.1492 397.9 1.3945 0.2711 -0.0832 0.4858 94.10 0.5689

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.45, Collinearity=0 & Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5412 0.1495 395.0 3.4763 0.0084 0.2365 0.8537 98.50 0.6172 EM 0.5450 0.1547 398.0 3.5008 0.0122 0.2393 0.8538 96.50 0.6145 PSM 0.5385 0.4156 395.6 3.5398 0.0176 0.2422 0.8499 93.70 0.6077 NNM 0.5432 0.1481 397.9 3.5085 0.0073 0.2402 0.8545 98.00 0.6143 RM 0.5412 0.1501 398.0 3.4925 0.0083 0.2380 0.8523 98.20 0.6143 MM 0.5439 0.1511 397.9 3.5193 0.0062 0.2423 0.8537 98.60 0.6114

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

211

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=0, Collinearity=0 & Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1451 0.1411 395.0 1.0233 0.3645 0.1343 0.4246 95.20 0.5589 EM 0.1501 0.1401 398.0 1.0632 0.3459 -0.1280 0.4281 94.90 0.5562 PSM 0.1498 0.1624 395.5 1.0735 0.3308 -0.1253 0.4248 90.70 0.5501 NNM 0.1513 0.1428 397.9 1.0706 0.3518 -0.1270 0.4295 95.60 0.5565 RM 0.1452 0.1420 398.0 1.0271 0.3621 -0.1332 0.4236 95.00 0.5568 MM 0.1564 0.1422 397.9 1.1056 0.3320 -0.1220 0.4347 94.50 0.5567

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D YD

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.10, Collinearity=0 & Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2526 0.1402 395.0 1.7387 0.1909 -0.0337 0.5388 96.30 0.5725 EM 0.2547 0.1469 398.0 1.5302 0.0752 -0.0576 0.4541 94.60 0.5697 PSM 0.2598 0.1617 395.6 1.8192 0.2011 -0.2133 0.5409 92.30 0.5622 NNM 0.2581 0.1392 397.9 1.7870 0.1797 -0.0262 0.5424 95.40 0.5684 RM 0.2521 0.1406 398.0 1.7426 0.1912 -0.0329 0.5372 96.00 0.5701 MM 0.2536 0.1373 397.9 1.7531 0.1913 -0.0313 0.5386 96.20 0.5679

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.45, Collinearity=0 & Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5998 0.1598 395.0 3.8323 0.0045 0.2925 0.9104 97.30 0.6178 EM 0.5958 0.1607 398.0 3.8392 0.0068 0.2923 0.9075 94.60 0.6152 PSM 0.5972 0.2141 395.6 3.8608 0.0078 0.2917 0.9003 95.30 0.6085 NNM 0.5990 0.1546 397.9 3.8286 0.0026 0.2908 0.9063 98.90 0.6156 RM 0.6008 0.1599 398.0 3.8507 0.0044 0.2941 0.9074 97.00 0.6151 MM 0.6011 0.1597 397.9 3.8347 0.0038 0.2919 0.9074 98.10 0.6155

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

212

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=0 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0042 0.1519 395.0 0.0297 0.4772 -0.2747 0.2832 93.60 0.5579 EM 0.0028 0.1465 398.0 0.0192 0.4782 -0.2801 0.2745 94.30 0.5546 PSM 0.0011 0.1771 396.0 0.0080 0.4237 -0.2755 0.2734 87.40 0.5489 NNM 0.0004 0.1572 397.9 0.0021 0.4549 -0.2776 0.2786 92.60 0.5562 RM 0.0046 0.1532 398.0 0.0329 0.4689 -0.2732 0.2825 93.20 0.5557 MM 0.0035 0.1505 397.9 0.0241 0.4742 -0.2811 0.2742 93.00 0.5553

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.10 Collinearity=0 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1085 0.1530 395.0 0.7446 0.4108 -0.1776 0.3944 94.50 0.5720 EM 0.1019 0.1474 398.0 0.7002 0.4338 -0.1835 0.3874 95.00 0.5709 PSM 0.1045 0.1717 396.0 0.7334 0.3882 -0.1767 0.3857 89.80 0.5624 NNM 0.1086 0.1494 397.9 0.7472 0.4056 -0.1765 0.3936 93.60 0.5701 RM 0.1081 0.1534 398.0 0.7456 0.4100 -0.1767 0.3929 94.40 0.5696 MM 0.1016 0.1521 397.9 0.6997 0.4258 -0.1833 0.3865 93.90 0.5698

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=0 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4408 0.1526 395.2 2.7837 0.0378 0.1278 0.7448 97.70 0.6170 EM 0.4435 0.1555 398.0 2.8297 0.0445 0.1347 0.7487 95.50 0.6140 PSM 0.4332 0.3942 395.5 2.8463 0.0507 0.1355 0.7432 93.50 0.6077 NNM 0.4388 0.1540 397.9 2.8046 0.0361 0.1307 0.7448 97.60 0.6141 RM 0.4406 0.1528 398.0 2.7956 0.0376 0.1291 0.7433 97.50 0.6142 MM 0.4423 0.1505 397.9 2.7975 0.0358 0.1296 0.7439 97.00 0.6143

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

213

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=0 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1018 0.1494 395.2 0.7172 0.4245 -0.1776 0.3812 92.60 0.5588 EM 0.0996 0.1388 398.0 0.7052 0.4219 -0.1781 0.3774 95.30 0.5554 PSM 0.1073 0.1683 395.6 0.7681 0.3943 -0.1673 0.3820 89.70 0.5493 NNM 0.1058 0.1417 397.9 0.4234 0.4233 -0.1719 0.3835 95.10 0.5554 RM 0.1026 0.1502 398.0 0.7259 0.4214 -0.1757 0.3809 92.70 0.5566 MM 0.1012 0.1416 397.9 0.7166 0.4366 -0.1768 0.3792 94.90 0.5560

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.10 Collinearity=0 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1978 0.1412 395.0 1.3766 0.2817 -0.0858 0.4847 95.90 0.5707 EM 0.2080 0.1462 398.0 1.4395 0.2599 -0.0763 0.4923 94.00 0.5686 PSM 0.2093 0.1640 395.5 1.4685 0.3943 -0.0716 0.4901 90.40 0.5619 NNM 0.2059 0.1410 397.9 1.4247 0.2699 -0.0785 0.4902 95.50 0.5687 RM 0.2003 0.1424 398.0 1.3882 0.2784 -0.0837 0.8444 95.50 0.5682 MM 0.2004 0.1400 397.9 1.3865 0.2732 -0.0839 0.4848 96.40 0.5687

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=0 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5542 0.1321 395.2 3.5434 0.0065 0.2464 0.8620 98.00 0.6156 EM 0.5503 0.1557 398.0 3.5315 0.0132 0.2436 0.8570 95.10 0.6134 PSM 0.5469 0.1499 395.6 3.5497 0.0121 0.2433 0.8505 94.70 0.6072 NNM 0.5545 0.1297 397.9 3.5560 0.0055 0.2476 0.8613 98.40 0.6137 RM 0.5549 0.1326 398.0 3.5636 0.0063 0.2485 0.8813 97.70 0.6128 MM 0.5534 0.1290 397.9 3.5504 0.0056 0.2466 0.8602 97.80 0.6136

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

214

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=0 Collinearity=0 and Correlation=.90 1000 Replications Average % Method YD* SD Df t-value p-value LCLM UCLM Coverage Width NOM 0.1451 0.1411 395.0 1.0233 0.3645 0.1343 0.4246 95.20 0.5589 EM 0.1501 0.1401 398.0 1.0632 0.3459 -0.1280 0.4281 94.90 0.5562 PSM 0.1498 0.1624 395.5 1.0735 0.3308 -0.1253 0.4248 90.70 0.5501 NNM 0.1513 0.1428 397.9 1.0706 0.3518 -0.1270 0.4295 95.60 0.5565 RM 0.1452 0.1420 398.0 1.0271 0.3621 -0.1332 0.4236 95.00 0.5568 MM 0.1564 0.1422 397.9 1.1056 0.3320 -0.1220 0.4347 94.50 0.5567

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=0 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2493 0.1450 395.0 1.7387 0.1909 -0.0337 0.5388 96.30 0.5725 EM 0.2549 0.1420 398.0 1.5302 0.0752 0.0576 0.4540 94.60 0.5697 PSM 0.2565 0.2011 395.6 1.8192 0.2011 -0.2133 0.5409 92.30 0.5622 NNM 0.2531 0.1460 397.9 1.7870 0.1797 -0.0262 0.5423 95.40 0.5684 RM 0.2495 0.1454 398.0 1.7426 0.1912 -0.0329 0.5371 96.00 0.5701 MM 0.2479 0.1459 397.9 1.7531 0.1913 -0.0313 0.5385 96.20 0.5679

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=0 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.6010 0.1312 395.0 3.8408 0.0039 0.2927 0.9093 98.70 0.6166 EM 0.6033 0.1566 398.0 3.8688 0.0053 0.2962 0.9104 95.80 0.6142 PSM 0.5945 0.1615 395.6 3.8539 0.0085 0.2903 0.9104 94.90 0.6083 NNM 0.6034 0.1305 397.9 3.8679 0.0033 0.2962 0.9107 97.90 0.6145 RM 0.6009 0.1309 398.0 3.8585 0.0038 0.2941 0.9077 98.80 0.6136 MM 0.5996 0.1349 397.9 3.8387 0.0041 0.2920 0.9072 97.20 0.6152

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

215

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=0 Colinearity=0 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0004 0.1491 395.0 0.0025 0.4722 -0.2796 0.2788 94.40 0.5584 EM 0.0004 0.1413 398.0 0.0033 0.5024 -0.2773 0.2784 95.00 0.5559 PSM 0.0012 0.1729 395.6 0.0097 0.4304 -0.2732 0.2756 88.60 0.5488 NNM 0.0014 0.1530 397.9 0.0100 0.4727 -0.2765 0.2792 94.00 0.5557 RM 0.0006 0.1502 398.0 0.0039 0.4690 -0.2788 0.2775 94.10 0.5563 MM 0.0028 0.1549 397.9 0.0184 0.4789 -0.2810 0.2755 91.10 0.5565

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.10 Collinearity=0 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0995 0.1467 395.0 0.6873 0.4356 -0.1863 0.3853 94.60 0.5716 EM 0.0953 0.1497 398.0 0.6602 0.4232 -0.1894 0.3799 94.60 0.5693 PSM 0.1040 0.1700 395.6 0.7306 0.3903 -0.1769 0.3851 90.20 0.5620 NNM 0.1051 0.1506 397.9 0.7283 0.4096 -0.1796 0.3899 93.90 0.5695 RM 0.1002 0.1480 398.0 0.6947 0.4302 -0.1846 0.3848 94.70 0.5694 MM 0.1061 0.1513 397.9 0.7349 0.4159 -0.1785 0.3907 93.70 0.5692

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=0 and Correlation=.99 1000 Replications Average Coverage Method YD* SD df t-value p-value LCLM UCLM % Width NOM 0.4364 0.1566 398.0 2.7966 0.0461 0.1289 0.7424 95.00 0.6134 EM 0.4362 0.1542 395.5 2.8446 0.0480 0.1347 0.7411 94.00 0.6063 PSM 0.4355 0.1358 397.9 2.8985 0.0371 0.1363 0.7495 97.70 0.6132 NNM 0.4384 0.1336 398.0 2.8277 0.0361 0.1389 0.7468 96.70 0.6129 RM 0.4435 0.1351 397.9 2.8081 0.0391 0.1310 0.7447 96.00 0.6137 MM 0.4385 0.1331 395.2 2.7837 0.0378 0.1278 0.7448 96.90 0.6157

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

216

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=0 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0942 0.1435 398.0 0.6660 0.4333 -0.1836 0.3720 94.90 0.5556 EM 0.0991 0.1587 395.6 0.7097 0.4045 -0.1753 0.3734 92.40 0.5488 PSM 0.0974 0.1429 397.9 0.6897 0.4366 -0.1808 0.3756 94.90 0.5565 NNM 0.0938 0.1435 398.0 0.6639 0.4360 -0.1844 0.3719 94.30 0.5564 RM 0.0991 0.1404 397.9 0.7008 0.4216 -0.1789 0.3771 95.40 0.5561 MM 0.0936 0.1431 398.0 0.6660 0.4333 -0.1836 0.3720 94.90 0.5556

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.10 Collinearity=0 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2001 0.1511 398.0 1.3812 0.2741 -0.0850 0.4852 94.50 0.5703 EM 0.0998 0.1621 395.6 1.3788 0.2888 -0.0843 0.4784 92.60 0.5628 PSM 0.2050 0.1453 397.9 1.4155 0.2712 -0.0802 0.4903 95.60 0.5705 NNM 0.2000 0.1458 398.0 1.3833 0.2812 -0.0847 0.4849 95.20 0.5697 RM 0.2000 0.1394 397.9 1.3654 0.2746 -0.0873 0.4835 96.20 0.5709 MM 0.2000 0.1447 395.1 1.3761 0.2826 0.0861 0.4859 95.60 0.5720

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Cas2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=0 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5417 0.1560 398.0 3.5486 0.0125 0.2467 0.8619 94.20 0.6147 EM 0.5452 0.1588 395.5 3.5881 0.0111 0.2486 0.8564 94.60 0.6070 PSM 0.5442 0.1300 397.9 3.5564 0.0063 0.2483 0.8640 97.80 0.6144 NNM 0.5467 0.1339 398.0 3.5170 0.0074 0.2417 0.8566 97.80 0.6142 RM 0.5484 0.1341 397.9 3.5665 0.0052 0.2500 0.8654 97.20 0.6134 MM 0.5474 0.1340 395.0 3.5017 0.0076 0.2404 0.8582 97.90 0.6170

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

217

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=0 Collinearity=0 and Correlation=.99 1000 Replications Average

% Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1536 0.1390 398.0 1.0908 0.3450 -0.1242 0.4313 95.20 0.5555 EM 0.1459 0.1659 395.5 1.0479 0.3376 -0.1283 0.4201 89.80 0.5584 PSM 0.1540 0.1486 397.9 1.0937 0.3375 -0.1237 0.4316 94.00 0.5554 NNM 0.1457 0.1466 398.0 1.0328 0.3540 -0.1321 0.4234 93.20 0.5556 RM 0.1527 0.1450 397.9 1.0840 0.3472 -0.1249 0.4303 95.60 0.5552 MM 0.1456 0.1461 395.0 1.0291 0.3547 0.1331 0.4247 93.70 0.5577

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=0 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2505 0.1384 398.0 1.7334 0.1947 -0.0341 0.5351 96.90 0.5692 EM 0.2518 0.1525 396.0 1.7644 0.1982 -0.0294 0.5330 93.70 0.5624 PSM 0.2524 0.1337 397.9 1.7479 0.1847 -0.0322 0.5370 97.00 0.5692 NNM 0.2482 0.1389 398.0 1.7155 0.1969 -0.0367 0.5331 96.40 0.5799 RM 0.2475 0.1382 397.9 1.7127 0.1970 -0.0374 0.5324 96.20 0.5698 MM 0.2479 0.1381 395.0 1.7062 0.1991 -0.0382 0.5341 97.00 0.5723

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=0 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.6011 0.1595 398.0 3.7871 0.0073 0.2839 0.8993 94.50 0.6153 EM 0.5998 0.1559 396.0 3.8626 0.0057 0.2909 0.8967 95.90 0.6058 PSM 0.6023 0.1303 397.9 3.8245 0.0032 0.2896 0.9044 98.10 0.6148 NNM 0.5922 0.1271 398.0 3.8293 0.0041 0.2903 0.9049 97.60 0.6145 RM 0.6053 0.1333 397.9 3.8238 0.0028 0.2897 0.9051 98.60 0.6153 MM 0.5927 0.1273 395.0 3.8106 0.0040 0.2887 0.9060 97.60 0.6174

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

218

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=.40 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0002 0.1387 398.0 0.0013 0.5125 -0.2789 0.2785 94.80 0.5575 EM 0.0003 0.2133 396.5 0.0009 0.3719 -0.2775 0.2781 85.40 0.5556 PSM 0.0024 0.1411 397.9 0.0163 0.5118 -0.2754 0.2801 94.20 0.5550 NNM 0.0066 0.1414 398.0 0.0452 0.5099 -0.2851 0.2719 94.90 0.5569 RM 0.0025 0.1438 397.9 0.0172 0.4985 -0.2758 0.2808 94.20 0.5566 MM 0.0064 0.1410 395.1 0.0449 0.5137 -0.2862 0.2729 95.00 0.5591

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.10Collinearity=.40 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1112 0.1460 398.0 0.6482 0.4468 -0.1911 0.3792 95.20 0.5703 EM 0.1017 0.1982 396.6 0.6343 0.3474 -0.1929 0.3764 85.20 0.5694 PSM 0.1039 0.1387 397.9 0.6928 0.4320 -0.1844 0.3855 95.50 0.5699 NNM 0.1048 0.1427 398.0 0.6670 0.4233 -0.1882 0.3816 95.50 0.5698 RM 0.1083 0.1443 397.9 0.7021 0.4275 -0.1830 0.3866 94.80 0.5696 MM 0.1045 0.1427 395.0 0.6622 0.4250 -0.1897 0.3824 95.50 0.5721

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=.40 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4436 0.1548 398.0 2.8408 0.0465 0.1360 0.7489 95.20 0.6130 EM 0.4435 0.2126 395.5 2.8539 0.0480 0.1347 0.7411 82.60 0.6110 PSM 0.4398 0.1553 397.9 2.8324 0.0426 0.1345 0.7479 95.30 0.6133 NNM 0.4342 0.1621 398.0 2.8208 0.0474 0.1328 0.7463 94.00 0.6134 RM 0.4411 0.1593 397.9 2.8031 0.0394 0.1378 0.7508 95.10 0.6131 MM 0.4340 0.1618 395.0 2.8086 0.0468 0.1315 0.7477 95.40 0.6162

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

219

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=.40 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0919 0.1370 398.0 0.6499 0.4448 -0.1862 0.3700 95.80 0.5562 EM 0.0997 0.2042 395.6 0.7128 0.3576 -0.1773 0.3765 85.40 0.5538 PSM 0.0998 0.1399 397.9 0.7096 0.4264 -0.1776 0.3771 95.00 0.5547 NNM 0.0982 0.1350 398.0 0.6955 0.4352 -0.1801 0.3765 96.30 0.5566 RM 0.0935 0.1398 397.9 0.6403 0.4423 -0.1873 0.3684 95.00 0.5557 MM 0.0984 0.1343 395.1 0.6938 0.4376 -0.1809 0.3778 96.10 0.5587

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.10 Collinearity=.40 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2018 0.1485 398.0 1.3979 0.2750 -0.0826 0.4869 95.20 0.5695 EM 0.2090 0.2011 396.6 1.3596 0.2698 -0.0879 0.4812 87.60 0.5691 PSM 0.2008 0.1495 397.9 1.4102 0.2698 -0.0807 0.4889 94.70 0.5696 NNM 0.2008 0.1486 398.0 1.3833 0.2700 -0.0846 0.4846 94.50 0.5692 RM 0.2015 0.1467 397.9 1.4001 0.2684 -0.0824 0.4876 95.00 0.5701 MM 0.2008 0.1475 395.0 1.3777 0.2715 -0.0858 0.4858 95.10 0.5716

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2 t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=.40 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5495 01724 398.0 3.5044 0.0118 0.2398 0.8542 95.30 0.6144 EM 0.5378 0.1845 396.5 3.5306 0.0350 0.2431 0.8577 84.40 0.6146 PSM 0.5468 0.1433 397.9 3.5285 0.0153 0.2436 0.8585 95.10 0.6149 NNM 0.5489 0.1492 398.0 3.4996 0.0122 0.2390 0.8537 94.80 0.6147 RM 0.5494 0.1502 397.9 3.5071 0.0142 0.2402 0.8552 93.80 0.6150 MM 0.5489 0.1484 395.0 3.4795 0.0124 0.2370 0.8544 95.60 0.6174

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

220

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=0 Collinearity=.40 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1536 0.1390 398.0 1.0908 0.3450 -0.1242 0.4313 95.20 0.5555 EM 0.1459 0.1659 395.5 1.0479 0.3376 -0.1283 0.4201 89.80 0.5584 PSM 0.1540 0.1486 397.9 1.0937 0.3375 -0.1237 0.4316 94.00 0.5554 NNM 0.1457 0.1466 398.0 1.0328 0.3540 -0.1321 0.4234 93.20 0.5556 RM 0.1527 0.1450 397.9 1.0840 0.3472 -0.1249 0.4303 95.60 0.5552 MM 0.1456 0.1461 395.0 1.0291 0.3547 0.1331 0.4247 93.70 0.5577

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=.40 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2549 0.1420 398.0 1.7074 0.2036 -0.0379 0.5316 95.50 0.5695 EM 0.2565 0.2011 396.6 1.7227 0.2362 -0.0354 0.5336 80.00 0.5690 PSM 0.2531 0.1460 397.9 1.7166 0.1998 -0.0364 0.5325 95.30 0.5689 NNM 0.2495 0.1454 398.0 1.6922 0.2100 -0.0397 0.5288 95.20 0.5685 RM 0.2479 0.1459 397.9 1.6786 0.2122 -0.0419 0.5269 94.30 0.5688 MM 0.2493 0.1450 395.0 1.6823 0.2119 -0.0413 0.5295 95.60 0.5709

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Cas2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=.40 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5958 0.1607 398.0 3.7677 0.0079 0.2811 0.8961 95.00 0.6150 EM 0.5972 0.2141 396.0 3.7962 0.0316 0.2849 0.8997 84.70 0.6147 PSM 0.5990 0.1546 397.9 3.7857 0.0083 0.2847 0.9012 95.30 0.6165 NNM 0.6008 0.1599 398.0 3.8240 0.0076 0.2897 0.9045 94.80 0.6148 RM 0.6011 0.1597 397.9 3.8055 0.0073 0.2867 0.9015 95.10 0.6147 MM 0.5998 0.1598 395.0 3.8066 0.0077 0.2884 0.9060 94.70 0.6176

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

221

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=0Collinearity=.40 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0091 0.1433 398.0 0.0652 0.4993 -0.2874 0.2691 95.80 0.5564 EM 0.0038 0.2048 396.5 0.0252 0.3848 -0.2800 0.2724 83.90 0.5524 PSM 0.0029 0.1492 397.9 0.0179 0.4738 -0.2810 0.2753 95.30 0.5562 NNM 0.0079 0.1459 398.0 0.0548 0.4956 -0.2862 0.2703 93.40 0.5565 RM 0.0089 0.1440 397.9 0.0625 0.4947 -0.2870 0.2692 95.40 0.5562 MM 0.0073 0.1456 395.0 0.0501 0.4980 -0.2866 0.2720 93.40 0.5585

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.10Collinearity=.40 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1050 0.1459 398.0 0.7009 0.4358 -0.1859 0.3862 94.80 0.5696 EM 0.0996 0.1729 396.5 0.6805 0.3546 -0.1859 0.3814 84.50 0.5673 PSM 0.1041 0.1461 397.9 0.6871 0.4150 -0.1856 0.3839 93.80 0.5695 NNM 0.1046 0.1509 398.0 0.6847 0.4219 -0.1856 0.3835 94.90 0.5691 RM 0.1037 0.1489 397.9 0.6940 0.4123 -0.1844 0.3849 94.00 0.5693 MM 0.1049 0.1494 395.0 0.6809 0.4140 -0.1870 0.3846 95.10 0.5715

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=.40 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4433 0.1560 398.0 2.8478 0.0454 0.1366 0.7507 95.30 0.6139 EM 0.4495 0.1719 396.6 2.7627 0.0735 0.1234 0.7360 88.10 0.6127 PSM 0.4434 0.1396 397.9 2.8113 0.0413 0.1312 0.7457 96.50 0.6145 NNM 0.4422 0.1375 398.0 2.8071 0.0410 0.1307 0.7440 96.20 0.6133 RM 0.4392 0.1425 397.9 2.7997 0.0446 0.1298 0.7452 95.90 0.6142 MM 0.4425 0.1375 395.1 2.7867 0.0425 0.1281 0.7443 95.70 0.6154

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

222

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=.40 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1076 0.1434 398.0 0.6595 0.4428 -0.1845 0.3712 96.00 0.5557 EM 0.1040 0.1678 396.6 0.7029 0.3662 -0.1782 0.3758 84.30 0.5540 PSM 0.1037 0.1446 397.9 0.6702 0.4274 -0.1831 0.3724 95.50 0.5555 NNM 0.1059 0.1461 398.0 0.6966 0.4298 -0.1789 0.3764 95.20 0.5553 RM 0.1016 0.1432 397.9 0.6992 0.4237 -0.1789 0.3765 94.50 0.5554 MM 0.1060 0.1451 395.0 0.6748 0.4332 -0.1797 0.3778 95.30 0.5575

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.10 Collinearity=.40 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1978 0.1498 398.0 1.3586 0.2839 -0.0883 0.4823 94.10 0.5707 EM 0.1982 0.1667 396.6 1.4243 0.2528 -0.0785 0.4881 85.40 0.5666 PSM 0.1975 0.1405 397.9 1.3642 0.2765 -0.0875 0.4828 95.70 0.5704 NNM 0.1978 0.1422 398.0 1.3706 0.2742 -0.0864 0.4836 94.80 0.5700 RM 0.2013 0.1492 397.9 1.3398 0.2851 -0.0907 0.4792 95.50 0.5699 MM 0.1978 0.1419 395.0 1.3670 0.2757 -0.0873 0.4851 95.10 0.5724

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=.40 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5465 0.1508 398.0 3.5044 0.0118 0.2398 0.8542 95.30 0.6144 EM 0.5460 0.1660 396.5 3.5306 0.0350 0.2431 0.8577 84.40 0.6146 PSM 0.5474 0.1349 397.9 3.5285 0.0153 0.2436 0.8585 95.10 0.6149 NNM 0.5451 0.1349 398.0 3.4996 0.0122 0.2390 0.8537 94.80 0.6147 RM 0.5493 0.1288 397.9 3.5071 0.0142 0.2402 0.8552 93.80 0.6150 MM 0.5452 0.1346 395.0 3.4795 0.0124 0.2370 0.8544 95.60 0.6174

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

223

Case 2: -test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=0 Collinearity=.40 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1492 0.1452 398.0 1.0908 0.3450 -0.1242 0.4313 95.20 0.5555 EM 0.1464 0.1726 395.5 1.0479 0.3376 -0.1283 0.4201 89.80 0.5584 PSM 0.1485 0.1440 397.9 1.0937 0.3375 -0.1237 0.4316 94.00 0.5554 NNM 0.1518 0.1462 398.0 1.0328 0.3540 -0.1321 0.4234 93.20 0.5556 RM 0.1449 0.1471 397.9 1.0840 0.3472 -0.1249 0.4303 95.60 0.5552 MM 0.1520 0.1450 395.0 1.0291 0.3547 0.1331 0.4247 93.70 0.5577

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: -test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=.40 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2522 0.1412 398.0 1.7352 0.1987 -0.0339 0.5360 94.30 0.5699 EM 0.2437 0.1720 396.6 1.8244 0.2118 -0.0208 0.5463 85.20 0.5672 PSM 0.2473 0.1368 397.9 1.7773 0.1834 -0.0278 0.5414 94.90 0.5692 NNM 0.2501 0.1433 398.0 1.7177 0.1995 -0.0362 0.5325 95.10 0.5687 RM 0.2495 0.1482 397.9 1.7129 0.1998 -0.0373 0.5317 94.00 0.5690 MM 0.2494 0.1430 395.1 1.1705 0.2007 -0.0374 0.5317 95.20 0.5710

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Cas2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=.40 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.6033 0.1566 398.0 3.8607 0.0056 0.2954 0.9108 95.60 0.6154 EM 0.5945 0.1615 396.6 3.8259 0.0171 0.2884 0.9016 87.80 0.6132 PSM 0.6034 0.1305 397.9 3.8585 0.0041 0.2951 0.9098 97.40 0.6147 NNM 0.6009 0.1309 398.0 3.8162 0.0045 0.2886 0.9036 97.00 0.6150 RM 0.5996 0.1349 397.9 3.7944 0.0057 0.2851 0.8999 97.00 0.6148 MM 0.6010 0.1312 395.0 3.7979 0.0044 0.2871 0.9049 97.70 0.6178

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

224

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=.40 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0056 0.1365 398.0 0.0395 0.5222 -0.2832 0.2721 95.80 0.5553 EM 0.0046 0.1966 396.5 0.0326 0.3904 -0.2809 0.2717 83.90 0.5526 PSM 0.0024 0.1439 397.9 0.0183 0.4863 -0.2755 0.2804 95.30 0.5559 NNM 0.0000 0.1501 398.0 0.0007 0.4796 -0.2776 0.2776 93.40 0.5552 RM 0.0013 0.1407 397.9 0.0093 0.4973 -0.2792 0.2768 95.40 0.5560 MM 0.0000 0.1491 395.0 0.0008 0.4841 -0.2787 0.2787 93.40 0.5574

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.10 Collinearity=.40 and Correlation=.90 1000 Replications Average Coverage Method YD* SD df t-value p-value LCLM UCLM % age Width NOM 0.0971 0.1417 398.0 0.6701 0.4326 -0.1877 0.3819 95.50 0.5696 EM 0.1012 0.2016 396.6 0.7057 0.3571 -0.1814 0.3837 84.80 0.5630 PSM 0.1059 0.1476 397.9 0.7335 0.4159 -0.1782 0.3900 94.70 0.5681 NNM 0.1006 0.1543 398.0 0.6975 0.4179 -0.1836 0.3845 93.90 0.5685 RM 0.1011 0.1483 397.9 0.6992 0.4083 -0.1836 0.3858 94.70 0.5693 MM 0.1007 0.1532 395.0 0.6955 0.4199 -0.1847 0.3861 94.10 0.5708

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=.40 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4435 0.1555 398.0 2.7920 0.0498 0.1287 0.7430 94.00 0.6143 EM 0.4332 0.3942 396.5 2.8569 0.0749 0.1375 0.7494 85.10 0.6119 PSM 0.4388 0.1540 397.9 2.8486 0.0383 0.1374 0.7511 96.00 0.6137 NNM 0.4406 0.1528 398.0 2.7703 0.0452 0.1251 0.7393 96.10 0.6143 RM 0.4423 0.1505 397.9 2.8224 0.0398 0.1331 0.7473 95.90 0.6142 MM 0.4408 0.1526 395.1 2.7576 0.0459 0.1235 0.7407 95.50 0.6169

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

225

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=.40 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0908 0.1366 398.0 0.6435 0.4411 -0.1873 0.3689 96.00 0.5562 EM 0.0995 0.1928 396.6 0.7068 0.3668 -0.1773 0.3765 85.70 0.5555 PSM 0.0997 0.1432 397.9 0.7015 0.4265 -0.1793 0.3784 93.90 0.5577 NNM 0.0969 0.1412 398.0 0.6827 0.4408 -0.1816 0.3754 94.70 0.5571 RM 0.1013 0.1433 397.9 0.7167 0.4227 -0.1771 0.3797 94.20 0.5568 MM 0.0969 0.1409 395.0 0.6799 0.4434 -0.1828 0.3765 94.90 0.5593

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.10 Collinearity=.40 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1983 0.1399 398.0 1.3586 0.2839 -0.0883 0.4824 94.10 0.5707 EM 0.2023 0.2065 396.6 1.4243 0.2528 -0.0785 0.4881 85.40 0.5666 PSM 0.1994 0.1497 397.9 1.3642 0.2765 -0.0876 0.4828 95.70 0.5704 NNM 0.1957 0.1454 398.0 1.3706 0.2742 -0.0864 0.4836 94.80 0.5700 RM 0.1980 0.1467 397.9 1.3398 0.2851 -0.0907 0.4792 95.50 0.5699 MM 0.1956 0.1453 395.0 1.3670 0.2757 -0.0873 0.4851 95.10 0.5724

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=.40 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5503 0.1557 398.0 3.5026 0.0144 0.2400 0.8556 93.80 0.6156 EM 0.5469 0.1499 396.5 3.6343 0.0250 0.2586 0.8707 88.60 0.6121 PSM 0.5545 0.1297 397.9 3.5183 0.0097 0.2421 0.8574 96.70 0.6154 NNM 0.5549 0.1326 398.0 3.5196 0.0094 0.2423 0.8565 96.50 0.6143 RM 0.5534 0.1290 397.9 3.5709 0.0078 0.2501 0.8646 96.60 0.6145 MM 0.5542 0.1321 395.0 3.5038 0.0095 0.2408 0.8578 96.70 0.6171

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

226

Cas2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=0 Collinearity=.40 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1503 0.1418 398.0 1.0636 0.3468 -0.1274 0.4279 95.20 0.5553 EM 0.1568 0.2005 396.5 1.1196 0.3122 -0.1198 0.4335 84.70 0.5533 PSM 0.1481 0.1471 397.9 1.0459 0.3457 -0.1303 0.4265 94.20 0.5568 NNM 0.1486 0.1447 398.0 1.0538 0.3505 -0.1291 0.4263 94.40 0.5554 RM 0.1486 0.1458 397.9 1.0551 0.3254 -0.1287 0.4268 94.30 0.5552 MM 0.1482 0.1442 395.1 1.0469 0.2715 -0.1305 0.4269 94.40 0.5574

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=.40 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2547 0.1469 398.0 1.7352 0.1987 -0.0339 0.5360 94.30 0.5699 EM 0.2598 0.1617 396.6 1.8244 0.2118 -0.0208 0.5463 85.20 0.5672 PSM 0.2581 0.1392 397.9 1.7773 0.1834 -0.0278 0.5414 94.90 0.5692 NNM 0.2521 0.1406 398.0 1.7177 0.1995 -0.0362 0.5325 95.10 0.5687 RM 0.2536 0.1373 397.9 1.7129 0.1998 -0.0373 0.5317 94.00 0.5690 MM 0.2526 0.1402 395.1 1.1705 0.2007 -0.0374 0.5317 95.20 0.5710

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=.40 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.6052 0.1577 398.0 3.875 0.0055 0.2977 0.9126 94.60 0.6149 EM 0.6063 0.1970 396.6 3.9071 0.0144 0.3006 0.9120 88.20 0.6113 PSM 0.6051 0.1442 397.9 3.8719 0.0051 0.2975 0.9126 97.00 0.6151 NNM 0.6004 0.1441 398.0 3.8449 0.0047 0.2931 0.9077 96.60 0.6145 RM 0.6028 0.1430 397.8 3.8583 0.0048 0.2953 0.9104 96.30 0.6150 MM 0.6009 0.1437 395.0 3.8300 0.0046 0.2921 0.9097 97.00 0.6175

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

227

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=.40 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0071 0.1469 398.0 0.0503 0.4856 -0.2850 0.2709 94.20 0.5558 EM 0.0068 0.1469 396.5 0.0516 0.4009 -0.2832 0.2696 84.90 0.5529 PSM 0.0063 0.1503 397.9 0.0448 0.4795 -0.2840 0.2715 92.80 0.5555 NNM 0.0001 0.1473 398.0 0.0005 0.4915 -0.2779 0.2778 93.50 0.5557 RM 0.0035 0.1481 397.9 0.0247 0.4737 -0.2745 0.2815 94.80 0.5561 MM 0.0001 0.1458 395.0 0.0015 0.4976 -0.2791 0.2788 93.90 0.5579

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.10 Collinearity=.40 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1039 0.1464 398.0 0.6865 0.4263 -0.1858 0.3845 95.00 0.5704 EM 0.1036 0.1979 396.5 0.7229 0.3632 -0.1796 0.3868 83.60 0.5664 PSM 0.1019 0.1416 397.9 0.7043 0.4294 -0.1828 0.3866 95.60 0.5695 NNM 0.0987 0.1488 398.0 0.6818 0.4255 -0.1864 0.3837 94.30 0.5700 RM 0.1008 0.1468 397.9 0.6973 0.4219 -0.1842 0.3857 94.00 0.5699 MM 0.0989 0.1485 395.2 0.6805 0.4260 -0.1873 0.3851 94.90 0.5724

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=.40 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4364 0.1566 398.0 2.8330 0.0454 0.1349 0.7497 95.30 0.6147 EM 0.4362 0.1542 396.6 2.8209 0.0797 0.1318 0.7439 87.10 0.6121 PSM 0.4355 0.1358 397.9 2.8128 0.0456 0.1316 0.7461 95.30 0.6144 NNM 0.4384 0.1336 398.0 2.7721 0.0485 0.1256 0.7405 95.60 0.6145 RM 0.4435 0.1351 397.9 2.7744 0.0449 0.1259 0.7414 96.50 0.6155 MM 0.4385 0.1331 395.0 2.7562 0.0492 0.1236 0.7409 95.80 0.6173

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

228

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=.40 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1039 0.1464 398.0 0.7222 0.4297 -0.1759 0.3801 95.60 0.5560 EM 0.1036 0.1979 396.5 0.7596 0.3419 -0.1693 0.3838 83.10 0.5531 PSM 0.1019 0.1416 397.9 0.7188 0.4240 -0.1767 0.3800 95.20 0.5568 NNM 0.0987 0.1488 398.0 0.7021 0.4175 -0.1785 0.3773 94.50 0.5558 RM 0.1008 0.1468 397.9 0.7069 0.4319 -0.1785 0.3778 94.50 0.5563 MM 0.0989 0.1485 395.0 0.6971 0.4204 -0.1799 0.3780 94.70 0.5580

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.10 Collinearity=.40 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1978 0.1445 398.0 1.3694 0.2779 -0.0869 0.4818 94.30 0.5680 EM 0.1960 0.1932 396.5 1.3663 0.2824 -0.0870 0.4790 85.00 0.5661 PSM 0.1973 0.1395 397.9 1.3655 0.2707 -0.0873 0.4819 96.10 0.5692 NNM 0.2017 0.1425 398.0 1.3938 0.2748 -0.0828 0.4861 95.20 0.5689 RM 0.2022 0.1440 397.9 1.3983 0.2719 -0.0825 0.4869 95.20 0.5695 MM 0.2013 0.1419 395.0 1.3860 0.2774 -0.0842 0.4869 95.30 0.5712

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=.40 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5414 0.1549 398.0 3.4674 0.0143 0.2341 0.8486 95.20 0.6145 EM 0.5412 0.2006 396.5 3.4903 0.0316 0.2359 0.8465 88.20 0.6106 PSM 0.5465 0.1484 397.9 3.4882 0.0124 0.2372 0.8518 95.30 0.6146 NNM 0.5506 0.1486 398.0 3.5273 0.0122 0.2434 0.8578 96.40 0.6143 RM 0.5453 0.1396 397.9 3.4866 0.0119 0.2370 0.8516 96.70 0.6145 MM 0.5500 0.1481 395.1 3.5082 0.0121 0.2415 0.8585 96.40 0.6171

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

229

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=0 Collinearity=.40 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1582 0.1482 398.0 1.1120 0.3326 -0.1192 0.4191 93.80 0.5560 EM 0.1432 0.1940 396.5 1.0245 0.3312 -0.1327 0.4191 83.70 0.5518 PSM 0.1489 0.1400 397.9 1.0556 0.3519 -0.1288 0.4266 95.40 0.5555 NNM 0.1513 0.1407 398.0 1.0724 0.3504 -0.1266 0.4291 95.10 0.5556 RM 0.1543 0.1446 397.9 1.0934 0.3463 -0.1236 0.4321 94.00 0.5557 MM 0.1512 0.1404 395.0 1.0678 0.3497 -0.1277 0.4302 95.10 0.5579

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=.40 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2356 0.1523 398.0 1.7352 0.1987 -0.0339 0.5360 94.30 0.5699 EM 0.2462 0.1927 396.6 1.8244 0.2118 -0.0208 0.5463 85.20 0.5672 PSM 0.2483 0.1444 397.9 1.7773 0.1834 -0.0278 0.5414 94.90 0.5692 NNM 0.2449 0.1435 398.0 1.7177 0.1995 -0.0362 0.5325 95.10 0.5687 RM 0.2474 0.1454 397.9 1.7129 0.1998 -0.0373 0.5317 94.00 0.5690 MM 0.2445 0.1426 395.1 1.1705 0.2007 -0.0374 0.5317 95.20 0.5710

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=.40 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5981 0.1510 398.0 3.875 0.0055 0.2977 0.9126 94.60 0.6149 EM 0.5992 0.1973 396.6 3.9071 0.0144 0.3006 0.9120 88.20 0.6113 PSM 0.5934 0.1400 397.9 3.8719 0.0051 0.2975 0.9126 97.00 0.6151 NNM 0.5930 0.1461 398.0 3.8449 0.0047 0.2931 0.9077 96.60 0.6145 RM 0.5945 0.1446 397.8 3.8583 0.0048 0.2953 0.9104 96.30 0.6150 MM 0.5924 0.1453 395.0 3.8300 0.0046 0.2921 0.9097 97.00 0.6175

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

230

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=.90 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0060 0.1435 398.0 0.0412 0.5026 -0.2716 0.2836 93.50 0.5552 EM 0.0086 0.4216 397.9 0.0731 0.1963 -0.2615 0.2788 57.40 0.5402 PSM 0.0093 0.1426 397.9 0.0664 0.4904 -0.2685 0.2871 95.40 0.5557 NNM 0.0000 0.1476 398.0 0.0004 0.4865 -0.2776 0.2777 93.90 0.5569 RM 0.0008 0.1416 397.8 0.0049 0.5005 -0.2783 0.2767 94.70 0.5550 MM 0.0000 0.1462 395.0 0.0004 0.4907 -0.2787 0.2788 94.40 0.5575

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.10 Collinearity=.90 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0996 0.1388 398.0 0.7189 0.4180 -0.1820 0.3889 95.30 0.5699 EM 0.1073 0.1683 398.0 0.7176 0.2076 -0.1788 0.3772 52.10 0.5559 PSM 0.1058 0.1417 397.9 0.6912 0.4343 -0.1851 0.3846 95.70 0.5697 NNM 0.1026 0.1502 398.0 0.7049 0.4218 -0.1831 0.3865 93.30 0.5695 RM 0.1012 0.1416 397.9 0.7128 0.4155 -0.1816 0.3872 94.60 0.5688 MM 0.1018 0.1494 395.0 0.7047 0.4218 -0.1838 0.3880 95.60 0.5718

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=.90 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4435 0.1555 398.0 2.7853 0.0485 0.1274 0.7418 94.80 0.6144 EM 0.4332 0.3942 397.9 2.8867 0.1369 0.1385 0.7405 55.70 0.6120 PSM 0.4388 0.1540 397.9 2.7857 0.0458 0.1274 0.7407 96.30 0.6133 NNM 0.4406 0.1528 398.0 2.8156 0.0494 0.1318 0.7456 94.30 0.6138 RM 0.4423 0.1505 397.9 2.8256 0.0446 0.1333 0.7465 94.90 0.6132 MM 0.4408 0.1526 395.0 2.7576 0.0488 0.1305 0.7471 94.50 0.6166

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

231

Case t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=.90 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1000 0.1382 398.0 0.6362 0.4418 -0.1877 0.3677 94.60 0.5554 EM 0.0920 0.1427 397.9 0.5574 0.1936 -0.1953 0.3462 56.20 0.5415 PSM 0.0977 0.1394 397.9 0.6940 0.4391 -0.1794 0.3750 95.90 0.5544 NNM 0.1035 0.1448 398.0 0.7091 0.4243 -0.1776 0.3773 94.20 0.5549 RM 0.0987 0.1394 397.9 0.6981 0.4216 -0.1789 0.3758 95.60 0.5547 MM 0.1033 0.1448 395.0 0.7088 0.4253 -0.1783 0.3787 94.20 0.5570

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.10 Collinearity=.90 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2080 0.1462 398.0 1.3854 0.2770 -0.0841 0.4846 96.00 0.5688 EM 0.2093 0.1640 397.9 1.3454 0.1911 -0.0890 0.4646 59.60 0.5535 PSM 0.2059 0.1410 397.9 1.3751 0.2787 -0.0856 0.4828 94.60 0.5684 NNM 0.2003 0.1424 398.0 1.4047 0.2693 -0.0814 0.4883 96.10 0.5697 RM 0.2004 0.1400 397.9 1.3524 0.2840 -0.0891 0.4800 95.90 0.5691 MM 0.1978 0.1412 395.0 1.3966 0.2700 -0.0829 0.4891 96.40 0.5720

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=.90 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5503 0.1557 398.0 3.5269 0.0150 0.3496 0.7471 94.20 0.6140 EM 0.5469 0.1499 397.9 3.6389 0.1059 0.2525 0.8504 59.30 0.5979 PSM 0.5545 0.1297 397.9 3.5088 0.0140 0.2404 0.8545 94.80 0.6141 NNM 0.5549 0.1326 398.0 3.5151 0.0136 0.2410 0.8545 94.80 0.6135 RM 0.5534 0.1290 397.9 3.5077 0.0120 0.2340 0.8533 94.40 0.6133 MM 0.5542 0.1321 395.0 3.4983 0.0135 0.2395 0.8558 95.40 0.6163

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

232

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=0 Collinearity=.90 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1501 0.1401 398.0 1.0669 0.3514 -0.1272 0.4272 94.50 0.5544 EM 0.1498 0.1624 397.9 1.0802 0.1823 -0.1232 0.4164 52.80 0.5396 PSM 0.1513 0.1428 397.9 1.0612 0.3510 -0.1277 0.4271 95.00 0.5548 NNM 0.1452 0.1420 398.0 1.0473 0.3509 -0.1297 0.4253 94.30 0.5550 RM 0.1564 0.1422 397.9 1.0491 0.3582 -0.1298 0.4252 94.50 0.5550 MM 0.1451 0.1411 395.1 1.0441 0.3503 -0.1306 0.4265 94.70 0.5571

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=.90 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2547 0.1469 398.0 1.7176 0.2037 -0.0363 0.5328 95.40 0.5690 EM 0.2598 0.1617 397.9 1.7212 0.1735 -0.0386 0.5176 57.60 0.5563 PSM 0.2581 0.1392 397.9 1.7059 0.2024 -0.0380 0.5326 95.80 0.5707 NNM 0.2521 0.1406 398.0 1.7178 0.2022 -0.0363 0.5335 94.50 0.5699 RM 0.2536 0.1373 397.9 1.7054 0.2021 -0.0373 0.5325 96.20 0.5706 MM 0.2526 0.1402 395.1 1.7128 0.2023 -0.0372 0.5350 94.90 0.5723

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2:t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=.90 and Correlation=0 1000 Replications Average % Method YD* SD DF t-value p-value LCLM UCLM Coverage Width NOM 0.6000 0.1536 398.0 3.875 0.0055 0.2977 0.9126 94.60 0.6149 EM 0.5960 0.1522 396.6 3.9071 0.0144 0.3006 0.9120 88.20 0.6114 PSM 0.5986 0.1236 397.9 3.8719 0.0051 0.2975 0.9126 97.00 0.6151 NNM 0.6017 0.1357 398.0 3.8449 0.0047 0.2931 0.9077 96.60 0.6146 RM 0.5997 0.1315 397.8 3.8583 0.0048 0.2953 0.9104 96.30 0.6151 MM 0.6017 0.1355 395.0 3.8300 0.0046 0.2921 0.9097 97.00 0.6176

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

233

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=.90 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0035 0.1374 398.0 0.0250 0.5049 -0.2746 0.2816 96.10 0.5562 EM 0.0145 0.3990 397.9 0.1079 0.2085 -0.2551 0.2843 58.80 0.5392 PSM 0.0050 0.1412 397.9 0.0354 0.5061 -0.2726 0.2826 95.10 0.5552 NNM 0.0076 0.1457 398.0 0.0521 0.4907 -0.2700 0.2852 93.80 0.5552 RM 0.0056 0.1462 397.8 0.0387 0.4878 -0.2721 0.2833 94.20 0.5554 MM 0.0070 0.1453 395.0 0.0475 0.4922 -0.2718 0.2857 94.20 0.5575

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.10 Collinearity=.90 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1001 0.1430 398.0 0.6767 0.4251 -0.1868 0.3828 95.20 0.5696 EM 0.1092 0.4059 397.9 0.6685 0.2054 -0.1832 0.3704 51.00 0.5536 PSM 0.1041 0.1444 397.9 0.6759 0.4310 -0.1869 0.3822 95.00 0.5691 NNM 0.1035 0.1440 398.0 0.6631 0.4225 -0.1889 0.3810 95.50 0.5699 RM 0.0974 0.1450 397.9 0.6675 0.4362 -0.1893 0.3796 95.40 0.5689 MM 0.1037 0.1438 395.0 0.6625 0.4261 -0.1898 0.3824 95.60 0.5722

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=.90 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4329 0.1572 398.0 2.7756 0.0496 0.1259 0.7398 94.50 0.6139 EM 0.4333 0.4319 397.9 2.8553 0.1341 0.1322 0.7343 53.70 0.6121 PSM 0.4368 0.1554 397.9 2.7814 0.0493 0.1268 0.7409 94.70 0.6139 NNM 0.4376 0.1483 398.0 2.7984 0.0433 0.1296 0.7440 96.50 0.6144 RM 0.4376 0.1518 397.9 2.8079 0.0421 0.1309 0.7443 95.20 0.6134 MM 0.4369 0.1481 395.0 2.7872 0.0436 0.1283 0.7471 96.30 0.6171

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

234

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=.90 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0963 0.1415 398.0 0.6816 0.4360 -0.1816 0.3741 93.90 0.5557 EM 0.1124 0.4097 397.9 0.8211 0.2003 -0.1595 0.3843 51.50 0.5438 PSM 0.1040 0.1381 397.9 0.7361 0.4172 -0.1741 0.3820 95.30 0.5561 NNM 0.0988 0.1460 398.0 0.7005 0.4279 -0.1787 0.3766 93.60 0.5553 RM 0.0965 0.1446 397.9 0.6849 0.4284 -0.1812 0.3742 94.70 0.5554 MM 0.0989 0.1451 395.0 0.6976 0.4273 -0.1799 0.3775 94.10 0.5574

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.10 Collinearity=.90 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1944 0.1404 398.0 1.3432 0.2826 -0.0905 0.4795 94.90 0.5700 EM 0.2150 0.4224 397.9 1.5225 0.1797 -0.0638 0.4937 56.90 0.5575 PSM 0.1980 0.1489 397.9 1.3682 0.2866 -0.0868 0.4829 94.90 0.5697 NNM 0.1966 0.1446 398.0 1.3578 0.2737 -0.0883 0.4816 94.40 0.5699 RM 0.1972 0.1434 397.9 1.3617 0.2766 -0.0877 0.4821 95.60 0.5698 MM 0.1973 0.1438 395.0 1.3573 0.2726 -0.0887 0.4834 94.70 0.5721

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2 t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=.90 and Correlation=.75 1000 Replications Average % Method YD* SD Df t-value p-value LCLM UCLM Coverage Width NOM 0.5450 0.1547 398.0 3.4898 0.0119 0.2376 0.8524 96.10 0.6148 EM 0.5385 0.4156 397.9 3.6787 0.1038 0.2525 0.8593 55.00 0.6163 PSM 0.5432 0.1481 397.9 3.4748 0.0117 0.2356 0.8509 96.60 0.6153 NNM 0.5412 0.1501 398.0 3.4673 0.0120 0.2338 0.8489 95.60 0.6151 RM 0.5439 0.1511 397.9 3.4835 0.0122 0.2364 0.8513 95.90 0.6149 MM 0.5412 0.1495 395.0 3.4473 0.0118 0.2324 0.8502 95.60 0.6178

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

235

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=0 Collinearity=.90 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1488 0.1466 398.0 1.0505 0.3529 -0.1296 0.4271 94.90 0.5568 EM 0.1439 0.4109 397.9 1.0439 0.1801 -0.1278 0.4155 58.00 0.5532 PSM 0.1469 0.1418 397.9 1.0384 0.3542 -0.1314 0.4252 94.50 0.5566 NNM 0.1497 0.1417 398.0 1.0613 0.3480 -0.1281 0.4276 94.70 0.5554 RM 0.1437 0.1454 397.9 1.0143 0.3565 -0.1281 0.4276 95.20 0.5556 MM 0.1497 0.1422 395.0 1.0566 0.3473 -0.1305 0.4269 95.00 0.5579

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=.90 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2532 0.1409 398.0 1.6586 0.2134 -0.0447 0.5251 94.20 0.5698 EM 0.2455 0.4291 397.9 1.8012 0.1836 -0.0253 0.5331 55.90 0.5584 PSM 0.2587 0.1416 397.9 1.6895 0.2090 -0.0404 0.5299 96.30 0.5703 NNM 0.2646 0.1396 398.0 1.6226 0.2175 -0.0500 0.5200 95.50 0.5700 RM 0.2641 0.1444 397.9 1.6726 0.2038 -0.0427 0.5325 95.20 0.5752 MM 0.2645 0.1386 395.0 1.6207 0.2163 -0.0504 0.5219 97.70 0.5723

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=.90 and Correlation=.75 1000 Replications Average Coverage Method YD* SD df t-value p-value LCLM UCLM % age Width NOM 0.6000 0.1536 398.0 3.8497 0.0065 0.2933 0.9075 94.80 0.6142 EM 0.5960 0.1522 396.6 3.9287 0.0855 0.2987 0.9040 55.10 0.6053 PSM 0.5986 0.1236 397.9 3.8695 0.0078 0.2960 0.9099 95.70 0.6139 NNM 0.6017 0.1357 398.0 3.8707 0.0059 0.2963 0.9107 95.40 0.6044 RM 0.5997 0.1315 397.8 3.8496 0.0067 0.2938 0.9084 95.60 0.6146 MM 0.6017 0.1355 395.0 3.8537 0.0060 0.2950 0.9115 97.60 0.6165

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

236

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=.90 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0038 0.1364 398.0 0.0262 0.5140 -0.2742 0.2817 95.80 0.5559 EM 0.0076 0.4255 397.9 0.0463 0.1994 -0.2638 0.2791 57.70 0.5429 PSM 0.0069 0.1535 397.9 0.0468 0.4770 -0.2710 0.1543 92.80 0.5553 NNM 0.0080 0.1446 398.0 0.0559 0.4845 -0.2701 0.2860 95.00 0.5561 RM 0.0003 0.1498 397.8 0.0115 0.4750 -0.2784 0.2776 93.50 0.5560 MM 0.0077 0.1435 395.0 0.0537 0.4916 -0.2715 0.2868 95.40 0.5583

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.10 Collinearity=.90 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0967 0.1452 398.0 0.6791 0.4300 -0.1862 0.3831 95.40 0.5693 EM 0.1127 0.4044 397.9 0.6635 0.2057 -0.1826 0.3742 51.10 0.5568 PSM 0.1002 0.1440 397.9 0.6967 0.4335 -0.1837 0.3850 94.10 0.5687 NNM 0.0957 0.1434 398.0 0.6850 0.4331 -0.1852 0.3836 95.30 0.5688 RM 0.1054 0.1393 397.9 0.6873 0.4273 -0.1853 0.3839 95.30 0.5692 MM 0.0956 0.1429 395.0 0.6803 0.4343 -0.1867 0.3845 95.30 0.5712

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=.90 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4412 0.1428 398.0 2.7928 0.0479 0.1286 0.7421 94.40 0.6135 EM 0.4259 0.4164 397.9 3.0297 0.1195 0.1560 0.7727 55.80 0.6027 PSM 0.4366 0.1478 397.9 2.8176 0.0410 0.1324 0.7461 96.00 0.6137 NNM 0.4416 0.1456 398.0 2.8005 0.0425 0.1296 0.7431 96.70 0.6135 RM 0.4362 0.1480 397.9 2.7487 0.0505 0.1217 0.7351 94.70 0.6134 MM 0.4418 0.1453 395.1 2.7882 0.0431 0.1282 0.7444 96.90 0.6162

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

237

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=.90 and Correlation=.90 1000 Replications Average Coverage Method YD* SD df t-value p-value LCLM UCLM % age Width NOM 0.0967 0.1452 398.0 0.6867 0.4143 -0.1808 0.3742 94.40 0.5550 EM 0.1127 0.4044 397.9 0.8943 0.2075 -0.1483 0.3936 52.10 0.5419 PSM 0.1002 0.1440 397.9 0.7108 0.4093 -0.1772 0.3776 93.70 0.5548 NNM 0.0957 0.1434 398.0 0.6734 0.4177 -0.1824 0.3718 94.50 0.5542 RM 0.1054 0.1393 397.9 0.7470 0.4115 -0.1724 0.3832 95.50 0.5556 MM 0.0956 0.1429 395.1 0.6697 0.4195 -0.1835 0.3728 94.40 0.5563

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.10 Collinearity=.90 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1962 0.1474 398.0 1.4565 0.2586 -0.0743 0.4947 94.60 0.5690 EM 0.1952 0.4132 397.9 1.4613 0.1827 -0.0711 0.4832 51.70 0.5543 PSM 0.1998 0.1476 397.9 1.4182 0.2672 -0.0796 0.4890 94.00 0.5686 NNM 0.1991 0.1440 398.0 1.4432 0.2540 -0.0760 0.4934 94.30 0.5694 RM 0.2027 0.1456 397.9 1.4382 0.2578 -0.0768 0.4922 94.90 0.5690 MM 0.1996 0.1442 395.0 1.4368 0.2543 -0.0772 0.4944 94.00 0.5716

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=.90 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5499 0.1452 398.0 3.5241 0.0137 0.2425 0.8571 94.30 0.6146 EM 0.5389 0.4044 397.9 3.5331 0.1037 0.2367 0.8409 54.10 0.6042 PSM 0.5535 0.1491 397.9 3.5482 0.0110 0.2464 0.8607 96.00 0.6143 NNM 0.5487 0.1525 398.0 3.5123 0.0123 0.2412 0.8561 95.80 0.6149 RM 0.5509 0.1496 397.9 3.5336 0.0111 0.2400 0.8578 95.90 0.6138 MM 0.5489 0.1519 395.0 3.4983 0.0122 0.2401 0.8577 95.70 0.6176

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

238

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=0 Collinearity=.90 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1450 0.1402 398.0 1.0289 0.3601 -0.1326 0.4226 94.80 0.5552 EM 0.1768 0.4022 397.9 1.3069 0.1927 -0.0938 0.4474 59.80 0.5412 PSM 0.1513 0.1343 397.9 1.0703 0.3556 -0.1262 0.4283 95.80 0.5551 NNM 0.1493 0.1360 398.0 1.0609 0.3614 -0.1282 0.4268 95.40 0.5550 RM 0.1547 0.1365 397.9 1.0951 0.3432 -0.1230 0.4324 95.90 0.5554 MM 0.1501 0.1356 395.0 1.0622 0.3606 -0.1284 0.4287 95.60 0.5571

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=.90 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2532 0.1409 398.0 1.6586 0.2134 -0.0447 0.5251 94.20 0.5698 EM 0.2455 0.4291 397.9 1.8012 0.1836 -0.0253 0.5331 55.90 0.5584 PSM 0.2587 0.1416 397.9 1.6895 0.2090 -0.0404 0.5299 96.30 0.5703 NNM 0.2646 0.1396 398.0 1.6226 0.2175 -0.0500 0.5200 95.50 0.5700 RM 0.2641 0.1444 397.9 1.6726 0.2038 -0.0427 0.5325 95.20 0.5752 MM 0.2645 0.1386 395.0 1.6207 0.2163 -0.0504 0.5219 97.70 0.5723

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=.90 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5903 0.1520 398.0 3.8480 0.0062 0.2930 0.9066 95.50 0.6136 EM 0.5706 0.4035 397.9 3.9037 0.0830 0.2949 0.8970 55.30 0.6021 PSM 0.5901 0.1464 397.9 3.8670 0.0066 0.2956 0.9090 96.40 0.6134 NNM 0.5989 0.1482 398.0 3.8645 0.0057 0.2964 0.9115 96.80 0.6151 RM 0.5927 0.1477 397.9 3.8708 0.0042 0.2970 0.9114 96.50 0.6144 MM 0.5990 0.1456 395.0 3.8501 0.0056 0.2955 0.9133 96.90 0.6178

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

239

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=.90 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0019 0.1432 398.0 0.0139 0.4980 -0.2761 0.2799 94.80 0.5560 EM 0.0068 0.4047 397.9 0.0626 0.2144 -0.2788 0.2654 59.60 0.5442 PSM 0.0041 0.1409 397.9 0.0277 0.5060 -0.2823 0.2742 95.00 0.5565 NNM 0.0073 0.1477 398.0 0.0514 0.4678 -0.2856 0.2708 94.20 0.5564 RM 0.0022 0.1430 397.8 0.0156 0.4902 -0.2759 0.2803 95.20 0.5562 MM 0.0074 0.1471 395.0 0.0509 0.4701 -0.2715 0.2719 94.40 0.5584

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.10 Collinearity=.90 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0945 0.1426 398.0 0.6508 0.4275 -0.1900 0.3786 96.00 0.5686 EM 0.0933 0.4059 397.9 0.6761 0.2063 -0.1839 0.3705 52.50 0.5544 PSM 0.1004 0.1467 397.9 0.6935 0.4246 -0.1841 0.3848 95.50 0.5689 NNM 0.0992 0.1492 398.0 0.6854 0.4128 -0.1852 0.3835 93.70 0.5687 RM 0.0989 0.1473 397.9 0.6845 0.4286 -0.1851 0.3829 94.00 0.5680 MM 0.0986 0.1489 395.0 0.6788 0.4169 -0.1869 0.3842 94.10 0.5711

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=.90 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4404 0.1559 398.0 2.7729 0.0486 0.1254 0.7398 95.40 0.6144 EM 0.4267 0.3940 397.9 2.8098 0.1402 0.1271 0.7307 55.60 0.6036 PSM 0.4360 0.1490 397.9 2.8037 0.0466 0.1303 0.7442 95.50 0.6139 NNM 0.4440 0.1520 398.0 2.8004 0.0463 0.1299 0.7442 95.40 0.6143 RM 0.4391 0.1569 397.9 2.7915 0.0459 0.1283 0.7433 95.40 0.6150 MM 0.4440 0.1507 395.0 2.7879 0.0459 0.1285 0.7456 95.30 0.6171

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

240

Case 2 t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=.90 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0967 0.1452 398.0 0.6867 0.4143 -0.1808 0.3742 94.40 0.5550 EM 0.1127 0.4044 397.9 0.8943 0.2075 -0.1483 0.3936 52.10 0.5419 PSM 0.1002 0.1440 397.9 0.7108 0.4093 -0.1772 0.3776 93.70 0.5548 NNM 0.0957 0.1434 398.0 0.6734 0.4177 -0.1824 0.3718 94.50 0.5542 RM 0.1054 0.1393 397.9 0.7470 0.4115 -0.1724 0.3832 95.50 0.5556 MM 0.0956 0.1429 395.1 0.6697 0.4195 -0.1835 0.3728 94.40 0.5563

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.10 Collinearity=.90 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1962 0.1474 398.0 1.4565 0.2586 -0.0743 0.4947 94.60 0.5690 EM 0.1952 0.4132 397.9 1.4613 0.1827 -0.0711 0.4832 51.70 0.5543 PSM 0.1998 0.1476 397.9 1.4182 0.2672 -0.0796 0.4890 94.00 0.5686 NNM 0.1991 0.1440 398.0 1.4432 0.2540 -0.0760 0.4934 94.30 0.5694 RM 0.2027 0.1456 397.9 1.4382 0.2578 -0.0768 0.4922 94.90 0.5690 MM 0.1996 0.1442 395.0 1.4368 0.2543 -0.0772 0.4944 94.00 0.5716

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=.90 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5414 0.1592 398.0 3.4681 0.0147 0.2341 0.8486 94.60 0.6145 EM 0.5416 0.4233 397.9 3.5630 0.1027 0.2400 0.8432 52.90 0.6032 PSM 0.5458 0.1465 397.9 3.4954 0.0117 0.2385 0.8532 95.80 0.6147 NNM 0.5422 0.1525 398.0 3.4728 0.0128 0.2348 0.8496 94.90 0.6148 RM 0.5415 0.1496 397.9 3.4666 0.0119 0.2339 0.8491 96.40 0.6152 MM 0.5426 0.1529 395.0 3.4590 0.0130 0.2337 0.8515 95.20 0.6178

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

241

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=0 Collinearity=.90 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1536 0.1390 398.0 1.1077 0.3357 -0.1215 0.4351 94.50 0.5566 EM 0.1459 0.1659 397.9 1.3667 0.1953 -0.0855 0.4578 58.70 0.5433 PSM 0.1540 0.1486 397.9 1.1099 0.3527 -0.1214 0.4346 94.40 0.5560 NNM 0.1457 0.1466 398.0 1.1199 0.3379 -0.1197 0.4361 95.80 0.5558 RM 0.1527 0.1450 397.9 1.0843 0.3319 -0.1250 0.4306 94.60 0.5556 MM 0.1456 0.1461 395.0 1.1162 0.3391 -0.1207 0.4373 95.90 0.5580

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=.90 and Correlation=.99 1000 Replications Average % Method YD* df t-value p-value LCLM UCLM Coverage Width NOM 0.2487 398.0 1.7149 0.2009 -0.0364 0.5338 94.20 0.5703 EM 0.2187 397.9 1.5574 0.1926 -0.0584 0.4958 53.00 0.5542 PSM 0.2488 397.9 1.7199 0.1975 -0.0357 0.5332 96.10 0.5689 NNM 0.2499 398.0 1.7279 0.1949 -0.0349 0.5346 95.50 0.5696 RM 0.2514 397.9 1.7345 0.1916 -0.0336 0.5363 95.40 0.5699 MM 0.2494 395.0 1.7176 0.1965 -0.0366 0.5354 95.60 0.5719

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 2: t-test Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=.90 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.6045 0.1575 398.0 3.8732 0.0068 0.2970 0.9119 95.00 0.6149 EM 0.5950 0.3955 396.6 3.9143 0.0898 0.2943 0.8956 52.90 0.6013 PSM 0.5991 0.1435 397.9 3.8468 0.0045 0.2924 0.9058 96.40 0.6134 NNM 0.5959 0.1440 398.0 3.8206 0.0053 0.2888 0.9029 96.40 0.6141 RM 0.5986 0.1432 397.8 3.8336 0.0050 0.2914 0.9059 97.20 0.6145 MM 0.5963 0.1430 395.0 3.8057 0.0053 0.2878 0.9048 96.80 0.6169

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

242

Appendix F

Case 3: Mixed Covariates YD t-test, 95% Confidence Interval & Probability of Coverage Statistics

243

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=0 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0030 0.1388 398.0 0.0242 0.4977 -0.2809 0.2749 95.70 0.5558 EM 0.0022 0.2591 397.7 0.0159 0.3116 -0.2734 0.2777 74.50 0.5510 PSM 0.0006 0.1436 397.9 0.0041 0.5013 -0.2784 0.2771 94.90 0.5555 NNM 0.0010 0.1432 398.0 0.0062 0.4884 -0.2785 0.2765 95.00 0.5551 RM 0.0040 0.1417 397.9 0.0292 0.5076 -0.2819 0.2738 94.80 0.5558 MM 0.0011 0.1426 395.0 0.0066 0.4920 -0.2797 0.2775 95.40 0.5572

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.10 Collinearity=0 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1022 0.1501 398.0 0.7069 0.4101 -0.1826 0.3870 94.70 0.5696 EM 0.1005 0.2582 397.7 0.7012 0.3050 -0.1809 0.3818 72.80 0.5627 PSM 0.1024 0.1484 397.9 0.7091 0.4256 -0.1820 0.3868 94.80 0.5683 NNM 0.1009 0.1477 398.0 0.7005 0.4218 -0.1835 0.3854 94.30 0.5689 RM 0.1008 0.1452 397.9 0.6966 0.4252 -0.1839 0.3855 94.80 0.5694 MM 0.1012 0.1476 395.0 0.6993 0.4920 -0.1845 0.3867 94.40 0.5712

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=0 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4412 0.1539 398.0 2.8362 0.0441 0.1348 0.7475 94.90 0.6125 EM 0.4462 0.2623 397.7 2.8855 0.1095 0.1414 0.7509 73.80 0.6095 PSM 0.4392 0.1517 397.9 2.8208 0.0428 0.1327 0.7457 95.90 0.6130 NNM 0.4366 0.1549 398.0 2.8073 0.0478 0.1304 0.7428 95.30 0.6125 RM 0.4390 0.1524 397.9 2.8221 0.0438 0.1327 0.7453 95.70 0.6126 MM 0.4365 0.1544 395.0 2.7939 0.0480 0.1288 0.7441 95.30 0.6153

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

244

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=0 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1079 0.1453 398.0 0.7634 0.4089 -0.1701 0.3858 94.90 0.5558 EM 0.1057 0.2567 397.7 0.7534 0.3061 -0.1708 0.3823 73.50 0.5530 PSM 0.1015 0.1416 397.9 0.7197 0.4365 -0.1765 0.3795 94.80 0.5560 NNM 0.1050 0.1403 398.0 0.7453 0.4309 -0.1729 0.3829 95.30 0.5559 RM 0.1089 0.1407 397.9 0.7711 0.4109 -0.1689 0.3867 95.50 0.5556 MM 0.1047 0.1405 395.0 0.7402 0.4327 -0.1743 0.3837 95.40 0.5580

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.10 Collinearity=0 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1967 0.1495 398.0 1.3596 0.2855 -0.0882 0.4813 94.30 0.5695 EM 0.1958 0.2520 397.7 1.3687 0.2562 -0.0876 0.4789 74.10 0.5665 PSM 0.1996 0.1424 397.9 1.3795 0.2710 -0.0852 0.4844 95.60 0.5696 NNM 0.2022 0.1437 398.0 1.3989 0.2805 -0.0824 0.4868 95.70 0.5692 RM 0.2020 0.1449 397.9 1.3988 0.2708 -0.0824 0.4864 94.50 0.5688 MM 0.2023 0.1429 395.0 1.3940 0.2822 -0.0834 0.4881 96.30 0.5714

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3:tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=0 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5505 0.1529 398.0 3.5159 0.0112 0.2423 0.8587 95.7 0 0.6164 EM 0.5444 0.2690 397.7 3.4983 0.0602 0.2375 0.8513 75.20 0.6138 PSM 0.5455 0.1628 397.9 3.4875 0.0154 0.2375 0.8535 93.80 0.6160 NNM 0.5479 0.1565 398.0 3.5089 0.0133 0.2403 0.8554 95.00 0.6150 RM 0.5469 0.1595 397.9 3.4964 0.0160 0.2391 0.8546 95.00 0.6155 MM 0.5482 0.1561 395.0 3.4952 0.0134 0.2393 0.8572 95.60 0.6179

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

245

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=0 Collinearity=0 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1497 0.1478 398.0 1.0510 0.3445 -0.1282 0.4276 94.30 0.5558 EM 0.1461 0.2574 397.7 1.0404 0.2882 -0.1305 0.4228 70.70 0.5533 PSM 0.1492 0.1450 397.9 1.0544 0.3499 -0.1290 0.4273 94.90 05563 NNM 0.1473 0.1500 398.0 1.0404 0.3477 -0.1307 0.4253 93.20 0.5560 RM 0.1458 0.1402 397.9 1.0295 0.3606 -0.1323 0.4238 95.20 0.5561 MM 0.1482 0.1490 395.0 1.0429 0.3481 -0.1310 0.4273 93.50 0.5583

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=0 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2502 0.1497 398.0 1.7335 0.2000 -0.0339 0.5343 94.40 0.5682 EM 0.2519 0.2650 397.7 1.7579 0.2132 -0.0308 0.5347 72.60 0.5656 PSM 0.2538 0.1476 397.9 1.7565 0.1962 -0.0307 0.5384 94.50 05692 NNM 0.2516 0.1489 398.0 1.7424 0.2018 -0.0326 0.5357 94.90 0.5683 RM 0.2566 0.1462 397.9 1.7732 0.1854 -0.0283 0.5414 95.10 0.5697 MM 0.2515 0.1475 395.0 1.7349 0.2009 -0.0338 0.5368 95.00 0.5706

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=0 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5983 0.1654 398.0 3.8307 0.0085 0.2910 0.9056 93.40 0.6146 EM 0.5896 0.2631 397.7 3.8036 0.0428 0.2838 0.8953 76.20 0.6114 PSM 0.5986 0.1639 397.9 3.8320 0.0067 0.2911 0.9060 94.60 0.6149 NNM 0.6021 0.1633 398.0 3.8567 0.0066 0.2948 0.9095 93.30 0.6147 RM 0.5967 0.1619 397.9 3.8199 0.0067 0.2893 0.9041 93.90 0.6148 MM 0.6018 0.1630 395.0 3.8373 0.0069 0.2930 0.9104 93.80 0.6174

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

246

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=0 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0068 0.1379 398.0 0.0485 0.5101 -0.2713 0.2849 95.40 0.5562 EM 0.0128 0.2136 397.7 0.0936 0.3655 -0.2607 0.2863 80.90 0.5471 PSM 0.0082 0.1577 397.9 0.0597 0.4639 -0.2699 0.2864 92.20 0.5564 NNM 0.0090 0.1536 398.0 0.0631 0.4784 -0.2690 0.2869 92.90 0.5559 RM 0.0011 0.1518 397.9 0.0764 0.4770 -0.2670 0.2883 92.60 0.5563 MM 0.0086 0.1525 395.0 0.0604 0.4802 -0.2705 0.2877 93.30 0.5582

Y =Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.10 Collinearity=0 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1011 0.1417 398.0 0.7010 0.7784 -0.1833 0.3854 95.60 0.5587 EM 0.1112 0.2185 397.7 0.7784 0.3311 -0.1693 0.3917 80.40 0.5610 PSM 0.1067 0.1540 397.9 0.7367 0.4089 -0.1779 0.3913 93.80 0.5692 NNM 0.1084 0.1539 398.0 0.7503 0.4059 -0.1760 0.3928 92.80 0.5688 RM 0.1071 0.1531 397.9 0.7418 0.4125 -0.1773 0.3916 92.70 0.5689 MM 0.1083 0.1529 395.0 0.7467 0.4084 -0.1773 0.3939 93.10 0.5712

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=0 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4375 0.1606 398.0 2.8470 0.0443 0.1367 0.7507 94.50 0.6140 EM 0.4379 0.2251 397.7 2.8227 0.0855 0.1307 0.7362 85.00 0.6055 PSM 0.4360 0.1402 397.9 2.8404 0.0382 0.1359 0.7495 97.00 0.6136 NNM 0.4362 0.1504 398.0 2.8184 0.0440 0.1319 0.7459 96.30 0.6138 RM 0.4363 0.1497 397.9 2.8055 0.0436 0.1305 0.7446 95.90 0.6141 MM 0.4369 0.1509 395.0 2.8012 0.0444 0.1304 0.7471 96.20 0.6167

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

247

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=0 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1047 0.1424 398.0 0.7265 0.4313 -0.1754 0.3998 94.10 0.5552 EM 0.0886 0.2158 397.7 0.6387 0.3457 -0.1853 0.3618 79.50 0.5471 PSM 0.0982 0.1472 397.9 0.6968 0.4216 -0.1795 0.3757 94.10 0.5552 NNM 0.0993 0.1504 398.0 0.7265 0.4105 -0.1752 0.3795 93.70 0.5447 RM 0.1022 0.1469 397.9 0.7539 0.4136 -0.1713 0.3839 93.70 0.5553 MM 0.0992 0.1490 395.0 0.7231 0.4136 -0.1763 0.3806 93.60 0.5569

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.10 Collinearity=0 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2000 0.1420 398.0 1.3839 0.2820 -0.0849 0.4849 95.60 0.5698 EM 0.2030 0.2195 397.7 1.4211 0.2575 -0.0784 0.4846 80.50 0.5630 PSM 0.2027 0.1499 397.9 1.4003 0.2762 -0.0823 0.4879 94.30 0.5702 NNM 0.2024 0.1445 398.0 1.3999 0.2781 -0.0826 0.4873 95.40 0.5699 RM 0.2017 0.1475 397.9 1.3943 0.2813 -0.0832 0.4866 94.10 0.5698 MM 0.2021 0.1434 395.0 1.3962 0.2777 -0.0840 0.4882 95.70 0.5722

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=0 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5500 0.1621 398.0 3.5231 0.0156 0.2427 0.8574 93.80 0.6147 EM 0.5438 0.2057 397.7 3.5244 0.0319 0.2391 0.8465 86.40 0.6073 PSM 0.5452 0.1411 397.9 3.4845 0.0106 0.2372 0.8532 97.50 0.6161 NNM 0.5510 0.1433 398.0 3.5228 0.0094 0.2430 0.8590 97.60 0.6160 RM 0.5450 0.1415 397.9 3.4809 0.0107 0.2364 0.8520 96.90 0.6156 MM 0.5509 0.1431 395.0 3.5059 0.0095 0.2415 0.8603 97.80 0.6188

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

248

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=0 Collinearity=0 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1452 0.1449 398.0 1.0983 0.3430 -0.1229 0.4322 95.20 0.5551 EM 0.1475 0.2069 397.7 1.0542 0.3024 -0.1279 0.4194 78.10 0.5473 PSM 0.1528 0.1502 397.9 1.0785 0.3444 -0.1260 0.4280 94.80 0.5660 NNM 0.1488 0.1457 398.0 1.0320 0.3444 -0.1322 0.4234 92.40 0.5558 RM 0.1584 0.1509 397.9 1.0870 0.3532 -0.1242 0.4308 94.60 0.5550 MM 0.1496 0.1457 395.0 1.0223 0.3461 -0.1340 0.4238 93.00 0.5579

Y =Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=0 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2010 0.1423 398.0 1.7757 0.1894 -0.0281 0.5409 96.10 0.5690 EM 0.2500 0.2134 397.7 1.7722 0.2163 -0.0284 0.5339 84.10 0.5623 PSM 0.2504 0.1434 397.9 1.7515 0.1936 -0.0315 0.5385 96.30 0.5700 NNM 0.2489 0.1409 398.0 1.7069 0.2074 -0.0379 0.5319 95.70 0.5698 RM 0.2520 0.1437 397.9 1.7233 0.1959 -0.0355 0.5335 95.40 0.5690 MM 0.2485 0.1415 395.0 1.7013 0.2063 -0.0389 0.5333 95.10 0.5722

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=0 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.6084 0.1543 398.0 3.8403 0.0077 0.2928 0.9079 93.90 0.6151 EM 0.5890 0.2295 397.7 3.8531 0.0209 0.2905 0.8978 84.20 0.6073 PSM 0.5957 0.1458 397.9 3.8678 0.0041 0.2966 0.8697 97.20 0.6141 NNM 0.6071 0.1470 398.0 3.8721 0.0043 0.2973 0.9119 96.40 0.6146 RM 0.6021 0.1447 397.9 3.8097 0.0050 0.2874 0.9017 96.70 0.6143 MM 0.6073 0.1460 395.0 3.8544 0.0042 0.2959 0.9133 96.60 0.6174

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

249

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=0 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0003 0.1462 398.0 0.0016 0.4855 -0.2786 0.2781 94.40 0.5567 EM 0.0094 0.2117 397.7 0.0626 0.3870 -0.2643 0.2830 80.80 0.5473 PSM 0.0048 0.1526 397.9 0.0320 0.4750 -0.2731 0.2828 93.40 0.5560 NNM 0.0044 0.1588 398.0 0.0301 0.4626 -0.2736 0.2825 91.80 0.5561 RM 0.0043 0.1516 397.9 0.0283 0.4757 -0.2739 0.2825 93.60 0.5564 MM 0.0047 0.1584 395.0 0.0320 0.4639 -0.2744 0.2838 91.80 0.5582

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.10 Collinearity=0 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0967 0.1460 398.0 0.6677 0.4192 -0.1879 0.3812 95.70 0.5691 EM 0.1009 0.2094 397.7 0.7099 0.3453 -0.1793 0.3812 79.80 0.5606 PSM 0.1053 0.1517 397.9 0.7295 0.4051 -0.1791 0.3897 93.70 0.5689 NNM 0.1045 0.1507 398.0 0.7212 0.4221 -0.1802 0.3892 93.30 0.5694 RM 0.1062 0.1561 397.9 0.7359 0.4089 -0.1784 0.3908 93.40 0.5692 MM 0.1047 0.1501 395.0 0.7197 0.4232 -0.1811 0.3906 93.60 0.5717

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=0 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4408 0.1544 398.0 2.8417 0.0415 0.1362 0.7508 96.30 0.6146 EM 0.4429 0.2208 397.7 2.8538 0.0772 0.1357 0.7415 85.10 0.6058 PSM 0.4444 0.1514 397.9 2.8678 0.0377 0.1401 0.7539 95.50 0.6138 NNM 0.4387 0.1473 398.0 2.8441 0.0427 0.1368 0.7518 95.80 0.6150 RM 0.4402 0.1478 397.9 2.8590 0.0376 0.1392 0.7537 96.30 0.6145 MM 0.4385 0.1470 395.0 2.8327 0.0424 0.1356 0.7535 95.90 0.6178

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

250

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=0 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0992 0.1430 398.0 0.7017 0.4291 -0.1783 0.3769 94.60 0.5552 EM 0.0948 0.2049 397.7 0.6829 0.3435 -0.1788 0.3684 81.80 0.5472 PSM 0.0945 0.1479 397.9 0.6598 0.4235 -0.1847 0.3724 93.30 0.5560 NNM 0.1004 0.1468 398.0 0.7109 0.4152 -0.1778 0.3785 93.70 0.5563 RM 0.0978 0.1476 397.9 0.6901 0.4177 -0.1803 0.3759 93.80 0.5662 MM 0.0999 0.1361 395.0 0.7047 0.4162 -0.1794 0.3792 94.20 0.5586

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.10 Collinearity=0 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1980 0.1441 398.0 1.3720 0.2792 -0.0864 0.4825 95.10 0.5689 EM 0.1963 0.2014 397.7 1.3795 0.2767 -0.0841 0.4767 84.20 0.5608 PSM 0.2033 0.1440 397.9 1.4060 0.2714 -0.0814 0.4880 95.00 0.5694 NNM 0.2068 0.1486 398.0 1.4312 0.2647 -0.0778 0.4913 94.10 0.5691 RM 0.2036 0.1444 397.9 1.4111 0.2705 -0.0808 0.4880 94.30 0.5688 MM 0.2072 0.1480 395.0 1.4286 0.2653 -0.0785 0.4930 94.80 0.5715

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.45Collinearity=0 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5428 0.1638 398.0 3.4760 0.0171 0.2354 0.8502 93.10 0.6148 EM 0.5459 0.2104 397.7 3.5527 0.0355 0.2428 0.8489 85.10 0.6061 PSM 0.5511 0.1389 397.9 3.5337 0.0083 0.2439 0.8583 96.50 0.6143 NNM 0.5517 0.1436 398.0 3.5304 0.0095 0.2440 0.8595 96.90 0.6155 RM 0.5523 0.1447 397.9 3.5388 0.0090 0.2450 0.8596 96.40 0.6146 MM 0.5516 0.1429 395.0 3.5138 0.0094 0.2424 0.8607 96.80 0.6183

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

251

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=0 Collinearity=0 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1426 0.1408 398.0 1.0538 0.3462 -0.1289 0.4258 93.10 0.6148 EM 0.1479 0.2119 397.7 1.1052 0.2960 -0.1200 0.4264 85.10 0.6061 PSM 0.1455 0.1451 397.9 1.0604 0.3527 -0.1278 0.4265 96.50 0.6143 NNM 0.1508 0.1532 398.0 1.1236 0.3350 -0.1190 0.4351 96.90 0.6155 RM 0.1457 0.1481 397.9 1.0899 0.3351 -0.1236 0.4306 96.40 0.6146 MM 0.1509 0.1517 395.0 1.1190 0.3380 -0.1201 0.4361 96.80 0.6183

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=0 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2517 0.1433 398.0 1.7371 0.0076 -0.0333 0.5366 94.70 0.5699 EM 0.2560 0.2011 397.7 1.7990 0.0092 -0.0243 0.5363 82.40 0.5606 PSM 0.2498 0.1482 397.9 1.7254 0.0077 -0.0351 0.5348 94.60 0.5699 NNM 0.2585 0.1463 398.0 1.7814 0.0076 -0.0267 0.5436 94.50 0.5703 RM 0.2484 0.1468 397.9 1.7178 0.0079 -0.0362 0.5330 94.60 0.5692 MM 0.2581 0.1461 395.1 1.7717 0.0080 -0.0282 0.5444 95.00 0.5726

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=0 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.6031 0.1579 398.0 3.8617 0.0061 0.2965 0.9104 94.70 0.6149 EM 0.6154 0.2086 397.7 3.9908 0.0158 0.3118 0.9190 85.80 0.6072 PSM 0.6079 0.1420 397.9 3.9042 0.0035 0.3031 0.9186 97.00 0.6155 NNM 0.6073 0.1411 398.0 3.8879 0.0033 0.2999 0.9147 97.00 0.6148 RM 0.6078 0.1397 397.9 3.3888 0.0037 0.3002 0.9154 96.60 0.6152 MM 0.6074 0.1398 395.1 3.8708 0.0033 0.2985 0.9161 97.30 0.6176

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

252

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=0 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0055 0.1386 398.0 0.0391 0.4912 -0.2723 0.2833 95.00 0.5556 EM 0.0016 0.2124 397.7 0.0134 0.3095 -0.2710 0.2742 80.30 0.5451 PSM 0.0068 0.1520 397.9 0.0480 0.4731 -0.2708 0.2844 92.90 0.5552 NNM 0.0049 0.1517 398.0 0.0350 0.4855 -0.2731 0.2830 93.60 0.5562 RM 0.0042 0.1516 397.9 0.0227 0.4559 -0.2740 0.2823 91.30 0.5563 MM 0.0050 0.1511 395.0 0.0355 0.4774 -0.2741 0.2842 93.80 0.5583

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.10 Collinearity=0 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0949 0.1435 398.0 0.6349 0.4434 -0.1927 0.3765 96.20 0.5592 EM 0.1010 0.1994 397.7 0.7129 0.3447 -0.1789 0.3810 82.80 0.5599 PSM 0.1043 0.1503 397.9 0.7188 0.4097 -0.1805 0.3891 94.90 0.5696 NNM 0.1009 0.1565 398.0 0.6957 0.4195 -0.1840 0.3858 92.60 0.5698 RM 0.0980 0.1571 397.9 0.6777 0.4039 -0.1866 0.3825 93.20 0.5691 MM 0.1011 0.1554 395.0 0.6942 0.4182 -0.1849 0.3871 93.00 0.5721

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=0 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4385 0.1475 398.0 2.8135 0.0414 0.1317 0.7455 95.90 0.6138 EM 0.4345 0.2013 397.7 2.8268 0.0766 0.1316 0.7373 86.80 0.6057 PSM 0.4406 0.1457 397.9 2.8227 0.0375 0.1333 0.7478 96.50 0.6145 NNM 0.4376 0.1421 398.0 2.8033 0.0371 0.1302 0.7450 96.90 0.6148 RM 0.4361 0.1439 397.9 2.7981 0.0410 0.1291 0.7431 95.70 0.6140 MM 0.4370 0.1406 395.0 2.7866 0.0371 0.1281 0.7458 97.40 0.6176

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

253

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=0 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0977 0.1362 398.0 0.6922 0.4434 -0.1807 0.3758 95.80 0.5565 EM 0.0959 0.2092 397.7 0.6622 0.3575 -0.1806 0.3650 81.60 0.5456 PSM 0.1003 0.1480 397.9 0.7105 0.4301 -0.1778 0.3784 93.40 0.5562 NNM 0.0966 0.1478 398.0 0. 6831 0.4277 -0.1816 0.3747 93.40 0.5563 RM 0.1024 0.1474 397.9 0.7260 0.3983 -0.1756 0.3803 94.80 0.5559 MM 0.0968 0.1473 395.0 0.6753 0.4393 -0.1833 0.3872 94.70 0.5584

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.10Collinearity=0 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1914 0.1451 398.0 1.3239 0.2871 -0.0930 0.4759 95.90 0.5690 EM 0.1937 0.1999 397.7 1.3642 0.2894 -0.0862 0.4736 85.70 0.5598 PSM 0.2025 0.1490 397.9 1.3995 0.2666 -0.0822 0.4871 93.80 0.5693 NNM 0.2063 0.1479 398.0 1.4290 0.2730 -0.0780 0.4905 94.80 0.5685 RM 0.2075 0.1470 397.9 1.4356 0.2659 -0.0771 0.4921 94.50 0.5692 MM 0.2064 0.1471 395.0 1.4240 0.2717 -0.0790 0.4918 95.10 0.5709

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=0 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5504 0.1582 398.0 3.5205 0.0144 0.2425 0.8582 95.10 0.6157 EM 0.5533 0.2079 397.7 3.5941 0.0310 0.2497 0.8568 84.80 0.6070 PSM 0.5564 0.1434 397.9 3.5560 0.0085 0.2484 0.8644 96.60 0.6159 NNM 0.5566 0.1492 398.0 3.5554 0.0100 0.2482 0.8648 96.20 0.6164 RM 0.5552 0.1508 397.9 3.5467 0.0096 0.2470 0.8635 95.60 0.6165 MM 0.5561 0.1488 395.0 3.5356 0.0105 0.2464 0.8656 96.40 0.6192

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

254

Case 3:t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=0 Collinearity=0 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1583 0.1438 398.0 1.1211 0.3264 -0.1194 0.4359 94.90 0.5553 EM 0.1564 0.2033 397.7 1.1348 0.3012 -0.1159 0.4287 81.60 0.5447 PSM 0.1562 0.1393 397.9 1.1084 0.3455 -0.1214 0.4338 95.50 0.5552 NNM 0.1549 0.1431 398.0 1.0978 0.3454 -0.1229 0.4326 95.70 0.5555 RM 0.1539 0.1438 397.9 1.0897 0.3427 -0.1243 0.4320 94.10 0.5663 MM 0.1556 0.1423 395.0 1.0988 0.3442 -0.1232 0.4345 95.70 0.5577

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=0 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2499 0.1433 398.0 1.7265 0.2050 -0.0349 0.5347 95.40 0.5696 EM 0.2519 0.2025 397.7 1.7742 0.2272 -0.0280 0.5318 84.70 0.5598 PSM 0.2476 0.1419 397.9 1.7124 0.1992 -0.0370 0.5322 95.40 0.5693 NNM 0.2468 0.1472 398.0 1.7092 0.2062 -0.0375 0.5312 94.70 0.5687 RM 0.2483 0.1397 397.9 1.7162 0.1927 -0.0364 0.5330 95.60 0.5695 MM 0.2469 0.1461 395.0 1.7019 0.2059 -0.0387 0.5323 94.80 0.5510

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=0 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.6026 0.1553 398.0 3.8559 0.0071 0.2951 0.9101 95.20 0.6150 EM 0.6063 0.2025 397.7 3.9431 0.0188 0.3032 0.9094 86.90 0.6062 PSM 0.6013 0.1379 397.9 3.8496 0.0039 0.2938 0.9089 97.60 0.6151 NNM 0.6056 0.1399 398.0 3.8729 0.0042 0.2977 0.9133 97.40 0.6156 RM 0.5980 0.1414 397.9 3.8301 0.0049 0.2906 0.9053 97.20 0.6147 MM 0.6054 0.1394 395.0 3.8538 0.0043 0.2961 0.9146 97.60 0.6185

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

255

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=.40 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0055 0.1386 398.0 0.0191 0.4989 -0.2802 0.2750 94.60 0.5553 EM 0.0016 0.2124 397.7 0.0380 0.3239 -0.2802 0.2703 70.40 0.5505 PSM 0.0068 0.1520 397.9 0.0150 0.4899 -0.2795 0.2751 94.20 0.5546 NNM 0.0049 0.1517 398.0 0.0177 0.5062 -0.2797 0.2746 95.20 0.5543 RM 0.0042 0.1516 397.9 0.0175 0.4880 -0.2801 0.2755 95.50 0.5556 MM 0.0050 0.1511 395.0 0.0207 0.5032 -0.2812 0.2752 95.30 0.5565

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.10 Collinearity=.40 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1020 0.1463 398.0 0.7068 0.4277 -0.1818 0.3858 94.90 0.5677 EM 0.1030 0.2653 397.7 0.7101 0.2928 -0.1798 0.3857 71.70 0.5655 PSM 0.1001 0.1526 397.9 0.6911 0.4277 -0.1842 0.3845 93.90 0.5687 NNM 0.1022 0.1492 398.0 0.7100 0.4248 -0.1820 0.3864 94.00 0.5644 RM 0.0997 0.1442 397.9 0.6886 0.4307 -0.1850 0.3844 95.10 0.5695 MM 0.1021 0.1487 395.0 0.7073 0.4289 -0.1831 0.3876 94.00 0.5707

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=.40 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4399 0.1540 398.0 2.8252 0.0431 0.1334 0.6415 94.90 0.5677 EM 0.4462 0.2688 397.7 2.8868 0.1146 0.1493 0.7504 71.70 0.5655 PSM 0.4475 0.1566 397.9 2.8770 0.0420 0.1411 0.7538 93.90 0.5687 NNM 0.4461 0.1567 398.0 2.8646 0.0442 0.1395 0.7527 94.00 0.5644 RM 0.4457 0.1564 397.9 2.8371 0.0459 0.1353 0.7488 95.10 0.5695 MM 0.4455 0.1560 395.0 2.8489 0.0445 0.1377 0.7536 94.00 0.5707

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

256

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=.40 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1020 0.1463 398.0 0.7074 0.4384 -0.1782 0.3779 94.40 0.5661 EM 0.1030 0.2653 397.7 0.7525 0.2833 -0.1710 0.3817 70.80 0.5627 PSM 0.1001 0.1526 397.9 0.7018 0.4259 -0.1785 0.3771 94.80 0.5556 NNM 0.1022 0.1492 398.0 0.7534 0.4067 -0.1711 0.3842 95.90 0.5553 RM 0.0997 0.1442 397.9 0.7437 0.4199 -0.1727 0.3827 94.80 0.5554 MM 0.1021 0.1487 395.0 0.7468 0.4095 -0.1726 0.3848 95.60 0.5774

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.10 Collinearity=.40 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2102 0.1467 398.0 1.4548 0.2578 -0.0740 0.4945 94.20 0.5685 EM 0.1931 0.2575 397.7 1.3468 0.2628 -0.0886 0.4749 75.40 0.5636 PSM 0.1986 0.1418 397.9 1.3746 0.2891 -0.0859 0.4832 96.30 0.5692 NNM 0.2013 0.1401 398.0 1.3935 0.2676 -0.0831 0.4857 95.30 0.5688 RM 0.1989 0.1420 397.9 1.3778 0.2828 -0.0854 0.4832 95.50 0.5686 MM 0.2018 0.1400 395.0 1.3914 0.2675 -0.0837 0.4874 95.30 0.5711

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=.40 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5549 0.1565 398.0 3.5530 0.0117 0.2475 0.8822 94.60 0.6147 EM 0.5553 0.2727 397.7 3.5879 0.0621 0.2500 0.8601 72.10 0.6106 PSM 0.5571 0.1560 397.9 3.5665 0.0116 0.2496 0.8645 95.90 0.6149 NNM 0.5583 0.1561 398.0 3.5583 0.0111 0.2509 0.5687 95.10 0.6148 RM 0.5599 0.1551 397.9 3.5885 0.0112 0.2526 0.8672 95.50 0.6147 MM 0.5586 0.1557 395.0 3.5630 0.0110 0.2498 0.8673 95.60 0.6175

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

257

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias.15, ES=0 Collinearity=.40 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1535 0.1378 398.0 1.0865 0.3453 -0.1240 0.4311 95.50 0.5551 EM 0.1598 0.2549 397.7 1.1487 0.2763 -0.1152 0.4347 71.30 0.5499 PSM 0.1532 0.1390 397.9 1.0874 0.3468 -0.1244 0.4307 95.10 0.5551 NNM 0.1460 0.1413 398.0 1.0368 0.3617 -0.1312 0.4232 95.30 0.5544 RM 0.1445 0.1430 397.9 1.0244 0.3595 -0.1332 0.4221 94.70 0.5553 MM 0.1460 0.1410 395.0 1.0326 0.3651 -0.1413 0.4243 95.30 0.5566

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=.40 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2500 0.1485 398.0 1.7289 0.2032 -0.0345 0.5344 95.00 0.5689 EM 0.2447 0.2333 397.7 1.7062 0.2240 -0.0380 0.5235 75.70 0.5616 PSM 0.2521 0.1394 397.9 1.7433 0.1905 -0.0232 0.5365 96.30 0.5688 NNM 0.2535 0.1460 398.0 1.7548 0.1922 -0.0308 0.5378 94.50 0.5686 RM 0.2526 0.1467 397.9 1.7489 0.1939 -0.0316 0.5368 94.50 0.5684 MM 0.2530 0.1451 395.0 1.7447 0.1934 -0.0324 0.5384 95.00 0.5708

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=.40 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.6026 0.1553 398.0 3.8495 0.0075 0.2935 0.9081 94.90 0.6146 EM 0.6063 0.2025 397.7 3.9403 0.0378 0.3056 0.9179 74.60 0.6123 PSM 0.6013 0.1379 397.9 3.8494 0.0073 0.2935 0.9080 94.50 0.6145 NNM 0.6056 0.1399 398.0 3.3675 0.0071 0.2964 0.9102 93.50 0.6144 RM 0.5980 0.1414 397.9 3.8371 0.0093 0.2920 0.9071 94.10 0.6151 MM 0.6054 0.1394 395.0 3.8507 0.0070 0.2951 0.9126 93.70 0.6175

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

258

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=.40 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0068 0.1449 398.0 0.0494 0.4912 -0.2704 0.2841 94.40 0.5545 EM 0.0026 0.2352 397.7 0.0597 0.3095 -0.2850 0.2680 76.10 0.5530 PSM 0.0008 0.1438 397.9 0.0057 0.4852 -0.2771 0.2788 95.20 0.5559 NNM 0.0038 0.1473 398.0 0.0288 0.4855 -0.2736 0.2814 94.10 0.5550 RM 0.0039 0.1476 397.9 0.0496 0.4854 -0.2704 0.2843 94.80 0.5547 MM 0.0039 0.1469 395.0 0.0289 0.4871 -0.2746 0.2824 94.20 0.5570

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.10 Collinearity=.40 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1000 0.1455 398.0 0.6909 0.4247 -0.1840 0.3841 95.00 0.5681 EM 0.1013 0.2340 397.7 0.7113 0.3190 -0.1793 0.3817 77.60 0.5610 PSM 0.0986 0.1515 397.9 0.6781 0.4205 -0.1864 0.3818 94.50 0.5682 NNM 0.1047 0.1759 398.0 0.7292 0.4103 -0.1793 0.3888 93.00 0.5581 RM 0.0998 0.1504 397.9 0.6910 0.4351 -0.1845 0.3840 93.20 0.5685 MM 0.1035 0.1504 395.0 0.7133 0.4170 -0.1820 0.3887 93.70 0.5508

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=.40 and Correlation=.75 1000 Replications Average % Method YD* STD df t-value p-value LCLM UCLM Coverage Width NOM 0.4355 0.1626 398.0 2.7970 0.0521 0.1289 0.7419 93.60 0.6130 EM 0.4312 0.2421 397.7 2.7910 0.1009 0.1270 0.7353 79.50 0.6083 PSM 0.4425 0.1484 397.9 2.8402 0.0391 0.1358 0.7492 96.20 0.6134 NNM 0.4375 0.1543 398.0 2.8081 0.0483 0.1308 0.7442 94.40 0.6133 RM 0.4417 0.1514 397.9 2.8344 0.0428 0.1349 0.7484 94.60 0.6135 MM 0.4379 0.1540 395.0 2.7974 0.0482 0.1297 0.7460 94.50 0.6162

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

259

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=.40 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0998 0.1428 398.0 0.7047 0.4296 -0.1783 0.3780 94.80 0.5563 EM 0.1027 0.2399 397.7 0.7319 0.3078 -0.1725 0.3779 75.50 0.5504 PSM 0.1009 0.1470 397.9 0.7284 0.4172 -0.1753 0.3815 93.60 0.5568 NNM 0.1002 0.1483 398.0 0.7086 0.4198 -0.1776 0.3780 93.50 0.5556 RM 0.1005 0.1497 397.9 0.7105 0.4148 -0.1775 0.3784 92.80 0.5559 MM 0.1000 0.1477 395.0 0.7043 0.4238 -0.1789 0.3789 93.80 0.5578

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.10Collinearity=.40 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2044 0.1453 398.0 1.4104 0.2671 -0.0809 0.4898 95.80 0.5707 EM 0.1928 0.2371 397.7 1.3435 0.2593 -0.0898 0.4754 76.10 0.5652 PSM 0.2079 0.1495 397.9 1.4342 0.2563 -0.0771 0.4929 93.50 0.5700 NNM 0.2096 0.1455 398.0 1.4495 0.2589 -0.0752 0.4945 94.30 0.5697 RM 0.2011 0.1485 397.9 1.3889 0.2790 -0.0839 0.4861 94.70 0.5700 MM 0.2097 0.1447 395.0 1.4444 0.2596 -0.0763 0.4956 94.70 0.5719

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=.40 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5467 0.1616 398.0 3.4932 0.0140 0.2386 0.8546 95.80 0.6160 EM 0.5460 0.2427 397.7 3.5253 0.0482 0.2407 0.8512 76.10 0.6105 PSM 0.5501 0.1474 397.9 3.4545 0.0115 0.2323 0.8478 93.50 0.6155 NNM 0.5508 0.1409 398.0 3.5226 0.0082 0.2430 0.8586 94.30 0.6156 RM 0.5504 0.1466 397.9 3.5195 0.0110 0.2425 0.8582 94.70 0.6157 MM 0.5506 0.1405 395.0 3.5049 0.0083 0.2413 0.8597 94.70 0.6184

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

260

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=0 Collinearity=.40 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1522 0.1443 398.0 1.0773 0.3460 -0.1253 0.4298 95.80 0.5552 EM 0.1587 0.2339 397.7 1.1458 0.2880 -0.1146 0.4345 76.10 0.5491 PSM 0.1549 0.1401 397.9 1.0968 0.3361 -0.1229 0.4326 93.50 0.5555 NNM 0.1532 0.1479 398.0 1.0849 0.3354 -0.1246 0.4310 94.30 0.5556 RM 0.1509 0.1401 397.9 1.0698 0.3391 -0.1269 0.4287 94.70 0.5556 MM 0.1529 0.1465 395.0 1.0786 0.3380 -0.1260 0.4318 94.70 0.5578

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=.40 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2505 0.1446 398.0 1.7354 0.1993 -0.0337 0.5346 95.80 0.5683 EM 0.2554 0.2295 397.7 1.7819 0.2128 -0.0266 0.5374 76.10 0.5640 PSM 0.2516 0.1393 397.9 1.7454 0.1865 -0.0322 0.5364 93.50 0.5687 NNM 0.2529 0.1365 398.0 1.7539 0.1892 -0.0311 0.5380 94.30 0.5691 RM 0.2531 0.1463 397.9 1.7514 0.1948 -0.0316 0.5378 94.70 0.5694 MM 0.2532 0.1359 395.0 1.7435 0.1904 -0.0327 0.5387 94.70 0.5614

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=.40 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.6100 0.1543 398.0 3.9077 0.0053 0.3028 0.9172 95.70 0.6144 EM 0.5890 0.2295 397.7 3.8173 0.0309 0.2845 0.8933 82.40 0.6088 PSM 0.5947 0.1458 397.9 3.8004 0.0063 0.2861 0.9010 96.10 0.6149 NNM 0.6071 0.1470 398.0 3.8921 0.0055 0.2999 0.9143 95.70 0.6144 RM 0.6021 0.1447 397.9 3.8545 0.0042 0.2947 0.9096 95.10 0.6148 MM 0.6073 0.1460 395.0 3.8755 0.0050 0.2987 0.9158 95.90 0.6172

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

261

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=.40 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0070 0.1449 398.0 0.5113 0.0521 -0.2707 0.2855 95.10 0.5562 EM 0.0080 0.2354 397.7 0.0201 0.3572 -0.2791 0.2716 78.70 0.5507 PSM 0.0021 0.1515 397.9 0.0471 0.4969 -0.2715 0.2849 93.79 0.5564 NNM 0.0027 0.1568 398.0 0.0040 0.4762 -0.2778 0.2789 93.70 0.5567 RM 0.0006 0.1557 397.9 0.0306 0.5000 -0.2739 0.2827 93.70 0.5566 MM 0.0032 0.1558 395.0 0.0001 0.4777 -0.2795 0.2794 94.10 0.5589

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.10 Collinearity=.40 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1040 0.1444 398.0 0.7377 0.4299 -0.1736 0.3819 95.60 0.5555 EM 0.1023 0.2324 397.7 0.7034 0.3327 -0.1764 0.3718 78.40 0.5482 PSM 0.1055 0.1506 397.9 0.7733 0.4057 -0.1685 0.3868 94.40 0.5553 NNM 0.1022 0.1556 398.0 0.7778 0.4081 -0.1682 0.3874 92.70 0.5456 RM 0.1008 0.1539 397.9 0.7209 0.4080 -0.1765 0.3794 93.70 0.5559 MM 0.1018 0.1549 395.0 0.7716 0.4105 -0.1697 0.3880 92.60 0.5577

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=.40 and Correlation=.90 1000 Replications Average Coverage Method YD* SD t-value p-value LCLM UCLM % age Width NOM 0.4418 0.1538 398.0 2.8294 0.0456 0.1344 0.7493 95.20 0.6149 EM 0.4434 0.2232 397.7 2.8740 0.0838 0.1396 0.7472 82.60 0.6076 PSM 0.4408 0.1450 397.9 2.8247 0.0399 0.1336 0.7479 95.90 0.6143 NNM 0.4406 0.1519 398.0 2.8248 0.0439 0.1336 0.7526 95.20 0.6140 RM 0.4453 0.1476 397.9 2.8522 0.0411 0.1379 0.7523 95.80 0.6146 MM 0.4408 0.1466 395.1 2.8136 0.0442 0.1325 0.7492 94.10 0.6167

*YD= LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage

262

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=.40 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1041 0.1417 398.0 0.7377 0.4290 -0.1736 0.3819 94.90 0.5555 EM 0.0978 0.2312 397.7 0.7034 0.3327 -0.1764 0.3718 74.40 0.5482 PSM 0.1091 0.1458 397.9 0.7734 0.4057 -0.1686 0.3867 94.30 0.5553 NNM 0.1096 0.1480 398.0 0.7778 0.4081 -0.1681 0.3875 93.90 0.5556 RM 0.1015 0.1494 397.9 0.7208 0.4080 -0.1674 0.3794 93.50 0.5558 MM 0.1091 0.1477 395.0 0.7716 0.4105 -0.1697 0.3881 93.80 0.5578

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.10 Collinearity=.40 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2086 0.1487 398.0 1.4408 0.2609 -0.0761 0.4934 94.70 0.5695 EM 0.1998 0.2323 397.7 1.3975 0.2600 -0.0817 0.4813 77.40 0.5630 PSM 0.2064 0.1470 397.9 1.4290 0.2716 -0.0779 0.4907 94.80 0.5686 NNM 0.2018 0.1439 398.0 1.3946 0.2718 -0.0830 0.4865 94.70 0.5695 RM 0.2036 0.1472 397.9 1.4054 0.2566 -0.0811 0.4883 93.50 0.5694 MM 0.2021 0.1428 395.0 1.3915 0.2723 -0.0837 0.4880 95.10 0.5717

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t--tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=.40 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5552 0.1583 398.0 3.5498 0.0123 0.2475 0.8630 94.10 0.6154 EM 0.5473 0.2333 397.8 3.5391 0.0434 0.2425 0.8522 79.90 0.6097 PSM 0.5528 0.1485 397.9 3.5363 0.0116 0.2450 0.8605 96.40 0.6155 NNM 0.5537 0.1462 398.0 3.5430 0.0091 0.2461 0.8613 97.00 0.6152 RM 0.5543 0.1490 397.9 3.5458 0.0095 0.2465 0.8620 93.30 0.6155 MM 0.5532 0.1461 395.1 3.5346 0.0093 0.2443 0.8622 97.20 0.6179

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

263

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.00 Collinearity=.40 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1500 0.1440 398.0 1.0636 0.3416 -0.1279 0.4279 95.00 0.5558 EM 0.1467 0.2309 397.7 1.0495 0.2954 -0.1278 0.4211 76.10 0.5489 PSM 0.1506 0.1535 397.9 1.0662 0.3385 -0.1270 0.4282 93.10 0.5552 NNM 0.1524 0.1484 398.0 1.0757 0.3465 -0.1261 0.4309 92.80 0.5570 RM 0.1459 0.1453 397.9 1.0252 0.3567 -0.1329 0.4225 94.00 0.5554 MM 0.1527 0.1478 395.1 1.0739 0.3498 -0.1368 0.4323 92.80 0.5591

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=.40 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2502 0.1456 398.0 1.7277 0.2003 -0.0349 0.5353 95.40 0.5702 EM 0.2494 0.2278 397.7 1.7548 0.2163 -0.0319 0.5306 79.80 0.5626 PSM 0.2474 0.1449 397.9 1.7108 0.2019 -0.0375 0.5323 94.40 0.5698 NNM 0.2524 0.1434 398.0 1.7453 0.1957 -0.0323 0.5372 95.50 0.5695 RM 0.2465 0.1474 397.9 1.7078 0.2058 -0.0379 0.5309 94.40 0.5688 MM 0.2524 0.1420 395.1 1.7382 0.1951 -0.0335 0.5384 95.60 0.5719

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=.40 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.6057 0.1513 398.0 3.8804 0.0059 0.2985 0.9130 95.20 0.6135 EM 0.6108 0.2290 397.7 3.9510 0.0233 0.3064 0.9150 82.40 0.6086 PSM 0.6041 0.1445 397.9 3.8680 0.0051 0.2966 0.9115 96.40 0.6149 NNM 0.6049 0.1423 398.0 3.8721 0.0043 0.2972 0.9125 96.60 0.5153 RM 0.6029 0.1480 397.9 3.8534 0.0047 0.2945 0.9100 97.10 0.6155 MM 0.6044 0.1422 395.0 3.8520 0.0042 0.2954 0.9134 96.90 0.6180

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

264

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=.40 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0074 0.1410 398.0 0.0613 0.5049 -0.2861 0.2690 95.00 0.5551 EM 0.0085 0.2258 397.7 0.0717 0.3488 -0.2850 0.2651 78.10 0.5501 PSM 0.0040 0.1428 397.9 0.0286 0.4989 -0.2820 0.2740 95.10 0.5560 NNM 0.0005 0.1491 398.0 0.0066 0.4862 -0.2790 0.2769 93.70 0.5569 RM 0.0044 0.1515 397.8 0.0565 0.4703 -0.2852 0.2695 92.80 0.5547 MM 0.0006 0.1479 395.0 0.0030 0.4903 -0.2795 0.2885 94.00 0.5580

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.10 Collinearity=.40 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0982 0.1457 398.0 0.6804 0.4376 -0.1859 0.3823 95.50 0.5682 EM 0.0979 0.2357 397.7 0.6892 0.2618 -0.1647 0.3943 77.40 0.5629 PSM 0.0971 0.1490 397.9 0.6695 0.4323 -0.1878 0.3820 94.60 0.5699 NNM 0.1066 0.1482 398.0 0.7362 0.4084 -0.1782 0.3913 94.40 0.5695 RM 0.1047 0.1479 397.9 0.7221 0.4185 -0.1799 0.3894 95.20 0.5694 MM 0.1059 0.1483 395.1 0.7287 0.3918 -0.1800 0.3917 94.40 0.5717

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.4Collinearity=.40 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4412 0.1541 398.0 2.8268 0.0440 0.1340 0.7485 95.70 0.6145 EM 0.4438 0.2303 397.9 2.8752 0.0849 0.1399 0.7478 81.50 0.6079 PSM 0.4416 0.1513 397.9 2.8274 0.0412 0.1344 0.7488 95.60 0.6144 NNM 0.4445 0.1515 398.0 2.8486 0.0399 0.1374 0.7514 95.70 0.6140 RM 0.4448 0.1533 397.9 2.8519 0.0409 0.1379 0.7517 96.70 0.6138 MM 0.4440 0.1513 395.0 2.8337 0.0408 0.1357 0.7524 96.00 0.6167

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

265

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=.40 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1010 0.1457 398.0 0.7194 0.4137 -0.1764 0.3784 93.30 0.5548 EM 0.1084 0.2268 397.7 0.7829 0.3261 -0.1660 0.3829 77.60 0.5489 PSM 0.1003 0.1492 397.9 0.7125 0.4093 -0.1774 0.3780 92.60 0.5554 NNM 0.0990 0.1539 398.0 0.7010 0.4109 -0.1791 0.3770 93.00 0.5561 RM 0.1001 0.1488 397.9 0.7106 0.4193 -0.1775 0.3778 94.10 0.5553 MM 0.0989 0.1537 395.0 06980 0.4132 -0.1802 0.3781 93.10 0.5583

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.10 Collinearity=.40 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1978 0.1427 398.0 1.3672 0.2771 -0.0871 0.4827 95.70 0.5698 EM 0.2035 0.2245 397.8 1.4138 0.2457 -0.0783 0.4833 78.60 0.5617 PSM 0.2039 0.1498 397.9 1.4091 0.2712 -0.0807 0.4885 94.50 0.5692 NNM 0.2034 0.1537 398.0 1.4036 0.2706 -0.0817 0.4885 92.80 0.5702 RM 0.1998 0.1442 397.9 1.3803 0.2809 -0.0848 0.4843 95.10 0.5691 MM 0.2035 0.1529 395.0 1.3991 0.2701 -0.0827 0.4898 92.40 0.5726

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=.40 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5455 0.1535 398.0 3.4919 0.0122 0.2380 0.8530 95.10 0.6150 EM 0.5426 0.2284 397.7 3.5033 0.0439 0.2373 0.8480 81.40 0.6107 PSM 0.5536 0.1478 397.9 3.5360 0.0089 0.2455 0.8616 96.50 0.6161 NNM 0.5539 0.1411 398.0 3.5449 0.0083 0.2463 0.8616 97.10 0.6153 RM 0.5514 0.1434 397.9 3.5113 0.0093 0.2434 0.8594 96.90 0.6160 MM 0.5539 0.1407 395.0 3.5289 0.0086 0.2449 0.8630 97.10 0.6181

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

266

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.00 Collinearity=.40 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1518 0.1398 398.0 1.0774 0.3433 -0.1259 0.4296 95.20 0.5555 EM 0.1539 0.2258 397.7 1.1072 0.2800 -0.1213 0.4291 77.90 0.5504 PSM 0.1520 0.1447 397.9 1.0758 0.3411 -0.1263 0.4304 94.30 0.5567 NNM 0.1489 0.1471 398.0 1.0554 0.3565 -0.1287 0.4265 93.20 0.5552 RM 0.1491 0.1409 397.9 1.0556 0.3596 -0.1286 0.4267 95.80 0.5553 MM 0.1486 0.1464 395.0 1.0497 0.3559 -0.1300 0.4272 93.90 0.5572

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=.40 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2551 0.1450 398.0 1.8311 0.1786 -0.0197 0.5499 94.30 0.5696 EM 0.2500 0.2331 397.7 1.7542 0.2190 -0.0311 0.5310 76.30 0.5621 PSM 0.2531 0.1445 397.9 1.7504 0.1906 -0.0315 0.5377 94.10 0.5693 NNM 0.2535 0.1466 398.0 1.7874 0.1895 -0.0261 0.5432 94.60 0.5693 RM 0.2501 0.1494 397.9 1.7987 0.1898 -0.0246 0.5448 93.70 0.5694 MM 0.2509 0.1460 395.0 1.7825 0.1881 -0.0270 0.5447 94.90 0.5717

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=.40 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.6064 0.1599 398.0 3.8861 0.0079 0.2991 0.9137 95.20 0.6147 EM 0.6036 0.2176 397.7 3.9129 0.0217 0.2994 0.9077 84.80 0.6083 PSM 0.6059 0.1404 397.9 3.8793 0.0039 0.2984 0.9134 97.70 0.6150 NNM 0.6077 0.1486 398.0 3.8938 0.0055 0.3005 0.9150 96.00 0.6145 RM 0.6067 0.1453 397.9 3.8867 0.0047 0.2995 0.9141 97.00 0.6146 MM 0.6078 0.1484 395.1 3.8763 0.0059 0.2991 0.9165 96.60 0.6174

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

267

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=.90 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0003 0.1469 398.0 0.0009 0.4942 -0.2843 0.1969 94.70 0.5692 EM 0.0037 0.3246 397.9 0.0156 0.2533 -0.3053 0.2579 60.20 0.5632 PSM 0.0070 0.1457 397.9 0.0498 0.4935 -0.2918 0.2778 95.20 0.5696 NNM 0.0014 0.1483 398.0 0.0098 0.4922 -0.2826 0.2855 95.00 0.5681 RM 0.0030 0.1450 397.8 0.0186 0.4951 -0.2816 0.2877 95.50 0.5693 MM 0.0015 0.1475 395.0 0.0102 0.4953 -0.2837 0.2868 95.10 0.5705

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.10 Collinearity=.90 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1019 0.1485 398.0 0.7055 0.4251 -0.1828 0.3865 95.00 0.5693 EM 0.1081 0.3372 397.9 0.7572 0.2470 -0.1721 0.3884 58.60 0.5605 PSM 0.1020 0.1483 397.9 0.7062 0.4234 -0.1824 0.3864 94.40 0.5688 NNM 0.1031 0.1488 398.0 0.7131 0.4258 -0.1815 0.3877 93.80 0.5692 RM 0.0976 0.1450 397.9 0.6691 0.4194 -0.1878 0.3806 95.00 0.5684 MM 0.1028 0.1479 395.0 0.7084 0.4275 -0.1829 0.3886 93.80 0.5715

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=.90 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4468 0.1581 398.0 2.8637 0.0431 0.1397 0.7539 94.50 0.6142 EM 0.4491 0.3271 397.9 2.9203 0.1293 0.1452 0.7530 64.00 0.6078 PSM 0.4426 0.1559 397.9 2.8387 0.0424 0.1357 0.7495 95.60 0.6138 NNM 0.4375 0.1543 398.0 2.8045 0.0458 0.1305 0.7444 95.30 0.6149 RM 0.4429 0.1525 397.9 2.8374 0.0380 0.1357 0.7501 94.90 0.6144 MM 0.4379 0.1532 395.0 2.7948 0.0452 0.1295 0.7463 95.50 0.6168

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

268

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=.90 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0989 0.1415 398.0 0.7012 0.4359 -0.1786 0.3764 94.10 0.5550 EM 0.1075 0.3322 397.9 0.7742 0.2490 -0.1664 0.3814 60.70 0.5478 PSM 0.0993 0.1351 397.9 0.7052 0.4312 -0.1782 0.3769 95.90 0.5551 NNM 0.0967 0.1408 398.0 0.6855 0.4248 -0.1808 0.3744 94.70 0.5552 RM 0.0973 0.1430 397.9 0.6890 0.4172 -0.1802 0.3748 93.60 0.5550 MM 0.0972 0.1407 395.0 0.6862 0.4231 -0.1815 0.3758 95.20 0.5573

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.10 Collinearity=.90 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2001 0.1448 398.0 1.3826 0.2730 -0.0846 0.4848 94.70 0.5694 EM 0.2038 0.3348 397.9 1.4280 0.2209 -0.0775 0.4851 59.30 0.5626 PSM 0.1970 0.1434 397.9 1.3590 0.2769 -0.0883 0.4821 95.40 0.5704 NNM 0.2018 0.1462 398.0 1.3930 0.2724 -0.0834 0.4871 94.30 0.5705 RM 0.2044 0.1463 397.9 1.4122 0.4870 -0.0806 0.4896 95.30 0.5702 MM 0.2013 0.1458 395.1 1.3841 0.2734 -0.0850 0.4877 94.40 0.5727

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=.90 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5508 0.1577 398.0 3.5333 0.0130 0.2444 0.8576 95.30 0.6136 EM 0.5503 0.3463 397.9 3.5830 0.0878 0.2471 0.8534 62.70 0.6063 PSM 0.5468 0.1572 397.9 3.5052 0.0139 0.2397 0.8538 95.10 0.6141 NNM 0.5538 0.1549 398.0 3.5562 0.0115 0.2472 0.8604 95.50 0.6132 RM 0.5543 0.1570 397.9 3.5579 0.0119 0.2478 0.8609 94.80 0.6131 MM 0.5539 0.1544 395.0 3.5401 0.0117 0.2459 0.8619 96.00 0.6160

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

269

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.00 Collinearity=.90 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1534 0.1415 398.0 1.0859 0.3494 -0.1243 0.4311 94.60 0.5554 EM 0.1640 0.3240 397.9 1.1817 0.2226 -0.1097 0.4378 60.10 0.5475 PSM 0.1531 0.1462 397.9 1.0866 0.3399 -0.1245 0.4309 94.70 0.5553 NNM 0.1575 0.1463 398.0 1.1154 0.3382 -0.1204 0.4354 94.10 0.5559 RM 0.1571 0.1452 397.9 1.1099 0.3353 -0.1211 0.4353 94.70 0.5564 MM 0.1569 0.1458 395.0 1.1065 0.3409 -0.1222 0.4359 94.30 0.5581

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=.90 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2522 0.1451 398.0 1.7451 0.1955 -0.0323 0.5368 95.30 0.5691 EM 0.2524 0.3301 397.9 1.7797 0.2004 -0.0288 0.5337 61.00 0.5625 PSM 0.2533 0.1483 397.9 1.7491 0.1989 -0.0318 0.5385 95.00 0.5703 NNM 0.2503 0.1444 398.0 1.7322 0.1939 -0.0344 0.5350 95.20 0.5694 RM 0.2502 0.1477 397.9 1.7284 0.2003 -0.0345 0.5448 94.90 0.5693 MM 0.2500 0.1440 395.0 1.7232 0.1955 -0.0358 0.5359 94.80 0.5717

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=.90 and Correlation=0 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5995 0.1545 398.0 3.8329 0.0082 0.2916 0.9075 95.60 0.6159 EM 0.5977 0.3395 397.9 3.8762 0.0684 0.2934 0.9019 63.70 0.6085 PSM 0.5997 0.1547 397.9 3.8350 0.0079 0.2919 0.9075 94.90 0.6156 NNM 0.5987 0.1588 398.0 3.8294 0.0069 0.2909 0.9164 95.20 0.6155 RM 0.5989 0.1565 397.9 3.8367 0.0067 0.2915 0.9063 95.20 0.6148 MM 0.5981 0.1584 395.1 3.8089 0.0072 0.2890 0.9073 95.40 0.6183

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

270

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=.90 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0024 0.1442 398.0 0.0174 0.4895 -0.2805 0.2757 94.70 0.5563 EM 0.0071 0.2983 397.9 0.0488 0.2805 -0.2679 0.2821 65.90 0.5501 PSM 0.0006 0.1455 397.9 0.0045 0.4843 -0.2774 0.2786 94.20 0.5560 NNM 0.0036 0.1486 398.0 0.0260 0.4838 -0.2747 0.2819 93.90 0.5566 RM 0.0038 0.1458 397.8 0.0280 0.4866 -0.2741 0.2818 94.90 0.5559 MM 0.0039 0.1483 395.1 0.0241 0.4849 -0.2760 0.2827 93.70 0.5587

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Table Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.10 Collinearity=.90 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1010 0.1420 398.0 0.6962 0.4352 -0.1834 0.3855 95.80 0.5689 EM 0.1148 0.3006 397.9 0.8130 0.2618 -0.1647 0.3943 63.80 0.5590 PSM 0.1014 0.1466 397.9 0.7016 0.4236 -0.1829 0.3857 94.70 0.5686 NNM 0.1026 0.1485 398.0 0.7110 0.4207 -0.1818 0.387 94.10 0.5688 RM 0.1017 0.1457 397.9 0.7020 0.4247 -0.1826 0.3861 95.00 0.5687 MM 0.1020 0.1480 395.0 0.7043 0.4223 -0.1834 0.3876 94.40 0.5711

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=.90 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4376 0.1573 398.0 2.8073 0.0483 0.1308 0.7444 94.70 0.6136 EM 0.4413 0.2993 397.9 2.8731 0.1150 0.1375 0.7450 68.20 0.6075 PSM 0.4415 0.1514 397.9 2.8302 0.0411 0.1343 0.7487 96.40 0.6143 NNM 0.4388 0.1557 398.0 2.8132 0.0452 0.1317 0.7459 95.60 0.6142 RM 0.4371 0.1616 397.9 2.8024 0.0467 0.1299 0.7443 95.90 0.6144 MM 0.4380 0.1546 395.1 2.7961 0.0451 0.1296 0.7517 96.10 0.6170

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

271

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=.90 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1091 0.1389 398.0 0.7706 0.4097 -0.1692 0.3874 95.30 0.5556 EM 0.1178 0.2953 397.9 0.8447 0.2661 -0.1567 0.3923 65.00 0.5490 PSM 0.1028 0.1428 397.9 0.7282 0.4147 -0.1749 0.3805 95.00 0.5554 NNM 0.1038 0.1414 398.0 0.7346 0.4276 -0.1741 0.3817 95.10 0.5558 RM 0.1014 0.1439 397.9 0.7188 0.4148 -0.1764 0.3792 94.20 0.5557 MM 0.1034 0.1406 395.0 0.7292 0.4303 -0.1756 0.3824 95.30 0.5580

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.10 Collinearity=.90 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2061 0.1439 398.0 1.4263 0.2639 -0.0783 0.4905 94.70 0.5688 EM 0.2111 0.2945 397.9 1.4802 0.2387 -0.0694 0.4917 65.30 0.5611 PSM 0.1991 0.1493 397.9 1.3766 0.2759 -0.0854 0.4836 94.20 0.5691 NNM 0.2032 0.1502 398.0 1.4090 0.2666 -0.0809 0.4873 94.10 0.5683 RM 0.2047 0.1479 397.9 1.4192 0.2612 -0.0796 0.4891 94.70 0.5687 MM 0.2036 0.1500 395.0 1.4057 0.2655 -0.0818 0.4889 94.00 0.5707

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=.90 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5364 0.1534 398.0 3.4350 0.0161 0.2290 0.8438 94.20 0.6148 EM 0.5566 0.2907 397.9 3.6200 0.0802 0.2530 0.8603 66.10 0.6072 PSM 0.5508 0.1448 397.9 3.5273 0.0124 0.2435 0.8582 95.30 0.6147 NNM 0.5485 0.1437 398.0 3.5082 0.0133 0.2407 0.8563 95.50 0.6155 RM 0.5488 0.1504 397.9 3.5113 0.0142 0.2412 0.8564 95.60 0.6152 MM 0.5486 0.1433 395.0 3.4930 0.0137 0.2394 0.8577 95.50 0.6183

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

272

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=0 Collinearity=.90 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1539 0.1459 398.0 1.0904 0.3473 -0.1240 0.4319 93.80 0.5559 EM 0.1565 0.2973 397.9 1.1301 0.2541 -0.1172 0.4303 65.30 0.5475 PSM 0.1481 0.1440 397.9 1.0429 0.3604 -0.1309 0.4253 94.80 0.5562 NNM 0.1496 0.1406 398.0 1.0587 0.3415 -0.1284 0.4276 94.80 0.5560 RM 0.1480 0.1426 397.9 1.0431 0.3460 -0.1305 0.4247 94.70 0.5552 MM 0.1491 0.1401 395.1 1.7312 0.3445 -0.1299 0.4282 94.90 0.5581

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=.90 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2539 0.1456 398.0 1.7539 0.1952 -0.0312 0.5391 95.80 0.5703 EM 0.2541 0.3090 397.9 1.7803 0.2062 -0.0273 0.5355 65.20 0.5628 PSM 0.2542 0.1446 397.9 1.7520 0.1950 -0.0314 0.5398 95.20 0.5712 NNM 0.2513 0.1477 398.0 1.7373 0.2011 -0.0335 0.5362 94.30 0.5697 RM 0.2577 0.1438 397.9 1.7790 0.1845 -0.0274 0.5430 95.10 0.5704 MM 0.2515 0.1472 395.0 1.7312 0.2016 -0.0345 0.5375 94.30 0.5720

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=.90 and Correlation=.75 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5997 0.1548 398.0 3.8342 0.0062 0.2918 0.9076 95.40 0.6158 EM 0.6023 0.3056 397.9 3.9352 0.0511 0.3027 0.9137 69.80 0.6110 PSM 0.6011 0.1462 397.9 3.8413 0.0056 0.2931 0.9091 96.40 0.6160 NNM 0.6015 0.1522 398.0 3.8423 0.0065 0.2994 0.9156 95.60 0.6164 RM 0.6075 0.1471 397.9 3.8378 0.0045 0.2994 0.9156 96.60 0.6162 MM 0.6016 0.1523 395.1 3.8246 0.0067 0.2919 0.9112 95.40 0.6193

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

273

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=.90 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0020 0.1459 398.0 0.0144 0.4975 -0.2759 0.2799 94.60 0.5558 EM 0.0063 0.3039 397.9 0.0395 0.2671 -0.2793 0.2686 63.20 0.5479 PSM 0.0068 0.1538 397.9 0.0068 0.4719 -0.2791 0.2769 93.50 0.5560 NNM 0.0014 0.1490 398.0 0.0627 0.4790 -0.2691 0.2866 94.80 0.5557 RM 0.0005 0.1540 397.8 0.0073 0.4830 -0.2766 0.2794 92.20 0.5559 MM 0.0071 0.1482 395.0 0.0570 0.4823 -0.2709 0.2869 95.20 0.5579

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.10 Collinearity=.90 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1040 0.1444 398.0 0.7377 0.4290 -0.1736 0.3819 95.60 0.5555 EM 0.1023 0.2324 397.7 0.7034 0.3327 -0.1765 0.3718 78.40 0.5483 PSM 0.1055 0.1457 397.9 0.7733 0.4057 -0.1686 0.3867 94.40 0.5553 NNM 0.1022 0.1555 398.0 0.7778 0.4081 -0.1681 0.3875 92.70 0.5556 RM 0.1008 0.1538 397.9 0.7209 0.4080 -0.1764 0.3794 93.70 0.5558 MM 0.1017 0.1549 395.0 0.7716 0.4105 -0.1698 0.3880 92.60 0.5578

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.44 Collinearity=.90 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.4455 0.1503 398.0 2.8581 0.0409 0.1386 0.7525 94.40 0.6139 EM 0.4550 0.2953 397.9 2.9535 0.1067 0.1510 0.7590 70.70 0.6079 PSM 0.4471 0.1553 397.9 2.8671 0.0419 0.1401 0.7541 93.30 0.6140 NNM 0.4486 0.1531 398.0 2.8841 0.0381 0.1423 0.7551 95.10 0.6127 RM 0.4455 0.1458 397.9 2.8577 0.0372 0.1385 0.2724 96.20 0.6139 MM 0.4487 0.1529 395.0 2.8709 0.0387 0.1409 0.7565 95.70 0.6156

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

274

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=0 Collinearity=.90 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1128 0.1420 398.0 0.8012 0.4099 -0.1651 0.3906 95.00 0.5557 EM 0.1123 0.2942 397.9 0.8146 0.2720 -0.1612 0.3858 65.50 0.5471 PSM 0.1033 0.1445 397.9 0.7318 0.4239 -0.1746 0.3813 94.50 0.5559 NNM 0.1067 0.1486 398.0 0.7583 0.4121 -0.1708 0.3842 94.30 0.5551 RM 0.1071 0.1429 397.9 0.7606 0.4181 -0.1706 0.3849 95.00 0.5555 MM 0.1075 0.1476 395.0 0.7608 0.4146 -0.1711 0.3862 94.10 0.5573

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.10 Collinearity=.90 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2036 0.1351 398.0 1.4090 0.2699 -0.0809 0.4883 95.90 0.5693 EM 0.1953 0.2960 397.9 1.3639 0.2439 -0.0868 0.4755 67.20 0.5622 PSM 0.1998 0.1419 397.9 1.3805 0.2696 -0.0851 0.4848 95.00 0.5700 NNM 0.2084 0.1426 398.0 1.4444 0.2570 -0.0758 0.4945 95.40 0.5703 RM 0.2040 0.1470 397.9 1.4095 0.2693 -0.0808 0.4889 94.90 0.5698 MM 0.2083 0.1421 395.1 1.4386 0.2574 -0.0770 0.4956 95.20 0.5726

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.45 Collinearity=.90 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5505 0.1492 398.0 3.5219 0.0123 0.2428 0.8581 95.80 0.6153 EM 0.5478 0.2934 397.9 3.5422 0.0724 0.2416 0.8498 70.20 0.6083 PSM 0.5513 0.1441 397.9 3.5308 0.0107 0.2439 0.8587 96.50 0.6148 NNM 0.5482 0.1521 398.0 3.5123 0.0131 0.2408 0.8555 95.90 0.6146 RM 0.5512 0.1535 397.9 3.5341 0.0112 0.2442 0.8583 95.20 0.6141 MM 0.5472 0.1495 398.0 3.4587 0.0150 0.2327 0.8477 96.00 0.6150

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

275

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=0 Collinearity=.90 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1498 0.1443 398.0 1.0600 0.3532 -0.1284 0.4280 94.90 0.5564 EM 0.1578 0.2974 397.9 1.1385 0.2485 -0.1161 0.4317 63.40 0.5478 PSM 0.1556 0.1471 397.9 1.1021 0.3410 -0.1222 0.4335 94.10 0.5557 NNM 0.1602 0.1492 398.0 1.1344 0.3310 -0.1175 0.4380 93.20 0.5555 RM 0.1581 0.1498 397.9 1.1193 0.3338 -0.1201 0.4364 92.80 0.5564 MM 0.1603 0.1486 395.1 1.1302 0.3319 -0.1186 0.4391 93.90 0.5577

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=.90 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2452 0.1507 398.0 1.6930 0.2094 -0.0398 0.5302 93.90 0.5700 EM 0.2496 0.2953 397.9 1.7637 0.2181 -0.0309 0.5300 67.90 0.5609 PSM 0.2458 0.1503 397.9 1.6955 0.2140 -0.0396 0.5311 94.00 0.5708 NNM 0.2439 0.1479 398.0 1.7191 0.2040 -0.0361 0.5348 95.10 0.5709 RM 0.2494 0.1489 397.9 1.7219 0.2147 -0.0358 0.5346 95.20 0.5704 MM 0.2495 0.1466 395.0 1.7126 0.2034 -0.0372 0.5361 95.10 0.5734

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.45 Collinearity=.90 and Correlation=.90 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5903 0.1607 398.0 3.7924 0.0081 0.2851 0.9009 94.00 0.6158 EM 0.5957 0.3062 397.9 3.8570 0.0531 0.2911 0.9003 70.50 0.6092 PSM 0.6067 0.1549 397.9 3.8734 0.0063 0.2984 0.9150 95.70 0.6166 NNM 0.6019 0.1514 398.0 3.8446 0.0060 0.2937 0.9101 95.50 0.6164 RM 0.5965 0.1528 397.9 3.8121 0.0061 0.2885 0.9045 95.80 0.6160 MM 0.6022 0.1512 395.0 3.8292 0.0065 0.2926 0.9118 95.80 0.6192

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

276

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=0 Collinearity=.90 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0020 0.1459 398.0 0.0144 0.4975 -0.2759 0.2799 94.60 0.5558 EM 0.0063 0.3039 397.9 0.0395 0.2671 -0.2793 0.2686 63.20 0.5479 PSM 0.0068 0.1538 397.9 0.0068 0.4719 -0.2791 0.2769 93.50 0.5560 NNM 0.0014 0.1490 398.0 0.0627 0.4790 -0.2691 0.2866 94.80 0.5557 RM 0.0005 0.1540 397.8 0.0073 0.4830 -0.2766 0.2794 92.20 0.5559 MM 0.0071 0.1482 395.0 0.0570 0.4823 -0.2709 0.2869 95.20 0.5579

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, ES=.10, Collinearity=.90 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.0972 0.1423 398.0 0.6692 0.4371 -0.1884 0.3824 96.40 0.5709 EM 0.0970 0.2840 397.9 0.6795 0.2632 -0.1838 0.3777 67.20 0.5615 PSM 0.0975 0.1505 397.9 0.6719 0.4075 -0.1876 0.3825 94.30 0.5701 NNM 0.1012 0.1469 398.0 0.6970 0.4214 -0.1841 0.3865 94.80 0.5706 RM 0.0967 0.1485 397.9 0.6649 0.4325 -0.1887 0.3821 95.30 0.5709 MM 0.1016 0.1456 395.0 0.6969 0.4253 -0.1848 0.3881 95.30 0.5730

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=0, & ES=.44, Collinearity=.90 and Correlation=.99 1000 Replications Average % Method YD* STD df t-value p-value LCLM UCLM Coverage Width NOM 0.4491 0.1573 398.0 2.8764 0.0428 0.1417 0.7566 95.40 0.6148 EM 0.4484 0.2993 397.9 2.9178 0.1173 0.1450 0.7518 68.40 0.6068 PSM 0.4481 0.1514 397.9 2.8703 0.0404 0.1409 0.7552 95.30 0.6143 NNM 0.4420 0.1557 398.0 2.8287 0.0433 0.1345 0.7496 95.10 0.6151 RM 0.4492 0.1616 397.9 2.8764 0.0431 0.1417 0.7568 94.20 0.6150 MM 0.4427 0.1546 395.0 2.8205 0.0427 0.1338 0.7517 95.70 0.6179

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

277

Case 3: t-tests Statistics for 1000 Replications by Matching Method when Bias=.10, ES=0, Collinearity=.90 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1009 0.1460 398.0 0.7239 0.4248 -0.1757 0.3799 94.90 0.5556 EM 0.0984 0.2855 397.9 0.7137 0.2766 -0.1747 0.3715 67.10 0.5462 PSM 0.1006 0.1479 397.9 0.7222 0.4237 -0.1758 0.3794 94.80 0.5553 NNM 0.1020 0.1466 398.0 0.7337 0.4222 -0.1743 0.3806 94.30 0.5549 RM 0.1017 0.1471 397.9 0.7320 0.4148 -0.1745 0.3804 94.00 0.5550 MM 0.1023 0.1477 395.1 0.7322 0.4225 -0.1751 0.3820 94.30 0.5570

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.10, ES=.10, Collinearity=.90 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2017 0.1459 398.0 1.3920 0.2719 -0.0833 0.4867 95.20 0.5700 EM 0.2042 0.3002 397.9 1.4295 0.2424 -0.0775 0.4858 65.60 0.5633 PSM 0.2024 0.1478 397.9 1.3979 0.2689 -0.0825 0.4873 94.90 0.5699 NNM 0.2012 0.1465 398.0 1.3882 0.2711 -0.0836 0.4861 94.80 0.5698 RM 0.2036 0.1437 397.9 1.4066 0.2675 -0.0815 0.4888 95.50 0.5703 MM 0.2011 0.1457 395.0 1.3813 0.2742 -0.0850 0.4872 94.90 0.5722

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for 1000 Replications by Matching Method when Bias=.10, ES=.45, Collinearity=.90 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.5460 0.1534 398.0 3.4902 0.0125 0.2380 0.7540 95.20 0.6158 EM 0.5553 0.2907 397.9 3.6144 0.0681 0.2521 0.8586 70.80 0.6066 PSM 0.5511 0.1448 397.9 3.5244 0.0105 0.2434 0.8588 96.50 0.6155 NNM 0.5448 0.1437 398.0 3.4825 0.0097 0.2370 0.8525 97.00 0.6154 RM 0.5482 0.1504 397.9 3.5100 0.0109 0.2409 0.8555 95.60 0.6157 MM 0.5454 0.1433 395.0 3.4711 0.0096 0.2363 0.8547 96.90 0.6182

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

278

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=0, Collinearity=.90 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.1546 0.1426 398.0 1.0970 0.3577 -0.1232 0.4324 95.20 0.5555 EM 0.1550 0.2901 397.9 1.1171 0.2641 -0.1189 0.4289 66.70 0.5479 PSM 0.1526 0.1407 397.9 1.0798 0.3430 -0.1254 0.4307 95.00 0.5561 NNM 0.1547 0.1471 398.0 1.0957 0.3356 -0.1231 0.4325 94.10 0.5556 RM 0.1556 0.1462 397.9 1.1047 0.3510 -0.1218 0.4331 94.00 0.5549 MM 0.1551 0.1463 395.0 1.0939 0.3355 -0.1238 0.4340 94.40 0.5578

Y = Y LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D D

Case 3: t-tests Statistics for the average of 1000 Replications by Matching Method when Bias=.15, ES=.10 Collinearity=.90 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.2492 0.1467 398.0 1.7222 0.2049 -0.0357 0.5342 95.90 0.5699 EM 0.2514 0.2883 397.9 1.7768 0.1981 -0.0294 0.5321 66.00 0.5615 PSM 0.2454 0.1429 397.9 1.6965 0.1971 -0.0395 0.5304 95.30 0.5699 NNM 0.2549 0.1495 398.0 1.7587 0.1931 -0.0305 0.5404 94.60 0.5710 RM 0.2444 0.1470 397.9 1.6880 0.2037 -0.0405 0.5294 94.70 0.5700 MM 0.2548 0.1487 395.0 1.7505 0.1923 -0.0319 0.5415 94.20 0.5734

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

Case 3: t-tests Statistics for 1000 Replications by Matching Method when Bias=.15, ES=.45, Collinearity=.90 and Correlation=.99 1000 Replications Average % Method YD* SD df t-value p-value LCLM UCLM Coverage Width NOM 0.6047 0.1467 398.0 3.8641 0.0049 0.2969 0.9125 94.80 0.6156 EM 0.6023 0.2883 397.9 3.9167 0.0553 0.2988 0.9058 69.70 0.6071 PSM 0.6019 0.1429 397.9 3.8523 0.0044 0.2945 0.9093 96.80 0.6148 NNM 0.6037 0.1495 398.0 3.8635 0.0047 0.2963 0.9112 95.70 0.6150 RM 0.5995 0.1470 397.9 3.8378 0.0045 0.2921 0.9070 97.40 0.6149 MM 0.6033 0.1487 395.1 3.8437 0.0057 0.2944 0.9122 95.90 0.6178

Y = LCLM=Lower CL Mean UCLM=Upper CL Mean Coverage = 95% CI probability of coverage * D

279