DOCSLIB.ORG

Understanding Meta-Analyses. a Consumer's Guide to Aims, Problems, Evaluation and Developments

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Understanding Meta-Analyses. a Consumer's Guide to Aims, Problems, Evaluation and Developments Plath, Ingrid Understanding meta-analyses. A consumer's guide to aims, problems, evaluation and developments Baden-Baden : Nomos 1992, 136 S. - (Studien zum Umgang mit Wissen; 7) Quellenangabe/ Citation: Plath, Ingrid: Understanding meta-analyses. A consumer's guide to aims, problems, evaluation and developments. Baden-Baden : Nomos 1992, 136 S. - (Studien zum Umgang mit Wissen; 7) - URN: urn:nbn:de:0111-opus-7367 - DOI: 10.25656/01:736 http://nbn-resolving.org/urn:nbn:de:0111-opus-7367 http://dx.doi.org/10.25656/01:736 Nutzungsbedingungen Terms of use Gewährt wird ein nicht exklusives, nicht übertragbares, persönliches und We grant a non-exclusive, non-transferable, individual and limited right to beschränktes Recht auf Nutzung dieses Dokuments. Dieses Dokument ist using this document. ausschließlich für den persönlichen, nicht-kommerziellen Gebrauch This document is solely intended for your personal, non-commercial use. Use bestimmt. Die Nutzung stellt keine Übertragung des Eigentumsrechts an of this document does not include any transfer of property rights and it is diesem Dokument dar und gilt vorbehaltlich der folgenden Einschränkungen: conditional to the following limitations: All of the copies of this documents must Auf sämtlichen Kopien dieses Dokuments müssen alle retain all copyright information and other information regarding legal Urheberrechtshinweise und sonstigen Hinweise auf gesetzlichen Schutz protection. You are not allowed to alter this document in any way, to copy it for beibehalten werden. Sie dürfen dieses Dokument nicht in irgendeiner Weise public or commercial purposes, to exhibit the document in public, to perform, abändern, noch dürfen Sie dieses Dokument für öffentliche oder distribute or otherwise use the document in public. kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, aufführen, vertreiben oder anderweitig nutzen. Mit der Verwendung dieses Dokuments erkennen Sie die By using this particular document, you accept the above-stated conditions of Nutzungsbedingungen an. use. Kontakt / Contact: peDOCS DIPF | Leibniz-Institut für Bildungsforschung und Bildungsinformation Informationszentrum (IZ) Bildung E-Mail: [email protected] Internet: www.pedocs.de Contents Preface 7 1 Introdnction 8 1.1 Traditional reviewing 10 2 What is meta-analysis? 11 2.1 Aims and functions of meta-analysis 12 2.2 The value of the meta-analytic approach 14 3 Criticized aspects of meta-analysis 16 3.1 Objectivity 16 3.2 Sampling bias 17 3.2.1 Publication bias 17 3.2.2 Selection bias 19 3.2.3 The quality of primary studies 20 3.3 Quantitative and statistical aspects 22 3.3.1 Mixing apples and oranges 22 3.3.2 Effect sizes: non-independence and other aspects 24 3.3.3 Critique of techniques and focus of the statistical analysis 25 4 Issnes of reliability and valldity 29 4.1 Reliability 29 4.2 Validity 31 4.2.1 Internal validity 32 4.2.2 Statistical conclusion validity 33 4.2.3 Construct validity 34 4.2.4 External validity 34 5 Guidelines for evalnating meta-analyses 37 5.1 Recommendations on how to evaluate a meta-analysis 40 5.2 Construct and extern al validity 41 5.2.1 Problem formulation and hypothesis selection 41 5.2.2 Sampling and selection of studies 42 5.2.3 Presenting the characteristics of the primary studies 43 5.2.4 Interpreting results 44 5 5.3 Internal and statistical conclusion validity 45 5.3.1 Coding 45 5.3.2 Statistical analysis 46 5.4 Ofwhat use is the evaluation? 48 6 Evaluation of a sampIe of meta-analyses 49 6.1 Method 49 6.1.1 SampIe description 49 6.1.2 Coding procedure 50 6.1.3 Data evaluation 51 6.2 Results 51 6.2.1 Theoretical framework 55 6.2.2 Sampling 56 6.2.3 Coded study characteristics 58 6.2.4 Data analysis 60 6.2.5 Interpretation 64 6.3 Do critiques and replication attempts affect evaluations? 66 6.4 Is the quality of meta-analyses improving? 67 6.4.1 Problematic issues: sampIe size and representativeness 67 6.4.2 Problematic issues: co ding procedure 69 6.4.3 Possible trends in the reporting quality of meta-analyses 70 6.5 Resurne 75 7 Improving tbe quality and utility of meta-analyses 77 7.1 Combining qualitative and quantitative reviewing approaches 78 7.2 Taking communication quality into account 79 7.3 Knowledge synthesis and practical relevance 80 7.4 Reviews of reviews -- meta-synthesis? 85 7.5 Concluding remarks 87 Glossary 91 References 98 Appendix 109 1 Introduction Practitioners have several paths open to them when wanting to find practical, scientifi­ cally based solutions to problems encountered during the course of their work or wishing to gain insight into the present state of research in a particular field of interest. They can ask experts for advice. They can search databases for relevant empirical studies. They can consult published reviews or books. All these approaches have their pros and cons, especially concerning their feasibility, representativeness, objectivity, reliability and validity. Traditionalliterature reviews have long served as a relatively convenient source of in­ formation. Since 1976 a specific form of review, called among other things quantitative synthesis, quantitative research integration or meta-analysis, has enjoyed increasing popularity with scientific reviewers. KULIK (1984) estimates that after eight years about a 1000 papers have been published on the topic, about a third of which are actual reviews. Since then the number has been steadily increasing. Propagated as an objective, scientific way of research integration, newcomers optimis­ tically approach this type of review in the hope of finding weH founded answers to their questions. Accustomed to traditional reviews the novice is likely to be slightly over­ whelmed. Confronted with masses of quantitative data, numerous statistical analyses, controversial discussions of statistical issues involved and relatively limited substantive information, the review will possibly cause more confusion than transmit actual informa­ tion. Newcomers can react to this state of affairs in several ways. They can stick to tradi­ tional reviews, despite the weaknesses they might have, prefering the more easily intel­ ligible narrative report. They can stick to meta-analyses, skipping the complicated parts to read the conclusions, trusting the expert's evaluations without being able to recon­ struct how these were reached. They can also decide to delve into the methodology of quantitative syntheses, becoming semi-experts themse1ves and thus able to grasp the finer points of the review and its conclusions. The last alternative is clearly what meta-analysts expect of potential consumers. To quote BANGERT-DROWNS (1986, p.388): "Readers, researchers, editors and re­ search reviewers need to be better informed if they are to be intelligent consumers and critics of meta-analytic reviews." This sounds simpler than it iso The relevant methodolo­ gicalliterature is usually directed at readers interested in conducting their own meta-ana­ lyses. Information specifically useful to consumers of reviews is gleaned mainly as by-product from the numerous method,ological papers spread over diverse journals and a few introductory books. 8 Instead of becoming simpler the subject tends to become more complicated and com­ plex as one progresses more deeply into it. One is confronted with methodological argu­ ments, controversies, criticisms, praises and doubts. What was hailed as the approach to clarify and systematically analyze the diversity of empirical research findings in particu­ lar domains, is now ironically adding to the confusion. Under these circumstances the question arises whether a quantitative review still has relevance for a practitioner. Does one really, as WALBERG (1984) suggests, have to gain insight into the explicit, quantitative, empirical techniques used in the current main­ stream of theory and research if one wishes to use or understand it? Can one expect prac­ titioners additionally to take this task upon themselves or is it the expert's responsibility to make scientific knowledge available to users in an acceptable form? Why should the consumer undertake the arduous task of studying the technical details of the approach if the simpler alternative of tradition al reviews exists? No guidelines were ever required to read these and having done so one usually had the impression of having gained some in­ sight into the theory and findings of the reviewed domain. This is not always the case with meta-analyses. Rather , one feels that to understand and profit from these one should already have been weIl versed in the methodology and the theory of the subject covered by the integration. What previous knowledge does the consumer really need to be able to work effec­ tively with meta-analytic reviews and to critically evaluate their quality? It is contended, that highly detailed knowledge of the technical and statistical issues is not absolutely necessary for this purpose, rather a clear conception of the aims or intentions of this type of reviewing and problems encountered, limiting or preventing their attainment. This knowledge is not only useful for working with quantitative syntheses but also encour­ ages a more sensitive handling oftraditional reviews. Having in part experienced the quandary described above and deciding to resolve it by working through the relevant sources to penetrate the details of the approach, the aims of the present report are: to focus on the needs of the potential consumer, to sum­ marize scattered methodological information considered essential, to present recom­ mendations for the critical evaluation of the approach, to discuss its practical relevance and to describe the impressions gained studying actual meta-analyses on specific educa­ tional topics. Hopefully, this will help reduce the apparently widespread lack of familiar­ ity with the approach which appears to be a matter of general concern (WANOUS, SULLIVAN & MALINAK, 1989). Since consumers rather than potential conductors of meta-analyses are the primary focus, technical and statistical details have intention­ ally been kept at a minimum, yet the use of and reference to technical concepts and procedures is unavoidable.
Load more

Recommended publications
  • Internal and External Validity: Can You Apply Research Study Results to Your Patients? Cecilia Maria Patino1,2,A, Juliana Carvalho Ferreira1,3,B
    J Bras Pneumol. 2018;44(3):183-183 CONTINUING EDUCATION: http://dx.doi.org/10.1590/S1806-37562018000000164 SCIENTIFIC METHODOLOGY Internal and external validity: can you apply research study results to your patients? Cecilia Maria Patino1,2,a, Juliana Carvalho Ferreira1,3,b CLINICAL SCENARIO extent to which the results of a study are generalizable to patients in our daily practice, especially for the population In a multicenter study in France, investigators conducted that the sample is thought to represent. a randomized controlled trial to test the effect of prone vs. supine positioning ventilation on mortality among patients Lack of internal validity implies that the results of the with early, severe ARDS. They showed that prolonged study deviate from the truth, and, therefore, we cannot prone-positioning ventilation decreased 28-day mortality draw any conclusions; hence, if the results of a trial are [hazard ratio (HR) = 0.39; 95% CI: 0.25-0.63].(1) not internally valid, external validity is irrelevant.(2) Lack of external validity implies that the results of the trial may not apply to patients who differ from the study STUDY VALIDITY population and, consequently, could lead to low adoption The validity of a research study refers to how well the of the treatment tested in the trial by other clinicians. results among the study participants represent true fi ndings among similar individuals outside the study. INCREASING VALIDITY OF RESEARCH This concept of validity applies to all types of clinical STUDIES studies, including those about prevalence, associations, interventions, and diagnosis. The validity of a research To increase internal validity, investigators should ensure study includes two domains: internal and external validity.
    [Show full text]
  • As an "Odds Ratio", the Ratio of the Odds of Being Exposed by a Case Divided by the Odds of Being Exposed by a Control
    STUDY DESIGNS IN BIOMEDICAL RESEARCH VALIDITY & SAMPLE SIZE Validity is an important concept; it involves the assessment against accepted absolute standards, or in a milder form, to see if the evaluation appears to cover its intended target or targets. INFERENCES & VALIDITIES Two major levels of inferences are involved in interpreting a study, a clinical trial The first level concerns Internal validity; the degree to which the investigator draws the correct conclusions about what actually happened in the study. The second level concerns External Validity (also referred to as generalizability or inference); the degree to which these conclusions could be appropriately applied to people and events outside the study. External Validity Internal Validity Truth in Truth in Findings in The Universe The Study The Study Research Question Study Plan Study Data A Simple Example: An experiment on the effect of Vitamin C on the prevention of colds could be simply conducted as follows. A number of n children (the sample size) are randomized; half were each give a 1,000-mg tablet of Vitamin C daily during the test period and form the “experimental group”. The remaining half , who made up the “control group” received “placebo” – an identical tablet containing no Vitamin C – also on a daily basis. At the end, the “Number of colds per child” could be chosen as the outcome/response variable, and the means of the two groups are compared. Assignment of the treatments (factor levels: Vitamin C or Placebo) to the experimental units (children) was performed using a process called “randomization”. The purpose of randomization was to “balance” the characteristics of the children in each of the treatment groups, so that the difference in the response variable, the number of cold episodes per child, can be rightly attributed to the effect of the predictor – the difference between Vitamin C and Placebo.
    [Show full text]
  • Introduction to Difference in Differences (DID) Analysis
    Introduction to Difference in Differences (DID) Analysis Hsueh-Sheng Wu CFDR Workshop Series June 15, 2020 1 Outline of Presentation • What is Difference-in-Differences (DID) analysis • Threats to internal and external validity • Compare and contrast three different research designs • Graphic presentation of the DID analysis • Link between regression and DID • Stata -diff- module • Sample Stata codes • Conclusions 2 What Is Difference-in-Differences Analysis • Difference-in-Differences (DID) analysis is a statistic technique that analyzes data from a nonequivalence control group design and makes a casual inference about an independent variable (e.g., an event, treatment, or policy) on an outcome variable • A non-equivalence control group design establishes the temporal order of the independent variable and the dependent variable, so it establishes which variable is the cause and which one is the effect • A non-equivalence control group design does not randomly assign respondents to the treatment or control group, so treatment and control groups may not be equivalent in their characteristics and reactions to the treatment • DID is commonly used to evaluate the outcome of policies or natural events (such as Covid-19) 3 Internal and External Validity • When designing an experiment, researchers need to consider how extraneous variables may threaten the internal validity and external validity of an experiment • Internal validity refers to the extent to which an experiment can establish the causal relation between the independent variable and
    [Show full text]
  • Randomized Controlled Trials, Development Economics and Policy Making in Developing Countries
    Randomized Controlled Trials, Development Economics and Policy Making in Developing Countries Esther Duflo Department of Economics, MIT Co-Director J-PAL [Joint work with Abhijit Banerjee and Michael Kremer] Randomized controlled trials have greatly expanded in the last two decades • Randomized controlled Trials were progressively accepted as a tool for policy evaluation in the US through many battles from the 1970s to the 1990s. • In development, the rapid growth starts after the mid 1990s – Kremer et al, studies on Kenya (1994) – PROGRESA experiment (1997) • Since 2000, the growth have been very rapid. J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 2 Cameron et al (2016): RCT in development Figure 1: Number of Published RCTs 300 250 200 150 100 50 0 1975 1980 1985 1990 1995 2000 2005 2010 2015 Publication Year J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 3 BREAD Affiliates doing RCT Figure 4. Fraction of BREAD Affiliates & Fellows with 1 or more RCTs 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 1980 or earlier 1981-1990 1991-2000 2001-2005 2006-today * Total Number of Fellows and Affiliates is 166. PhD Year J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 4 Top Journals J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 5 Many sectors, many countries J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 6 Why have RCT had so much impact? • Focus on identification of causal effects (across the board) • Assessing External Validity • Observing Unobservables • Data collection • Iterative Experimentation • Unpack impacts J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 7 Focus on Identification… across the board! • The key advantage of RCT was perceived to be a clear identification advantage • With RCT, since those who received a treatment are randomly selected in a relevant sample, any difference between treatment and control must be due to the treatment • Most criticisms of experiment also focus on limits to identification (imperfect randomization, attrition, etc.
    [Show full text]
  • Why Internal Validity Is Not Prior to External Validity
    Contributed Paper PSA 2012 (draft), p. 1 Why internal validity is not prior to external validity Johannes Persson & Annika Wallin Lund University, Sweden Corresponding author: [email protected] [Abstract: We show that the common claim that internal validity should be understood as prior to external validity has, at least, three epistemologically problematic aspects: experimental artefacts, the implications of causal relations, and how the mechanism is measured. Each aspect demonstrates how important external validity is for the internal validity of the experimental result.] 1) Internal and external validity: perceived tension and claimed priority Donald T. Campbell introduced the concepts internal and external validity in the 1950s. Originally designed for research related to personality and personality change, the use of this conceptual pair was soon extended to educational and social research. Since then it has spread to many more disciplines. Without a doubt the concepts captures two features of research scientists are aware of in their daily practice. Researchers aim to make correct inferences both about that which is actually studied (internal validity), for instance in an experiment, and about what the results ‘generalize to’ (external validity). Whether or not the language of internal and external validity is used in their disciplines, the tension between these two kinds of inference is often experienced. In addition, it is often claimed that one of the two is prior to the other. And the sense in which internal validity is often claimed to be prior to external validity is both temporal and epistemic, at least. For instance, Francisco Guala claims that: “Problems of internal validity are chronologically and epistemically antecedent to problems of external validity: it does not make much sense to ask whether a result is valid outside the experimental circumstances unless we are confident that it does therein” (Guala, 2003, 1198).
    [Show full text]
  • Experimental and Quasi-Experimental Designs for Research
    CHAPTER 5 Experimental and Quasi-Experimental Designs for Research l DONALD T. CAMPBELL Northwestern University JULIAN C. STANLEY Johns Hopkins University In this chapter we shall examine the validity (1960), Ferguson (1959), Johnson (1949), of 16 experimental designs against 12 com­ Johnson and Jackson (1959), Lindquist mon threats to valid inference. By experi­ (1953), McNemar (1962), and Winer ment we refer to that portion of research in (1962). (Also see Stanley, 19S7b.) which variables are manipulated and their effects upon other variables observed. It is well to distinguish the particular role of this PROBLEM AND chapter. It is not a chapter on experimental BACKGROUND design in the Fisher (1925, 1935) tradition, in which an experimenter having complete McCall as a Model mastery can schedule treatments and meas~ In 1923, W. A. McCall published a book urements for optimal statistical efficiency, entitled How to Experiment in Education. with complexity of design emerging only The present chapter aspires to achieve an up­ from that goal of efficiency. Insofar as the to-date representation of the interests and designs discussed in the present chapter be­ considerations of that book, and for this rea­ come complex, it is because of the intransi­ son will begin with an appreciation of it. gency of the environment: because, that is, In his preface McCall said: "There afe ex­ of the experimenter's lack of complete con­ cellent books and courses of instruction deal­ trol. While contact is made with the Fisher ing with the statistical manipulation of ex; tradition at several points, the exposition of perimental data, but there is little help to be that tradition is appropriately left to full­ found on the methods of securing adequate length presentations, such as the books by and proper data to which to apply statis­ Brownlee (1960), Cox (1958), Edwards tical procedure." This sentence remains true enough today to serve as the leitmotif of 1 The preparation of this chapter bas been supported this presentation also.
    [Show full text]
  • Validity 1 VALIDITY of QUANTITATIVE RESEARCH Recall
    Validity 1 VALIDITY OF QUANTITATIVE RESEARCH Recall “the basic aim of science is to explain natural phenomena. Such explanations are called theories” (Kerlinger, 1986, p. 8). Theories have varying degrees of truth. Validity is the best approximation to the truth or falsity of propositions (Cook & Campbell, 1979). Validity is at best approximate or tentative “since one can never know what is true. At best, one can know what has not yet been ruled out as false” (Cook & Campbell, 1979, p. 37). And, as we have seen in examining the logic of hypothesis testing, statistical power, and the validity of outcome measures, we don’t really prove that something is false. In other words, we never really prove a null hypothesis; we only fail to reject it. “Experimental results never ‘confirm’ or ‘prove’ a theory -- rather the successful theory is tested and escapes being disconfirmed” (Campbell & Stanley, 1963, p. 35). In other words, we fail to reject the null hypothesis. “Varying degrees of ‘confirmation’ are conferred upon a theory through the number of plausible rival hypotheses available to account for the data. The fewer such plausible rival hypotheses remaining, the greater the degree of ‘confirmation’” (Campbell & Stanley, 1963, p. 36). Thus, research is a field of varying degrees of certainty. [Analogy: Monet’s garden.] And, “continuous, multiple experimentation is more typical of science than once-and-for- all definitive experiments” (Campbell & Stanley, 1963, p. 3). [Analogy: Brush strokes in the painting.] Threats to validity are plausible rival hypotheses (i.e., other ways of explaining the results rather than the author(s) hypothesis).
    [Show full text]
  • Study Designs and Their Outcomes
    © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC CHAPTERNOT FOR SALE 3 OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION © JonesStudy & Bartlett Designs Learning, LLC and Their Outcomes© Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION “Natural selection is a mechanism for generating an exceedingly high degree of improbability.” —Sir Ronald Aylmer Fisher Peter Wludyka © Jones & Bartlett Learning, LLC © Jones & Bartlett Learning, LLC NOT FOR SALE OR DISTRIBUTION NOT FOR SALE OR DISTRIBUTION OBJECTIVES ______________________________________________________________________________ • Define research design, research study, and research protocol. • Identify the major features of a research study. • Identify© Jonesthe four types& Bartlett of designs Learning,discussed in this LLC chapter. © Jones & Bartlett Learning, LLC • DescribeNOT nonexperimental FOR SALE designs, OR DISTRIBUTIONincluding cohort, case-control, and cross-sectionalNOT studies. FOR SALE OR DISTRIBUTION • Describe the types of epidemiological parameters that can be estimated with exposed cohort, case-control, and cross-sectional studies along with the role, appropriateness, and interpreta- tion of relative risk and odds ratios in the context of design choice. • Define true experimental design and describe its role in assessing cause-and-effect relation- © Jones &ships Bartlett along with Learning, definitions LLCof and discussion of the role of ©internal Jones and &external Bartlett validity Learning, in LLC NOT FOR evaluatingSALE OR designs. DISTRIBUTION NOT FOR SALE OR DISTRIBUTION • Describe commonly used experimental designs, including randomized controlled trials (RCTs), after-only (post-test only) designs, the Solomon four-group design, crossover designs, and factorial designs.
    [Show full text]
  • Randomized Clinical Trials
    15 Randomized clinical trials One pill makes you larger, and one pill makes you small. And the ones your mother gives you, don’t do anything at all. Grace Slick, The Jefferson Airplane: White Rabbit, from Surrealistic Pillow, 1967 Learning objectives In this chapter you will learn: ● the unique features of randomized clinical trials (RCTs) ● how to undertake critical interpretation of RCTs The randomized clinical trial (RCT) is the ultimate paradigm of clinical research. Many consider the RCT to be the most important medical development of the twentieth century, as their results are used to dictate clinical practice. Although these trails are often put on a pedestal, it is important to realize that as with all experiments, there may be flaws in the design, implementation, and interpretation of these trials. The competent reader of the medical literature should be able to evaluate the results of a clinical trial in the context of the potential biases introduced into the research experiment, and determine if it contains any fatal flaws Introduction The clinical trial is a relatively recent development in medical research. Prior to the 1950s, most research was based upon case series or uncontrolled observations. James Lind, a surgeon in the British Navy, can claim credit for performing the first recorded clinical trial. In 1747, aboard the ship Salisbury, he took 12 sailors with scurvy and divided them into six groups of two each. He made sure they were similar in every way except for the treatment they received for scurvy. Dr. Lind found that the two sailors who were given oranges and lemons got better while the other ten did not.
    [Show full text]
  • Introduction to Meta-Analysis • in 1952, Hans J
    What got all this started? The two events that seem to have defined & stimulated meta-analysis in Psychology Introduction to Meta-Analysis • In 1952, Hans J. Eysenck reviewed the available literature and concluded that there were no favorable effects of psychotherapy – guess how that went • a bit of history over… • definitions, strengths & weaknesses – 20 additional years of empirical research “failed to • what studies to include ??? resolve the debate” • “choosing” vs. “coding & comparing” studies • In 1978, Gene V. Glass statistically aggregated the • what information to code along with each effect size ??? findings of 375 psychotherapy outcome studies – Glass (and colleague Smith) concluded that psychotherapy did indeed work – Glass called his method “meta-analysis” The Emergence of Meta-Analysis The statistical ideas behind meta-analysis predate Glass’s work… R. A. Fisher (1944) • “When a number of quite independent tests of significance have been made, it sometimes happens that although few or none can be claimed individually as significant, yet the aggregate gives an impression that the probabilities are on the whole lower than would often have been obtained by chance” • Source of the idea of aggregating probability values W. G. Cochran (1953) • Discusses a method of averaging means across independent studies • Laid-out much of the statistical foundation that modern meta- analysis is built upon (e.g., inverse variance weighting and homogeneity testing) The Logic of Meta-Analysis When is meta-analysis applicable? • Traditional methods
    [Show full text]
  • Policy Evaluation, Randomized Controlled Trials, and External Validity – a Systematic Review
    RUHR ECONOMIC PAPERS Jörg Peters Jörg Langbein Gareth Roberts Policy Evaluation, Randomized Controlled Trials, and External Validity – A Systematic Review #589 Imprint Ruhr Economic Papers Published by Ruhr-Universität Bochum (RUB), Department of Economics Universitätsstr. 150, 44801 Bochum, Germany Technische Universität Dortmund, Department of Economic and Social Sciences Vogelpothsweg 87, 44227 Dortmund, Germany Universität Duisburg-Essen, Department of Economics Universitätsstr. 12, 45117 Essen, Germany Rheinisch-Westfälisches Institut für Wirtschaftsforschung (RWI) Hohenzollernstr. 1-3, 45128 Essen, Germany Editors Prof. Dr. Thomas K. Bauer RUB, Department of Economics, Empirical Economics Phone: +49 (0) 234/3 22 83 41, e-mail: [email protected] Prof. Dr. Wolfgang Leininger Technische Universität Dortmund, Department of Economic and Social Sciences Economics – Microeconomics Phone: +49 (0) 231/7 55-3297, e-mail: [email protected] Prof. Dr. Volker Clausen University of Duisburg-Essen, Department of Economics International Economics Phone: +49 (0) 201/1 83-3655, e-mail: [email protected] Prof. Dr. Roland Döhrn, Prof. Dr. Manuel Frondel, Prof. Dr. Jochen Kluve RWI, Phone: +49 (0) 201/81 49-213, e-mail: [email protected] Editorial Office Sabine Weiler RWI, Phone: +49 (0) 201/81 49-213, e-mail: [email protected] Ruhr Economic Papers #589 Responsible Editor: Manuel Frondel All rights reserved. Bochum, Dortmund, Duisburg, Essen, Germany, 2015 ISSN 1864-4872 (online) – ISBN 978-3-86788-684-0 The working papers published in the Series constitute work in progress circulated to stimulate discussion and critical comments. Views expressed represent exclusively the authors’ own opinions and do not necessarily reflect those of the editors.
    [Show full text]
  • The External Validity of Laboratory Experiments: Qualitative Rather Than Quantitative Effects1 by Judd Kessler and Lise Vesterlund
    The External Validity of Laboratory Experiments: Qualitative rather than Quantitative Effects1 By Judd Kessler and Lise Vesterlund 1: Introduction Laboratory experiments are used to address a wide variety of questions within economics, including whether behavior is consistent with the predictions and assumptions of theory and how various mechanisms and institutions affect the behavior of economic agents (see Roth 1987 for an overview). The experimental laboratory has become an integral part of the field of economics and a productive dialog now exists between theory, laboratory experiments, and field studies. Results from laboratory experiments are being published in the leading economics journals and many top departments now have experimental laboratories. Recently, however, a set of papers by Levitt and List (2006, 2007a, 2007b, 2008) has questioned what we can learn from laboratory studies. At the center of their critique is the statement that “the critical assumption underlying the interpretation of data from lab experiments is that the insights gained can be extrapolated to the world beyond” (Levitt and List 2007a, p. 153) and the subsequent argument that there are “many reasons to suspect that these laboratory findings might fail to generalize to real markets” (Levitt and List 2008, p. 909), suggesting that the critical assumption about extrapolation may not hold. Specifically, the authors highlight five factors that differ between the laboratory and economic environments of interest. 2 They recognize that external validity also is a concern for field experiments and for naturally occurring data, however their statement that “field experiments avoid many of the important obstacles to generalizability faced by lab experiments” (Levitt and List 2008, p.
    [Show full text]