Systematic Reviews in the Social Sciences a PRACTICAL GUIDE
Total Page:16
File Type:pdf, Size:1020Kb
Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_01_pre-toc Final Proof page 3 4.8.2005 1:07am Systematic Reviews in the Social Sciences A PRACTICAL GUIDE Mark Petticrew and Helen Roberts Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_01_pre-toc Final Proof page 1 4.8.2005 1:07am Systematic Reviews in the Social Sciences Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_01_pre-toc Final Proof page 2 4.8.2005 1:07am Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_01_pre-toc Final Proof page 3 4.8.2005 1:07am Systematic Reviews in the Social Sciences A PRACTICAL GUIDE Mark Petticrew and Helen Roberts Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_01_pre-toc Final Proof page 4 4.8.2005 1:07am ß 2006 by Mark Petticrew and Helen Roberts BLACKWELL PUBLISHING 350 Main Street, Malden, MA 02148-5020, USA 9600 Garsington Road, Oxford OX4 2DQ, UK 550 Swanston Street, Carlton, Victoria 3053, Australia The right of Mark Petticrew and Helen Roberts to be identified as the Authors of this Work has been asserted in accordance with the UK Copyright, Designs, and Patents Act 1988. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photo- copying, recording or otherwise, except as permitted by the UK Copyright, Designs, and Patents Act 1988, without the prior permission of the publisher. First published 2006 by Blackwell Publishing Ltd 1 2006 Library of Congress Cataloging-in-Publication Data Petticrew, Mark. Systematic reviews in the social sciences : a practical guide/Mark Petticrew and Helen Roberts. p. cm. Includes bibliographical references and index. ISBN-13: 978-1-4051-2110-1 (hard cover: alk. paper) ISBN-10: 1-4051-2110-6 (hard cover: alk. paper) ISBN-13: 978-1-4051-2111-8 (pbk. : alk. paper) ISBN-10: 1-4051-2111-4 (pbk. : alk. paper) 1. Social sciences—Research—Methodology. 2. Social sciences—Statistical methods. I. Roberts, Helen, 1949–II. Title. H62.P457 2005 300’.72’3—dc22 2005011632 A catalogue record for this title is available from the British Library. Set in 10.5pt/12.5pt Bembo by SPI Publisher Services, Pondicherry, India Printed and bound in the United Kingdom by TJ International, Padstow, Cornwall The publisher’s policy is to use permanent paper from mills that operate a sustainable forestry policy, and which has been manufactured from pulp processed using acid-free and elementary chlorine-free practices. Furthermore, the publisher ensures that the text paper and cover board used have met acceptable environmental accreditation standards. For further information on Blackwell Publishing, visit our website: www.blackwellpublishing.com Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_02_toc Final Proof page 5 4.8.2005 1:07am Contents Foreword – William R. Shadish vi Acknowledgments x Preface xiii Chapter 1 Why do we need systematic reviews? 1 Chapter 2 Starting the review: Refining the question and defining the boundaries 27 Chapter 3 What sorts of studies do I include in the review? Deciding on the review’s inclusion/exclusion criteria 57 Chapter 4 How to find the studies: The literature search 79 Chapter 5 How to appraise the studies: An introduction to assessing study quality 125 Chapter 6 Synthesizing the evidence 164 Chapter 7 Exploring heterogeneity and publication bias 215 Chapter 8 Disseminating the review 247 Chapter 9 Systematic reviews: Urban myths and fairy tales 265 Glossary 277 Appendix 1 The review process (and some questions to ask before starting a review) 284 Appendix 2 MOOSE Guidelines 288 Appendix 3 Example of a flow diagram from a systematic review 291 Appendix 4 Example data extraction form 293 Appendix 5 Variations in the quality of systematic reviews 296 Bibliography 298 Index 324 Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_03_posttoc Final Proof page 6 4.8.2005 1:06am Foreword The idea of evidence-based practice is not new. For example, nearly 60 years ago, the scientist-practitioner model of clinical psychology in the US was based in part on the idea that clinical psychologists would scientifically investigate the effectiveness of their own treatments, and use what proved to be effective. Around 100 years ago, the British Army asked Sir Karl Pearson to review the evidence about the effects of a new typhoid vaccine, to help them decide whether to adopt that vaccine more widely.1 Such relatively isolated examples are common for hundreds of years. What is new is the widespread acceptance that evidence-based practice is practice whose time has come. Whether it be through the Cochrane Collaboration, the Camp- bell Collaboration, the What Works Clearinghouse in the US Department of Education, or any of a host of similar efforts, the idea that practices and policies that are supported by scientific evidence can be identified and disseminated has now been institutionalized. That in itself is a remarkable accomplishment. In this context, this book by Petticrew and Roberts is a timely gem, full of detailed history, pithy quotes, marvelous examples, compelling argu- ments, practical advice, and transparent demonstrations of how to do one of the most common ways of putting the evidence into evidence-based practice – the systematic review. Practical advice leaps out at the reader from every chapter, with novel information I have not seen in related works. Let me start with reference to the key justification for systematic reviews, that ‘‘single studies taken in isolation are often seriously misleading’’ (chapter 1, this volume). Or as I tend to put it: Never trust the results of single studies. One need only graph a frequency distribution of effect sizes to see that some studies will show unusually positive effects, and some un- usually negative effects, substantially by virtue of chance. Yet the public, the media, and even other scientists routinely place great faith in single studies. Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_03_posttoc Final Proof page 7 4.8.2005 1:06am FOREWORD vii As a result, ineffective or inaccurate policies, practices, or ideas are adopted, accurate or effective ones are dismissed, and all parties get confused. The analogy that Petticrew and Roberts make with a survey is a good one. We would never take seriously the results of a sample survey where the sample was of a single person, and we all understand why – we will often get a different answer by asking a different person. The same is true in systematic reviews. We will often get a different answer when we look at a different study. In sample surveys, only a large sample is capable of giving us a reliable answer. In reviewing literature on a question, only the systematic review process is capable of helping to clarify where the answer really lies. Yet as Petticrew and Roberts point out in chapter 2 and elaborate in chapter 3, we have had an unnecessarily narrow view of what a systematic review can do – answer questions about cause and effect relationships. Systematic reviews need not only address what works, as important as that question is. Such reviews can address the evidence about nearly any kind of question, for example, about the prevalence of a problem, the correlation of two variables to each other, the implementation of a program, the meaning program stakeholders ascribe to a program or its context, the causal mediating processes that help us understand why it works, and program costs, to name just a few. Similarly, our understanding of the question of what works has often been too narrow. That question involves high quality evidence about cause and effect relationships, to be sure. But we are also informed about what works by, for example, knowledge that some stake- holder groups refuse to participate, that costs are too high to be practical, or that the need for the treatment is questionable. What is less clear is whether we have well-developed technologies for addressing some of these questions. The technological problem is twofold, how to integrate studies that are all of one kind (e.g., about need) with each other, and how to integrates studies of one kind (e.g., need) with studies of another kind (e.g., treatment effectiveness). The former problem is relat- ively unproblematic for most kinds of quantitative studies, but is far less clear for qualitative studies. The latter problem is almost completely unaddressed. It is this context that makes me so pleased to see the effort Petticrew and Roberts have put into both these problems. Many people talk about the value of integrating quantitative and qualitative studies, for example, but actual examples of such integration are few and far between. Petticrew and Roberts lay out the logic of such integrations, some of the methods that might be used for doing them, along with examples. And they don’t shy away from the difficulties such as a lack of standardized search terms to use in locating qualitative studies on a given topic. For many years, when critics have decried the narrow focus on experiments in systematic reviews of what Petticrew / Assessing Adults with Intellecutal Disabilities 1405121106_03_posttoc Final Proof page 8 4.8.2005 1:06am viii FOREWORD works, I have responded by saying we are working on that problem. Now I can actually point to the work. I was also pleased to see the huge number of references in chapter 5 about assessing the quality of different kinds of research designs. Just the compila- tion itself is a major contribution. Yet we should remember that quality can mean two different things: (a) quality of the implementation of the method (e.g., did the primary study author implement the qualitative method properly); and (b) quality of the inferences from the method (do the methods used support a causal inference).