Principles and Methods of Social Research

PRINCIPLES AND METHODS OF SOCIAL RESEARCH

Used to train generations of social scientists, this thoroughly updated classic text covers the latest research techniques and designs. Applauded for its comprehensive coverage, the breadth and depth of content is unparalleled. Through a multi-methodology approach,the text guides readers toward the design and conduct of social research from the ground up. Explained with applied examples useful to the social, behavioral, educational, and organizational sciences, the methods described are intended to be relevant to contemporary researchers.

The underlying logic and mechanics of experimental, quasi-experimental, and nonexperimental research strategies are discussed in detail. Introductory chapters covering topics such as validity and reliability furnish readers with a ﬁrm understanding of foundational concepts. Chapters dedicated to sampling, interviewing, questionnaire design, stimulus scaling, observational methods, content analysis, implicit measures, dyadic and group methods, and metaanalysis provide coverage of these essential methodologies.

The book is noted for its: •••
Emphasis on understanding the principles that govern the use of a method to facilitate the researcher’s choice of the best technique for a given situation. Use of the laboratory experiment as a touchstone to describe and evaluate field experiments,correlational designs, quasi experiments, evaluation studies, and survey designs. Coverage of the ethics of social research, including the power a researcher wields and tips on how to use it responsibly.

The new edition features:

•••••
A new co-author,Andrew Lac,instrumental in fine-tuning the book’s accessible approach and highlighting the most recent developments at the intersection of design and statistics. More learning tools, including more explanation of the basic concepts, more research examples, tables, and figures,and the addition of boldfaced terms,chapter conclusions,discussion questions,and a glossary. Extensive revision of Chapter 3 on measurement reliability theory that examines test theory,latent factors, factor analysis, and item response theory. Expanded coverage of cutting-edge methodologies, including mediation and moderation, reliability and validity, missing data, and more physiological approaches such as neuroimaging and fMRIs. A new web-based resource package that features PowerPoint presentations and discussion and exam questions for each chapter and,for students,chapter outlines and summaries,key terms,and suggested readings.

Intended as a text for graduate or advanced undergraduate courses in research methods (design) in psychology, communication, sociology, education, public health, and marketing, an introductory undergraduate course on research methods is recommended.

William D. Crano is the Oskamp Distinguished Chair and Head of the Department of Psychology at Clare-

mont Graduate University. Marilynn B. Brewer is Professor Emeritus at The Ohio State University and a Visiting Professor of Psychology at the University of New South Wales.

Andrew Lac is Research Professor of Psychology at Claremont Graduate University.

This page intentionally left blank

PRINCIPLES AND METHODS OF SOCIAL RESEARCH

Third Edition

William D. Crano, Marilynn B. Brewer, and Andrew Lac

Third edition published 2015 by Routledge 711 Third Avenue, New York, NY 10017

and by Routledge 27 Church Road, Hove, East Sussex BN3 2FA

Routledge is an imprint of the T a ylor & Francis Grou p , a n informa business

© 2015 Taylor & Francis The right of William D. Crano, Marilynn B. Brewer, and Andrew Lac to be identiﬁed as the authors of this work has been asserted by them in accordance with sections 77 and 78 of the Copyright, Designs and Patents Act 1988.

All rights reserved. No part of this book may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers.

T r ademark notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identiﬁcation and explanation without intent to infringe.

First edition published by Allyn & Bacon 1986 Second edition published by Lawrence Erlbaum Associates, Inc. 2002

Library of Congress Cataloging-in-Publication Data

Crano, William D., 1942– Principles and methods of social research / William D. Crano, Marilynn B. Brewer, Andrew Lac. — Third edition. pages cm
1. Social sciences—Research. 2. Social sciences—Methodology. I. Brewer, Marilynn B.,
1942– II. Lac, Andrew. III. Title. H62.C692 2014 300.72—dc23 2014012586

ISBN: 978-0-415-63855-5 (hbk) ISBN: 978-0-415-63856-2 (pbk) ISBN: 978-1-315-76831-1 (ebk)

Typeset in Bembo by Apex CoVantage, LLC

CONTENTS

Preface Acknowledgements About the Authors xi xvii xix

PART I

Introduction to Social Research Methods

1

1 Basic Concepts

3

4

Science and Daily Life Theories and Hypotheses From Theor y , C oncept, or Idea to Operation Role of Theory in Scientiﬁc Inquiry Conclusion and Overview Questions for Discussion References

56
15 18 19 19

2 Internal and External Validity

Causation

22

23 25 27 31 39 41 42 43

Phases of Research Distinguishing Internal and Externa l V a lidity Basic Issues of Interna l V a lidity Basic Issues of Externa l V a lidity Conclusion Questions for Discussion References

3 Measurement Reliability

Classical T e st Theory Contemporary T e st Theory Conclusion

45

45 53 61

vi Contents

Questions for Discussion References

62 62

4 Measurement Validity

T y pes of Measuremen t V a lidity Threats to Measuremen t V a lidity Conclusion

64

65 74 78 78 78

PART II

Research Design Strategies: Experiments,

Quasi-Experiments, and Nonexperiments

81

5 Designing Experiments: Variations on the Basics

Basi c V a riations in Experimental Design Expanding the Number of Experimental T r eatments Blocked Designs: Incorporating a Nonexperimental Factor Repeated-Measures Designs and Counterbalancing Conclusion

83

83 86 94 96 98 99 99

6 Constructing Laboratory Experiments

Steps for Constructing an Experiment T y pes of Experimental Manipulations Manipulation Checks

101

101 104 109 110 111 112 120 120 121

Assignment of Participants to Conditions: Randomization Procedures Realism in an Experiment Social Simulations and Analogue Experiments Conclusion Questions for Discussion References

7 External Validity of Laboratory Experiments

Generalizability Across Participants Experimenter Expectancy and Bias Three Faces of Externa l V a lidity Conclusion

125

126 131 135 138 138 139

8 Conducting Experiments Outside the Laboratory

Research Settings and Issues o f V a lidity Constructing a Field Experiment The Internet as a Site for Experimental Research Conclusion

141

142 143 149 155 155 156

Contents vii

9 Nonexperimental Research: Correlational Design and Analyses

159

160 166 167 170 172 181 182 183

Analyzing and Interpreting Nonexperimental Research Multiple Regression Uses and Misuses of Correlational Analysis Multi-Level Models Structural Equation Models Conclusion Questions for Discussion References

10 Quasi-Experiments and Evaluation Research

Program Evaluation Research Quasi-Experimental Methods Use of Archival Data in Longitudinal Research Conclusion

185

186 189 211 213 213 213

PART III

Data Collecting Methods

217

11 Survey Studies: Design and Sampling

Selection vs. Assignment Random Sampling

219

219 224 233 236 241 244 246 246 247

Nonrandom Sampling More Sampling Issues T y pes of Survey Studies Missing Data Conclusion Questions for Discussion References

12 Systematic Observational Methods

Three Aspects of Naturalism Observer Involvement in the Naturalistic Setting: The Participatory–
Nonparticipatory Distinction
Coding Observations

250

250 253 260 274 275 276

Conclusion Questions for Discussion References

13 Interviewing

280

280 281 286 290 293 296 298

Modes of Administration: Face-to-Face and T e lephone Developing the Interview Interview Structure Conducting the Interview Interviewer Characteristics: Establishing Rapport Group Interviews and Focus Groups Conclusion

viii Contents

298 299

14 Content Analysis

Conducting a Content Analysis Representative Examples Conclusion

303

304 314 319 319 320

15 Questionnaire Design and Scale Construction

323

323 326 338 338 339

Questionnaires Constructing Rating Scales Conclusion Questions for Discussion References

16 Indirect and Implicit Measures of Cognition and Affect

Indirect Measures

342

343 344 350

Information Processing: Attention and Memory Priming: Processing Without Awareness or Intent Social Psychophysiology: Physiological T r aces of Affect and Cognitive Processing
Conclusion Questions for Discussion

358 363 364 366

References

17 Scaling Stimuli: Social Psychophysics

Scaling Stimuli

370

371 372 377 382 382 383

T e chniques for Stimulus Scaling Multidimensional Scaling Models Conclusion Questions for Discussion References

18 Methods for Assessing Dyads and Groups

Dyadic Designs

385

385 387 391 396 398 402 402 403

Deriving Dyadic and Group Leve l V a riables Designs to Study Group Structures Designs to Study Multiple Groups Measuring Group Process and Outcomes Conclusion Questions for Discussion References

Contents ix

PART IV

Concluding Perspectives

407

19 Synthesizing Research Results: Meta-Analysis

Stages in the Meta-Analysis Process Interpreting the Meta-Analysis Conclusion

409

411 422 424 424 425

20 Social Responsibility and Ethics in Social Research

Ethics of Research Practices

428

428 435 438 440 441 441 442

The Regulatory Context of Research Involving Human Participants Ethics of Data Reporting Ethical Issues Related to the Products of Scientiﬁc Research Conclusion Questions for Discussion References

Glossary Suggested Additional Readings Author Index

445 459 467 477

Subject Index

This page intentionally left blank

PREFACE

When this book’s “great-grandfather” first saw the light of day, William Crano and Marilynn Brewer were fresh out of graduate school, former students of Donald T. Campbell, one of social science’s preeminent research methodologists. Then and now we recognized the need for a clear understanding of the proper application of the scientific method to the study of human social behavior. As before, many people today still contend that science has nothing to say about the beliefs,values,and behavioral characteristics that define us as human. From this ultimately pessimistic point of view, love, hate, identity, alienation, prejudice, reason, discrimination, altruism, attitudes, intergroup relations, and a host of other distinctly human characteristics and properties are defined as beyond the reach of science. The argument goes, “Such things are better left to poets or priests or philosophers—they are not appropriate subjects of scientific study.” We disagree.
We disagree because we are convinced that in the short space of this book’s life, good social research has added immensely to our store of knowledge about all of these difficult-to-research but ultimately understandable issues. Complex aspects of human social life are subject to good scientific analysis, which when applied properly produces knowledge that leads to a better understanding of the human condition. This book is designed to help lay the necessary foundation for this quest. It is firmly grounded in the scientific method, and provides the background knowledge necessary to design and implement good research. It is crucial, however, that the rigorous application of the scientific method is accompanied by creativity and dedication if real progress is to be made in social research. Although we believe this book can supply the necessary grounding, you will need to bring your own creativity and motivation.
In our earliest rendition of this book, we focused strongly on the experimental method as a touchstone against which other research methodologies were to be judged. We still view the experiment as the preeminent means of developing strong causal inferences, but even in the original volume we recognized that the environment of the laboratory was quite restrictive, and so we expanded the field of play to include topics like interviewing and content analysis,topics that at the time were deemed “esoteric”inclusions in a book on research design. In the book’s later version,we expanded our consideration of the more traditional experimental models and extended our discussion to include new topics, including analogue experiments, meta-analysis, regression-based causal modeling, survey sampling, evaluation research, dyadic analysis, and considerations of experimenter expectancy effects. Emerging methods in social cognition also made the chapter list, along with an augmented emphasis on external validity and generalizability,the use of field research accompanied

xii Preface

by relatively novel techniques like propensity analysis, and other methods of intervention and evaluation that fit with our growing realization that good science owes a debt to the society that nurtured it, a concern Campbell advanced insistently in his later years.
The basic structure of our book has been designed to build logically from fundamental research strategies (experimental and quasi-experimental methods, correlational designs, etc.) to the tactics of data collection and the logistics of proper research procedures. We have presented this material as clearly as we could, and believe a conscientious student should not have problems understanding the concepts and advice included across the various chapters. To facilitate comprehension, important terms throughout the chapters are printed in boldface type, indicating a word or concept that has been defined and included in the glossary at the end of the book. We have been relatively selective in our use of citations throughout the book. The works that have been cited represent the best and most relevant work we could find. For conscientious students,then,we suggest that it might be wise to spend some time reflecting on these key contributions to methodological understanding.
A unique feature of this book is its comprehensiveness. We do not know of a book of this type that considers such a broad range of experimental, nonexperimental, correlational, quasiexperimental, and evaluation research designs, and that has combined these with data collection methods ranging from survey sampling to interviewing to content analysis, scale development, and methods for assessing dyads and groups. There is much to do in developing a working understanding of the intricacies of good methodological design, and we believe that this presentation will facilitate its readers in doing so.

Content Overview

The volume is broken into four major,multi-chapter sections. Part I is focused on the basics,the fundamental introductory concepts that must be well in hand if the rest of the material is to be properly learned and profitably employed. Crucial issues of reliability and validity occupy the initial section, as it should occupy the initial considerations of any research method. We emphasize the basics in the first chapter,the nature of the scientific method,operationalization,the place of theory in a developing science, and ideas of causation, moderation, and mediation. The second chapter leans heavily on Campbell’s and Stanley’s classic exposition of threats to internal and external validity in experimentation. Chapter 3 is a major reworking of measurement reliability theory. It includes discussion of classic and contemporary test theory,latent factors,exploratory and confirmatory factor analysis,and item response theory. In Chapter 4 we draw heavily on our earlier discussion of measurement validity. Here, we discuss the many faces of validity, the multi-trait multi-method matrix, and the various threats to validity that must be considered when developing trustworthy measures.
Part II is concerned with important features of experimental design. Experimentation is a fundamental feature of the book, as it provides a disciplined way of thinking about research that we believe is productive and generative. This mindset infuses our discussion of research methods that do not involve the strict control and behavioral constriction of the experiment. Chapter 5 is focused on true experimental designs,factorialization,and interpreting interaction effects. Chapter 6 is concerned with the nuts and bolts of actually designing and conducting an experiment. It deals with various forms of manipulations and manipulation checks,analogue and simulation studies,and mundane and experimental realism. Chapter 7 develops this latter theme; it highlights generalizability, or external validity, with a special emphasis on the nature of the individuals who serve as the participants in our studies, the possibly distorting effects of experimenter expectancies on the research results, and ways to avoid them.
In the later sections of Part II, we expand the canvas to consider research outside the friendly confines of the laboratory. Chapter 8 deals with the special issues involved in field experimentation,

Preface xiii

including research that makes use of the Internet as a source of both information and research participants. We discuss the types of replications that are most likely to advance progress, and come down firmly on the side of conceptual rather than exact replications. Chapter 9 moves from experimentation to correlational designs, including considerations of multiple regression, path analysis, and latent structural equation approaches, which are used to specify and evaluate the adequacy of complex multivariate models of causation.
Chapter 10 rounds out Part II with a discussion of quasi-experimental designs and evaluation research. This chapter is a unique entry not often included in many introductions to research design in the social sciences. However, these methods add value to the social scientist’s arsenal of tools and should be well understood, as they greatly expand the range of phenomena that can be investigated in a rigorous manner. We noted in an earlier volume that there was never a single,right way to support a position. That being the case, it is important to have a command of a variety of methods beyond the experiment that allow different features of a phenomenon to be investigated in different ways. To hold out for the experiment over all other design alternatives suggests a shallow understanding of the research process, because some questions are not ideally studied with experimental methods, just as some almost demand them. To manipulate a research question so that it “fits” an experimental approach is backwards. It is far better to fit the appropriate method to the question at issue, and often that method is not an experiment.
Part III of the book is concerned with data collection methods used to supply the raw material for the design possibilities discussed in Parts I and II. Here we learn about the many techniques in which grist for the social scientist’s mill can be collected as reliably and validly as possible. Chapter 11 concerns the survey in all its various forms, with a focus on sampling methodologies. In addition, we consider various forms of missing data and their implications for interpretation of results. Chapter 12 provides an overview of systematic observational methods and ways to develop systems to code the often unruly data of field observations. Chapter 13 is focused on the various forms of interviewing,ranging from the unsystematic exploratory interview to the highly structured,scheduled interview, and the interviewing tactics designed to capture the most useful, reliable, and valid data from this method. Chapter 14 deals with content analysis and factors that must be considered when contemplating either a deductive or inductive form of communication analysis. In Chapter 15, we focus on the proper design of items and responses for questionnaires and scales, using Thurstone’s, Guttman’s, Likert’s, or Osgood’s measurement approaches.
Chapter 16 is an exposition of techniques used to measure implicit thoughts and feelings. It describes the use of implicit approaches,ranging from the thematic apperception test to the implicit associations test. The chapter also considers more physiologically oriented approaches, including event-related potential (brain waves), neuroimaging (positron emission tomography, or PET scans), and functional magnetic resonance imaging (fMRI). Chapter 17 discusses psychophysical methods that have been in use in psychology since the 1800s. Psychophysical scales were among the earliest contributions of the pioneers of psychology, and include measures derived from pair comparison and rank order scaling. These techniques are valuable in and of themselves, and serve as a lead into the more modern techniques of multidimensional scaling analysis.
Data describing dyadic and group interactions are the special focus of Chapter 18. This chapter outlines some of the ways in which this inherently social form of data can be understood and the designs that can be used to capture this understanding. Round-robin designs, sociometric assessments, and social network analysis are considered in this final chapter of Part III.
Part IV of the text consists of only two chapters, but both of them are crucial to the research enterprise. The first of these, Chapter 19, deals not with a specific research design method or technique, but rather with meta-analysis, a means of quantitatively synthesizing data across a host of studies that have all focused on the same relationship. As a way of summarizing the collected