Response Effects in Surveys norc NATIONAL OPINION RESEARCH CENTER MONOGRAPHS IN SOCIAL RESEARCH 1. JAMES A. DAVIS/ Great Aspirations 2. JAMES A. DAVIS/ Undergraduate Career Decisions 3. NORMAN M. BRADBURN AND DAVID CAPLOVITZ I Reports on Happiness 4. JOHN w. c. JOHNSTONE AND RAMON J. RIVERA I Volunteers for Learning 5. JAMES A. DAVIS/ Education for Positive Mental Health 6. ANDREW M. GREELEY AND PETER H. ROSSI I The Education of Catholic Americans 7. SEYMOUR WARKOV /Lawyers in the Making 8. THEODORE M. NEWCOMB AND EVERETT K. WILSON (EDS.) j College Peer Groups 9. WALTER L. WALLACE/ Student Culture 10. SEYMOUR SUDMAN /Reducing the Cost of Surveys 11. JACOB J. FELDMAN/ The Dissemination of Health Information 12. PETER. H. ROSSI AND BRUCE J. BIDDLE (EDS.) I The New Media and Education 13. ANDREW M. GREELEY/ The Changing Catholic College 14. ROBERT L. CRAIN/ The Politics of School Desegregation 15. NORMAN M. BRADBURN/ The Structure of Psychological Well-Being 16. SEYMOUR SUDMAN AND NORMAN M. BRADBURN I Response Effects in Surveys Response Effects in Surveys A Review and Synthesis by SEYMOUR SUDMAN and NORMAN M. BRADBURN ALDINE PUBLISHING COMPANY Chicago Copyright© 1974 by National Opinion Research Center All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the publisher. First published 1974 by ALDINE Publishing Company 529 South Wabash Avenue Chicago, lllinois 60605 Second printing, 1977 Library of Congress Catalog Card Number 73-89510 ISBN 0-202-30270-9 Designed by Greer Allen Printed in the United States of America Preface Since surveys are the most common m,ethod of data collection in the social sciences, there is great interest in response effects that may distort survey results. In addition, because an interview is a controlled social interaction between two people, response effects may be intrinsically interesting and not merely nuisances to be eliminated. Such interest in response effects is demonstrated by the large number of studies that have been conducted and that are . analyzed in this monograph. Initially we had planned to conduct experiments to measure response effects in threatening situations. Peter Rossi in particular, and our colleagues at NORC, however, strongly urged us to wait until we did a comprehensive review of the literature and could better see the gaps that needed filling. We agreed, and as is often the case in research, the project snowballed as we uncovered new sources and refined our analysis. We do not claim that the bibliog­ raphy that resulted from this review is complete. We believe that few U.S. sources have been overlooked, but we suspect that there are many foreign studies of whiCh we were unaware, beyond the number of studies from Britain, France, Germany, and Sweden that are included. Our analysis differs from the usual review of literature that describes the results of different studies. Drawing on our survey research backgrounds, we coded each study and treated the separ­ ate results as individual respondents would be treated in a survey. That is, we show distributions and statistics for combined studies rather than for individual studies. Obviously, this procedure de­ pends heavily on the very careful coding of individual studies. We are grateful to Neil Shover, Said Atamna, and Dan Waung for their conscientious efforts, both in coding and in tracking down the ref­ erences used in the analysis. v vi Response Effecrs in Surveys The final preparation of the bibliography and subject index was done with great skill by Mary Sp;;ieth, who also edited parts of the initial draft of the manuscript. The final manuscript was edited by Paige Wickland and proofread by Susan Campbell. The final typing -of the manuscript was capably handled by Nella Siefert. We owe special thanks to Charles Cannell at the Survey Re­ search Center, University of Michigan, not only for his helpful suggestions throughout the project, but especially for making avail­ able to us an unpublished bibliography on survey methodology that was the nucleus for our bibliography'. Other colleagues who read the manuscript and made very helpful suggestions were Barbara Bailer of the Bureau of the Census; James A. Davis, Eve Weinberg, and Jam es Murray of the National Opinion Research Center; Barbara Dohrenwend of the City Uni­ versity of New York; Robert Ferber of the University of Illinois, Urbana-Champaign; and Peter Rossi of Johns Hopkins University. We also wish to express our deep gratitude to the Alexander von Humboldt Stiftung, Bad Godesberg, West Germany, whose gener­ ous support of the junior author during the academic year 1970-71 enabled him to collect data from German sources as well as work on the manuscript; to Professor Dr. E. K. Scheuch, Herr H. Stege­ mann, and the staff of the Zentral Archiv fiir Empirische For­ schung, Universitiit KOln, who provided excellent facilities and congenial colleagueship which enriched our study; to Professor Dr. Elizabeth Noelle-Neumann, Institut filr Demoskopie Allensbach, both for the data made available to us and for her warm support of our efforts; to Professor Dr. K. C. Behrens and Dipl.-Kfm. A. Grommann of the Freie Universitat Berlin and Dr. K. Schreiber, Berlin-Test, Berlin, for allowing us to analyze unpublished data on response effects. Several tables in this monograph originally appeared in other publications. We acknowledge with thanks permission to reprint granted by the American Sociological Association, the Columbia University Press, the Journal Press, and the University of Chicago Press. This project was funded under National Science Foundation Grant GS-2008. We acknowledge with gratitude the financial and vii Prefa.ce moral support from the staff of the National Science Foundation, without whose help this and other NORC studies in research methodology would not have been possible. SEYMOUR SUDMAN Survey Research Laboratory University of Illinois, Urbana-Champaign NORMAN M. BRADBURN National Opinion Research Center University of Chicago Contents v Preface xiii List of Tables CHAPTER 1 1. Introduction 2 Development of the Conceptual Framework 4 Model of the Interview 7 Task Variables 13 Role of the Interviewer I 6 Respondent Role Behavior 17 Summary 18 The Dependent Variable: Response Effect 21 Coding the Independent Variables 23 Example of the Treatment of Data 25 Analysis 27 Organization of the Report 28 2. Task Variables 30 Conditions of the Interview and Method of Administration 33 Position of Question and Deliberate Bias or Deception 35 Degree of Structure · 36 Question Length 36 Problems of Threat, Self-Presentation, and Saliency of Questions 40 Method of Administration and Problems of Self-Presentation 46 Self-Presentation and the Structure, Length, Difficulty, and Position of Questions 51 Respondent and Interviewer Characteristics and Threatening Questions 53 Saliency and Self-Presentation 54 Topic of Study ix x Response Effects in Surveys 62 Subject of Report 65 Summary 67 3. Effects of Time and Memory Factors on Response 67 Introduction 68 A Simple Model of the Effect of Time on Memory 68 Omissions 69 Telescoping 74 Illustrations of the Memory Model 79 Response Effect by Length of Rernll Period and Subject of Study 80 Aided Recall and the Use of Records 82 Aided Recall, Use of Records, and Length of the Recall Period 83 Bounded Recall Procedures 85 Age of Respondent 85 Conditions of the interview 86 Method of Administration 87 Threat of the interview or Question 89 Position of Question in the Questionnaire 90 Closed-ended and Open-ended Questions 92 Summary 93 4. The Effect of Respondent and interviewer Characteristics on Response to Attitudinal Questions 95 Characteristics of Respondents 98 "Don't Know" or "No Opinion" Responses JOO Combinations of Characteristics 102 Respondent Characteristics and Conditions of the interview 109 Characteristics of Interviewers 112 interviewer Characteristics and Conditions of the interview 117 Interactions between Respondent and interviewer Characteristics 119 Social Class Differences: The Katz Results 122 Age Interactions 125 Sex interactions 126 Race interactions 128 Education Interactions 132 Summary 137 Conclusions and /111plications for interviewer Selection xi Contents 140 5. Summary and Implications for Further Research l 41 Non-Threatening Behavioral, Questions 142 Threatening Behaviaral Questions 743 Attitudes I 45 Suggested Additional Research 148 The Coding of Studies as a Research Strategy APPENDICES 151 A. Coding Instructions 168 B. Full Distribution of Responses from Table 2.I 177 BIBLIOGRAPHY 247 SUBJECT INDEX TO BIBLIOGRAPHY List of Tables TABLE 31 2.1 Mean Response Effect in Standard Deviation Units by Selected Task Variable 32 2.2 Mean Response Effect in Standard Deviation Units by Location of Interview and Method of Administration 34 2.3 Mean Response Effect in Standard Deviation Units by Position of Question Relative to Related Questions and in Total Questionnaire 35 2.4 Mean Response Effect in Standard Deviation Units by Condition of Interview and Structure of Questions 37 2.5 Mean Response Effect in Standard Deviation Units by Form of Interview and Length of Questions 41 2.6 Estimated Net Underreporting for Self-Report Deviant Behavioral Items 42 2.7 Comparison of Women's Responses to Health Questions by Self-Administered, Telephone, and Face-to-Face Interviews 44 . 2.8 Attitudes of Employees to Working