The Essential Role of Language in Survey Research Edited by Mandy Sha and Tim Gabel RTI Press 4642.indb 1 11-04-2020 3:32:43 PM © 2020 RTI International. All rights reserved. Credit The RTI Press mission is to disseminate must be provided to the author and source of the information about RTI research, analytic publication when the content is quoted. No part of tools, and technical expertise to a this publication may be reproduced in any form by any national and international audience. RTI electronic or mechanical means without permission Press publications are peer-reviewed by in writing from the publisher. RTI International is a at least two independent substantive registered trademark and a trade name of Research experts and one or more Press editors. Triangle Institute. The RTI logo is a registered RTI International is an independent, trademark of Research Triangle Institute. nonprofit research institute dedicated A free digital download of this book is available under to improving the human condition. We a CC-BY-NC-ND license from rti.org/rti-press. combine scientific rigor and technical expertise in social and laboratory Library of Congress Control Number: 2020935779 sciences, engineering, and international ISBN 978-1-934831-23-6 development to deliver solutions to the (refers to print version) critical needs of clients worldwide. RTI Press publication No. BK-0023-2004 https://doi.org/10.3768/rtipress.bk.0023.2004 www.rti.org/rtipress Cover design: Dayle Johnson This publication is part of the RTI Press Book series. RTI International 3040 East Cornwallis Road, PO Box 12194 Research Triangle Park, NC 27709-2194, USA [email protected] www.rti.org 4642.indb 2 11-04-2020 3:32:43 PM To the power of collaboration. 4642.indb 3 11-04-2020 3:32:43 PM 4642.indb 4 11-04-2020 3:32:43 PM Contents List of Figures and Tables vii Acknowledgments xi Foreword xiii Co-editors’ Preface xv Part I Language Influences on Survey Responses and Data Quality 1. The Effect of Language of Survey Administration on the Response Formation Process 3 Emilia Peytcheva 2. Seeking Clarifications for Problematic Questions: Effects of Interview Language and Respondent Acculturation 23 Evgenia Kapousouz, Timothy P. Johnson, and Allyson L. Holbrook 3. A Longitudinal Perspective on the Effects of Household Language on Data Quality in the American Community Survey 47 Heather Kitada Smalley 4. Quantitative Evaluation of Response Scale Translation Through a Randomized Experiment of Interview Language With Bilingual English- and Spanish-Speaking Latino Respondents 75 Sunghee Lee, Mengyao Hu, Mingnan Liu, and Jennifer Kelley 5. Language Differences Between Interviewers and Respondents in African Surveys 101 Charles Q. Lau, Stephanie Eckman, Luis Sevilla Kreysa, and Benjamin Piper 6. Void of the Voiceless: An Analysis of Residents With a Language Barrier in Germany, France, and the United Kingdom 117 Nicholas Heck-Grossek and Sonila Dardha Part II Survey Questionnaire Development and Implementation 7. Pretesting Methods in Cross-Cultural Research 129 Eva Aizpurua 8. Cross-Cultural Comparison of Focus Groups as a Research Method 151 Mandy Sha, Hyunjoo Park, Yuling Pan, and Jennifer Kim 4642.indb 5 11-04-2020 3:32:43 PM vi Contents 9. Hmong and Chinese Qualitative Research Interview Questions: Assumptions and Implications of Applying the Survey Back Translation Method 181 Maichou Lor and Chenchen Gao 10. Sociocultural Issues in Adapting Spanish Health Survey Translation: The Case of the Quality of Well-Being Scale (QWB-SA) 203 Nereida Congost-Maestre and Maichou Lor 11. Nura Knows You Better: Redesigning Conversations With Artificial Intelligence 221 Arundati Dandapani 12. Scaling the Smileys: A Multicountry Investigation 231 Aaron Sedley, Yongwei Yang, and Joseph M. Paxton Afterword: Future Directions in Multinational, Multiregional, and Multicultural (3MC) Survey Research 243 Julie de Jong, Kristen Cibelli Hibben, Jennifer Kelley, and Dorothée Behr About the Authors 257 Index 261 4642.indb 6 11-04-2020 3:32:43 PM List of Figures and Tables Figures Figure 1-1. Effect of language on survey response formation processes 4 Figure 2-1. Distribution of the index of all problematic questions, by race 36 Figure 3-1. A reproduction of the language questions from the 2017 American Community Survey 55 Figure 3-2. Segment of R code specifying a survey design object using the survey package 58 Figure 3-3. Trends in language diversity and prevalence and distribution of non-English speakers across the country, 2006–2017 59 Figure 3-4. Choropleth map of Public Use Microdata Areas colored by the most popular language group other than English, using 2017 Public Use Microdata Sample data 60 Figure 3-5. Choropleth map of Public Use Microdata Areas shaded by the number of limited English proficiency households, using 2017 Public Use Microdata Sample data 61 Figure 3-6. Conditional distributions for response mode across each language group and year have significant differences 61 Figure 3-7. Conditional distributions on both household language and English proficiency exhibit significantly different modes of response distributions 62 Figure 3-8. Average allocation rates across household language groups split by mode of response and level of English proficiency with bands for 95 percent confidence intervals 63 Figure 4-1. Distribution of Spanish and English proficiency on speaking, reading, and writing, by interview language 83 Figure 4-2. Percentage relative difference for items with frequency and quantity scales between Spanish and English interviews, by Spanish translation version 86 Figure 5-1. Taxonomy of language choice, by Afrobarometer country 106 Figure 5-2. Coefficients from multilevel logistic regression models predicting language choice 111 4642.indb 7 11-04-2020 3:32:43 PM viii List of Figures and Tables Figure 8-1. Percentages of linguistic features across the languages: Interaction direction and speech 164 Figure 8-2. Percentages of linguistic features across the languages: Types of answers and back channeling 166 Figure 9-1. Back translation process 183 Figure 11-1. A chatbot screen (mock-up by the author) where Nura, a chatbot, is enticing the user with its friendly tone to engage in some product research for audio short stories at Generation1.ca 222 Figure 11-2. Chatbot Wizu initiates a conversation with a guest about a customer experience hotel stay survey 226 Figure 11-3. When prompted by AI chatbot Wizu, a customer reports on the hotel stay service experience candidly 226 Figure 12-1. Smartphone screenshot example of emoji-based answer scales 232 Figure 12-2. Smiley faces used for 5-point satisfaction scale 233 Figure 12-3. Smartphone screenshots of meaning interpretation questions 235 Figure 12-4. Smartphone screenshots of numeric value questions, by condition 236 Figure 12-5. Words and phrases associated with the happiest face 238 Figure 12-6. Words and phrases associated with the unhappiest face 238 Figure 12-7. Median numeric values assigned to faces 239 Tables Table 2-1. Explanation of verbal behavior codes 34 Table 2-2. Explanation of independent variables 35 Table 2-3. Descriptive statistics of continuous independent variables 36 Table 2-4. Distribution of dichotomous independent variables 36 Table 2-5. Distribution of the factor variable “education” 36 Table 2-6. Logistic hierarchical and hierarchical linear models examining requests for clarifications to problematic questions 38 Table 3-1. American Community Survey modes of survey administration and languages per mode 53 4642.indb 8 11-04-2020 3:32:43 PM List of Figures and Tables ix Table 3-2. Coefficient estimates for main effects and interactions of household languages and response modes both treated as factors from 2006 through 2012 64 Table 3-3. Coefficient estimates for main effects and interactions of household languages and response modes both treated as factors from 2013 through 2017 65 Table 4-1. Proportion of “strongly agree” for family cohesion questions, by interview language 85 Table 5-1. Language taxonomy 105 Table 5-2. Survey language in Kenya, Mozambique, and Cameroon, by language taxonomy 108 Table 6-1. Cross-tabulations of units with and without a language barrier and their dwelling or area characteristics 121 Table 7-1. Frequently used probes 132 Table 7-2. Examples of behavior codes 140 Table 8-1. Focus group composition 159 Table 8-2. Coding scheme definitions and objectives 160 Table 8-3. Coding scheme examples 161 Table 8-4. Percentage agreement between coders in each language 163 Table 8-5. Pairwise differences across the five languages and linguistic features 167 Table 8-6. English-language focus group interaction 169 Table 8-7. Chinese-language focus group interaction (Chinese transcript followed by its meaning in English) 171 Table 8-8. Vietnamese-language focus group discussion (Vietnamese transcript followed by its meaning in English) 172 Table 8-9. Korean-language focus group interaction (Korean transcript followed by its meaning in English) 173 Table 10-1. Example 1 209 Table 10-2. Example 2 209 Table 10-3. Example 3 210 Table 10-4. Example 4 and Example 5 211 Table 10-5. Example 6 212 Table 10-6. Example 7 212 Table 10-7. Example 8 213 Table 12-1. Deviation from ideal interval size 240 4642.indb 9 11-04-2020 3:32:43 PM 4642.indb 10 11-04-2020 3:32:43 PM Acknowledgments One year is an ambitious timeline to publish any book. Thanks to the power of collaboration among our 28 contributors, we were propelled to complete this book project within that time frame. We would like to acknowledge the American Association for Public Opinion Research (AAPOR) for providing a platform in the form of an annual conference where we cultivate ideas, foster collaborations, and champion each other’s efforts. The AAPOR conference short course by Daphna Oyserman and Norbert Schwarz ignited our passion for writing this book to share with a broad audience survey research theories, practices, and outlook in cross-cultural contexts.