Are data collected by NSOs biased? A comparison of Afrobarometer and Governance, Peace and Security surveys in Africa Thomas Calvo, Mireille Razafindrakoto, François Roubaud DIAL, LEDa, IRD, Université Paris-Dauphine, PSL Research University. November, 2018 Authors Keywords: Interviewer effect; Household survey; Governance; Perception; Africa. JEL classification codes: D02, C81, C83, O10, O55. Abstract The need to collect data on governance related issues has been growing since the 2010s. African countries are now leaders to collect governance, peace and security related household data. Yet, sensitive survey data collected by government dependent institutions have been recently pointed out as potentially biased due to self-censorship from respondents. This paper studies the potential response bias from sensitive surveys, here governance-related surveys, collected by public organisations. We originally combine Afrobarometer survey data, collected by claimed independent institutions and first-hand and harmonized Governance, Peace and Security (GPS-SHaSA) survey data, integrated in representative traditional household surveys conducted by the National Statistical Offices (NSOs). We identify more than twenty similarly worded questions regarding the democracy functioning, the trust in institutions and the perceived level of corruption of public servants. Estimating ordered logit, we first compare responses from Afrobarometer (AB) respondents according to the respondents’ perception of the survey sponsor identity. No systematic response bias is found between respondents who believe the government and those who believe an independent institution is behind the AB survey. The residual differences observed are explained by selection bias mitigated through propensity score matching. The absence of a systematic self- censorship is further evidenced through a RCT-like framework comparing responses from respondents interviewed by a NSO agent and those interviewed a declared independent agent. The results provide evidence on the capacity of government-related bodies to collect data on matters of national interest, at more precise scales than other existing data sources. 1 INTRODUCTION While the number of existing household surveys skyrockets in developed and developing countries, their quality and their underlying biases have been rarely studied. Common survey errors are mainly divided into sampling errors and non-sampling errors (Groves, 2004). The former are usually easier to observe than the latter as non-sampling errors take very different shapes from measurement errors to non-response biases. One is called the “threat of disclosure”: it corresponds to the risks and the potential costs a surveyed individual would take by honestly reporting their answers. One lecture of the threat of disclosure can be understood as the reluctance to let the interviewer and the survey sponsor identify the re spondent’s opinion. However, very little of the existing literature examines a response bias explained by the survey sponsor identity. Some experts still have expressed their reluctance to let public organisations collect sensitive data particularly in non-democratic countries , where “NSOs may suffer from being pe rceived as ‘agents of the state’ ” (UNDP, 2009). Sustainable Development Goals (SDGs) established the centrality of governance, peace and security issues for the developed and developing countries. For instance, Goal 16 of post- 2015 SDGs promotes “peaceful and inclusive societies for sustainable development, the provision of access to justice for all, and building effective, accountable institutions at all levels” (UN, 2015). Particularly exposed to armed conflict and political instability, African countries are substantially concerned by governance, peace and security issues. Between the fall of the Soviet bloc and 2009 about half of the armed conflict related fatalities were registered in the African continent (HSRP, 2012). Regarding corruption perceptions, among the 176 countries part of the Transparency International ranking, three quarter of Sub-Saharan African countries are below the median (TI, 2017). The need for related indicators is primordial in order to offer more adequate country specific responses. Yet, the measure of governance is still up-to-now a matter of fierce debate. Very different indicators have been used in the literature to fill the caveat in development economics with their own specific limitations (Arndt & Oman, 2006). They can be based upon objective measures as the existing laws, the political regime, upon administrative data or upon expert opinions. Survey-based measures on both governance and peace and security issues have also found a substantial interest from academics and policy makers. Presented as very sensitive questions, it is feared that government-related organisations as National Statistical Offices (NSO) collect an erroneous picture of citizens’ governance assessment. Respondents are expected to give more positive perceptions in order to hide their actual thoughts and avoid potential reprisals. This intuition holds even if NSOs are 2 supposed neutral and independent from the central authority. This paper aims at reducing the knowledge gap regarding the impact of the survey sponsor identity on responses to sensitive questions, and particularly governance related questions. To the best of our knowledge, directly comparable literature is limited to two studies. Both support a “fear -of-the- state bias” (Zimbalist, 2017) despite apparent insufficiencies in the analysis. This paper tries to disentangle for the first time the role of the survey sponsor through a comparative analysis of survey data collected by two different organisations: claimed independent collectors, the Afrobarometer (AB) network, and government-related collectors, the NSOs. The AB data is provided by an experienced and most importantly declared independent body created along with local partners constituted of mainly consulting groups, research institutes and universities. The network benefits from the technical support of the Michigan State University. The Governance, Peace and Security (GPS) data collected by the NSOs are survey modules part of the Strategy for the Harmonization of Statistics in Africa (SHaSA). Supported by the African Union Commission (AUC) and based upon a framework lastingly experimented (Herrera et al., 2007 ; Razafindrakoto & Roubaud, 2015), the GPS- SHaSA modules are integrated in national household surveys. Both datasets cover similar governance-related issues including corruption perceptions, trustfulness in public services and institutions and political preferences. These questionnaires are comparable across eight African countries. Even if they are different in some aspects both datasets offer enough similarities (sampling procedures, questions’ wording and period of data collection among others) to conduct an extensive comparison to study the potential response bias due to the survey sponsor identity. It is the first time sensitive data are compared using two different sources, including first-hand and original household survey data collected by NSOs. Through the systematic comparison of responses to more than twenty similarly worded questions, we first find within Afrobarometer samples that no systematic response bias due to the perceived survey sponsor can be evidenced. The existing differences in responses between AB respondents according to the perceived survey sponsor identity are attributed to a selection bias. Propensity score matching (PSM) mitigates the latter bias and confirm the absence of a difference of responses according to the perceived survey sponsor identity. The comparison of the responses to the selected questions between AB’s and NSO’s surveyed adults supports the absence of a ‘fear-of-the-state ’ b ias. Respondents to NSOs’ agents do not express a systematic more positive assessment of the national and local governance. Estimations shall not be biased by reverse causations or omitted variable biases as sampling methods in both surveys are based upon random stratification and random selection of households. Nevertheless, the results this 3 study emphasizes can only be externally valid if similar protocols than GPS-SHaSA framework are adopted when collecting sensitive data. The study does not intend to promote one survey sponsor over the other as both survey data are complementary and serve different purposes which are interdependent. Furthermore, this study does neither intend to dismiss social desirability biases in sensitive surveys. The paper is structured as follows. In section 2, we present the related literature. Then, in section 3 we present the data. Section 4 is dedicated to the empirical strategy description. Section 5 presents our results from survey comparisons and within AB analyses. We summarise our findings and add concluding remarks in section 6. 2 RELATED LITERATURE Different survey data biases have been unequally documented in the literature. Survey errors are mainly divided into sampling errors and non-sampling errors (Groves, 2004). Non- sampling errors can take the shape of non-response biases and measurement errors which can be both explained by the sensitive characteristics of a question. Tourangeau and Yan (2007) highlight three dimensions that characterize a question as sensitive. First, its intrusiveness dimension, meaning the embarrassment the question provokes. Second, the utility one finds to answer differently than she would actually think is another