Measuring the Rule of Law

Svend-Erik Skaaning, Assistant Professor, PhD Department of Political Science, University of Aarhus, Denmark [email protected]

1 Measuring the Rule of Law

Abstract

This article offers a comparative review of seven rule of law measures. The assessment demonstrates that the differences are not only a question of form, but also of appropriateness. The shortcomings are, among others, restrictions in scope and availability of disaggregate data, insufficient codebooks, and unjustified aggregation procedures.

Moreover, the indices tap into two different if related dimensions: quality of the legal system and societal order. Finally, statistical analyses show that if different indices are employed, the results change significantly. These findings suggest that more precaution is required in the development and employment of rule of law measures.

Keywords: rule of law; measurement; dataset assessment

2 Introduction Since the end of the Cold War, efforts to advance the rule of law have accompanied democracy promotion for both personal security reasons and to support political and economic development. Governance reform programs initiated by international financial institutions and national aid agencies have emphasized rule of law as a key component for human welfare, stability, and growth (Haggard et al. 2008, 206; Carothers 1998; 2006). Naturally, the issue has also received significant attention in social scientific research, and many scholars have turned to quantitative rule of law measures to facilitate their studies (e.g. Andrews and Montinola 2004; Apodaca 2004; Barro 1997; 2000; Joireman 2001; 2004; Rigobon and Rodrick 2005). The development of governance indicators advances our ability to monitor differences and changes in the quality of government across space and time. However, while rule of law violations around the world underline the existence of human rights problems, the creation of good measures is not straightforward. Such efforts only signify a step forward if the development is followed by careful attention to the value of the data in terms of their reliability, validity, and equivalence. However, in contrast to the measurement of democracy (cf. Bollen 1993; Bollen and Paxton 2000; Hadenius and Teorell 2005; Lauth 2004; Munck and Verkuilen 2002), no one has yet stepped back and systematically taken stock of the conceptualization and operationalization of the different rule of law indices framing such analyses. The present article addresses this neglected issue and provides a comparative review of seven up-to-date indices (see table 1 below) that reflect, according to their users and generators, the de facto level of rule of law in many different countries.1 The measures are provided by five organizations. The first is made available by the private operating Bertelsmann Foundation2 (BF). The work of the German foundation is funded mainly by income earned from its shares in the media corporation Bertelsmann AG. Its rule of law measure is part of a larger dataset, which is compiled to investigate the level and conditions of – and to advance – constitutional democracy combined with a socially responsible market economy. Similarly, Freedom House3 (FH) explicitly supports the expansion of freedom in the world. Freedom House4 is an independent nongovernmental organization with headquarters in Washington, D.C. It relies on grants and donations from private organizations and government agencies, first and foremost the US State Department, and is responsible for no less than three of the indices linked to three different projects, namely Freedom in the World (FW), Countries at the Crossroads (CC), and Nations in Transit (NT).

3 Global Integrity5 (GI) is another independent, non-profit organization placed in Washington, D.C. It is financed by a diverse mix of charitable foundations, governments, multilateral institutions, and the private sector. The aim of Global Integrity is to produce information on governance and corruption trends in order to improve democratic and accountable political rule. The Global Integrity index of rule of law and access to justice is included in a wider dataset on national-level anti-corruption mechanisms. The only rule of law measure, included in this review, that is supplied by a for-profit organization, the US-based Political Risk Services Group6 (PRS), is the law and order index that constitutes a fraction of the International Country Risk Guide (Political Risk Ratings). The PRS only aims to provide a comparative assessment of the political stability of the countries covered and not to improve governance around the world. In this respect, it differs from the other data generators, including the World Bank7 (WB) that has compiled the final measure put under direct scrutiny, namely the rule of law indicator from the Worldwide Governance Indicators dataset. The first part of the assessment of these indices follows the different steps of dataset assessment suggested by Gerardo Munck and Jay Verkuilen (2002). Hence, the article proceeds in five parts. The first four concern the indices’ focus and scope, conceptualization, measurement, and aggregation. The fifth evaluates whether the differences in measurement involve differences in the result when the rule of law indices are substituted for each other as dependent variables in cross- national analyses.

Focus and Scope Even though all the measures share a common focus on the rule of law, they are also marked by significant differences. One is whether the scores exclusively concern violations of rule of law committed by the government/state or if the assessments also take the society’s overall situation into consideration. Both possibilities are legitimate, but the focus depends on the particular research question, and therefore also which measure is more suitable. The scores provided by three measures – BF, GI, and one of the Freedom House ratings (CC) – only reflect the practices of the government/state and its agents. The WB and PRS indicators provide an evaluation of the general rule of law condition, while the other two Freedom House measures, FW and NT, take an intermediate position; they focus predominantly, but not exclusively, on violations committed by the government/state.

4 The information gathered in table 1 shows that none of the datasets contain scores for any years before 1984 and that more than half of them just cover four years or less. However, many of the indices show rather impressive country coverage. FW and WB simply cover all independent countries and some more or less autonomous territories. The people behind GI have the same intention, but countries are tentatively chosen according to three criteria: availability of experts and balance with regard to geography and basic freedom and civil liberties.

Table 1 about here

PRS is not linked to an explicit principle concerning the inclusion of countries. Nonetheless, it tends to be somewhat biased towards rich and large countries,8 while the novel and bi-annual BF is limited to all developing and transformation countries with more than two million inhabitants. CC and NT also focus on transformation countries. CC covers 60 countries around the world that Freedom House considers to be strategically important and at a critical crossroads in determining their political future; half of the countries are evaluated in odd years, half in even years. NT just covers all the former Communist states of Europe and Eurasia, except Mongolia. As the scope of the datasets is restricted in time and space, their value and relevance for particular research questions are limited. Especially the ability to track back in time the developments in respect for the rule of law is severely restricted. For example, none of the indices cover the whole period of the third wave of democratization (since 1975), not to speak of the post- WWII era; to mention just two demarcations often used in diachronic, empirical research. Furthermore, not all measures support direct comparisons with experiences of rich, long-enduring democracies (primarily OECD members) and/or small countries. Thus, more effort is certainly needed in expanding rule of law measures back in time. On the other hand, the scope limitations lower the risk of conceptual stretching (cf. Sartori 1970).

Conceptualization Regarding the attributes and their components, which the data generators have associated with rule of law, the concept is rather stretched as many different aspects are emphasized. But even though it is doubtful whether there is a correct specification, all definitions and specifications are not equally reasonable. They can be either too maximalist or too minimalist, or they can be infected by redundancy and conflation (Munck and Verkuilen 2002, 9-13).

5 As to the definition of rule of law, one way to address the different conceptualizations is to compare them with theoretically well-grounded classifications. According to Lon Fuller, in a country characterized by the rule of law, the law should fulfill several criteria: generality, publicity, prospectivity, clarity, non-contradictoriness, capability of compliance, stability, and congruence between declared rules and the acts of administrators (1981, 158; cf. Raz 1977, 196). In a discussion of previous attempts to reveal common principles associated with the concepts of constitutionalism, Rechtsstaat, and the rule of law, Hans-Joachim Lauth (2001, 33) recognizes these eight attributes. But he expands on the list even further to cover the fourteen attributes shown in table 2.

Table 2 about here

There are good reasons to acknowledge these elements as parts of the core concept. Even though the number of defining attributes appears rather overwhelming, they do not, by far, constitute an exhaustive list of all the elements that have been linked to the rule of law in legal and social scientific research. This is illustrated by the conceptual overview of all the indices shown in appendix A. It demonstrates the presence of interesting similarities as well as differences among the constitutive elements. Compared to the attributes presented in table 2, none of the measures cover all of them, at least not explicitly. While most measures emphasize judicial independence and equality before the law, the questions of whether the laws are publicly promulgated, clear, comprehensive, coherent, and stable are hardly addressed. On the other hand, quite a number of supplements to the list are also present among the attributes. As a result, it is possible to distinguish between at least three clusters of attributes, that is, conceptual dimensions of the rule of law. One dimension concerns the general quality of the legal system, where an independent and impartial judicial system with high integrity, respect for the decisions of the courts, due process, and equality before the law make up core attributes. A second dimension addresses the extent to which personal integrity rights, such as freedom from arbitrary arrests, police violence, and inhuman/degrading treatment, are violated by the government. Finally, a third dimension covers features related to whether order or open conflict, violence, upheavals, and crime characterize a country. Whereas all measures consider the functioning of the legal system, GI is the only dataset to focus exclusively on the first dimension. The practice of the legal system is the principal focal point of four other indices, namely BF, CC, NT, and FW, out of which only the latter addresses order

6 besides personal integrity rights. PRS pays attention to the legal system’s strength and impartiality and public respect for law obedience. The same applies to the covering of personal integrity by WB. One would think that the more dimensions an index captures, the higher the risk of maximalist conceptualizations. This is, however, not necessarily so. Actually, some of the most blatant examples of maximalism are not caused by a multidimensional view. One of the attribute components associated with FW demands an effective and democratic control of law enforcement officials. Likewise, NT emphasizes the importance of legislation that provides protections for fundamental political, civil, and human rights in addition to the state’s and nongovernmental actors’ respect for all of these fundamental rights in practice. In both cases, the conceptual domain of rule of law shows a considerable overlap with that of democracy.9 You can also question whether the level of crime and violence (PRS,10 WB), protection of property rights (CC, WB), and agents’ confidence in the rules of society (WB) should be constitutive parts of the rule of law concept. In general, one should not include attributes under the same conceptual umbrella if they are likely to be causally related. Regarding redundancy, the measures explicitly distinguish between different levels of abstraction; WB and NT only operate on two levels, though. The providers of the other indices have made efforts to sort the attributes and attribute components systematically, but they have not been equally successful in doing so. CC provides at least three redundant attribute components. For instance, if all persons are entitled to equal protection under the law, and all persons are equal before the courts and tribunals, then it is not possible to discriminate on the grounds of gender, ethnicity, or sexual orientation in the legal system.11 Similarly, concerning BF, if the separation of powers works, it means that the judiciary is independent. As to the problem of conflation, GI offers an illustration of the problem when it places the rather specific attribute ‘Judges are safe when adjudicating corruption cases’ on the same abstraction level as (e.g.) ‘Citizens have equal access to the justice system’. The conceptual bounds of the rule of law are obviously unsettled. The term ‘rule of law’ is certainly ‘used in a plethora of ways, often with different and even contradictory implications for both research and policy’ (Haggard et al. 2008, 220-221). An equivalent to the widely acknowledged – although disputed – definition of democracy (polyarchy) by Robert Dahl (1989, 221) has not yet been established. But the best answer to the problem is not necessarily to abandon the concept altogether and commence on a more disaggregated approach; only to focus, for example, on judicial independence (Ríos-Figuera and Staton 2008). Much leverage can be gained

7 by strengthening the conceptualization of the key concept that is here to stay, although essentially contested (Waldron 2002). In any case, researchers should pay attention to the conceptualization underlying the measures, since some of them may fit their research questions better than others. It can make a significant difference whether one is interested in, say, the security of persons or businesses. This final comment on the conceptualization is also worth keeping in mind when we move on to the question of measurement.

Measurement A clarification of the conceptual landscape is a prerequisite for proper measurement; that is, the operationalization of the rule of law. A codebook is necessary in order to record and make public the rules and choices guiding the coding process and to increase consistency, transparency, and replicability. As the quality of available information is often questionable and inconsistent across nations, over time, and vis-à-vis different aspects (cf. Bollen 1992, 189), it is extra important to establish equivalence through firm guidelines. As shown in table 3, all measures base their scores on a broad range of information. Information on the condition of rule of law is calibrated into numerical values according to predetermined sets of coding standards. This kind of subjective measurement has been heavily disputed because reliability problems are likely to arise owing to random and systematic measurement errors introduced by the raters who interpret the sources differently. Then again, rule of law and all its key traits are very difficult to measure through objective indicators, and the validity of the measures should generally have the highest priority (Munck and Verkuilen 2002, 18). The Freedom House measures and BF stand out for their narrative country reports, which accompany their numerical assessments, while GI takes the lead when it comes to a detailed description of the circumstances, sources, and review comments that have influenced the score of each component.

Table 3 about here

Even though a comprehensive coding manual is warranted, the Freedom House codebooks (FW, CC, NT) consist of little more than checklists defining the parameters of the indicators combined with mere standard descriptions of how to interpret the numerical scores assigned to the cases.12 Furthermore, the checklists used in the data construction have undergone changes over the years,

8 meaning that the diachronic, internal consistency of the scores is questionable. The guidelines outlined in the codebooks of BF and GI have become increasingly detailed, so that they now represent good backgrounds for the coding process. In contrast, it is questionable whether an outright PRS code manual even exists. According to the generator, the scores are assigned based on a series of pre-set questions for each component. However, neither the list of questions nor information on the methodology are made available, and both are likely to be inadequate. As to the use of coders, all datasets are based on the work of more than one coder to support intercoder reliability. Furthermore, in all cases except PRS, the scores are based on assessments made by “experts” rather than students, and the data are scrutinized through comprehensive review processes. Despite these positive features, however, statistical interrater reliability tests are missing. The overall percentage of agreement13 (or near-agreement) was published for the overall BF dataset, but only for the 2003 assignments. Among the guidelines found in the codebooks, we are often presented with the range and graduation of numerical scores to be assigned. Mostly the measurement levels of the examined measures are neither theoretically justified nor attached to discussions directed towards maximizing homogeneity within classes using a minimal number of distinctions (cf. Munck and Verkuilen 2002, 17-18). Variance truncation could pose a problem concerning the attribute components of GI and PRS, because of crude measurement scales. On the other hand, the more fine-grained options linked to the other measures are not unproblematic either; they make it more demanding to define criteria for each point and more difficult for the coders to employ them consistently.

Aggregation When the process of assignment is concluded, researchers often combine the disaggregate scores into overall indices. But this analytical step is usually not founded in theoretical and empirical justification, and the rule of law measures do not deviate from this trend. The data providers do not base their aggregations on explicit theory about the relationship between the attributes, just as they do not test the empirical dimensionality of their datasets in any systematic way.14 I have therefore carried out such tests myself, and the results (not reported) indicate that CC and BF are uni- dimensional and that GI is not. Unfortunately, the data generators of the remaining measures do not score the cases on a low level of abstraction, despite a working definition that disaggregates the main concepts (NT).15 Or they do not make their lower-level data publicly available, even if requested (FW, PRS). In this

9 way, these measures only allow their users to discriminate between countries according to their overall rule of law score without the possibility of digging a bit deeper into interesting relationships or making use of alternative aggregation procedures. It is telling that – despite the missing theoretical (and empirical) establishment of links between the constitutive attributes – all use the same aggregation rule, namely taking the simple average at the component level. Apparently, examinations of alternative aggregation procedures, such as weighting the features differently or to consider some of them as either necessary or sufficient (cf. Goertz 2006), have not been considered. Comparisons of competitive indices linked to similar concepts have often been limited to simple correlation tests, to clarify whether they tend to tap into the same latent phenomenon. Following up on this tradition, table 4 presents the bivariate correlations between the rule of law indices under scrutiny. The differences in conceptualization, sources, coding, and so on manifest themselves in some of the associations, but not all of them show the very high values one would expect from measures meant to reflect the same concept, at least in name. The correlation between CC and PRS even displays the opposite direction of the expected. Broadly speaking, CC, NT, and GI show the lowest statistical association with the other measures, whereas FW, WB, and – less so – BF are more strongly related with their rivals.

Table 4 about here

A closer look at the correlations indicates that the measures represent more than one principal dimension. The same impression evolves from a rotated factor analysis, which includes the four measures covering most countries and concerns the data from one of the only overlapping years (i.e. 2005). The results show two principal factors with eigenvalues above 1, explaining 68 per cent and 26 per cent of the variation, respectively. The first component tends to reflect the legal system’s quality, and it seems reasonable to understand the second as representing societal order, which would be in accordance with the distinctions suggested above in the section on conceptualization. WB is equally associated with both dimensions. This finding is not surprising because it reflects the fact that all the data from the original measures are used in the construction of WB, and that it is based on a very broad definition and range of sources.

Do the Differences Matter?

10 The many differences and similarities in the conceptualizations and measurements of rule of law naturally intrigue our curiosity of what would happen if these measures were employed interchangeably. It has recently been stated that robustness checks are particularly important when making claims about the effects of rule of law (Haggard et al. 2008, 222). Other researchers have already begun to take up this task (Ríos-Figuera and Staton 2008, 19-22). However, no checks on the solidity of the causes of the rule of law have so far been carried out. Thus, the four indices with the widest country coverage are used interchangeably to represent the dependent variable in two analyses. The first is a replication study of an interesting study by Sandra Joireman (2004), in which she used an institutional approach to investigate whether and how much the legal system influences the degree to which countries are characterized by the rule of law. More specifically, the hypothesis tested was whether common law systems show a better performance in this respect than civil law systems, as it has been suggested in the literature (cf. Hayek 1973; Eisenberg 1988; La Porta et al. 1999). Relying on the PRS measure, Joireman found that the proposition is only supported in those countries that have been colonized. The replication analysis employs the same method as the original study, namely the Mann- Whitney U-test. However, while Joireman focuses on the average rule of law level across years, the huge differences in diachronic coverage make it more suitable to focus on the scores for 2005, which is one of the two years of common coverage. Moreover, to increase the country coverage, the data used to distinguish common law from civil law countries, on the one hand, and between formerly colonized countries and those not colonized, on the other, come from La Porta et al. (1999).16 The findings, presented in table 5, demonstrate a significant difference in respect for the rule of law between common law and civil law countries if the WB or FW measures are used. However, the opposite conclusion is derived if we pin our faith on the PRS or BTI scores. Moreover, the same patterns in results emerge if the analysis only includes former colonies. This means that the differences in rule of law measurement do in fact make a difference, and the main conclusion of the original study by Joireman does not rest on solid empirical ground.

Table 5 about here

11 In the second analysis, the same four measure are used interchangeably as dependent variables, but another method is employed, that is, multiple OLS-regression. Furthermore, the model is expanded with additional explanatory factors used in other rule of law studies. These are: oil production (Barro 2000; Hansson and Olsson 2006), wealth (Barro 2000; Joireman 2004), country size (Hansson and Olsson 2006), ethno-religious fractionalization (Hayo and Voigt 2005; Weingast 1997; Hansson and Olsson 2006), communist past (Hoff and Stiglitz 2004; Sandholz and Taagepera 2005), and religion (Barro 2000; Hayo and Voigt 2005). The operationalization of the independent variables is presented in appendix B. Finally, to see whether the results depend on the case coverage, regressions are run for all countries and the countries included in all four datasets, respectively. The year in focus is still 2005. The results of the multiple regressions are summarized in table 6. A few explanatory variables consistently show a significant (oil production, wealth) or non-significant (ethno-religious fractionalization, common law) association with rule of law. On the other hand, the results linked to the remaining variables demonstrate interesting dissimilarities across case coverage and/or rule of law indices. For example, lifting the restriction of common inclusion of cases means that the variable Protestant turns significant for the WB and PRS indices, and insignificant for the BTI measure. Likewise, the inclusion of more countries means that communist past tends to have a negative rather than no impact on the rule of law, if the latter is measured by WB or FW

Table 6 about here

Not only do we see differences in whether the relationships are significant or not, the significant associations for some variables (colonized, communist past, Muslim) even show opposite directions. In other words, the association of the explanatory variables with rule of law ranges from positive over non-existing to negative, depending on the index used to operationalize the rule of law. Also worth mentioning are the positive, significant associations between the PRS index and communist past as well as Muslim, which contradict the theoretical expectations. Taken together, the mixed findings for many of the variables indicate that the results are not very robust. Previous general studies exploring the sources of rule of law have only used a single rule of law index. Their conclusions have thus depended heavily on the reliability and validity of the scores employed. The results from this study indicate that this problem is far from trivial, as the differences in measurement and coverage of the rule of law indices have considerable consequences

12 for the substantial findings – and much more so than in the related field of democratization studies (cf. Bollen and Paxton 2000; Casper and Tufis 2003; Hadenius and Teorell 2005).

Conclusion The review of seven rule of law measures has lent support to some general conclusions. First, the indices differ significantly on all the parameters addressed in this article, that is, focus and scope, conceptualization, measurement, aggregation, and association with suggested explanatory factors. The assessment also demonstrated that often the differences were not equally (im)plausible. In general, the justifications made in relation to the index constructions were inadequate. Among the particular shortcoming were: considerable restrictions in the coverage of years and countries, conceptual pitfalls in the form of redundancy and weak theoretical foundation, limitations in the creation and availability of disaggregate data, insufficient codebooks, and missing references to sources undermined the operationalizations. As regards the aggregation procedures, they were characterized by the default option of using unweighted averages rather than being grounded in theory and tests of empirical dimensionality. In fact, the rule of law measures empirically (and conceptually) tapped into at least two distinct dimensions; one is reflecting the quality of the legal system and the other is capturing the level of order in society. The considerable differences were also underlined by replication studies and a multiple-regression analysis, which showed that replacing the rule of law measures – as the dependent variable – changed the results significantly. Future research can benefit in three ways from the explications and critical points put forward in this assessment. First, the findings provide implicit and explicit suggestions for improving the existing measures. Second, we can learn from the advantages and disadvantages of previous measures when we set out to construct new datasets and indices related to good governance in general and rule of law in particular. Third, the shortcomings of – and differences in – the leading measures call for re-examinations of the many studies that, in one way or another, have used them as rule of law indicators. All in all, the findings suggest that we have to be more careful when developing and applying rule of law indicators, but not that we should give up our search for better measurement tools.

13 Appendix A: Attributes Linked to the Rule of Law Measures

Measure Attributes Attribute Components Working separation of powers Checks and balances Independent judiciary Rule of Law Legal or political penalties for power abusing officials (BF) Civil rights Protection of civil liberties and possibility of seeking redress for violations

Judiciary not subject to interference Judges appointed and dismissed in a fair and unbiased manner Independent judiciary Judges rule fairly and impartially Governmental authorities comply with judicial decisions which are enforced Private concerns comply with judicial decisions which are enforced

Protection of defendants’ rights, including presumption of innocence Access to independent, competent legal counsel Rule of law in civil and Fair, public, and timely hearing by competent, independent, and impartial tribunal Prosecutors independent of political control and influence criminal matters Prosecutors independent of powerful private legal/illegal interests Rule of Law Effective and democratic control of law enforcement officials Law enforcement officials free from the influence of non-state actors (FW) Protection from political No arbitrary arrests and detentions and no fabrication or planting of evidence terror, unjustified No beating of detainees or use of excessive force or torture imprisonment, exile, torture, Conditions in pretrial facilities and prisons humane and freedom from war and Citizens have means of effective petition and redress insurgencies Population not subjected to physical harm, forced removal etc. due to conflict/war Various distinct groups have equality before the law Laws, policies, and practices Violence against such groups not widespread and perpetrators brought to justice No legal/de facto discrimination in employment, housing etc. against such groups guarantee equal treatment Women enjoy full equality in law and practice Non-citizens enjoy basic human rights Independent, impartial, and non-discriminating administration of justice Judges/magistrates protected from interference by executive and legislative Independent judiciary Governmental authorities comply with judicial decisions Judges appointed, promoted, and dismissed in a fair and unbiased manner Judges appropriately trained Presumption of innocence Fair, public, and timely hearing by competent, independent, and impartial tribunal Primacy of rule of law in civil Rights and access to independent counsel and criminal matters Access to independent counsel if beyond means when serious felonies Prosecutors independent of political direction and control Rule of Law Public officials and ruling party actors prosecuted for abuse of power (CC) Accountability of security Effective and democratic control of the police, military, and security service Police, military, and security services refrain from interference in politics forces and military to civilian Police, military, and security services accountable for abuses of power authorities Members of police, military, and security services respect human rights

Right to own property alone and in association with others Protection of property rights Adequate enforcement of property rights and contracts Protection from arbitrary/unjust deprivation of property

All persons entitled to equal protection under the law Equal treatment under the law All persons equal before the courts and tribunals No discrimination on the grounds of gender, ethnicity, or sexual orientation Judicial Legislation provides protections for fundamental political, civil, and human rights Framework and Independence The state and nongovernmental actors respect fundamental rights in practice (NT) Independence and impartiality in the interpretation and enforcement of constitution Equality before the law Effective reform of the criminal code/criminal law

14 Suspects and prisoners protected in practice against arbitrary arrest, detention without trial, searches without warrants, torture, and abuse, and excessive delays in the criminal justice system Judges are appointed in a fair and unbiased manner, and they have adequate legal training before assuming the bench Judges rule fairly and impartially, and courts are free of political control and influence Legislative, executive, and other governmental authorities comply with judicial decisions, and judicial decisions are enforced effectively

Appeals mechanism for In law, there is a general right of appeal challenging criminal In practice, appeals are resolved within a reasonable time period judgments In practice, citizens can use the appeals mechanism at a reasonable cost Judgments in the criminal In practice, judgments in the criminal system follow written law system follow written law Judicial decisions are enforced In practice, judicial decisions are enforced by the state by the state In law, the independence of the judiciary is guaranteed Rule of Law and The judiciary is able to act In practice, national-level judges are protected from political interference Access to Justice independently In law, the distributing cases to national-level judges is transparent and objective (GI) In law, national-level judges protected from removal without relevant justification Judges are safe when In practice, no judges physically harmed because of adjudicating corruption cases adjudicating corruption cases In practice, no judges killed because of adjudicating corruption cases In practice, judicial decisions are not affected by racial or ethnic bias In practice, women have full access to the judicial system Citizens have equal access to In law, provision of legal counsel for defendants who cannot afford it In practice, provision of adequate legal counsel for defendants who cannot afford it the justice system In practice, citizens with median yearly income can afford to bring a legal suit In practice, a typical small retail business can afford to bring a legal suit In practice, all citizens have access to a court of law, regardless of location

Law and Order Law Strength and impartiality of the legal system (PRS) Order Popular observance of the law Agents have confidence in the rules of society Agents abide by the rules of society High quality contract enforcement Rule of Law High quality police (WB) High quality courts Low likelihood of crime Low likelihood of violence

15 Appendix B: Operationalization of Explanatory Factors

Oil production. Oil production is operationalized by using IMF’s (2007) list of hydro-carbon rich countries (2000-2005) found in the Guide on Resource Transparency. These countries rely heavily on oil production for government revenues and receive the value of 1 and the rest a 0 meaning that the variable is treated as a dichotomy. Wealth. Wealth is measured by a standard wealth indicator, namely (natural log of) GDP per capita (PPP) using data from the Penn World Tables for the year 2004. Country size. The (natural) log of a country’s total area in square kilometres is used to measure this variable, based on data from the World Development Indicators provided by the World Bank (2008). Ethno-religious fractionalization. Data on the degree of heterogeneity are taken from Alesina et al. (2003), who have constructed scores of ethnic and religious fractionalization. The values of the indices correspond to the probability that two randomly selected people belong to different ethnic and religious groups, respectively; meaning that a high value on the index corresponds to a high degree of fractionalization in a given country (2003, 158-160). As the same logic applies to both ethnic and religious fractionalization, a combined measure has been constructed based on the maximum value of the two original indices for each country. Communist past. A dummy variable for communist and post-communist countries has been constructed. Dominant religion. Following the procedure used by M. Steven Fish (2002), two dummy variables have been created distinguishing between countries where Islam and Protestantism are the dominant (plurality or majority) religions, respectively.

16 References Alesina, Alberto, Arnaud Devleeschauwer, William Easterly, Sergio Kurlat, and Romain Wacziarg. 2003. Fractionalization. Journal of Economic Growth 8:155-94. Andrews, Josephine T., and Gabriella R. Montinola. 2004. Veto Players and the Rule of Law in Emerging Democracies. Comparative Political Studies 37:55-87. Apodaca, Clair. 2004. The Rule of Law and Human Rights. Judicature 87:292-99. Barro, Robert (1997). Determinants of Economic Growth: A Cross-Country Empirical Study. Cambridge, Mass: MIT Press. Barro, Robert. 2000. Rule of Law, Democracy, and Economic Performance. In 2000 Index of

Economic Freedom, eds. Gerald P. O’Driscoll Jr., Kim R. Holmes, and Melanie Kirkpatrick.

Washington and New York: Heritage Foundation and Wall Street Journal.

Bollen, Kenneth A. 1992. Political Rights and Political Liberties. In Human Rights and Statistics: Getting the Record Straight, ed. Thomas B. Jabine, and Richard P. Claude. Philadelphia: University of Pennsylvania Press. Bollen, Kenneth A. 1993. Liberal Democracy: Validity and Method Factors in Cross-National Measures. American Journal of Political Science 37:1207-30. Bollen, Kenneth A., and Pamela Paxton. 2000. Subjective Measures of Political Democracy. Comparative Political Studies 33:58-86. Carothers, Thomas. 1998. The Rule of Law Revival. Foreign Affairs 77:95-106. Carothers, Thomas. 2006. Promoting the Rule of Law Abroad: The Search for Knowledge. Washington, DC: Carnegie Endowment for International Peace. Casper, Gretchen, and Claudio Tufis. 2003. Correlation versus Interchangeability: The Limited Robustness of Empirical Findings on Democracy using Highly Correlated Datasets. Political Analysis 11:196-203. Chomsky, Noam, and Edward S. Herman. 1988. Manufacturing Consent. New York: Pantheon Books. Dahl, Robert A. 1989. Democracy and Its Critics. New Haven: Yale University Press. Eisenberg, Melvin A. 1988. The Nature of the Common Law. Cambridge, Mass.: Harvard University Press. Fish, M. Steven. 2002. Islam and Authoritarianism. World Politics 55:4-37.

17 Fuller, Lon L. 1981. The Principles of Social Order. Durham: Duke University Press. Goertz, Gary. 2006. Social Science Concepts: A User’s Guide. Princeton: Princeton University

Press.

Hadenius, Axel, and Jan Teorell. 2005. Assessing Alternative Indices of Democracy. IPSA

Committee on Concepts and Methods Working Paper Series (Political Concepts No. 6).

Haggard, Stephan, Andrew MacIntyre, and Lydia Tiede. 2008. .“The Rule of Law and Economic Development. Annual Review of Political Science 11:205-34. Hansson, Gustav, and Ola Olsson. 2006. Country Size and the Rule of Law: Resuscitating Montesquieu. Working Paper 200, School of Economics: Göteborg University. Hayek, Friedrich A. 1973. Law, Legislation, and Liberty. Chicago: University of Chicago Press. Hayo, Bernd, and Stefan Voigt. 2005. Explaining de facto judicial independence. Philipps- Universität Marburg, Working Paper No. 200507. Hoff, Karla, and Joseph Stiglitz. 2004. After The Big Bang? Obstacles to the Emergence of Rule of

Law in Post-Communist Societies. American Economic Review 94:753-63.

Joireman, Sandra F. 2001. Inherited Legal Systems and Effective Rule of Law: Africa and the Colonial Legacy. Journal of Modern African Studies.39:571-96. Joireman, Sandra F. 2004. Colonization and the Rule of Law: Comparing the Effectiveness of Common Law and Civil Law Countries. Constitutional Political Economy 15:315-38. La Porta, Rafael, Florencio Lopez-de-Silanes, Andrei Shleifer, and Robert Vishny. 1999. The Quality of Government. Journal of Law, Economics, and Organization 15:222-79. Lauth, Hans-Joachim. 2001. Rechtsstaat, Rechtssysteme und Demokratie. In Rechtsstaat und

Demokratie, ed. Michael Becker, Hans-Joachim Lauth, and Gert Pickel. Opladen: Leske +

Budrich.

Lauth, Hans-Joachim. 2004. Demokratie und Demokratiemessung: Eine konzeptionelle Grundlegung für den interkulturellen Vergleich. Wiesbaden: Verlag für Sozialwissenschaften. Munck, Gerardo, and Jay Verkuilen. 2002. Conceptualizing and Measuring Democracy: Evaluating Alternative Indices. Comparative Political Studies 35:5-34. Raz, Joseph. 1977. The Rule of Law and Its Virtue. Law Quarterly Review 93:185-211.

18 Rigobon, Roberto, and Dani Rodrick. 2005. Rule of Law, Democracy, Openness, and Income. The Economics of Transitions 13:533-64. Ríos-Figueroa, Julio, and Jeffrey K. Staton. 2008. Unpacking the Rule of Law: A Review of Judicial

Independence Measures. IPSA Committee on Concepts and Methods Working Paper Series

(Political Concepts No. 21).

Sandholtz, Wayne, and Rein Taagepera. 2005. Corruption, Culture, and Communism. International Review of Sociology 15:109-31. Sartori, Giovanni. 1970. Concept Misformation in Comparative Politics. American Political Science Review 64:1033-53. Waldron, Jeremy. 2002. Is the Rule of Law an Essentially Contested Concept (in Florida). Law & Philosophy 21:137-64. Weingast, Barry R. 1997. The Political Foundations of Democracy and the Rule of Law. American Political Science Review 91: 245-63.

19 Table 1 Generator, Dataset, and Scope of the Rule of Law Measures Generator Dataset Index Countries17 Years 2003, 2005, Bertelsmann Foundation Bertelsmann Transformation Index Rule of Law (BF) 125 2007

Freedom House Freedom in the World Rule of Law (FW) 193 2005-2007

Freedom House Countries at the Crossroads Rule of Law (CC) 30(60) 2003-2006

Judicial Framework and Freedom House Nations in Transit 29 1996-2007 Independence (NT) Rule of Law and Access 2003, 2006, Global Integrity Global Integrity Index 55 to Justice (GI) 2007

The PRS Group International Country Risk Guide Law and Order (PRS) 139 1984-2007 1996, 1998, World Bank Worldwide Governance Indicators Rule of Law (WB) 212 2000, 2002- 2007

20 Table 2 Principles of the Rule of Law 1. General laws and not ad personam 2. Publicly promulgated laws 3. Prohibition against retrospective laws 4. Clear and comprehensive laws 5. No contradictory or inconsistent laws 6. No law should be impossible to respect 7. Relative stability in the lawmaking 8. Proportionality in the law 9. Equality before the law 10. Precedence of law, no one is above the law 11. Independent and effective juridical control 12. Due process 13. Liability to pay compensation for any damage caused 14. Absence of arbitrary state action and promotion of legality

21 Table 3 Sources and Coders

Measure Sources Coders

Expert. Review by expert from Rule of Law (BF) Broad range of information country/region, regional coordinator, and academic board Researcher/outside consultant. Broad range of information, including information from news Review by analysts and reports, academic analyses, organizations, professionals Rule of Law (FW) academics with regional contacts, and visits (general list of reports used for the survey) expertise Scholar/analyst. Review by regional expert committee and Broad range of information (references in narrative report) Rule of Law (CC) internal Freedom House committee Judicial Framework and Expert. Review by academic Broad range of information (references in narrative report) Independence (NT) advisors and Freedom House Rule of Law and Access to Justice In country reporters/researchers. Broad range of information (precise references in report) (GI) Review by Global Integrity

PRS staff with special country Broad range of information Law and Order (PRS) expertise

Second-hand scores from eleven representative and thirteen (no basic coding) Rule of Law (WB) non-representative sources

22 Table 4 Correlations between the Rule of Law Indices Rule of Rule of Rule of Judicial Rule of Law Law Rule of Law Law Law Framework and and Access to and Law (BF) (FW) (CC) Independence Justice Order (WB) (NT) (GI) (PRS) 1.00 0.93 0.45 -0.92 0.24 0.19 0.68 Rule of Law (BF) (359) (244) (56) (81) (58) (303) (359) 0.93 1.00 0.71 -0.95 0.45 0.52 0.80 Rule of Law (FW) (244) (578) (60) (83) (89) (416) (577) 0.45 0.71 1.00 -0.86 0.06 -0.22 0.31 Rule of Law (CC) (56) (60) (120) (17) (14) (94) (120) Judicial Framework and -0.92 -0.95 -0.86 1.00 -0.47 -0.36 -0.92 Independence (NT) (81) (83) (17) (331) (25) (210) (248) Rule of Law and Access 0.24 0.45 0.06 -0.47 1.00 0.29 0.60 to Justice (GI) (58) (89) (14) (25) (114) (100) (114) 0.19 0.52 -0.22 -0.36 0.29 1.00 0.78 Law and Order (PRS) (303) (416) (94) (210) (100) (3167) (1240) 0.68 0.80 0.31 -0.92 0.60 0.78 1.00 Rule of Law (WB) (359) (577) (120) (248) (114) (1240) (1800) Factor loadings (factor 1) 0.99 0.98 - - - -0.10 0.55 Factor loadings (factor 2) -0.04 -0.04 - - - 1.00 0.60 Note: Results refer to bivariate Pearson’s r correlations (n in parentheses) and a principal component factor analysis using Oblique rotation.

23 Table 5 Mean Comparisons – Common Law vs. Civil Law N Mean Rank Sum of Ranks Mann-Whitney U Asymp. Sig. 138 (civil law) 96.11 13263.5 WB2005 (all) 72 (common law) 123.49 8891.5 3672.5 0.002** 98 (civil law) 70.44 6903.5 PRS 2005 (all) 41 (common law) 68.94 2826.5 1965.5 0.839 128 (civil law) 91.18 11671.0 FW2005 (all) 64 (common law 107.14 6857.0 3415.0 0.060* 91 (civil law) 58.91 5360.5 BTI2005 (all) 28 (common law) 63.55 1779.5 1174.5 0.533 118 (civil law) 80.03 9443.5 WB2005 (col.) 67 (common law) 115.84 7761.5 2422.5 0.000** 81 (civil law) 57.80 4682.0 PRS 2005 (col.) 37 (common law) 63.22 2339.0 1361.0 0.420 108 (civil law) 75.05 8105.5 FW2005 (col.) 59 (common law) 100.38 5922.5 2219.5 0.001** 86 (civil law) 53.98 4642.0 BTI2005 (col.) 901.0 0.343 24 (common law) 60.96 1463.0 Note: *p<0.1, **p<0.01 (two-tailed).

24 Table 6 Summary of Regression Results with Rule of Law Indices as Dependent Variable

WB05 PRS05 FW05 BTI05

Oil Production - (-) - (-) - (-) - (-)

Wealth + (+) + (+) + (+) + (+)

Country Size - (-) ns (ns) - (ns) ns (ns)

Ethno-religious ns (ns) ns (ns) ns (ns) ns (ns) Fractionalization

Not Colonized + (ns) + (+) ns (-) - (-)

Common Law ns (ns) ns (ns) ns (ns) ns (ns)

Communist Past - (ns) + (+) - (ns) ns (ns)

Muslim ns (ns) + (+) - (-) - (-)

Protestant + (ns) + (ns) + (+) ns (+)

Note: + denotes a positive, significant association; - denotes a negative, significant association; ns denotes an insignificant association (p>0.1); bracketed results refer to the analysis with restricted (i.e., common) country coverage.

25 Notes

26 1 All descriptive information on the datasets is found on the webpages of the providers if no other source or reference is mentioned.

2 http://www.bertelsmann-transformation-index.de

3 http://www.freedomhouse.org

4 This organization has faced serious critique; it has been accused of having a right-wing bias and to overstate the level of freedom in ‘US-friendly’ countries (Bollen 1992, 205; Chomsky and Herman 1988).

5 http://www.globalintegrity.org

6 http://www.prsgroup.com

7 http://info.worldbank.org/governance/wgi2007

8 This impression is supported by the results (not reported) of a simple statistical test showing that, on average, the countries included in the sample are significantly larger (measured by area) and richer (measured by GDP/capita) than the ones excluded.

9 This critique also applies to the third Freedom House measure, since CC, among others, builds upon the attributes effective and democratic control of the police, military, and security service and that police, military, and security services refrain from interference in politics.

10 PRS is also characterized by the opposite pitfall, namely excessive minimalism, as popular observance of the law is not enough to secure order (the state and foreign powers could also ruin it). PRS is generally defined in unspecific and vague terms, which make it difficult to get a grip on its intension. On the other hand, the simplicity of this measure minimizes the risks of conceptual conflation and redundancy.

11 Similarly, if there is adequate enforcement of property rights and contracts, protection from arbitrary/unjust deprivation of property is in place, and if an independent, impartial, and non-discriminating administration of justice exists, judges/magistrates do not experience interference by executive and legislative.

12 This point also applies to GI.

13 This simple measure has several more sophisticated alternatives, such as Cohen’s kappa and Krippendorf’s alpha.

14 WB constitutes an exception to the latter point, as it is based on the use of an unobserved components model on the data from different sources (including the scores of the other measures examined in this article). The values of WB are weighted averages of the underlying data, with weights reflecting the precision of the individual data sources.

15 No aggregation rule is applied in the construction of the Bertelsmann rule of law scale for 2003, as the cases are not coded at a disaggregate level.

16 The few missing values have been replaced by scores based on information from CIA’s The World Fact Book, https://www.cia.gov/library/publications/the-world-factbook. 17 The number of countries concerns the latest year covered. Most datasets have increased the number of countries covered during their lifetime.