How to Estimate Better Left-Right Positions by Scaling Party Manifesto Data

Putting Text in Context: How to Estimate Better Left-Right Positions by Scaling Party Manifesto Data

Kenneth Benoit§ Thomas D¨aubler∗

§LSE and Trinity College Dublin

∗University of Mannheim and MZES

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data Questions

I How should we understand the “left-right dimension”?

I How should we construct a left-right index using content analysis results from political documents (here: manifestos)?

I What does this say about the past, present, and future of measuring left-right?

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data I a posteriori: estimate a scale using inductive methods from data (higher-dimensional measurements)

I One common application: estimating left-right from category counts based on human-coding of policy documents

I The CMP/MARPOR’s main deliverable is its ﬁxed scale: RILE “Its a priori, deductive nature is important in allowing its application in all places at all times without the qualiﬁcations about content or context which apply to inductive scales. It is a substantively invariant measure whose numeric values always carry the same meaning.” (Budge and Meyer 2013: 88)

Locating Parties on a Left-Right Dimension

I Two approaches

I a priori: specify a ﬁxed scale in advance

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data I One common application: estimating left-right from category counts based on human-coding of policy documents

Locating Parties on a Left-Right Dimension

I Two approaches

I a priori: specify a ﬁxed scale in advance I a posteriori: estimate a scale using inductive methods from data (higher-dimensional measurements)

Kenneth Benoit and Thomas Däubler Scaling Manifesto Data I The CMP/MARPOR’s main deliverable is its fixed scale: RILE “Its a priori, deductive nature is important in allowing its application in all places at all times without the qualifications about content or context which apply to inductive scales. It is a substantively invariant measure whose numeric values always carry the same meaning.” (Budge and Meyer 2013: 88)

Locating Parties on a Left-Right Dimension

I Two approaches

I a priori: specify a ﬁxed scale in advance I a posteriori: estimate a scale using inductive methods from data (higher-dimensional measurements)

I One common application: estimating left-right from category counts based on human-coding of policy documents

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data Locating Parties on a Left-Right Dimension

I Two approaches

I a priori: specify a ﬁxed scale in advance I a posteriori: estimate a scale using inductive methods from data (higher-dimensional measurements)

I One common application: estimating left-right from category counts based on human-coding of policy documents

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data I Let yij represent the counts of coded text units from i = 1, ..., N documents, falling into category j = 1, ..., J.

yij ∼ Neg.Binomial(µij , φj )

log(µij ) = αi + ζj + λj θi

I Interpretation: αi the variable length of the document (total length) ζj the baseline frequency of a category (issue coverage) λj responsiveness of the category to the latent variable θi the latent variable (e.g. left-right position) 1/φj overdispersion parameter

An IRT Model for Counts of Unordered Categorical Outcomes

I IRT modelling (Rasch 1960, Bock 1972)

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data An IRT Model for Counts of Unordered Categorical Outcomes

I IRT modelling (Rasch 1960, Bock 1972)

I Let yij represent the counts of coded text units from i = 1, ..., N documents, falling into category j = 1, ..., J.

yij ∼ Neg.Binomial(µij , φj )

log(µij ) = αi + ζj + λj θi

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data Parameters and Outcomes

Expected category counts (α = 1 and all ζ = 1) Item response category functions (any α, all ζ = 1) 150 1.0

λ = − 1 λ = 1 λ = − 1 λ = 1 0.8 100 0.6 0.4 50 λ = − 0.5 λ = 0.5 Expected number of items Expected number

λ = − 0.5 λ = 0.5 Expected probability/proportion 0.2 λ = 0 λ = 0 0 0.0

−3 −2 −1 0 1 2 3 −3 −2 −1 0 1 2 3

Latent position θ Latent position θ

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data Identiﬁcation

I There are ﬁve fundamental indeterminacies

I The following constraints constitute one way of identifying the model:

ζj = 0

λ¯j = 0

θi ∼ N(0, 1)

λj > λj0

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data Inference

I Bayesian approach (using HMC in Stan) with priors as follows:

αi ∼ N(µα, σα)

ζj ∼ N(µζ , σζ )

λj ∼ N(0, σλ)

θi ∼ N(0, 1)

µα ∼ N(0, 5)

µζ ∼ N(0, 5)

σα ∼ Half-Cauchy(0, 5)

σζ ∼ Half-Cauchy(0, 5)

σλ ∼ Half-Cauchy(0, 5) 1/φ ∼ Uniform(0, 200)

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data The CMP/MARPOR Data

I Human coding of 3000+ election manifestos from 50+ countries

I Standard coding scheme has 56 categories, some directed, others not

I Frequently used “canned” RILE index

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data And Now for Some Results...

I Sample of 2288 manifestos from post-1972 democracies

I The most and least frequent categories (ζ values):

Top 10 Categories Bottom 10 Categories Category ζ Category ζ Welfare State Expansion + 2.20 Marxist Analysis + -3.37 Environm. Protection + 1.72 Education Limitation + -3.22 Technology and Infrastr. + 1.70 Labour Groups - -2.71 Education Expansion + 1.66 National Way of Life - -2.18 Political Authority + 1.63 Centralisation + -1.86 Social Justice + 1.53 Foreign Special Rel. - -1.46 Gov. and Admin. Eﬃc. + 1.50 Protectionism - -1.45 Non-Econ. Demogr. Gr. + 1.37 Constitutionalism - -1.29 Democracy + 1.35 Corporatism + -1.23 Farmers + 1.25 Keynesian Dem. Man. + -1.15

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data Left-Right Positions in Italy: CMP Rile

2008 PdL People of Freedom ● IdV List Di Pietro − Italy of Values ● PD Democratic Party ● LN Northern League ● UdC Union of the Center ●

2013 UdC Union of the Center ● CD Democratic Centre ● FDI−CDN Brothers of Italy ● SVP South Tyrolean People’s Party ● PdL People of Freedom ● SC Civic Choice ● 3L Labour and Freedom List ● PD Democratic Party ● Autonomy Progress Federalism ● SEL Left Ecology Freedom ● RC Civil Revolution ● M5S Five Star Movement ●

−40 −20 0 20 40 CMP rile

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data Left-Right Positions in Italy: Poisson IRT

2008 PdL People of Freedom ● UdC Union of the Center ● PD Democratic Party ● IdV List Di Pietro − Italy of Values ● LN Northern League ●

2013 PdL People of Freedom ● FDI−CDN Brothers of Italy ● SC Civic Choice ● CD Democratic Centre ● 3L Labour and Freedom List ● SVP South Tyrolean People’s Party ● UdC Union of the Center ● Autonomy Progress Federalism ● RC Civil Revolution ● M5S Five Star Movement ● SEL Left Ecology Freedom ● PD Democratic Party ●

−2 −1 0 1 2 Scaling Estimates of Theta

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data Left-Right Positions in Italy: NegBin IRT

2008 PdL People of Freedom ● PD Democratic Party ● UdC Union of the Center ● IdV List Di Pietro − Italy of Values ● LN Northern League ●

2013 FDI−CDN Brothers of Italy ● UdC Union of the Center ● CD Democratic Centre ● PdL People of Freedom ● SC Civic Choice ● SVP South Tyrolean People’s Party ● 3L Labour and Freedom List ● Autonomy Progress Federalism ● PD Democratic Party ● RC Civil Revolution ● SEL Left Ecology Freedom ● M5S Five Star Movement ●

−2 −1 0 1 2 Scaling Estimates of Theta

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data Validation with Expert Survey Placements

r = 0.82 W Europe

2 r = 0.55 E Europe r = 0.89 Pacific & N America ● ● ● r = 0.75 All ●

1 ●

● ● ● ● ●

● ● 0 ● ● ●

●

−1 ● Scaling Estimates of Theta

●

● −2 −3

2 4 6 8 10

Expert Survey Estimates of Left−Right

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data Country-by-Country Validation (vs. CMP Rile)

● ● 1.0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●

● ● 0.8 ● ● ● ● ●

●

● 0.6

● Corr. of IRT w experts of IRT Corr.

● ● ● 0.4

● 0.2

0.2 0.4 0.6 0.8 1.0 Corr. of CMP rile w experts

Romania not shown (r for rile -.55, r for IRT theta -.44)

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data λˆ (post-1972 democracies)

Right Welfare State Limitation + ● Free Enterprise + ● Traditional Morality + ● Military + ● National Way of Life + ● Economic Orthodoxy + ● Protectionism − ● Incentives + ● Law and Order + ● Political Authority + ● Social Harmony + ● Constitutionalism + ● Freedom and Hum. Rights + ● Neutral Labour Groups − ● Education Limitation + ● Multiculturalism − ● Gov. and Admin. Effic. + ● Productivity + ● Farmers + ● Centralisation + ● European Integration + ● Technology and Infrastr. + ● Middle Class, Profess. Gr. + ● Economic Goals ● Constitutionalism − ● Foreign Special Rel. + ● Corporatism + ● Internationalism − ● Culture + ● Political Corruption − ● Decentralisation + ● Non−Econ. Demogr. Gr. + ● Foreign Special Rel. − ● Keynesian Dem. Man. + ● Underpriv. Minority Gr. + ● Environm. Protection + ● European Integration − ● Multiculturalism + ● Social Justice + ● Traditional Morality − ● Anti−Growth Economy + ● National Way of Life − ● Marxist Analysis + ● Left Education Expansion + ● Protectionism + ● Welfare State Expansion + ● Market Regulation + ● Internationalism + ● Economic Planning + ● Democracy + ● Labour Groups + ● Controlled Economy + ● Peace + ● Anti−Imperial./−Colonial. ● Nationalisation + ● Military − ●

−3 −2 −1 0 1 2 3

λ^ λˆ (post-1972 democracies) 3 3 R = 0.37 Rile RIGHT R = 0.64 Rile RIGHT 2 2 702

505 414

1 1 608 401 303 305704

j 110 603

^ 601

λ 410 104 411304704 j 303605 410402

605 ^ 402 505702 λ 606 407 104 604 506408 409 408 507 701403504502 414 507 602 108 416501503404405703108302109204 110506 0 406 0 415 404703201102101 413 107606305101 301504411304302 412 202706 608601 401 413 503202706 603 701 109 301 Pacific 203 105 409705201 407 412 403107406 E Europe 415 106 204 102 103 405 203 501 502 103 607 416 −1 −1 105 705 602 604607 106 −2 −2

Rile LEFT Rile LEFT −3 −3

−3 −2 −1 0 1 2 3 −3 −2 −1 0 1 2 3 ^ ^ Western Europe λj Western Europe λj 3 3 R = 0.49 Rile RIGHT R = −0.097 Rile RIGHT 2 2

603

1 505 1 401 406 j j ^ λ 606303 408 412 ^ 101 104 λ 304 102204605 107 602 201203109 702 103105 503 703201101 110304704 402 416106404202110305102203108302109303204 414 702 0 415 103 106301107305506108 608507 0 415 409607405 608601507401 105 404607202 408 407414 413 602 502606704 505 409705405411 601 604501504706411 407 104 412701706703410 301 402 604503 302 705 506 603 413 403504406 East Far 701 410605 N America 416501 403 502 −1 −1 −2 −2

Rile LEFT Rile LEFT −3 −3

−3 −2 −1 0 1 2 3 −3 −2 −1 0 1 2 3 ^ ^ Western Europe λj Western Europe λj Take Me to Another Dimension

I In the two-dimensional case, we model the mean of the counts as:

log(µij ) = αi + ζj + λ1j θ1i + λ2j θ2i

I The model requires 13 constraints for identiﬁcation

I Mean of positions zero in each dimension I “Dimension founders” I 1 single, 1 single, 1 double, 2 further zero constraints on λdj I Reference category for ζj

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data ˆ λd for 2D-model (post-1972 democracies)

Pure Economic Economic and Social ● Labour Groups − ● Education Limitation + ● Internationalism − Free Enterprise + ● Political Authority + ● ● Welfare State Limitation + Foreign Special Rel. − ● Economic Orthodoxy + ● Anti−Imperial./−Colonial. ● Protectionism − ● Incentives + ● European Integration − ● Productivity + ● Military + ● ● Technology and Infrastr. + Centralisation + ● Economic Goals ● ● Economic Planning + ● Constitutionalism + Education Expansion + ● Foreign Special Rel. + ● ● Middle Class, Profess. Gr. + ● ● Gov. and Admin. Effic. + Corporatism + ● Protectionism + ● Social Justice + Welfare State Expansion + ● Constitutionalism − ● ● Market Regulation + Internationalism + ● ● Keynesian Dem. Man. + ● Labour Groups + ● Peace + Controlled Economy + ● European Integration + ● Anti−Growth Economy + ● Farmers + ● Nationalisation + ● Culture + ● Marxist Analysis + ● Military − ● Economic and Social Environm. Protection + ● ● Military + Decentralisation + ● Farmers + ● Decentralisation + ● European Integration + ● Pure Social ● Culture + ● ● National Way of Life + Gov. and Admin. Effic. + ● Foreign Special Rel. + ● Multiculturalism − Constitutionalism − ● Traditional Morality + ● ● Constitutionalism + Law and Order + ● ● Environm. Protection + ● Internationalism + ● Social Harmony + Centralisation + ● Freedom and Hum. Rights + ● Peace + ● Democracy + ● Social Justice + ● Non−Econ. Demogr. Gr. + ● Foreign Special Rel. − ● ● Political Authority + ● Underpriv. Minority Gr. + Military − ● Traditional Morality − ● ● European Integration − Multiculturalism + ● Anti−Imperial./−Colonial. ● ● Internationalism − ● National Way of Life −

−3 −2 −1 0 1 2 3 −3 −2 −1 0 1 2 3 β^ β^

(a) Economic Dimension (b) Social Dimension Validation with Expert Survey Placements - 2D

Economic Social

r = 0.73 ● ● ● ● ●

2 ● ● r = 0.53 ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ●● ●● 2 ● ● ● ● ● ● ● ● ●●● ● ● ● ●● ● ●● ●● ● ● ● ● ● ● ● ●● ● ●● ●●●● ●●● ●● ● ● ● ● ● ● ● ● ● ●● ●● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●●●●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ●●●● ●● ● ● ●●● ● ●● ● ● ●●● ● ● ● ●● ●● ●● ● ●● ●●● ● ●●●● ● ●● ● ● ● ● ● ● ●● ●● ●●●● ● ●●●●● ●●● ●●● ● ● ●● ● ● ●● ● ● ● ●● ● ● ●● ●●●● ●●● ●●●●●●●● ●●●●●●● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ●●●● ●● ● ●●● ● ●● ●● ● ● ● ● ●●● ● ● ●●● ● ● ●● ●● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ●● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ●● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●●● ●● ●● ● ●● ●● ● ●● ●● ●● ●● ●● ● ● ● ●● ● ● ● ●● ●●● ● ● ●● ● ● ●●● ● ● ● ● ●●●● ● ● ● ● ●●● ● ●● ●● ● ● ● ●● ●● ● ●● ● ●●● ● 0 ●●●● ● ● ● ●●●● ●●● ●● ● ●● ● ● ● ● ● ● ●●●●●● ●● ● ●●● ● ● ● ● ● ● ●● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●●● ●● ● ● ● ● ●●●●●● ●● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ●● ●● ● ● ● ● ● ● ● ● ● ●● ●●● ●●●●● ● ● ● ● ● ● ● ●● ●● ● ●● ● ● ● ● ● ● ● ●●● ● ● ● ● ●●●● ●● ● ● ● ● ● ● ● ●●● ● ●●●● ●● ● ● ● ● ●●●●● ● ●● ● ● ● ● ●● ●●● ● ●● ● ● ●●● ● ● ●● ● ●● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●●● ● ●●● ● ●● ● ● ● ● ● ● ● ●● ● ● ●●●●● ● ●●● ● ● ●●●● ● ●●● ● ● ● ● ● ● ● ● ● ● ● 0 ● ● ● ● ●● ●●● ● ● ● ●● ● ● ● ●● ●● ●● ● ●● ●● ● ● ●●●● ● ● ● ● ● ● ● ● ●●●●● ● ● ● ● ● ●● ●●● ● ● ●● ● ●● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ●● ● ● ● ● ●● ● ● ●●●●● ●● ● ● ● ● ● ●● ● ● ● ●● ●● ● ●● ●● ●●●● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ●●● ● ● ● ●● ● ●● ●●● ● ● ● ●●● ● ●● ● ● ● ● ● ●●●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●●● ● ●● ● ● ●● ● ● ● ● ●●● ●● ● ● ● ●●● ●● ● ● ● ● ● ● ●● ● ● ●● ● −2 ●●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● −2 ● ● ● ●

● ● ●

−4 ● Scaling Estimates of Theta (Social Dimension) ● Scaling Estimates of Theta (Economic Dimension) −4

2 4 6 8 10 0 2 4 6 8 10

Expert Survey Estimates of Taxes vs Spending Expert Survey Estimates of Social Dimension

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data Using Alternative Items - Belgian CAP data

2003 VB ● NVA ● VLD ● SPIRIT ● CD&V ● SPA ● AGALEV ●

1995 PRL ● VLD ● VB ● VU ● CVP ● PSC ● PS ● SP ● AGALEV ● ECOLO ●

1991 VB ● VU ● PVV ● PRL ● CVP ● PSC ● PS ● SP ● ECOLO ● AGALEV ●

1999 VB ● PRL ● VLD ● VU ● CD&V ● PSC ● PS ● ECOLO ● AGALEV ● SP ●

−3 −2 −1 0 1 2

Scaling Estimates of Theta

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data Validation of Belgian CAP Data Results

−1.5 −0.5 0.5 1.0 −20 −15 −10 −5 0 5 7 Expert Surveys 6 5

0.83 0.84 0.79 4 3 2

●

● 1.0 ● Agendas Project IRT

● ● 0.5

● ●●

● 0.46 0.43 ● ● −0.5

● ●

● −1.5

● ● 0.5 ● ● ● ● ● ● ● ● ● ● CMP IRT ● ● ● ●

● ● 0.0 ● ● ● ● 0.84 ● ●

● ● −0.5

● ● −1.0

● ● ● 5

● ● ● ● ● ● RILE 0 ● ● ●

● ● ●

−5 ● ● ● ● ● ● ● ● ● ● ● ●

−10 ● ● ●

● ● ●

−15 ● ● ●

● ● ●

−20 ● ● ●

2 3 4 5 6 7 −1.0 −0.5 0.0 0.5

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data Conclusion

I IRT model as inductive approach for inferring a left-right dimension from category counts

I Left-right as a super-issue rather than deﬁned by content

I Superior to ﬁxed content-based index (RILE)

I Can incorporate a priori information explicitly

I Flexible in terms of extensions:

I Diﬀerential item functioning I Model the parameters with covariates I ...

Kenneth Benoit and Thomas D¨aubler Scaling Manifesto Data