Assessing Political News Quality: an Automated Comparison of Political News Quality Indicators Across German Newspapers with Different Modalities and Reach
Total Page：16
File Type：pdf, Size：1020Kb
Assessing political news quality: An automated comparison of political news quality indicators across German newspapers with different modalities and reach Nicolas Mattis Student number: 12283177 Research Master’s Thesis Graduate School of Communication University of Amsterdam Research Master in Communication Science Supervised by Dr. Anne Kroon Word count: 7,497 June 26th, 2020 Abstract In order to best perform their societal functions, news media must adhere to certain normative standards for news quality – especially when reporting about events with political significance. While various past studies have examined (political) news quality, they often differ in the indicators and operationalisations that they use, making it difficult to compare findings across studies. Hence, this thesis proposes a comprehensive framework for automatically measuring political news quality that is easily scalable and can be applied in various contexts as well as over longer timespans. It combines existing measures with newly developed classifiers that assess impartiality, thereby highlighting the potential that supervised machine learning has for journalism studies and providing a means for future studies to assess impartiality in an automated manner. Furthermore, this thesis generates new insights into differences in political news quality across German newspapers that differ in their reach (national vs. regional) and modality (online vs. offline). The results indicate that both modality and reach appear to affect newspapers’ performance in terms of political news quality indicators, even though these differences tend to not be particularly pronounced. While especially online newspapers performed comparably worse in terms of indicators such as actor diversity, impartiality, and emotionality, the results suggest that modality and reach alone are not sufficient to explain differences across news outlets. On the whole, this thesis highlights the potential that automated research methods have for future research into (political) news quality and urges scholars to employ and advance existing measures to provide a fuller picture of (political) news quality across countries, outlets and, maybe most importantly, over time. Keywords: Automated content analysis, News quality, Impartiality, Diversity, Supervised machine learning 1 Introduction Often referred to as the fourth estate, news media are widely regarded as crucial for well-functioning democracies (Jacobi, Kleinen-von Königslöw, & Ruigrok, 2016). Building on Locke (1967), Strömbäck (2005) argues that one can describe the relationship between news media and democracy as a social contract: Democracy creates the necessary conditions for news media to operate in, while news media contribute to democracy by providing relevant, high-quality information to both the public and the government, as well as by serving as a watchdog of a countries’ institutions. To live up to those standards and inform the public both accurately and fairly, news outlets need to adhere to certain normative news quality standards such as diversity and impartiality (Urban & Schweiger, 2014). Naturally, this begs the question how well newspapers in a given media market adhere to such standards. While there is ample (comparative) research on different news quality indicators such as diversity, negativity, and objectivity (e.g. Burggraaff & Trilling, 2017; Humprecht & Esser, 2018; Jacobi et al., 2016; Masini et al., 2018), studies often differ in their choice and operationalisation of these indicators. Hence, this thesis proposes a comprehensive framework for assessing news quality through an automated content analysis (ACA) by combining existing measures with newly developed classifiers that assess three key indicators of impartiality on the article level. ACA constitutes an efficient and affordable research methodology for the analysis of large bodies of data (Grimmer & Stewart, 2013) that can be applied to journalistic content in both an inductive and a deductive manner (Boumans & Trilling, 2016). Given that the field of journalism studies tends to largely neglect automated research methods (Boumans & Trilling, 2016), this thesis hopes to a) drive the field methodologically forward – by illustrating the potential of supervised machine learning (SML) and moving beyond mere case studies - and b) facilitate future comparative research by providing a means to assess and monitor important news quality indicators in an easily scalable and resource-efficient manner. 2 On a theoretical level, this thesis addresses concerns over an overall decrease in journalistic news quality, that a number of scholars have voiced since new technological affordances and increased economic pressures have begun transforming traditional newspaper markets (e.g. Burggraaff & Trilling, 2017; Humprecht & Esser, 2018; Jacobi et al., 2016; Jungnickel, 2011; McManus, 2009; Plasser, 2005). The underlying argument of those concerns is that the current transformation of the newspaper market results in a fierce competition for advertising revenue. In order to cope, newspapers attempt to boost their reach to attract advertisers, often at the expense of journalistic quality (McManus, 2009) – a process that scholars refer to as commercialisation (Jacobi et al., 2016) or tabloidization (Esser, 1999). Commercialisation is often assumed to be especially pronounced in online news content (e.g. Burggraaf & Trilling, 2017). However, existent research on the effects of modality is inconclusive as some researchers have found evidence for lower news quality online (e.g. Burggraaff & Trilling, 2017; Welbers, Van Atteveldt, Kleinnijenhuis, & Ruigrok, 2018), whereas others have found no notable differences (Ghersetti, 2014) or even contradictory ones (e.g. Humprecht & Esser, 2018). Other important factors that might affect news quality are the structure of a given media market (Esser & Umbricht, 2013) and the size of a newspaper (Masini et al., 2018). For example, Masini et al. (2018) claim that local newspapers can allocate fewer resources to quality reporting, especially about events on the national level. In light of these considerations, this thesis compares political news quality across German newspapers with different modalities (online vs. offline) and reach (national vs. regional). By unravelling the effects that those factors have, this thesis hopes to add to existing research by providing a clearer picture of political news quality in Germany. In the following, this thesis will a) lay out the theoretical underpinnings of an automated news quality measurement framework, b) apply it to a sample of German newspapers, c) present the differences across newspapers with different reach and modalities, and d) close with implications of ACA and suggestions for future research. 3 Theoretical Framework Political news quality and its indicators What constitutes good political news? The answer to this question will likely depend on who answers it. As Urban and Schweiger (2014) argue, a journalist might judge an article’s quality by the effort that it took to produce, whereas a reader might simply judge it by how enjoyable it is to read. This study builds on McQuail’s (1992) notion of the ‘marketplace of ideas’ and takes a normative perspective on the quality of news accordingly. Following Urban and Schweiger (2014), it posits that high-quality political news should provide accurate and impartial information that gives room to a wide variety of relevant actors and their positions in order to enhance the public’s understanding of important political matters as well as broader societal debates. This perspective builds on Strömbäck’s (2005) idea of a “participatory democracy”, the notion that citizens should (be able to) participate in all aspects of political life. Naturally, to do so effectively, citizens need to have access to high quality political information – not only during and before elections, but all year round. Over time, various media and journalism scholars have spelled out the elements that constitute (political) quality news. For example, Jungnickel (2011) identified seven quality criteria, namely lawfulness, accuracy, relevance, comprehensibility, transparency, impartiality, and diversity, with various sub-dimensions. Urban and Schweiger (2014) propose a somewhat similar, yet slightly more parsimonious model with six quality criteria: diversity, impartiality, relevance, comprehensibility, accuracy, and ethics. Although many analyses of news quality indicators have relied on manual content analyses (e.g. Esser & Umbricht, 2013; Masini et al., 2018; Ramírez de la Piscina, Gonzalez Gorosarri, Aiestaran, Zabalondo, & Agirre, 2015), several of those indicators can be assessed through ACA. In fact, a few studies have already done so (Burggraaff & Trilling, 2017; Jacobi et al., 2016). ACA constitutes a valuable research methodology in journalism studies as it a) significantly reduces the cost of traditional content analysis, b) provides a means to test hypotheses on a larger 4 scale, and c) potentially might even reveal insights that more traditional methods have missed (Boumans & Trilling, 2016). It also allows researcher to explore over-time developments with comparable ease. In the following, four core dimensions of an automated measurement approach as taken in this study, namely diversity, impartiality, emotionality, and comprehensibility are discussed. Diversity “[D]iversity in public affairs coverage is crucial because the news media are expected to create a mediated