Arxiv:1908.02322V1

DpgMedia2019: A Dutch News Dataset for Partisanship Detection

Chia-Lun Yeh,1,2 Babak Loni,∗3 Marielle¨ Hendriks,1 Henrike Reinhardt,1 and Anne Schuth1 1DPG Media, The Netherlands 2TU Delft, The Netherlands 3TezolMarket, Iran [email protected], [email protected] {marielle.hendriks, henrike.reinhardt, anne.schuth}@persgroep.net

Abstract contains a large number of articles that were labeled using the partisanship of publishers. The We present a new Dutch news dataset with la- second part contains a few hundreds of articles that beled partisanship. The dataset contains more were annotated by readers who were asked to read than 100K articles that are labeled on the publisher level and 776 articles that were crowd- each article and answer survey questions. In the sourced using an internal survey platform and following sections, we describe the collection and labeled on the article level. In this paper, we annotation of both parts of the dataset. document our original motivation, the collection and annotation process, limitations, and 2 Dataset description applications. DpgMedia20191 is a Dutch dataset that was col- 2 1 Introduction lected from the publications within DPG Media. We took 11 publishers in the Netherlands for In a survey across 38 countries, the Pew Research the dataset. These publishers include 4 national Center reported that the global public opposed publishers, Algemeen Dagblad (AD), de Volk- partisanship in news media (Center, 2018b). It is, skrant (VK), Trouw, and Het Parool, and 7 re- however, challenging to assess the partisanship gional publishers, de Gelderlander, Tubantia, Bra- of news articles on a large scale. We thus made bants Dagblad, Eindhovens Dagblad, BN/De Stem an effort to create a dataset of articles annotated PZC, and de Stentor. The regional publishers are with political partisanship so that content analysis collectively called Algemeen Dagblad Regionaal systems can benefit from it. (ADR). A summary of the dataset is shown in Ta- ble1. To construct a dataset of news articles labeled with partisanship, it is required that some annota- 2.1 Publisher-level data tors read each article and decide whether it is parti- We used an internal database that stores all articles san. This is an expensive annotation process. An- written by journalists and ready to be published to other way to derive a label for an article is by using collect the articles. From the database, we queried the partisanship of the publisher of the article. Pre- all articles that were published between 2017 and vious work has used this method (Potthast et al., 2019. We filtered articles to be non-advertisement. 2018; Kulkarni et al., 2018; Kiesel et al., 2019). We also filtered on the main sections so that the arXiv:1908.02322v1 [cs.CL] 6 Aug 2019 This labeling paradigm is premised on that parti- articles were not published under the sports and san publishers publish more partisan articles and entertainment sections, which we assumed to be non-partisan publishers publish more non-partisan less political. After collecting, we found that a lot articles. Although there would be non-partisan ar- of the articles were published by several publish- ticles published by partisan publishers (and vice ers, especially a large overlap existed between AD versa), and thus labeled wrongly, the assumption and ADR. To deal with the problem without los- ensures more information than noise. Once the ing many articles, we decided that articles that ap- partisanship of a publisher is known, the labels of peared in both AD and its regional publications be- all its articles are known, which is fast and cheap. longed to AD. Therefore, articles were processed We created a dataset of two parts. The first part 1The dataset is released at https://github.com/ This∗ research was carried out while this author was dpgmedia/partisan-news2019 working at DPG Media. 2https://www.persgroep.nl/ Label Publisher-level Article-level Number of articles 100K 766 Percentage of partisan articles 50% 26% Number of publishers 11 11 Annotation method audience-based crowdsource Inter-rater agreement X Krippendorf’s alpha: 0.18

Table 1: Summary of the two parts of dpgMedia2019 dateset. in the following steps: Publisher #annotators Score 1. Remove any article that was published by vk 780 -0.6372 more than one national publisher (VK, AD, trouw 491 -0.5438 Trouw, and Het Parool). This gave us a list of parool 410 -0.4341 unique articles from the largest 4 publishers. degelderlander 208 -0.2452 tubantia 236 -0.1864 2. Remove any article from ADR that over- drabantsdagblad 188 -0.1862 lapped with the articles from national pub- eindhovensdagblad 194 -0.0722 lishers. destem 151 -0.0464 3. Remove any article that was published by pzc 202 -0.0248 more than one regional publisher (ADR). ad 695 0.0302 destentor 55 0.0727 The process assured that most of the articles are unique to one publisher. The only exceptions were total: 3,610 mean: -0.2067 the AD articles, of which some were also pub- Table 2: Publisher, number of people of whom the po- lished by ADR. This is not ideal but acceptable litical leaning we know, and the computed partisanship as we show in the section 2.1.1 that AD and ADR score. publishers would have the same partisanship labels. In the end, we have 103,812 articles. media report from the Pew Research Center in 2.1.1 Annotation of publisher partisanship 2018 (Center, 2018a), which found that VK is left- To our knowledge, there is no comprehensive re- leaning and partisan while AD is less partisan. search about the partisanship of Dutch publishers. Table3 shows the final publisher-level dataset We thus adopted the audience-based method to de- of dpgMedia2019, with the number of articles and cide the partisanship of publishers. Within the sur- class distribution. vey that will be explained in section 2.2, we asked the annotators to rate their political leanings. The 2.2 Article-level data question asked an annotator to report his or her po- To collect article-level labels, we utilized a litical standpoints to be extreme-left, left, neutral, platform in the company that has been used by right, or extreme-right. We mapped extreme-left the market research team to collect surveys from to -2, left to -1, center to 0, right to 1, extremely- the subscribers of different news publishers. right to 2, and assigned the value to each anno- The survey works as follows: The user is first tator. Since each annotator is subscribed to one presented with a set of selected pages (usually 4 of the publishers in our survey, we calculated the pages and around 20 articles) from the print paper partisanship score of a publisher by averaging the the day before. The user can select an article scores of all annotators that subscribed to the pub- each time that he or she has read, and answer lisher. The final score of the 11 publishers are some questions about it. We added 3 questions listed in Table2, sorted from the most left-leaning to the existing survey that asked the level of to the most right-leaning. partisanship, the polarity of partisanship, and We decided to treat VK, Trouw, and Het Parool which pro- or anti- entities the article presents. as partisan publishers and the rest non-partisan. We also asked the political standpoint of the user. This result largely accords with that from the news The complete survey can be found in Appendices. Partisanship Partisan Non-partisan Publisher de Volkskrant Trouw Het Parool AD ADR Article Num. 11,761 21,614 19,498 40,029 10,910 Total 52,873 (50.9%) 50,939 (49.1%)

Table 3: Number of articles per publisher and class distribution of publisher-level part of dpgMedia2019.

right and conservative ones. This shows that these The reason for using this platform was two-fold. left-leaning annotators were able to identify their First, the platform provided us with annotators partisanship and rate the articles accordingly. with a higher probability to be competent with We suspected that the annotators would induce the task. Since the survey was distributed to bias in ratings based on their political leaning and subscribers that pay for reading news, it’s more we might want to normalize it. To check whether likely that they regularly read newspapers and this was the case, we grouped annotators based on are more familiar with the political issues and their political leaning and calculate the percentage parties in the Netherlands. On the other hand, of each option being annotated. In Figure1, we if we use crowdsourcing platforms, we need to grouped options and color-coded political leanings design process to select suitable annotators, for to compare whether there are differences in the an- example by nationality or anchor questions to notation between the groups. We observe that the test the annotator’s ability. Second, the platform ”extreme-right” group used less ”somewhat parti- gave us more confidence that an annotator had san”, ”partisan”, and ”extremely-partisan” annota- read the article before answering questions. Since tions. This might mean that articles that were con- the annotators could choose which articles to sidered partisan by other groups were considered annotate, it is more likely that they would rate an ”non-partisan” or ”impossible to decide” by this article that they had read and had some opinions group. We didn’t observe a significant difference about. between the groups. Figure2 shows the same for the second question. Interestingly, the ”extreme- The annotation task ran for around two months right” group gave a lot more ”right” and slightly in February to April 2019. We collected annota- more ”progressive” ratings than other groups. In tions for 1,536 articles from 3,926 annotators. the end, we decided to use the raw ratings. How to scale the ratings based on self-identified political 2.2.1 Annotation distributions leaning needs more investigation. For the first question, where we asked about the in- 2.2.2 Quality control and agreement analysis tensity of partisanship, more than half of the anno- The main question that we are interested in is tations were non-partisan. About 1% of the anno- the first question in our survey. In addition to tation indicated an extreme partisanship, as shown the 5-point Likert scale that an annotator could in Table4. For the polarity of partisanship, most choose from (non-partisan to extremely partisan), of the annotators found it not applicable or diffi- we also provided the option to choose ”impossi- cult to decide, as shown in Table5. For anno- ble to decide” because the articles could be about tations that indicated a polarity, the highest per- non-political topics. When computing inter-rater centage was given to progressive. Progressive and agreement, this option was ignored. The remain- conservative seemed to be more relevant terms in ing 5 ratings were treated as ordinal ratings. The the Netherlands as they are used more than their initial Krippendorff’s alpha was 0.142, using the counterparts, left and right, respectively. interval metric. To perform quality control, we de- As for the self-rated political standpoint of the vised some filtering steps based on the information annotators, nearly half of the annotators identified we had. These steps are as follows: themselves as left-leaning, while only around 20% were right-leaning. This is interesting because 1. Remove uninterested annotators: we as- when deciding the polarity of articles, left and pro- sumed that annotators that provided no infor- gressive ratings were given much more often than mation were not interested in participating in reasonably non- somewhat extremely impossible non-partisan partisan partisan partisan partisan to decide 52.85% 16.34% 10.54% 5.49% 0.91% 13.88%

Table 4: Distribution of annotations of the strength of partisanship.

left right progressive conservative others not applicable unknown 5.66% 2.74% 7.74% 2.78% 7.29% 54.81% 18.96%

Table 5: Distribution of annotations of the polarity of partisanship.

extreme-left left middle right extreme-right 1.14% 46.87% 32.71% 19.14% 0.14%

Table 6: Distribution of annotations of self-identiﬁed political standpoints.

intensity of partisanship

extreme-left left 50 middle right extreme-right

30 %

0 non-partisan reasonably non-partisan somewhat partisan partisan extremely partisan impossible to decide

Figure 1: Percentage of annotation grouped by political leaning and annotation for the intensity of partisanship.

polarity of partisanship 60 extreme-left left middle 50 right extreme-right

% 30

0 left right progressive conservative others not applicable unknown

Figure 2: Percentage of annotation grouped by political leaning and annotation for the polarity of partisanship.

the task. These annotators always rated ”not 2. Remove unreliable annotators: as we didn’t possible to decide” for Q1, ’not applicable’ have ”gold data” to evaluate reliability, we or ”unknown” for Q2, and provide no textual used the free text that an annotator provided comment for Q3. There were in total 117 un- in Q3 to compute a reliability score. The as- interested annotators and their answers were sumption was that if an annotator was able discarded. to provide texts with meaningful partisanship description, he or she was more reliable in Publisher #articles %partisan performing the task. To do this, we collected vk 166 27.11 the text given by each annotator. We filtered trouw 140 25.00 out text that didn’t answer the question, such parool 121 28.93 as symbols, ’no idea’, ’see above’, etc. Then degelderlander 46 19.57 we calculated the reliability score of annota- tubantia 34 41.18 tor i with equation1, where ti is the number brabantsdagblad 32 31.25 of clean texts that annotator i provided in to- eindhovensdagblad 20 35.00 tal and Ni is the number of articles that anno- destem 34 17.65 tator i rated. pzc 30 26.67 t + 1 score = i × (t + 1) (1) ad 133 24.06 i N i i destentor 10 0.00 We added one to ti so that annotators that gave no clean texts would not all end up with total: 766 mean: 26.24 a zero score but would have different scores Table 7: Number of articles and percentage of partisan based on how many articles they rated. In articles by publisher. other words, if an annotator only rated one article and didn’t give textual information, we considered he or she reliable since we had lit- article-level) of the datasets. In Table8, we listed tle information. However, an annotator that the length of articles of the two parts. The rea- rated ten articles but never gave useful textual son that this is important is to check whether there information was more likely to be unreliable. are apparent differences between the articles in the The reliability score was used to filter out an- two parts of the dataset. We see that the lengths are notators that rarely gave meaningful text. The comparable, which is desired. threshold of the filtering was decided by the Krippendorff’s alpha that would be achieved article length publisher-level article-level after discarding the annotators with a score below the threshold. Mean 470.1 471.2 SD 387.5 275.1 3. Remove articles with too few annotations: ar- 50% percentile 381.0 451.0 ticles with less than 3 annotations were discarded because we were not confident with a Table 8: Statistics of length of articles. label that was derived from less than 3 annotations. The second analysis is the relationship between publisher and article partisanship. We want to 4. Remove unreliable articles: if at least half of check whether the assumption of partisan publish- the annotations of an article were ”impossible ers publish more partisan articles is valid for our to decide”, we assumed that the article was dataset. To do this, we used the article-level la- not about issues of which partisanship could bels and calculated the percentage of partisan arti- be decided. cles for each publisher. This value was then com- Finally, we mapped ratings of 1 and 2 to non- pared with the publisher partisanship. We cal- partisan, and 3 to 5 to partisan. A majority vote culated Spearsman’s correlation between the pub- was used to derive the final label. Articles with lisher partisanship derived from the audience and no majority were discarded. In the end, 766 arti- article content. We take the absolute value of the cles remained, of which 201 were partisan. Table partisanship in table2 and that in table7. The cor- 7 shows the number of articles and the percentage relation is 0.21. This low correlation resulted from of partisan articles per publisher. The final alpha the nature of the task and publishers that were con- value is 0.180. sidered. The partisan publishers in DPG Media publish news articles that are reviewed by profes- 3 Analysis of the datasets sional editors. The publishers are often partisan In this section, we analyze the properties and re- only on a portion of the articles and on certain top- lationship of the two parts (publisher-level and ics. 4 Limitations Appendices

We identified some limitations during the process, We list the questions we asked in the partisanship which we describe in this section. annotation survey, in the original Dutch language When deciding publisher partisanship, the num- and an English translation. ber of people from whom we computed the score was small. For example, de Stentor is estimated to • Q1: Over het algemeen, hoe bevooroordeeld reach 275K readers each day on its official web- vindt u dit artikel? Een artikel dat bevooro- site. Deciding the audience leaning from 55 sam- ordeeld is, is voor of tegen een persoon of ples was subject to sampling bias. Besides, the groep. N.B. Een artikel kan gaan over con- scores differ very little between publishers. None troversile onderwerpen, zoals politiek, maar of the publishers had an absolute score higher than blijft redelijk neutraal. 1, meaning that even the most partisan publisher 1. Onbevooroordeeld was only slightly partisan. Deciding which pub- 2. Redelijk onbevooroordeeld lishers we consider as partisan and which not is 3. Enigszins bevooroordeeld thus not very reliable. 4. Bevooroordeeld The article-level annotation task was not as 5. Extreem bevooroordeeld well-defined as on a crowdsourcing platform. We included the questions as part of an existing sur- 6. Onmogelijk om te bepalen vey and didn’t want to create much burden to the • Q2: Als u vindt dat dit artikel bevooro- annotators. Therefore, we did not provide long de- ordeeld is, ten gunste van welke politieke scriptive text that explained how a person should richting vindt u dit artikel geschreven? (U annotate an article. We thus run under the risk of kunt meerdere antwoorden kiezen) annotator bias. This is one of the reasons for a low inter-rater agreement. 1. Links 2. Rechts 3. Progressief 5 Dataset Application 4. Conservatief This dataset is aimed to contribute to developing a 5. Anders, namelijk: OPEN partisan news detector. There are several ways that 6. Niet van toepassing op dit artikel the dataset can be used to devise the system. For 7. Ik weet het niet example, it is possible to train the detector using publisher-level labels and test with article-level la- • Q3: Als u vindt dat het artikel bevooro- bels. It is also possible to use semi-supervised ordeeld is, pro of anti wie of wat vindt learning and treat the publisher-level part as unsu- u dit artikel? (bijvoorbeeld pro-PVV, pro- pervised, or use only the article-level part. We also conservatieven, pro-kapitalist, anti-Trump, released the raw survey data so that new mecha- anti-moslims, anti-athest). (U kunt meerdere nisms to decide the article-level labels can be de- antwoorden kiezen) vised. – Pro: – Anti:

6 Acknowledgements • Q4: Hoe zou u uw eigen politieke standpunt bepalen? We would like to thank Johannes Kiesel and the colleagues from Factmata for providing us with 1. Extreemlinks the annotation questions they used when creating a 2. (gematigd-)links hyperpartisan news dataset. We would also like to 3. Neutraal thank Jaron Harambam, Judith Moller¨ for helping 4. (gematigd-)rechts us in asking the right questions for our annotations 5. Extreemrechts and Nava Tintarev for sharing her insights in the domain. Translated • Q1: Overall, how biased is this article? An Johannes Kiesel, Maria Mestre, Rishabh Shukla, Em- article that is biased is for or against a person manuel Vincent, Payam Adineh, David Corney, or group. Note that an article can talk about Benno Stein, and Martin Potthast. 2019. SemEval- 2019 Task 4: Hyperpartisan News Detection. In contentious topics, like politics, but remains Proceedings of The 13th International Workshop on fairly neutral. Semantic Evaluation (SemEval 2019). Association for Computational Linguistics. 1. Unbiased Vivek Kulkarni, Junting Ye, Steve Skiena, and 2. Fairly unbiased William Yang Wang. 2018. Multi-view models for 3. Somewhat biased political ideology detection of news articles. In Pro- 4. Biased ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 3518– 5. Extremely biased 3527. Association for Computational Linguistics. 6. Not possible to decide Martin Potthast, Johannes Kiesel, Kevin Reinartz, Ja- nek Bevendorff, and Benno Stein. 2018. A Stylo- • Q2: If you ﬁnd the article biased, which po- metric Inquiry into Hyperpartisan and Fake News. litical direction do you ﬁnd this article in fa- In 56th Annual Meeting of the Association for Com- vor of? (You can choose multiple answers) putational Linguistics (ACL 2018), pages 231–240. Association for Computational Linguistics. 1. Left 2. Right 3. Progressive 4. Conservative 5. Others 6. Not applicable to the article 7. I don’t know

• Q3: If you ﬁnd the article biased, in- dicate who or what the article is biased in favor of (’pro’) and/or against (’anti’)? (for example pro-PVV,pro-conservative, pro- capitalist, anti-Trump, anti-Muslims, anti- atheist). (You can have multiple answers) – Pro: – Anti:

• Q4: How would you determine your own political position? 1. Extreme-left 2. (moderate)left 3. Neutral 4. (moderate)right 5. Extreme-right

References Pew Research Center. 2018a. News media and political attitudes in the netherlands.

Pew Research Center. 2018b. Public globally want unbiased news coverage.