Assessing the Readability of Policy Documents on the Digital Single Market of the European Union

Jukka Ruohonen University of Turku, Finland Email: juanruo@utu.ﬁ

Abstract—Today, literature skills are necessary. Engineering fruit. For instance, anecdotal evidence hints that engineering and other technical professions are not an exception from this and computer science students are receptive to the grammar requirement. Traditionally, technical reading and writing have and vocabulary of law [6]. Yet questions remain whether been framed with a limited scope, containing documentation, specifications, standards, and related text types. Nowadays, and how well they are able to understand and translate law however, the scope covers also other text types, including legal, into technical specifications and implementations. In practice, policy, and related documents. Given this motivation, this paper collaboration between lawyers and engineers is often required evaluates the readability of 201 legislations and related policy for such translations [7]. To ease the translations, different documents in the European Union (EU). The digital single market formalization methods and tools have long been proposed [8]. (DSM) provides the context. Five classical readability indices provide the methods; these are quantitative measures of a text’s Eventually, such technical solutions may eliminate the need readability. The empirical results indicate that (i) generally a to have lawyers in situations requiring the engineering of Ph.D. level education is required to comprehend the DSM laws requirements from law. These requirements have increased and policy documents. Although (ii) the results vary across the particularly in the EU as regulation of the information technol- five indices used, (iii) readability has slightly improved over time. ogy sector has tightened; therefore, the DSM laws and policy Index Terms—readability index, comprehension, literature documents provide an ideal case study. But engineers and other skills, text mining, legal texts, law, digital single market, EU technical professions are obviously not the only ones required to read, interpret, and understand law and its requirements. The comprehension of law by ordinary citizens remains the I.INTRODUCTION most pressing issue. And existing results are not encouraging. Comprehension of different texts is a requirement of today’s For instance, even many legal counseling websites have been life. No matter of their background, citizens need to understand observed to be beyond the comprehension of those with weak many text types to participate in a society and manage their literacy skills [9]. The problem can be argued to contain two lives. At the same time, surveys around the world have parts. The first is intentional obfuscation. And, indeed, it is reported declines in literature skills, generally defined not only not difficult to find arguments that companies and their legal as the ability to read and write but to also understand, evaluate, representatives deliberately obfuscate documents in order to and use written texts [1]. There are many reasons behind the evade transparency requirements [10]. Nor is public sector declines. Technology is one factor. Although existing results administration immune to such arguments. The second part are not definite [2], smart phones and social media are among derives from the ways lawyers, academics, and other profes- the technological factors partially explaining the declines. sionals, including legislators, communicate in writing. The When the declines are coupled with other trends, such as topics covered by them are often complex and hence the today’s avalanches of misinformation and disinformation, the writing tends to be complex. Sometimes, though, complexity societal consequences may be ruinous in the long-run. only serves complexity. Lawyers, for instance, have long been arXiv:2102.11625v2 [cs.CY] 15 Sep 2021 Technical professions do not exempt from literature skills. accused of writing gobbledygook [11]. Other professionals, Programmers need to write documentation and understand including civil servants, are often no better in this regard. the documentation of other programmers. Engineers need to Therefore, simple and understandable writing has long been read and write technical specifications. Thus, language and seen as a part of good administration. The same applies to literature skills, critical thinking, and related “humanistic” law-making for which language improvements have long been skills have long been on the agenda in engineering education recommended [12]. This point motivates to ask a research programs [3]. There are even online courses specifically for question (RQ1): how readable are the DSM-specific laws and technical writing skills [4]. At the same time, the skill require- policy documents? The framing to the DSM can be justified ments have likely increased and expanded in their scope. with the earlier remarks on technical professions; these are Nowadays, engineers need to comprehend also legal docu- the laws and policy documents engineers specifically should ments, particularly in case legal counseling is not attainable, as be able to understand. The second research question (RQ2) is often the case in start-ups and small companies. To this and is about validity: are there statistical differences between five other ends, educators have long tried to expose students to dif- typical readability indices? The final RQ3 is longitudinal: has ferent text types [5]. To some extent, the attempts have beared the readability of the laws and documents improved over time?

Proceedings of the Eighth International Conference on eDemocracy & eGovernment (ICEDEG 2021), Quito (online), pp. 205–209, IEEE. This is the author’s copy. The publisher’s copy is available online at https://dx.doi.org/10.1109/ICEDEG52154.2021.9530996. II.RELATED WORK were directives, regulations, or communications (COM), staff The paper shares a long tradition of related work. Read- working documents (SWD), recommendations, decisions, or ability first became a research topic already in the 1920s. joint declarations (JOIN) of the European Commission or the Ever since, different quantitative indices have been proposed Council. Hence, references to informal documents, treaties, to gauge the perceived readability of a text. The ideal has corrigenda, court cases, decisions of non-legislative EU in- been simplicity. Plain language, plain prose, or plain English stitutions, and related document types were excluded. Both have been the terms used to describe this ideal. It has been current and deprecated laws were included in the dataset but endorsed by authors of both fact and fiction. Actually, the only insofar as these were explicitly linked on the web pages. classical assessments and opinions of both author types have The individual legislations and policy documents are enu- been highly similar, as becomes evident by comparing the merated in Fig.1. Of these, about 34% are decisions, 32% 1946 works of Flesch [13] and Orwell [14], respectively. The directives, 18% regulations, and the remaining communica- former author also developed a quantitative index for his ideal. tions, recommendations, and other document types. The high From the early 1970s onward, the Flesch’s classical read- amount of decisions is partially explained by the specifications ability index was met by multiple competing indices. Yet these for frequency bands used for electronic and mobile communi- competing indices never abandoned the ideal of simplicity. cation. Based on a subjective classification, as much as 35% of Simple is usually better when a topic is complex. Therefore, the laws and policy documents are about telecommunications. a classical application domain has been technical writing by Privacy and data protection (14%), copyright and intellectual engineers [15], [16]. But the indices have been also applied for property (14%), different contracts and justice in general (6%), many other purposes. Other application domains include the Internet governance (4%), and cyber security (3%) follow. evaluation of financial reports [17], websites [18] and their pri- IV. METHODS vacy policies [19], tweets by politicians who speak at the level Five traditional readability indices are used. The selection of 4–5:th graders [20], fake online reviews [21], and consent of these was practical; these are readily available from an forms for scientific research [22], to name some examples. existing Python package [30]. All measure the hypothetical Some previous work exists also for using indices to evaluate grade level required to comprehend a text. The grades are the readability laws and related legal documents. The results based on those used in the United States. In theory, there is a have not been surprising. The comprehension requirements upper limit in the grades—a graduate program in a university have been observed to be beyond the educational attainment of would be somewhere around the 22:th grade or so, but none most people—and particularly of those who would most need of the indices impose limits. Negative values are also possible. information about their rights [23], [24]. Even the instructions Thus, in general, the higher the score, the more difficult a text delivered to juries in common law legal systems have been is to comprehend. To ensure comparability, all scores from the measured to be beyond the literature skills of many adults [25]. indices are further truncated up towards the nearest integer. The decisions reached by some courts have also become more The first is the Flesch–Kincaid index [31]. It is defined as: difficult to read over time, which, as such, reflects the increas- ing complexity of many legal questions [26]. However, no # words # syllables g = 0.39 + 11.8 − 15.59 , previous research seems to exists in the EU context according 1 # sentences # words to a reasonable literature search. Regarding the engineering where g1 refers to a grade. The second is the SMOG index or context, it is also worth remarking the work that has focused on the “simple measure of gobbledygook” [32]. It is given by: the understandability of the general logic of law [27]. But it is & s ' difficult to understand a logic when a text describing the logic # polysyllables g = 1.0430 30 + 3.1291 . is difficult to understand. This point provides a justification 2 # sentences for evaluating the overall readability of legislations in the EU. The third is ARI, the automated readability index [30], [33]: III.DATA # characters # words The dataset covers the main documents on the digital single g = 4.71 + 0.5 − 21.43 . 3 # words # sentences market of the EU. Although there exists no single database that would cover everything about the DSM, the EU has The fourth is the Coleman-Liau index [34], as defined by: still provided a portal that provides summaries about key g = d0.0588α − 0.296β − 15.8e , topical policy areas. The area reserved for the DSM was 4 used to assemble the dataset by covering all documents except where α is the number of letters per a hundred words and archival material. Based on this portal [28], five domains of β the average number of sentences per 100 words. The fifth the DSM are present: general rules, electronic communication and final index is the Linsear Write readability formula. It is networks, personal data and privacy, copyright and audiovisual defined differently from the other four indices. In essence: for material, and data economy and data protection. The data each 100 word sample, easy (with two syllables or less) and collection resembled the so-called snowball sampling [29]: for hard (three or more syllables) are counted and scored (with each domain, all hyperlinks were visited, and the documents one and three points, respectively), after which the per-sample mentioned on the web pages were collected as long as these scores are divided by the number of sentences in the sample, Directive 98/34/EC COM(1998) 476 final Decision 2005/222/JHA Directive (EU) 2019/790 Directive 98/84/EC COM(2003) 198 final Decision 2005/513/EC Directive 1999/5/EC JOIN(2013) 1 final COM(2008) 593 final Decision 2006/771/EC Directive 1999/93/EC Recommendation (EU) 2020/1307 COM(2009) 111 final Decision 2007/344/EC Directive 2000/31/EC Recommendation 2007/879/EC COM(2009) 277 final Decision 2007/90/EC Directive 2001/29/EC Recommendation 2008/295/EC COM(2011) 878 final Decision 2007/98/EC Directive 2001/84/EC Recommendation 2008/850/EC COM(2012) 596 final Decision 2008/411/EC Directive 2002/19/EC Recommendation 2009/396/EC COM(2013) 627 final Decision 2008/477/EC Directive 2002/20/EC Recommendation 2010/572/EU COM(2013) 634 final Decision 2009/381/EC Directive 2002/21/EC Recommendation 2013/105/EC COM(2014) 228 final Decision 2009/449/EC Directive 2002/22/EC Recommendation 2013/466/EU COM(2014) 442 final Decision 2009/766/EC Directive 2002/58/EC Recommendation 2014/478/EU COM(2014) 72 final Decision 2010/166/EU Directive 2002/77/EC Regulation (EC) 1008/2008 COM(2015) 192 final Decision 2010/267/EU Directive 2004/48/EC Regulation (EC) 1211/2009 COM(2015) 680 final Decision 2010/368/EU Directive 2006/115/EC Regulation (EC) 1370/2007 COM(2016) 117 final Decision 2010/87/EU Directive 2006/116/EC Regulation (EC) 2006/2004 COM(2016) 179 final Decision 2011/251/EU Directive 2006/24/EC Regulation (EC) 45/2001 COM/2017/0725 final Decision 2011/485/EU Directive 2006/95/EC Regulation (EC) 544/2009 Decision (EU, Euratom) 2015/443 Decision 2011/667/EU Directive 2008/294/EC Regulation (EC) 717/2007 Decision (EU, Euratom) 2017/46 Decision 2011/829/EU Directive 2008/52/EC Regulation (EC) 733/2002 Decision (EU) 2015/1293 Decision 2012/471/EU Directive 2008/63/EC Regulation (EC) 874/2004 Decision (EU) 2015/750 Decision 2012/472/EU Directive 2009/114/EC Regulation (EEC) 3577/92 Decision (EU) 2016/2317 Decision 2012/688/EU Directive 2009/136/EC Regulation (EU) 1215/2012 Decision (EU) 2016/339 Decision 2013/195/EU Directive 2009/140/EC Regulation (EU) 1291/2013 Decision (EU) 2016/687 Decision 2013/275/EU Directive 2009/24/EC Regulation (EU) 2015/1986 Decision (EU) 2017/1483 Decision 2013/504/EU Directive 2009/43/EC Regulation (EU) 2015/2120 Decision (EU) 2017/191 Decision 2013/654/EU Directive 2009/81/EC Regulation (EU) 2015/758 Decision (EU) 2017/2077 Decision 2013/743/EU Directive 2010/13/EU Regulation (EU) 2016/679 Decision (EU) 2017/899 Decision 2013/752/EU Directive 2011/77/EU Regulation (EU) 2016/794 Decision (EU) 2018/1538 Decision 2014/221/EU Directive 2011/83/EU Regulation (EU) 2017/1001 Decision (EU) 2018/1927 Decision 2014/243/EU Directive 2012/28/EU Regulation (EU) 2017/1128 Decision (EU) 2018/1961 Decision 2014/276/EU Directive 2013/37/EU Regulation (EU) 2017/1563 Decision (EU) 2018/1962 Decision 2014/641/EU Directive 2013/40/EU Regulation (EU) 2017/1939 Decision (EU) 2018/1996 Decision 243/2012/EU Directive 2013/98/EC Regulation (EU) 2017/920 Decision (EU) 2018/254 Decision 626/2008/EC Directive 2014/23/EU Regulation (EU) 2018/1241 Decision (EU) 2018/637 Decision 672/2002/EC Directive 2014/24/EU Regulation (EU) 2018/1488 Decision (EU) 2018/661 Decision 676/2002/EC Directive 2014/25/EU Regulation (EU) 2018/1725 Decision (EU) 2019/154 Decision 91/287/EEC Directive 2014/26/EU Regulation (EU) 2018/1727 Decision (EU) 2019/165 Directive (EU) 2015/1535 Directive 2014/30/EU Regulation (EU) 2018/1807 Decision (EU) 2019/235 Directive (EU) 2016/1148 Directive 2014/35/EU Regulation (EU) 2018/1971 Decision (EU) 2019/236 Directive (EU) 2016/2102 Directive 2014/53/EU Regulation (EU) 2019/1150 Decision (EU) 2019/784 Directive (EU) 2016/680 Directive 2014/61/EU Regulation (EU) 2019/517 Decision (EU) 2019/785 Directive (EU) 2016/681 Directive 2016/943 Regulation (EU) 2019/881 Decision (EU) 2020/1426 Directive (EU) 2017/1564 Directive 87/372/EEC Regulation (EU) 2020/857 Decision (EU) 2020/590 Directive (EU) 2018/1808 Directive 88/301/EEC Regulation (EU) 386/2012 Decision (EU) 2020/636 Directive (EU) 2018/1972 Directive 91/250/EEC Regulation (EU) 531/2012 Decision 11 June 2019 Directive (EU) 2019/1024 Directive 93/13/EEC Regulation (EU) 593/2008 Decision 1247/2002/EC Directive (EU) 2019/2161 Directive 93/83/EEC Regulation (EU) 611/2013 Decision 2001/497/EC Directive (EU) 2019/770 Directive 93/98/EEC Regulation (EU) 910/2014 Decision 2002/622/EC Directive (EU) 2019/771 Directive 95/46/EC SWD(2015) 100 final Decision 2004/915/EC Directive (EU) 2019/789 Directive 96/9/EC SWD(2016) 308 final

Fig. 1. Legal and Policy Documents Included in the Dataset (n = 201)

Flesch−Kincaid SMOG ARI Coleman−Liau Linsear

100 100 100 100 100

80 80 80 80 80

60 60 60 60 60

Grade 40 Grade 40 Grade 40 Grade 40 Grade 40

20 20 20 20 20

0 0 0 0 0 Median = 28 Median = 25 Median = 36 Median = 16 Median = 72

Fig. 2. Readability Grades (g1, . . . , g5, n = 201) and further scaled (for more details see [30]). As with the other Frequency indices, a truncated grade, g5, is outputted from the arithmetic. Two additional points are worth briefly making about these 8 8 23 42 57 63 classical indices. First, there are many modifications to these, 70 as well as numerous alternatives. Hundreds of individual 60 variables were considered already in the 1970s for construct- 50 40 ing readability indices [35]. More recent modifications have Grade 30 focused on natural language processing and machine learning 20 methods [36], [37]. The second point follows: all readability indices have always been heavily criticized. The criticism and thus limitations are later on briefly discussed in the concluding < 1995

SectionVI. For the present purposes, it suffices to justify 1995 − 2000 2001 − 2005 2006 − 2010 2011 − 2015 2016 − 2020 the use of the five readability indices with an argument that modifications and more elegant methods seem uncalled for in the present application domain. As there is no prior empirical Fig. 3. Readability Grades Across Time (sum variable) work in the domain, the five indices serve to make a baseline. of overall complexity. For instance, local newspapers in the V. RESULTS United States have been observed to attain values around 12 The grades across the 201 documents are shown in Fig.2 for at maximum [39]. Another comparative example would be each of the five readability indices. There are three points to abstracts in psychology papers for which values around 17 make from the figure. First, the Linsear Write index does not have been reported [40]. Finally, the visualization in Fig.3 pass a commonsense validity test; the grades from the index suffices to answer to RQ3. Although the amount of laws show a fairly large variance, but the median of 72 is not a and policy documents has steadily grown over the years, the realistic value (grade) in practice. Second, there is another readability of these has slightly improved when compared potential validity concern about the Coleman-Liau index; to the 1980s and 1990s. A potential explanation may relate the small standard deviation of 1.34 seems surprising when to incremental law-making; many of the new laws enacted compared to the outputs from the other four indices. Third, amend or replace old laws, building upon their foundations. the previous two points translate into modest correlations with Nevertheless, the averages have still remained around 30 or so. the grades from the remaining three indices. In contrast, the grades from the Flesch-Kincaid, SMOG, and ARI indices VI.CONCLUSION are highly correlated (see TableI). By implication, a sum This short paper evaluated the readability of laws and policy variable constructed from these three indices (based on the documents related to the digital single market of the EU. In rowwise arithmetic means) attains high internal consistency. general, these are difficult to comprehend according to the For instance, Cronbach’s α-coefficient [38] is as large as 0.98. quantitative readability indices. Even though readability has slightly improved, the hypothetical grade level is still around TABLE I thirty. What could explain this result? A partial explanation CORRELATIONS (PEARSON) would be that the DSM laws and policies, in particular, simply are complex; many of them address highly technical topics. 1. 2. 3. 4. 5. Another partial explanation stems from the EU’s law-making 1. Flesch-Kincaid processes. After a law is finally enacted, it has gone through 2. SMOG 0.940 3. ARI 0.995 0.928 a heavy process of revisions, often including copy-pasted 4. Coleman-Liau −0.111 0.018 −0.098 snippets from politicians pressed by lobbyists [41]. While 5. Linsear 0.330 0.347 0.312 −0.232 copy-pasting is not unique to the EU [42], it is likely to increase gobbledygook. Another source relates to Regulation These observations are enough to answer to the first and 1/1958 according to which multiple European languages must second research questions. Regarding RQ2: there are notable be accounted for already during the law-making processes. differences between the five indices. Regarding RQ1: based A further potential explanation stems from the limitations of on the median of the sum variable, the overall readability the quantitative readability indices. In particular, the construct is somewhere near the 30:th grade. What does such a grade validity of these is often questionable; it is not entirely clear mean; how large is this value? Given that even the first quartile whether they measure what they intend to measure. It is not is about 27, it seems equitable to conclude that a completion necessary to elaborate the lengthy criticism in detail. It suffices of a graduate program is required to comprehend the DSM to note that the indices ignore grammar [15], and do not laws and policy documents—at least insofar as the quantitative address the fact that simplicity by itself does not necessarily readability indices convey their intended function. Even if guarantee comprehension [43]. Thus, developing a readability their validity is questioned, as can be done on many grounds, index specifically for law would be a good topic for further re- there is still a comparative viewpoint supporting the conclusion search; the newer machine learning methods may help with the task. Another topic would be to examine the comprehension of [22] F. Santel, I. Bah, K. Kim, J.-A. Lin, J. McCracken, and A. Teme, laws and policy documents by human subjects. A particularly “Assessing Readability and Comprehension of Informed Consent Ma- terials for Medical Device Research: A Survey of Informed Consents interesting question is the comprehension—or, possibly, lack from FDA’s Center for Devices and Radiological Health,” Contemporary thereof—among engineering and computer science students. Clinical Trials, p. 105831, 2019. The question is relevant because it has been computer science [23] R. N. Arkell and H. C. van Dyck, “Readability of Human Rights Material,” Canadian Community Law Journal – Revue Canadienne de that has been seen to enhance law and its practice, often with Droit, vol. 2, pp. 12–14, 1978. questionable consequences [44], and not the other way around. [24] L. M. Tan and G. Tower, “The Readability of Tax Laws: An Empirical Study in New Zealand,” Australian Tax Forum, vol. 9, no. 3, pp. 355– REFERENCES 372, 1992. [25] R. Small, J. Platania, and B. Cutler, “Assessing the Readability of Capital [1] OECD, PISA 2018 Results: What Students Know and Can Do, Volume I. Pattern Jury Instructions,” Jury Expert, vol. 25, no. 1, pp. 18–22, 2013. Paris: OECD Publishing, 2019. [26] R. Whalen, “Judicial Gobbledygook: The Readability of Supreme Court [2] L. Verheijen, W. Spooren, and A. van Kemenade, “Relationships Be- Writing,” Yale Law Journal Forum, vol. 125, pp. 200–211, 2015. tween Dutch Youths’ Social Media Use and School Writing,” Computers [27] D. R. Ploch, B. K. Dumas, G. B. Gray, B. J. MacLennan, and J. E. and Composition, vol. 56, p. 102574, 2020. Nolt, “Readability of the Law: Forms of Law for Building Legal Expert [3] M. Abdulwahed, W. Balid, M. O. Hasna, and S. Pokharel, “Skills of Systems,” Jurimetrics, vol. 33, no. 2, pp. 189–222, 1993. Engineers in Knowledge Based Economies: A Comprehensive Literature [28] European Union, “Digital Single Market,” 2021, Summaries of EU Review, and Model Development,” in Proceedings of IEEE International Legislation. Available online in February 2021: https://eur-lex.europa. Conference on Teaching, Assessment and Learning for Engineering eu/summary/chapter/31.html. (TALE 2013). Bali: IEEE, 2013, pp. 759–765. [29] P. Biernacki and D. Waldorf, “Snowball Sampling: Problems and Tech- [4] Google, Inc., “Technical Writing Courses,” 2021, Available online in niques of Chain Referral Sampling,” Sociological Methods & Research, January 2021: https://developers.google.com/tech-writing. vol. 10, no. 2, pp. 141–163, 1981. [5] M. W. Conley and A. Wise, “Comprehension for What? Preparing [30] S. Bansal and C. Aggarwal, “Textstat,” 2021, Version 0.7.0. Available Students for Their Meaningful Future,” Theory Into Practice, vol. 50, online in January: https://pypi.org/project/textstat/. pp. 93–99, 2011. [31] J. P. Kincaid, R. P. Fishburne, R. L. Rogers, and B. Chissom, “Derivation [6] M. Hildebrandt, Law for Computer Scientists and Other Folk. Oxford: of New Readability Formulas (Automated Readability Index, Fog Count Oxford University Press, 2020. and Flesch Reading Ease Formula) for Navy Enlisted Personnel,” [7] K. Hjerppe, J. Ruohonen, and V. Leppanen,¨ “The General Data Pro- 1975, Naval Technical Training Command, Research Branch Report tection Regulation: Requirements, Architectures, and Constraints,” in 8-75. Available online in February 2021: https://apps.dtic.mil/sti/pdfs/ Proceedings of the 27th IEEE International Requirements Engineering ADA006655.pdf. Conference (RE 2019). Jeju Island: IEEE, 2019, pp. 265–275. [32] G. H. McLaughlin, “SMOG Grading – a New Readability Formula,” [8] L. Mommers, W. Voermans, W. Koelewijn, and H. Kielman, “Under- Journal of Reading, vol. 12, no. 8, pp. 639–646, 1969. standing the Law: Improving Legal Knowledge Dissemination by Trans- [33] E. A. Smith and J. P. Kincaid, “Derivation and Validation of the lating the Contents of Formal Sources of Law,” Artificial Intelligence and Automated Readability Index for Use with Technical Materials,” Human Law, vol. 17, pp. 51–78, 2009. Factors, vol. 12, no. 5, pp. 457–564, 1970. [9] D. D. Dyson and K. Schellenberg, “Access to Justice: The Readability [34] M. Coleman and T. L. Liau, “A Computer Readability Formula Designed of Legal Services Corporation Legal Aid Internet Services,” Journal of for Machine Scoring,” Journal of Applied Psychology, vol. 60, pp. Poverty, vol. 21, no. 2, pp. 142–165, 2016. 283–284, 1975. [10] B. A. Rutherford, “Obfuscation, Textual Complexity and the Role of [35] E. B. Entin and G. R. Klare, “Factor Analyses of Three Correlation Ma- Regulated Narrative Accounting Disclosure in Corporate Governance,” trices of Readability Variables,” Journal of Reading Behavior, vol. 10, Journal of Management and Governance, vol. 7, pp. 187–210, 2003. no. 3, pp. 279–290, 1979. [11] H. Frooman, “Lawyers and Readability,” The Journal of Business [36] T. François and E. Miltsakaki, “Do NLP and Machine Learning Improve Communication, vol. 18, no. 4, pp. 45–51, 1973. Traditional Readability Formulas?” in Proceedings of the First Workshop [12] U. Karpen, “Instructions for Law Drafting,” European Journal of Law on Predicting and Improving Text Readability for Target Reader Popu- Reform, vol. 10, no. 2, pp. 163–182, 2008. lations (PITR 2012). Montreal: ACM, 2012, pp. 49–57. [13] R. Flesch, How to Write, Speak, and Think More Effectively. New [37] L. C. R. Timana,´ D. F. S. Lozano, and J. F. C. Garc´ıa, “Software York: Harper & Brothers, 1960 [1946]. to Determine the Readability of Written Documents by Implementing [14] G. Orwell, Politics and the English Language and Other Essays. Public a Variation of the Gunning Fog Index Using the Google Linguistic domain 2018 edition, 1946. Corpus,” in Proceedings of the International Conference on Applied [15] G. M. McClure, “Readability Formulas: Useful or Useless?” IEEE Technologies (ICAT 2019). Quito: Springer, 2020, pp. 409–420. Transactions on Professional Communication, vol. PC-30, no. 1, pp. [38] L. J. Cronbach, “Coefficient Alpha and the Internal Structure of Tests,” 12–15, 1987. Psychometrika, vol. 16, no. 3, pp. 297–334, 1951. [16] S. Zhou, H. Jeong, and P. A. Green, “How Consistent Are the Best- [39] B. Wasike, “Preaching to the Choir? An Analysis of Newspaper Read- Known Readability Equations in Estimating the Readability of Design ability vis-a-vis Public Literacy,” Journalism, vol. 19, no. 11, pp. 1570– Standards?” IEEE Transactions on Professional Communication, vol. 60, 1587, 2016. no. 1, pp. 97–111, 2017. [40] J. Stricker, A. Chasiotis, M. Kerwer, and A. Gunther,¨ “Scientific Ab- [17] Y. Sun, X. Wang, and Y. Yu, “Readability to Financial Report: A stracts and Plain Language Summaries in Psychology: A Comparison Comparative Study of Chinese and Foreign Countries,” in Proceedings of Based on Readability Indices,” PLoS ONE, vol. 15, no. 4, p. e0231160, the Seventh International Joint Conference on Computational Sciences 2020. and Optimization (CSO 2014). Beijing: IEEE, 2014, pp. 69–73. [41] J. Ruohonen, “David and Goliath: Privacy Lobbying in the European [18] E. K. Leong, M. T. Ewing, and L. F. Pitt, “E-Comprehension: Evalu- Union,” 2019, Archived manuscript, available online: arXiv:1906.01883. ating B2B Websites Using Readability Formulae,” Industrial Marketing [42] T. Allee and M. Elsig, “Are the Contents of International Treaties Management, vol. 31, pp. 125–131, 2020. Copied and Pasted? Evidence from Preferential Trade Agreements,” [19] B. Fabian, T. Ermakova, and T. Lentz, “Large-Scale Readability Analysis International Studies Quarterly, vol. 63, no. 3, pp. 603–613, 2019. of Privacy Policies,” in Proceedings of the International Conference on [43] J. Korunovska, B. Kamleitner, and S. Spiekermann, “The Challenges and Web Intelligence (WI 2017). Leipzig: ACM, 2017, pp. 18–25. Impact of Privacy Policy Comprehension,” in Proceedings of the Twenty- [20] O. Kayam, “The Readability and Simplicity of Donald Trump’s Lan- Eight European Conference on Information Systems (ECIS 2020). AIS, guage,” Political Studies Review, vol. 16, no. 1, pp. 73–78, 2017. 2020, pp. 1–17. [21] X. Wang, X. Zhang, C. Jiang, and H. Liu, “Identification of Fake [44] N. S. Goltz and G. Dondoli, “A Note on Science, Legal Research and Reviews Using Semantic and Behavioral Features,” in Proceedings of Artificial Intelligence,” Information & Communications Technology Law, the 4th International Conference on Information Management (ICIM vol. 28, no. 3, pp. 239–251, 2019. 2018). Oxford: IEEE, 2018, pp. 92–97.