OSIAN: Open Source International Arabic News Corpus - Preparation and Integration into the CLARIN-infrastructure
Imad Zeroual*, Dirk Goldhahn†, Thomas Eckart†, Abdelhak Lakhouaja* *Computer Sciences Laboratory, Mohamed First University, Morocco {mr.imadine, abdel.lakh}@gmail.com †Natural Language Processing Group, University of Leipzig, German {dgoldhahn, teckart}@informatik.uni-leipzig.de
Abstract growth of the ten most frequent online languages in the last 18 years. However, a few years ago, The World Wide Web has become a Arabic was considered relatively an under- fundamental resource for building large text resourced language that lacks the basic resources corpora. Broadcasting platforms such as news and corpora for computational linguistics, not a websites are rich sources of data regarding diverse topics and form a valuable foundation single modern standard Arabic tagged corpus was for research. The Arabic language is freely or publicly available. Since then, major extensively utilized on the Web. Still, Arabic progress has been made in building Arabic is relatively an under-resourced language in linguistic resources, primarily corpora (Zeroual terms of availability of freely annotated and Lakhouaja, 2018a); still, building valuable corpora. This paper presents the first version annotated corpora with a considerable size is of the Open Source International Arabic expensive, time-consuming, and requires News (OSIAN) corpus. The corpus data was appropriate tools. Therefore, many Arabic corpora collected from international Arabic news builders produce their corpora in a raw format. websites, all being freely available on the For building the Open Source International Web. The corpus consists of about 3.5 million articles comprising more than 37 million Arabic News (OSIAN) corpus, the typical sentences and roughly 1 billion tokens. It is procedures of the Leipzig Corpora Collection were encoded in XML; each article is annotated utilized. Furthermore, a language-independent with metadata information. Moreover, each Part-of-Speech (PoS) tagger, Treetagger, is adapted word is annotated with lemma and part-of- to annotate the OSIAN corpus with lemma and speech. The described corpus is processed, part-of-speech tags. archived and published into the CLARIN The prime motivation for building OSIAN infrastructure. This publication includes corpus is the lack of open-source Arabic corpora descriptive metadata via OAI-PMH, direct that can cope with the perspectives of Arabic access to the plain text material (available Natural Language Processing (ANLP) and Arabic under Creative Commons Attribution-Non- Commercial 4.0 International License - CC Information Retrieval (AIR), among other research BY-NC 4.0), and integration into the areas. Hence, we expect that the OSIAN corpus WebLicht annotation platform and can be used to answer relevant research questions CLARIN’s Federated Content Search FCS. in corpus linguistics, especially investigating variation and distinction between international and 1 Introduction national news broadcasting platforms with a diachronic and geographical perspective. The Arabic language is spoken by 422 million After this introduction, the remainder of the people, making it the fourth most used language on paper is structured as follows: In section 2, we 1 the Web . Its presence on the Web had the highest highlight the state-of-the-art of web-crawled
1 http://www.internetworldstats.com/stats7.htm 175 Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182 Florence, Italy, August 1, 2019. c 2019 Association1 for Computational Linguistics corpora of the Arabic language. Further, the • Open Source Arabic Corpora2 (OSAC) methodology and tools used to build the OSIAN (Saad and Ashour, 2010): It is a collection of corpus are presented in Section 3. In Section 4, the large and free accessible raw corpora. The OSIAN corpus is described in more detail, yet, OSAC corpus consists of web documents some data analyses are performed and discussed. extracted from over 25 Arabic websites Finally, Section 5 contains some concluding using the open source offline explorer, remarks and future work. HTTrack. The compilation procedure involves converting HTML/XML files into 2 Literature review UTF-8 encoding using “Text Encoding Converter” as well as removing the The World Wide Web is an important source for HTML/XML tags. The final version of the researchers interested in the compilation of very corpus comprises roughly 113 million large corpora. A recent survey (Zeroual and tokens. Besides, it covers several topics Lakhouaja, 2018b) reports that 51% of corpora are namely Economy, History, Education, constructed based, totally or partially, on Web Religion, Sport, Health, Astronomy, Law, content. Web corpora continue to gain relevance Stories, and Cooking Recipes. within the computational and theoretical • arTenTen (Arts et al., 2014): It is a member linguistics. Given their size and the variety of of the TenTen Corpus Family (Jakubíček et domains covered, using Web-derived corpora is al., 2013). The arTenTen is a web-derived another way to overcome typical problems faced corpus of Arabic crawled using Spiderling by statistical corpus-based studies such as data- (Suchomel et al., 2012) in 2012. The sparseness and the lack of variation. arTenTen corpus is partially tagged. i.e., one The web corpora continue to gain relevance sample of the corpus, comprises roughly 30 within the computational and theoretical million, is tagged using the Stanford Arabic linguistics. Given their size and the variety of part-of-speech tagger. While, another domains covered, using web-derived corpora is sample, contains over 115 million words, is another way to overcome typical problems faced tokenised, lemmatised, and part-of-speech by statistical corpus-based studies such as data- tagged using MADA system. All in all, the sparseness and the lack of variation. Besides, they arTenTen comprises 5.8 billion words but it can be used to evaluate different approaches for the can only be explored by paying a fee via the classification of web documents and content by Sketch Engine website3. text genre and topic area (e.g., (Chouigui et al., • ArabicWeb16: Since 2009, the ClueWeb09 2017)). Furthermore, web corpora have become a web crawl (Callan et al., 2009), that includes prime and well-established source for 29.2 million of Arabic pages, was considered lexicographers to create many large and various the only and largest Arabic web crawl dictionaries using specialised tools such as the available. However, in 2016, a new and corpus query and corpus management tool Sketch- larger crawl of today’s Arabic web is Engine (Kovář et al., 2016). Moreover, some publicly available. This web crawl is called completely new areas of research, for which they ArabicWeb16 (Swuaileh et al., 2016) and deal exclusively with web corpora, have emerged. comprises over 150M web pages crawled Indeed, the aim was to build, investigate, and over the month of January 2016. In addition analyse corpora based on online social networks to addressing the limitation of the posts, short messages, and online forum ClueWeb09, ArabicWeb16 covers both discussions. dialectal and Modern Standard Arabic. Publicly available Arabic web corpora are quite Finally, the total size of the compressed limited, which greatly impacts research and dataset of ArabicWeb16 is about 2TB and it development of Arabic NLP and IR. However, is available for download after filling a some research groups (Zaghouani, 2017) have request form4. shown potentials in building web-derived corpora • The GDELT Project5 is a free open platform in recent years. Among them are: for research and analysis of the global
2 https://sites.google.com/site/motazsite/corpora/osac 4 https://sites.google.com/view/arabicweb16 3 https://www.sketchengine.co.uk/ 5 https://www.gdeltproject.org/ 176
2
database. All the datasets released are free, direct access via a Web interface, LCC data is also open, and available for unlimited and offered for free download. unrestricted use for any academic, For each word the dictionaries contain: commercial, or governmental use. Also, it is • Word frequency information. possible to download the raw datafiles, visualize it, or analyse it at limitless scale. • Sample sentences. Recently, the GDELT Project is starting to • Statistically significant word co-occurrences create linguistic resources. In fact, 9.5 billion (based on left or right neighbours or whole words of worldwide Arabic news has been sentences). monitored over 14 months (February 2015 to June 2016) to make a trigram dataset for the • A semantic map visualizing the strongest Arabic language. Consequently, an Arabic word co-occurrences. trigram table of the 6,444,208 trigrams that • Part of speech information (partially). appeared more than 75 times is produced6. • Similar words and other semantic It is worth mentioning that larger corpora in the information (partially). region of billions of words are usually created by downloading texts from the web unselectively 3.1.2 Crawling and processing of data with respect to their text type or content. For corpus creation, an adapted version of the Therefore, the content of such corpora cannot be CURL-portal (Crawling Under-Resourced determined before their construction, thus, it is Languages8) (Goldhahn et al., 2016) of the LCC necessary to filter, clean, and evaluate it was utilized. CURL allows creating Web- afterwards. accessible and downloadable corpora by simply entering URLs into the portal. In order to build a 3 Methodology and tools balanced corpus of international Arabic news, the In this section, we describe the crawling, data have been drawn from a wide range of processing and annotation tasks alongside with the reliable sources around the world. Six million tools used. webpages were downloaded, three and a half million pages which contain Arabic text were 3.1 Data acquisition extracted and sub-corpora for several Arabic In a first step the data needs to be crawled from the speaking countries were created. World Wide Web. Since the crawled data are often The crawling was conducted in March 2018 duplicated or in other ways problematic, they need using Heritrix, the crawler of the Internet Archive. to be cleaned and filtered. Therefore, the following Further processing was carried out according to the processing steps were executed. language independent processing chain described in (Goldhahn et al., 2012) and involved steps as 3.1.1 Leipzig Corpora Collection extracting raw text from the Web ARChive file format, sentence separation and removal of non- The Leipzig Corpora Collection (LCC) (Goldhahn sentences using regular expressions. Finally, texts et al., 2012; Quasthoff et al., 2014) started as were extracted based on Web domain and assigned “Projekt Deutscher Wortschatz7” in the Nineties as to the respective country. Furthermore, since the a resource provider for digital texts in the German crawler writes the data in one large file, we language mostly based on newspaper articles and developed a tool for extracting the texts based on royalty-free text material. the Web domain. For each Web domain, the tool Today, the LCC offers corpus-based extracts and saves each article/page in a single file. monolingual full form dictionaries in more than Finally, these articles are assigned to the respective 200 languages mainly based on online accessible country. A list of the crawled Web domains, the text material, divided under several aspects like the number of articles extracted, and the countries year of acquisition, text genre, country of origin covered are provided in the Appendix “A”. and more. Since June 2006, LCC can be accessed at http://corpora.uni-leipzig.de. In addition to
6 https://goo.gl/MZZkDJ 8 http://curl.corpora.uni-leipzig.de/ 7 http://wortschatz.uni-leipzig.de 177
3
The number of articles extracted from the 4 The OSIAN Corpus crawled data is varying from one website to another. Some domains were only restricted by the Instead of using unselected data from the Web, the short duration of the crawling, whereas others ran aim of the OSIAN corpus is to build a balanced out of crawlable URLs early due to a low amount corpus in which the data must be drawn from a of crawlable resources, robots.txt-restrictions or wide range of reliable and open sources. Therefore, external links to other domains which were not this corpus is compiled based on 31 different followed. international Arabic news broadcasting platforms, all being freely available on the Web. 3.2 Corpus annotation We extracted six million webpages. After Among the widely used and relevant types of cleaning and filtering, we were left with about corpus annotations are e.g. lemma and part of three and half million articles comprising more speech. Lemmatization is a basic morphological than 37 million sentences and roughly 1 billion analysis to deal with derivation paradigms, tokens. whereas part-of-speech tagging is part of a further 4.1 Word length distribution syntactic analyses (i.e., parsing) to determine the sentence's syntactic structure. Both annotation The average length of words varies from 7 to 12 9 forms affect the performance of subsequent text letters in many languages . According to Mustafa analysis in NLP and IR. (2012), the average length of Arabic words in a For both part of speech tagging and normal text is five letters. When analyzing the lemmatization tasks, we used a previously adapted OSIAN corpus the length of 36% of the words is and well-established version of Treetagger for the above six letters, this percentage is increased to Arabic language (Imad and Abdelhak, 2016). 75% if duplicate words are considered. This makes Further, we improved this model and retrained it the corpus a good soil to evaluate techniques that using new linguistic resources namely the aim to reduce a word to its base form. Frequency Dictionary of Arabic (Buckwalter and It is worth mentioning that tokens with length Parkinson 2014). This frequency dictionary superior to 10 letters are not considered since news contains the top 5,000 words that were derived articles contain phrases written without space from a collection of representative corpora that characters between words as well as non-derived include 30 million words of both written texts and and concatenated words, such as ,Euro-Mediterranean/”األورومتوسطي“ Word Occurrence Percentage Occurrence Percentage length (Unique) (Duplicated) 2 4,180 0,03% 113,129,168 12,22% 3 45,723 0,28% 148,295,530 16,03% 4 412,528 2,52% 154,159,209 16,66% 5 1,550,485 9,48% 175,925,523 19,01% 6 2,877,426 17,59% 133,290,941 14,40% 7 3,353,777 20,50% 107,877,916 11,66% 8 2,864,584 17,51% 54,007,298 5,84% 9 1,919,115 11,73% 20,526,042 2,22% 10 1,196,370 7,31% 9,072,780 0,98% >10 2,137,492 13,06% 9,050,623 0,98% Total 16,361,680 100% 925,335,030 100%
Table 1: Word length statistics Electromagnetism, etc. This/”الكهرومغناطيسية“ .transcribed speech A sample of 10,000 words of the corpus has explains why we found more than two million been manually checked to evaluate the unique tokens that consist of over 11 letters which performance of Treetagger and the achieved is an irrational result for the Arabic language. accuracy rate is 95.02%.
9 http://www.ravi.io/language-word-lengths 178
4
Table 1 displays the percentage of words 4.3 Corpus format covered in the OSIAN corpus with respect to their The XML-format is used to facilitate the use of the lengths, including unique and duplicate words. corpus. This is the first version of the OSIAN 4.2 Word frequency list corpus which consists of separate directories for each country. Furthermore, each directory includes Calculating word frequencies enables us to the articles in XML format, where the sentences are indicate the distribution of words across the text lemmatized and PoS tagged. Moreover, the XML categories. Besides, it is feasible to produce word files contain metadata to provide information about frequency lists using the tokens’ PoS tags instead domain names, webpage location, and the date of of their orthographic status. extraction. For more illustration, Figure 1 presents Obviously, function words will be at the top of a sample of the XML files. the frequency wordlist. Nevertheless, the words Note that some Web domains include in their thematically organized in Table 2 are also among URLs the topic of the published articles like the the most frequent words. sample provided in Figure 1 where the word In the context of IR and corpus linguistics, many “Science and tech” appeared in the article’s URL. of the top frequently words have no value or effect This is another feature that can be used to classify on further analyses since they are typical in news the articles based on their topics, one among other World: techniques, to prepare them for classification and) ”العالم“ articles; examples include Government: topic detection. Unfortunately, not all the URLs) ”الحكومة“ ,(F=1,182,181; R=37 Negotiations: include such information; therefore, the topic label) ”مفاوضات“ F=667,862; R=73), and F=524,035; R=101). However, the words listed in remains “unknown” till a solution is found (using Table 2 are a result of the circumstances of the topic detection and tracking methods). Middle East in recent years, FIFA World Cup, and the Brexit, which make these words occur 4.4 CLARIN Integration frequently in various world news. Using LancsBox CLARIN10 (Common Language Resources and to analyze the corpus data, it was possible to Technology Infrastructure) is a European calculate frequencies of words that are obvious Research Infrastructure established in 2012 and took up the mission to create an online االتحاد “ ,(World Cup) ”كأس العالم“ collocates such as environment to provide access to language ”البيت األبيض“ European Union), and) ”األوربي (White House). Moreover, it is also possible to Theme Word Frequency (F) Rank (R) 81 608,176 ترامب (Trump, President of USA) 164 380,086 سلمان (Persons (Salman, King of Saudi 687 114,586 السيسي (El-Sisi, President of Egypt) 51 960,732 سوريا (Syria) 57 862,156 بريطانيا (Countries (United Kingdom 70 704,457 قطر (Qatar) 117 482,688 االنتخابات (Election) 134 434,376 بريكست (Topics (Brexit 188 349,873 كأس العالم (World Cup) 161 387,174 الناتو (NATO) 448 177,383 االتحاد األوربي (Organizations (European Union 648 124,762 البيت األبيض (White House)
Table 2: Relevant words from the frequency wordlist calculate statistical information about the resources (in written, spoken, or multimodal form) association, the strength of collocation, and the comparative frequencies of word forms in the overall data of the OSIAN corpus or in country- separated data.
10 https://www.clarin.eu/ 179
5
Figure 1: A sample of OSIAN corpus encoded in XML format for the support of scholars in the humanities and descriptive metadata via OAI-PMH11, direct access social sciences, and beyond (de Jong et al., 2018). to the plain text material (available under Creative Currently, CLARIN also offers advanced tools to Commons Attribution-NonCommercial 4.0 discover, explore, exploit, annotate, analyse, and International License - CC BY-NC 4.0), and combine such data sets wherever they are located. integration into the WebLicht annotation platform Unsurprisingly, a strong focus of CLARIN has and CLARIN’s Federated Content Search FCS. In been laid so far on resources for European the future, the corpus will be made available via the languages. The integration of more data for non- KonText advanced corpus query interface for the European languages will broaden and extend Manatee-open corpus search engine (as used in the possible research questions that users of the NoSketchEngine). This will enable compatibility infrastructure can approach. Among others, the with the FCS-QL specification v2.0 and will allow CLARIN centre at the University of Leipzig is querying text and annotation layers such as part of working on expanding available resources for a speech and lemmas. variety of languages with a dedicated focus on lesser-resourced ones. 5 Conclusion and future work Based on standard procedures and workflows that have been proven effective for “in-house” In this paper we presented a new open source resources, the OSIAN corpus is processed, corpus based on well-known and reliable archived and published into the CLARIN international broadcasting platforms. After infrastructure. This publication includes cleaning and filtering processes, the datasets are automatically annotated with lemma and PoS tags.
11 See for example http://hdl.handle.net/11022/0000-0007- C65C-3 180
6
At the moment, this corpus comprises roughly 1 Dirk Goldhahn, Maciej Sumalvico, and Uwe billion tokens that have been stored in a uniform Quasthoff. 2016. Corpus Collection for Under- XML format. The XML format of the OSIAN Resourced Languages with More than One Million corpus will be publicly available for download and Speakers. In Collaboration and Computing for Under-Resourced Languages (CCURL), pages 67– use in research. In addition, the current version and 73, Portorož. any updates of the OSIAN corpus can be found through the CLARIN research infrastructure, Zeroual Imad and Lakhouaja Abdelhak. 2016. connecting them to central services such as VLO Adapting a decision tree based tagger for Arabic. In proceedings of the International Conference on and FCS for metadata and content search. Information Technology for Organizations In the future, we will extend the OSIAN corpus Development (IT4OD), pages 1–6. IEEE. to cover more international Arabic news with a Miloš Jakubíček, Adam Kilgarriff, Vojtěch Kovář, diachronic and geographical perspective to make Pavel Rychlỳ, and Vít Suchomel. 2013. The tenten the corpus an ideal choice to explore language corpus family. In proceedings of the 7th change and variation. Additionally, we will aim to International Corpus Linguistics Conference CL, improve the accuracy of the used tools as well as pages 125–127. to adopt new and meaningful forms of annotation. Vojtěch Kovář, Vít Baisa, and Miloš Jakubíček. 2016. Regarding CLARIN-integration, FCS 2.0 and the Sketch Engine for bilingual lexicography. querying of annotation layers is planned to be International Journal of Lexicography, 29(3):339– supported. Furthermore, we will explore the usage 352. of the OSIAN corpus in corpus linguistics, ANLP, Suleiman H. Mustafa. 2012. Word stemming for and AIR. Arabic information retrieval: The case for simple light stemming. Abhath Al-Yarmouk: Science & References Engineering Series, 21(1):2012. Tressy Arts, Yonatan Belinkov, Nizar Habash, Adam Uwe Quasthoff, Dirk Goldhahn, and Thomas Eckart. Kilgarriff, and Vit Suchomel. 2014. arTenTen: 2014. Building large resources for text mining: The Arabic Corpus and Word Sketches. Journal of King Leipzig Corpora Collection. In Text Mining, pages Saud University - Computer and Information 3–24. Springer. Sciences, 26(4):357–371. Motaz K. Saad and Wesam Ashour. 2010. Osac: Open Tim Buckwalter and Dilworth Parkinson. 2014. A source arabic corpora. In proceeding of the 6th frequency dictionary of Arabic: Core vocabulary for International Conference on Electrical and learners. Routledge. Computer Systems (EECS’10), volume 10. Jamie Callan, Mark Hoy, Changkuk Yoo, and Le Zhao. Vít Suchomel and Jan Pomikálek. 2012. Efficient web 2009. Clueweb09 data set. Available at crawling for large text corpora. In proceedings of http://lemurproject.org/clueweb09/. the seventh Web as Corpus Workshop (WAC7), Amina Chouigui, Oussama Ben Khiroun, and Bilel pages 39–43. Elayeb. 2017. ANT Corpus: An Arabic News Text Reem Swuaileh, Mucahid Kutlu, Nihal Fathima, Collection for Textual Classification. In Tamer Elsayed, and Matthew Lease. 2016. proceedings of the 14th ACS/IEEE International ArabicWeb16: A New Crawl for Today’s Arabic Conference on Computer Systems and Applications Web. In proceedings of the 39th International ACM (AICCSA 2017), pages 135–142, Hammamet, SIGIR conference on Research and Development in Tunisia. Information Retrieval, pages 673–676. ACM. Franciska de Jong, Bente Maegaard, Koenraad De Wajdi Zaghouani. 2017. Critical survey of the freely Smedt, Darja Fišer, and Dieter Van Uytvanck. 2018. available Arabic corpora. arXiv preprint CLARIN: Towards FAIR and Responsible Data arXiv:1702.07835. Science Using Language Resources. In Proceedings of the Eleventh International Conference on Imad Zeroual and Abdelhak Lakhouaja. 2018a. Arabic Language Resources and Evaluation (LREC 2018), Corpus Linguistics: Major Progress, but Still a Long pages 3259–3264. Way to Go. In Intelligent Natural Language Processing: Trends and Applications, Studies in Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff. Computational Intelligence, pages 613–636. 2012. Building Large Monolingual Dictionaries at Springer, Cham. the Leipzig Corpora Collection: From 100 to 200 Languages. In LREC, pages 759–765.
181
7
Imad Zeroual and Abdelhak Lakhouaja. 2018b. Data science in light of natural language processing: An overview. Procedia Computer Science, 127:82–91.
A Appendices
Region or Web-domain Nb. of country articles International news.un.org 693,629 arabic.euronews.com ara.reuters.com namnewsnetwork.org arabic.sputniknews.com Middle-east aljazeera.net 366,211 alarabiya.net Algeria djazairess.com 588,514 Australia eltelegraph.com 4,614 Canada arabnews24.ca 30,135 halacanada.ca China arabic.cctv.com 1,365 Egypt alwatanalarabi.com 85,351 France france24.com 74,718 Iran alalam.ir 344,011 Iraq iraqakhbar.com 28,248 Germany dw.com 117,261 Jordon sarayanews.com 49,461 Morocco www.marocpress.com 188,045 Palestine al-ayyam.ps 81,495 Qatar raya.com 8,986 Russia arabic.rt.com 57,238 Saudi Arabia alwatan.com.sa 1,512 Sweden alkompis.se 33,790 Syria syria.news 36542 Tunisia www.turess.com 495,674 Turkey turkey-post.net 76,638 aa.com.tr UAE emaratalyoum.com 25,081 UK bbc.com 10,686 USA arabic.cnn.com 113,557
Table 1: List of crawled web-domains
182
8