Applying Lessons from 20 Years of African Language Technology
Total Page:16
File Type:pdf, Size:1020Kb
Looking forward by looking back: Applying lessons from 20 years of African language technology Martin Benjamin École Polytechnique Fédérale de Lausanne Lausanne, Switzerland [email protected] Mohomodou Houssouba Songhay.org Basel, Switzerland [email protected] Abstract This paper takes a frank look at what has and has not been achieved in African language technology during the past two decades. Several questions are addressed: What was the status of technology for African languages 20 years ago? What were the major initiatives during that time? What were their successes and failures? What can we learn from these experiences? How does this inform the work that we are planning going forward? Examining in particular the history of Swahili, it is argued that technology projects have often achieved their expressed aims, but have collectively not significantly advanced the normalization of African languages as operable within the technical sphere, even while Africa has become blanketed with mobile technology. It is argued that future projects will succeed only by asserting the goal that technology of 2035 must be fully operational in users' primary languages, and gearing policy, funding, and individual project efforts toward gathering and deploying linguistic data for a large number of African languages to meet cutting-edge technologies as they emerge. Keywords: HLT, localization, African languages, digital content, lexicography, Wikipedia, Swahili, equity, policy 1. African languages and technology in discuss the obstacles and opportunities those 1995 experiences reveal, and make suggestions for building In 1995, when graphical browsers were first opening those lessons into work going forward. the Internet to dial-up home-computer users in some ICT in 1995 was dominated by English, a Northern countries, African languages had virtually consequence of the locations of the predominant early no representation in the technical sphere. The developers and mass markets. European and Asian computer support department of a major American languages improved their resources over the next two university questioned when a project proposed for a decades in response to market forces, government large African language would ever be used by its initiatives, and academic interest. It is difficult to native speakers. At a time when Tanzania had just one remember that in 1995, rendering standard European telephone per each 300-odd inhabitants (and only diacritical characters such as “á” demanded 2000 mobile phones),1 and Mali had 17,164 phones in contortions such as “á”. African characters such the entire country,2 the notion that Africa would have as Yoruba “ọ̀” were literally impossible to type. With more than two-thirds of a billion mobile devices essentially no electronic wordlists for African twenty years later3 was unthinkable. Arguably, this languages, basic tools such as spellcheckers or OCR failure to imagine the future led to chronic could not exist. While the full text search of known unpreparedness; while the number of devices and Internet pages that became possible with the release reach of networks available to Africans have grown of WebCrawler in 1994 was in principle language exponentially, the ability to use those resources in and neutral, African language content was almost entirely for even the largest of the continent’s 2000 languages absent online and thus search was irrelevant. Before has barely gotten off the ground. In this paper, we the release of the Netscape graphical web browser at review some of the major technological developments the end of 1994, few organizations had even in support of African languages during that time, contemplated having a “homepage”, to say nothing of translated versions of their interface or content. Windows 95 was available to the brave4 for French, 1 Tanzania Communications Regulatory Authority, German, Italian, Russian, Arabic, Hebrew, Japanese, “Trends of Telephone Subscription”, https://www. Korean, and Traditional Chinese, and a Swedish tcra.go.tz/index.php/25-statistics, accessed 9/9/15 version of Word was spotted in Dar es Salaam by 2 The World Bank Databank, “Fixed Telephone 1996, but neither Microsoft nor most other Subcriptions”, http://data.worldbank.org/indicator/ corporations attempted localization for Swahili or IT.MLT.MAIN?page=3, accessed 9/9/15 other large African languages until a decade ago at the 3 The World Bank Databank, “World Development Indicators: Power and communications, Table 5.11”, 4 Setting up Windows 95 for Multiple Languages, http://wdi.worldbank.org/table/5.11, accessed 9/9/15 http://www.ficorp.com/multi/, accessed 9/9/15 earliest. Video game consoles had not yet appeared on select languages, including the creation of proposed the market in most African countries. Other hardware, ICT terminologies,6 assisting the production of such as keyboards and mobile phones, was still in the African language content. A third theme has been early days of supporting European languages other work on language data, including building lexicons than English (Diki-Kidiri 2007). In essence, “African and corpora (Badenhorst et al 2011, Chiarcos et al language technology” was oxymoronic in 1995, and 2011, Bosch 2014), and methods for data extraction few people envisioned that would change. and analysis such as part-of-speech tagging (De Pauw et al 2012), named entity recognition, morphological 2. Major initiatives analysis (Katushemererwe 2013, Littell et al 2014), The notion of supporting African languages was not and efforts toward machine translation (Gasser 2012). new in 1995. A Conference of Education Ministers Some work has been done on acoustic models (Gelas took place in Tehran in 1965, at the peak of the et al 2011), with advances by the Meraka Institute for independence movement, which stipulated the use of automatic speech recognition for the official South local languages to enact national literacy programs. African languages, likely to play a central role in Subsequent summits focused on specific tasks human/machine interaction in the next twenty years, intended to equip languages as instruments of social that illuminate both the possibilities of more complex development and governance. Yet, the development NLP applications that tailor extant technologies to of language resources held low priority in most African languages, and their paucity for most of the countries. Tanzania was the notable leader, with a continent (Barnard et al 2010). highly successful Swahili literacy campaign in the 1970s, conducted by an army of passionate teachers 3. Analysis: successes, failures, and armed with a few roughly-printed instructional texts. reasons for both Yet even the story of Swahili mostly left out The various projects above have often succeeded in Tanzania’s other 120 languages. Mali began a school their own terms, but have collectively done little to reform in 1962 that was intended to join four normalize African language use within technological “national” languages with French as languages of domains.7 Experiences with Swahili provide some instruction, expanded to 10 in 1982 and 13 by 1993, insight.8 with print resources emerging such as dictionaries, technical terminologies, and translations of important documents (Houssouba 2007). Were education 6 ministers to have had technologies available for their “ANLoc ICT Terminology Available to Download: nations’ languages in 1995, many would assuredly Completed Terminology Sets”, https://kamusi.org/ content/anloc-ict-terminology-available-download, have worked for their adoption. accessed 10/9/15 There was no home-grown technology industry, 7 however, outside of South Africa and Egypt. Africans The situation in South Africa is well-discussed in who could gain training opportunities in IT abroad Grover et al (2011). They find that, “whilst there is a tended to stay abroad. Without a grand vision of considerable level of HLT activity in South Africa, African language technology as something that could there are significant differences in the amount of be instigated locally, efforts were left to random activity across the eleven languages, and in general academic research centers, volunteers, and the language resources and applications currently available are of a very basic nature (p. 282). corporations operating in the interest of short-term 8 altruism and long-term profit. A parallel discussion of Mali could not be included Osborn (2010) presents a full overview of the for space constraints. Very briefly, localized products projects that grew after 1995. Much of the work from for Malian languages are mostly the fruit of volunteer that time centered on the enabling environment: initiatives: Firefox in Songhay, completed by a creating fonts and standardizing characters in Malian team, is the only browser available for a Unicode, restoring unusual diacritics in text national language, and the MAKDAS association has recognition for languages such as Gĩkũyũ, software produced word-processing software with embedded for keyboard input, detailing language parameters for keyboards for ten local languages (bataki.org). the Common Locales Data Repository, and, through Currently, the leading telecom company is partnering the African Network for Localisation (ANLoc), with Mozilla to localize Firefox OS for smartphones convening technologists and policy planners to into Bambara, the country’s main language, by the promote ongoing development (Sinha and Hyma end of 2015. Mali recently issued two policy 2013).5 Another strand