Are Multilingual Models Effective in Code-Switching? Genta Indra Winata, Samuel Cahyawijaya, Zihan Liu, Zhaojiang Lin, Andrea Madotto, Pascale Fung Center for Artificial Intelligence Research (CAiRE) The Hong Kong University of Science and Technology [email protected] Abstract Several approaches have been explored in code-switching representation learning in NLU. Multilingual language models have shown de- Character-level representations have been utilized cent performance in multilingual and cross- lingual natural language understanding tasks. to address the out-of-vocabulary issue in code- However, the power of these multilingual mod- switched text (Winata et al., 2018c; Wang et al., els in code-switching tasks has not been fully 2018b), while external handcrafted resources such explored. In this paper, we study the effective- as gazetteers list are usually used to mitigate ness of multilingual language models to under- the low-resource issue in code-switching (Aguilar stand their capability and adaptability to the et al., 2017; Trivedi et al., 2018); however, this mixed-language setting by considering the in- approach is very limited because it relies on the ference speed, performance, and number of pa- rameters to measure their practicality. We con- size of the dictionary and it is language-dependent. duct experiments in three language pairs on In another line of research, meta-embeddings named entity recognition and part-of-speech have been used in code-switching by combin- tagging and compare them with existing meth- ing multiple word embeddings from different lan- ods, such as using bilingual embeddings and guages (Winata et al., 2019a,b). This method multilingual meta-embeddings. Our findings shows the effectiveness of mixing word representa- suggest that pre-trained multilingual models tions in closely related languages to form language- do not necessarily guarantee high-quality rep- agnostic representations, and is considered very resentations on code-switching, while using meta-embeddings achieves similar results with effective in Spanish-English code-switched named significantly fewer parameters. entity recognition tasks, and significantly outper- forming mBERT (Khanuja et al., 2020) with fewer 1 Introduction parameters. Learning representation for code-switching has be- While more advanced multilingual language come a crucial area of research to support a greater models (Conneau et al., 2020) than multilin- variety of language speakers in natural language gual BERT (Devlin et al., 2019) have been pro- processing (NLP) applications, such as dialogue posed, their effectiveness is still unknown in code- system and natural language understanding (NLU). switching tasks. Thus, we investigate their effec- Code-switching is a phenomenon in which a per- tiveness in the code-switching domain and compare them with the existing works. Here, we would like arXiv:2103.13309v1 [cs.CL] 24 Mar 2021 son speaks more than one language in a conver- sation, and its usage is prevalent in multilingual to answer the following research question, “Which communities. Yet, despite the enormous number of models are effective in representing code-switching studies in multilingual NLP, only very few focus on text, and why?." code-switching. Recently, contextualized language In this paper, we evaluate the representation models, such as mBERT (Devlin et al., 2019) and quality of monolingual and bilingual word embed- XLM-R (Conneau et al., 2020) have achieved state- dings, multilingual meta-embeddings, and multi- of-the-art results on monolingual and cross-lingual lingual language models on five downstream tasks tasks in NLU benchmarks (Wang et al., 2018a; Hu on named entity recognition (NER) and part-of- et al., 2020; Wilie et al., 2020; Liu et al., 2020; Lin speech tagging (POS) in Hindi-English, Spanish- et al., 2020). However, the effectiveness of these English, and Modern Standard Arabic-Egyptian. multilingual language models on code-switching We study the effectiveness of each model by con- tasks remains unknown. sidering three criteria: performance, speed, and the Figure 1: Model architectures for code-switching modeling: (a) model using word embeddings, (b) model us- ing multilingual language model, (c) model using multilingual meta-embeddings (MME), and (d) model using hierarchical meta-embeddings (HME). number of parameters that are essential for practi- quired by each model over different sequence cal applications. Here, we set up the experimental lengths in a GPU. Thus, we are able to un- setting to be as language-agnostic as possible; thus, derstand which model to choose in a practical it does not include any handcrafted features. scenario. Our findings suggest that multilingual pre- 2 Representation Models trained language models, such as XLM-RBASE, achieves similar or sometimes better results than In this section, we describe multilingual models the hierarchical meta-embeddings (HME) (Winata that we explore in the context of code-switching. et al., 2019b) model on code-switching. On the Figure1 shows the architectures for a word em- other hand, the meta-embeddings use word and beddings model, a multilingual language model, subword pre-trained embeddings that are trained and the multilingual meta-embeddings (MME), and using significantly less data than mBERT and HME models. XLM-RBASE and can achieve on par performance to theirs. Thus, we conjecture that the masked lan- 2.1 Word Embeddings guage model is not be the best training objective for 2.1.1 FastText representing code-switching text. Interestingly, we In general, code-switching text contains a primary found that XLM-RLARGE can improve the perfor- language the matrix language (ML)) as well as a mance by a great margin, but with a substantial cost secondary language (the embedded language (EL)). in the training and inference time, with 13x more To represent code-switching text, a straightforward parameters than HME-Ensemble for only around idea is to train the model with the word embed- a 2% improvement. The main contributions of our dings of the ML and EL from FastText (Grave et al., work are as follows: 2018). Code-switching text has many noisy tokens • We evaluate the performance of word embed- and sometimes mixed words in the ML and EL that dings, multilingual language models, and mul- produce a “new word”, which leads to a high num- tilingual meta-embeddings on code-switched ber of out-of-vocabulary (OOV) tokens. To solve NLU tasks in three language pairs, Hindi- this issue, we utilize subword-level embeddings English (HIN-ENG), Spanish-English (SPA- from FastText (Grave et al., 2018) to generate the ENG), and Modern Standard Arabic-Egyptian representations for these OOV tokens. We conduct (MSA-EA), to measure their ability in repre- experiments on two variants of applying the word senting code-switching text. embeddings to the code-switching tasks: FastText (ML) and FastText (EL), which utilize the word • We present a comprehensive study on the ef- embeddings of ML and EL, respectively. fectiveness of multilingual models on a vari- 2.1.2 MUSE ety of code-switched NLU tasks to analyze the practicality of each model in terms of per- To leverage the information from the embeddings formance, speed, and number of parameters. of both the ML and EL, we utilize MUSE (Lample et al., 2018) to align the embeddings space of the • We further analyze the memory footprint re- ML and EL so that we can inject the information of the EL embeddings into the ML embeddings, such as mBERT, and resumes the pre-training of and vice versa. We perform alignment in two di- the upper layers of the model. The method aims rections: (1) We align the ML embeddings to the to increase the robustness of the model to various vector space of the EL embeddings (denoted as typography styles. MUSE (ML ! EL)); (2) We conduct the align- ment in the opposite direction, which aligns the EL 2.3 Multilingual Meta-Embeddings embeddings to the vector space of the ML embed- dings (denoted as MUSE (EL ! ML)). After the The MME model (Winata et al., 2019a) is formed embeddings alignment, we train the model with the by combining multiple word embeddings from dif- aligned embeddings for the code-switching tasks. ferent languages. Let’s define w to be a sequence of words with n elements, where w = [w1; : : : ; wn]. 2.2 Multilingual Pre-trained Models First, a list of word-level embedding layers is used (w) Pre-trained on large-scale corpora across numer- Ei to map words w into embeddings xi. Then, ous languages, multilingual language models (De- the embeddings are combined using one out of the vlin et al., 2019; Conneau et al., 2020) possess the following three methods: concat, linear, and self- ability to produce aligned multilingual representa- attention. We briefly discuss each method below. tions for semantically similar words and sentences, which brings them advantages to cope with code- Concat This method concatenates word embed- mixed multilingual text. dings by merging the dimensions of word represen- tations into higher-dimensional embeddings. This 2.2.1 Multilingual BERT is one of the simplest methods to join all embed- Multilingual BERT (mBERT) (Devlin et al., 2019), dings without losing information, but it requires a a multilingual version of the BERT model, is pre- larger activation memory than the linear method. trained on Wikipedia text across 104 languages CONCAT with a model size of 110M parameters. It has been xi = [xi;1; :::; xi;n]: (1) shown to possess a surprising multilingual abil- ity and to outperform existing strong models on Linear This method sums all word embeddings multiple zero-shot cross-lingual tasks (Pires et al., into single word embeddings with equal weight 2019; Wu and Dredze, 2019). Given its strengths without considering each embedding’s importance. in handling multilingual text, we leverage it for The method may cause a loss of information and code-switching tasks. may generate noisy representations. Also, though 2.2.2 XLM-RoBERTa it is very efficient, it requires an additional layer to project all embeddings into a single-dimensional XLM-RoBERTa (XLM-R) (Conneau et al., 2020) space if one embedding is larger than another.