Automatic Construction of Sememe Knowledge Bases via Dictionaries

Fanchao Qi1,2, Yangyi Chen2,4∗, Fengyu Wang1,2, Zhiyuan Liu1,2,3, Xiao Chen5, Maosong Sun1,2,3† 1Department of Computer Science and Technology, Tsinghua University, Beijing China 2Beijing National Research Center for Information Science and Technology 3Institute for Artificial Intelligence, Tsinghua University, Beijing, China 4Huazhong University of Science and Technology 5Huawei Noah’s Ark Lab [email protected]

Abstract word husband

A sememe is defined as the minimum seman- sense “married man” “carefully use” tic unit in linguistics. Sememe knowledge human economize bases (SKBs), which comprise words anno- sememe tated with sememes, enable sememes to be ap- family male spouse plied to natural language processing. So far a large body of research has showcased the Figure 1: Sememe annotations of the word “husband” unique advantages and effectiveness of SKBs in HowNet. in various tasks. However, most languages have no SKBs, and manual construction of SKBs is time-consuming and labor-intensive. of human, male and child, while the meaning To tackle this challenge, we propose a sim- of “girl” can be expressed by human, female ple and fully automatic method of building an child human male female SKB via an existing dictionary. We use this and , where , , and method to build an English SKB and a French child are predefined sememes. It is even deemed SKB, and conduct comprehensive evaluations that this sememe-based semantic system as well as from both intrinsic and extrinsic perspectives. the sememe set is universal among different lan- Experimental results demonstrate that the au- guages, in which case sememes are also named tomatically built English SKB is even supe- universal semantic primitives (Wierzbicka, 1996). rior to HowNet, the most widely used SKB Sememes are implicit in words and cannot be that takes decades to build manually. And both the English and French SKBs can bring directly used in natural language processing (NLP). obvious performance enhancement in multiple Dong and Dong(2006) make a seminal contribu- downstream tasks. All the code and data of tion and put the sememe-based semantic system this paper (except the copyrighted dictionaries) into practice. They define a set of about 2, 000 can be obtained at https://github.com/ sememes and use them to annotate senses of over thunlp/DictSKB. 100, 000 Chinese and English words, whereupon 1 Introduction a sememe knowledge base (SKB) named HowNet is built up. Figure1 illustrates an example of how A word is the smallest linguistic element that can words are annotated with sememes in HowNet. arXiv:2105.12585v2 [cs.CL] 3 Jun 2021 be used on its own with a particular meaning, but As a sememe-based lexical knowledge base, not the smallest semantic unit (O’Grady et al., HowNet is very different from most other lexi- 1997). The meaning of a word can be divided into cal knowledge bases like WordNet (Miller, 1998), smaller components. In linguistics, a sememe is de- which extensionally explain meanings of words by fined as the minimum semantic unit of human lan- word-level relations, e.g., hyponym and meronym. guages (Bloomfield, 1926). Some linguists believe In contrast, HowNet provides intensional defini- that meanings of all words can be expressed by a tions using infra-word sememes. This distinct- limited set of predefined sememes (Goddard and ness brings special advantages to HowNet. First, Wierzbicka, 1994). For example, the basic mean- the sememe-to-word semantic compositionality en- ing of “boy” can be expressed by the compositions dows HowNet with particular suitability for inte- ∗Work done during internship at Tsinghua University gration into neural networks (Qi et al., 2019a; Li † Corresponding author. Email: [email protected] et al., 2019). The sememes of a word can be re- garded as semantic labels and easily incorporated French SKB and conduct both intrinsic and extrin- into the neural processing unit of the word, e.g., a sic evaluations. In intrinsic evaluation, we find cell of RNN (Qin et al., 2020). Second, the nature that both the SKBs possess high sememe annota- that a limited set of sememes are used to express tion consistency, and the English SKB performs meanings of unlimited words makes HowNet very even better than the English part of HowNet. In useful in low-data regimes, e.g., improving embed- extrinsic evaluation, we apply the dictionary-based dings of rare words (Sun and Chen, 2016; Niu et al., SKBs to several sememe-incorporated models orig- 2017), where sememes serve as a bridge between inally designed for HowNet and carry out experi- high-frequency and rare words. Thus far a large ments on different downstream tasks. Experimen- body of research has demonstrated the usefulness tal results show that incorporating the SKBs can of HowNet in various NLP tasks (Qi et al., 2020b). bring consistent performance enhancement, and HowNet is distinctive and valuable, but it cov- the English SKB-incorporated models even outper- ers only two languages. Most languages have no form HowNet-incorporated models. These results SKBs like HowNet, which deprives NLP in those demonstrate the usefulness and effectiveness of the languages of benefit from sememes. An obvious dictionary-based SKBs as well as the feasibility of solution to this problem is to build an SKB for each building SKBs via dictionaries. language manually, but it is not realistic because it To conclude, our contributions are threefold: (1) would be unimaginably time-consuming and labor- discovering the similarity between sememes and intensive.1 To address the challenge, previous stud- words in the controlled defining vocabulary, which ies try to extend HowNet to other languages by au- is the first time as far as we know; (2) proposing to tomatically predicting sememes for words in those automatically build an SKB via a dictionary, which languages (Qi et al., 2018, 2020a). However, exist- can be achieved by a simple and quick process; ing methods are not effective enough, and manual and (3) building an English SKB and a French effort is necessary to ensure the correctness of their SKB based on dictionaries and demonstrating their sememe prediction results. effectiveness in multiple downstream tasks. In this paper, we explore a fully automatic way to build an SKB for a language via dictionaries 2 Related Work with a controlled defining vocabulary. A dictio- 2.1 HowNet and Its Applications nary, especially a learner’s dictionary, usually uses Since HowNet was published (Dong and Dong, a well-chosen list of words to construct all its defini- 2003), it has attracted considerable attention of tions, and the word list is named controlled defining NLP researchers. In the era of statistical NLP, it vocabulary (CDV) (Atkins and Rundell, 2008). A plays a very important role in various NLP tasks CDV is composed of high-frequency words that including word similarity computation (Liu and Li, not only cover the vast majority of texts but also 2002), word sense disambiguation (Zhang et al., form a semantic basis so as to express meanings 2005; Duan et al., 2007), text classification (Sun of all other words (Nation and Waring, 2004). To et al., 2007), sentiment analysis (Zhu et al., 2006; some extent, words in a CDV can fit the definition Fu et al., 2013), etc. of sememes (Wierzbicka, 1996). This discovery inspires us to utilize a dictionary to build an SKB When deep learning becomes the mainstream by regarding the words in its CDV as sememes. approach of NLP, the usefulness of HowNet is also proved in diverse tasks including word represen- We design a quite simple and quick process for tation learning (Sun and Chen, 2016; Niu et al., automatically building SKBs based on dictionar- 2017), language modeling (Gu et al., 2018), seman- ies. First, a sememe set is constructed based on the tic composition (Qi et al., 2019a), sequence mod- CDV of a dictionary by removing words that are eling (Qin et al., 2020), reverse dictionary (Zhang not suitable as sememes (e.g., stop words), then et al., 2020), word sense disambiguation (Hou et al., sememes of words are extracted from correspond- 2020), textual adversarial attacking (Zang et al., ing definitions, and finally an SKB composed of 2020) and backdoor attacking (Qi et al., 2021). words annotated with sememes is established. We adopt the process to build an English SKB and a 2.2 Expansion of HowNet

1The construction of HowNet takes several linguistic ex- To tackle the challenge that many new words are perts more than two decades. not contained in HowNet, Xie et al.(2017) present the task of lexical sememe prediction, aiming to 3 Building an SKB via a Dictionary expand HowNet by automatically predicting se- memes for new words. They propose two simple In this section, we detail the process of building an and effective sememe prediction methods inspired SKB via a dictionary. We take the building process Longman Dictionary by recommendation system. Jin et al.(2018) fur- of an English SKB based on of Contemporary English ther incorporate Chinese characters into sememe (LDOCE) (Bullon, 2006), prediction and achieve higher performance when a highly influential English learner’s dictionary, as predicting sememes for Chinese words. an example, and the building method can be readily generalized to other languages or dictionaries.2 Another research line focuses on extending HowNet to other languages. Qi et al.(2018) pro- 3.1 Constructing the Sememe Set pose the task of cross-lingual lexical sememe pre- We first construct the sememe set from the CDV of diction, aiming to extend HowNet to a new lan- LDOCE by removing some words. LDOCE uses an guage by predicting sememes for words in that approximately 2, 000-word CDV named Longman language. Qi et al.(2020a) present a more effi- Defining Vocabulary (Bullock, 2011), which is de- cient way to extend HowNet to other languages, veloped from General Service List (West, 1953), a i.e., building a multilingual SKB based on Babel- famous high-frequency word list for English learn- Net (Navigli and Ponzetto, 2012). BabelNet is ers. The CDV includes some stop words such as composed of multilingual synsets that contain syn- “that” and “to”, which bear insignificant meanings onyms in many languages. Words (synonyms) in a and are not suitable as sememes. Thus, we filter synset have the same meaning and hence the same them out according to the stop word list of NLTK sememes. Therefore, they propose to predict se- (Loper and Bird, 2002). But negators like “not” are memes for the multilingual synsets, by which all retained because they are critical to the meanings the words in synsets will obtain predicted sememes of words. In addition, according to previous work at the same time. (Xie et al., 2017; Qin et al., 2020), sememes that Limited by the accuracy of sememe prediction, are annotated to too many or too few words are usu- manual examination is necessary if we want to put ally uninformative and ineffective to downstream the above HowNet expansion methods into service. applications. Therefore, we count the frequencies In contrast, our proposed dictionary-based SKB of words in the CDV occurring in all definitions construction method is completely automatic and and empirically remove the most frequent 1% and can build a usable SKB very quickly. the infrequent 10%. So far we have obtained the sememe set that is composed of 2, 046 sememes. 2.3 Applications of Dictionaries 3.2 Extracting Sememes from Definitions Dictionaries are handy and high-quality resources for NLP research. A main application of dictio- Next, we extract sememes for each sense of each naries is word sense disambiguation, where dictio- word from its definition. We take the word “beauti- naries play the role of sense inventory, and their ful” as a running example to illustrate the process definitions provide abundant semantic information of sememe extraction, as shown in Figure2. for each sense (Lesk, 1986; Luo et al., 2018a,b; “beautiful” has two senses in LDOCE, and Kumar et al., 2019; Huang et al., 2019; Du et al., both of them are adjective. For each sense, we 2019; Blevins and Zettlemoyer, 2020). The seman- first use NLTK to normalize its definition includ- tic information in dictionary definitions is also used ing tokenization and lemmatization. For exam- to improve word representation learning (Tissier ple, the definition of its first sense is normalized et al., 2017; Bahdanau et al., 2017; Bosc and Vin- into a sequence of tokens: {“someone”, “or”, cent, 2018; Scheepers et al., 2018). In addition, “something”, “that”, “be”, “beautiful”, “be”, “ex- dictionary definitions are also utilized in reverse tremely”, “attractive”, “to”, “look”, “at”}. Then dictionary (Hill et al., 2016; Pilehvar, 2019; Zhang we remove the tokens that are not in the se- et al., 2020), knowledge graph embedding (Zhong meme set. In the above example, “someone”, et al., 2015; Xie et al., 2016), reading comprehen- “something”, “or”, “that”, “be”, “to” and “at” sion (Long et al., 2017), etc. As far as we know, are removed. So far we obtain the sememes this paper is the first work to utilize dictionaries to 2The building process and evaluation results of the French build SKBs. SKB are given in AppendixA andB. beautiful someone, or, something, someone or something that is that, be, beautiful, be, extremely beautiful is extremely attractive attractive extremely, attractive, to look at attractive to, look, at look beautiful good good very good or giving you great very, good, or, give, give give pleasure you, great, pleasure pleasure pleasure

Word Dictionary Definition Token Sequence Sememes Distilled Sememes

Text Normalization Remove Non-Sememes Distill Sememes

Figure 2: The process of extracting sememes from dictionary definitions for the word “beautiful”.

SKB #Word/Phrase #Sense #Sememe #AvgSem 2020). In a sememe-based word substitution strat- HowNet 50,879 111,519 2,187 2.26 egy (Zang et al., 2020), more sememes per sense EDSKB 70,218 105,160 2,046 6.03 mean fewer substitute words that share the same EDSKB∗ 70,218 105,160 1,6823 2.04 sememes can be found, which will decrease the Table 1: Statistics of EDSKB, its distilled version final adversarial attack success rate. To address EDSKB∗ and the English part of HowNet.4 #AvgSem this problem, we intend to craft an extra distilled denotes the average sememe number per sense. version of EDSKB by distilling its sememes of senses. of the first sense of “beautiful”: {beautiful, To this end, we need to determine the impor- extremely, attractive, look}. In a simi- tance of each sememe of a sense, and remove the lar way, we can obtain the sememes of its second relatively unimportant sememes. Here we resort sense: {good, give, pleasure}. to dependency parsing (Kubler et al., 2009). De- By repeating this process on all the words of pendency parsing is used to analyze syntactic struc- LDOCE, we obtain an English SKB that we name tures of a sentence by identifying the word that EDSKB. Its statistics are shown in Table1. another word is “dependent” on, e.g., the adjec- tive is dependent on the noun in an adjective-noun 3.3 Distilling Sememes of Senses phrase. We believe that the words with more de- pendents are more important in a definition. Hence, By comparison with HowNet, we find that the se- we define the importance score of a sememe for a meme set of EDSKB is smaller (EDSKB 2, 046 vs. sense as the number of the dependents of its origi- HowNet 2, 187), but its average sememe number nal word in the definition. per sense is much larger (EDSKB 6.03 vs. HowNet 2.26), which means the sememes of EDSKB are Then, we empirically remove the sememes utilized more fully and effectively. Moreover, anno- whose importance scores are below the highest tating a sense with more sememes can explain the importance score minus t for the senses having m sense more accurately and finely. Nevertheless, it or more sememes. Here t and m are two hyper- would also increase the distinguishability between parameters and are tuned to 1 and 4 respectively, different senses/words, which has an adverse effect based on the performance on the validation sets on some downstream tasks. For example, word- of downstream tasks, especially adversarial attack- level textual adversarial attacking conducts word ing. For example, for the first sense of “beautiful”, substitution to generate adversarial examples, in by using AllenNLP (Gardner et al., 2018) to con- which fewer substitute words usually lead to lower duct dependency parsing on its definition, we ob- attack success rates (Wang et al., 2019; Zang et al., tain the numbers of dependents of all the words in the definition. Correspondingly, we get the impor- 3 Notice that its sememe set shrinks because some sememes tance scores of the four sememes {beautiful, are not annotated to any senses anymore. 4The data of HowNet are obtained from OpenHowNet (Qi extremely, attractive, look}, which are et al., 2019b). 2, 0, 6 and 0 respectively. The highest impor- tance score is 6 and thus the sememes whose im- SKB MAP F1 portance scores are less than 5 are removed, i.e., HowNet 0.93 0.91 beautiful, extremely and look. Finally, EDSKB 0.88 0.86 the remaining sememe of the first sense of “beauti- EDSKB∗ 0.95 0.91 ful” is {attractive}. As for the second sense, it has only 3 sememes and all of them are retained Table 2: Sememe annotation consistency results. The (the sememe number threshold for sememe reduc- boldfaced results show statistically significant im- tion is m = 4). Therefore, its final sememes after provement over the best results from baselines with reduction are still {good, give, pleasure}. p<0.1 given by t-test, and the underlined results rep- resent having no significant difference.5 By repeating the above process on all the words, we obtain a distilled version of EDSKB (signified by EDSKB∗), whose average sememe number per more consistent. Correspondingly, the sememe an- sense is comparable with HowNet (2.04 vs. 2.26). notation consistency of an SKB is measured by Its detailed statistics are also shown in Table1. two sememe prediction evaluation metrics, namely Later experiments (on both English and French) mean average precision (MAP) and F1 score. show that the full version outperforms the distilled Table2 lists the evaluation results of sememe an- version in some downstream tasks while not in notation consistency. We can see that the distilled others. In practice, we can build both full and version of EDSKB has overall higher consistency distilled versions and conduct experiments to see than HowNet, and the full version of EDSKB yields which one is better in a specific task. It is affordable lower consistency results. It is not strange because to build and evaluate two versions. CCSA is based on sememe prediction and accord- ing to previous work (Qi et al., 2020a), senses with 4 Intrinsic Evaluation more sememes usually have lower prediction per- formance. Since the full version of EDSKB has In this section, we conduct an intrinsic evaluation to much more sememes per sense than HowNet, it is assess the sememe annotation consistency of ED- actually not fair to compare their consistency using SKB. Sememe annotation consistency measures CCSA. The distilled version of EDSKB has a sim- how compatible the sememe annotations for differ- ilar average sememe number as HowNet, and its ent words/senses are, e.g., whether two synonyms superior results can demonstrate the great consis- are annotated with exactly the same sememes. The tency of the dictionary-based SKB. sememe annotation consistency of an SKB not only reflects its intrinsic quality but also has impact on 5 Extrinsic Evaluation its effectiveness in downstream tasks. We evaluate both full and distilled versions of In this section, we conduct extrinsic evaluations EDSKB, and the English part of HowNet for com- to assess the effectiveness of EDSKB in down- parison. We adopt a sememe consistency assess- stream tasks. We pick three representative sememe- ment method named CCSA (Liu et al., 2020), incorporated neural network models that are used which is designed for HowNet originally but can for language modeling, sequence modeling and tex- be used for any SKB. This method is motivated by tual adversarial attacking tasks, respectively. All of the idea that semantically close senses should have them are originally designed for HowNet and have similar sememes, which conforms to the linguistic demonstrated efficacy on their respective tasks. definition of sememes. It actually implements a 5.1 Language Modeling sememe prediction process that predicts sememes In this subsection, we try to apply EDSKB to the for a small proportion of senses according to the task of language modeling. We use SDLM (Gu sememe annotations of the other senses. The se- et al., 2018), a sememe incorporation method for meme prediction method it adopts is based on col- language models, to incorporate EDSKB into two laborative filtering (Xie et al., 2017), and tends to representative language models based on recurrent predict the sememes that are annotated to seman- neural networks (RNNs). tically close senses to the target sense. Therefore, Language modeling is aimed at predicting the higher sememe prediction performance means the next word given previous context (Bengio et al., semantically close senses are annotated with more similar sememes, and the sememe annotations are 5The same is true for the following tables. 2003). Language models based on RNNs, espe- Dataset PTB WikiText-2 cially LSTMs (Hochreiter and Schmidhuber, 1997), Model Valid Test Valid Test are very popular, which use RNNs to encode the Tied LSTM 63.92 63.98 53.10 51.41 previous text into a vector and then feed the vec- +HowNet 58.93 58.95 48.83 47.28 tor to a classifier to predict the next word. SDLM +EDSKB 58.81 58.82 43.38 42.15 reforms the prediction process. Instead of directly +EDSKB∗ 60.17 60.15 45.18 42.59 predicting the next word, SDLM predicts sememes AWD-LSTM 58.89 59.24 45.29 44.13 first, then senses and finally the next word. +HowNet 58.95 58.92 46.84 45.29 +EDSKB 56.94 57.13 42.44 41.25 Base Models We use two representative LSTM- +EDSKB∗ 58.63 58.59 43.85 43.95 based language models as the base models into which EDSKB is incorporated by SDLM. Table 3: Perplexity results of different language models • Tied LSTM (Zaremba et al., 2014), which on the validation and test sets of PTB and WikiTex-2. enhances a vanilla two-layer LSTM language model by introducing dropout and weight tying. even the HowNet-incorporated models. These re- We use its large version whose word embedding sults demonstrate the effectiveness of EDSKB in and hidden vector sizes are 1, 500. language modeling. • AWD-LSTM (Merity et al., 2018), which adopts several regularization and optimization 5.2 Sequence Modeling strategies including DropConnect (Wan et al., In this subsection, we incorporate EDSKB into 2013) and non-monotonically triggered average RNNs to improve their sequence modeling abil- stochastic gradient descent, and is a very strong ity by SememeCell (Qin et al., 2020), a sememe baseline language model. Its hidden vector size incorporation method for enhancing RNNs. is 1, 150 and word embedding size is 400. SememeCell uses a special RNN cell to encode Baseline Methods In addition to the two original sememes of a word into a latent vector and trans- base models, we additionally use SDLM to incor- mits it to the corresponding RNN cell of the word, porate HowNet into the base models as baseline aiming to inject the semantic information of se- methods. memes into RNNs. It has demonstrated its effec- tiveness in improving the sequence modeling abil- Datasets We choose two benchmark language ity of RNNs in multiple downstream tasks, includ- modeling datasets for evaluation, namely Penn ing natural language inference, sentiment analysis Treebank (PTB) (Marcus et al., 1993) and and paraphrase detection (Qin et al., 2020). WikiText-2 (Merity et al., 2017). PTB consists of news stories from the Wall Street Journal. The Base Models Following Qin et al.(2020), we numbers of tokens in its training, validation and test choose two most representative RNNs, namely sets are 887, 521, 70, 390 and 78, 669, respectively. LSTM, GRU (Cho et al., 2014), and their bidi- WikiText-2 is made up of Wikipedia articles, and rectional versions (BiLSTM and BiGRU) as the it has 2, 088, 628, 217, 646 and 245, 569 tokens in base models, into which sememes are incorporated its training, validation and test sets. by SememeCell.

Experimental Settings In our experiments, we Baseline Methods In addition to the vanilla and use the official implementation of SDLM and its HowNet-incorporated RNNs, we also design an- default hyper-parameters as well as training meth- other two baseline methods. ods. The evaluation metric is perplexity. The lower • +Pseudo. RNNs incorporated with either ED- perplexity a language model computes, the better SKB or HowNet have a little more parameters the language model is. than vanilla RNNs. To eliminate the possible effect brought by more parameters, we build Experimental Results Table3 lists the perplex- a pseudo-SKB named Pseudo. Specifically, for ity results on the two datasets. We observe that each sense in EDSKB, we substitute its sememes the models incorporated with EDSKB, especially with the same number of meaningless labels. the full version, consistently outperform the two The labels are randomly sampled from a label base models without sememe incorporation and set with the same size as the sememe set of ED- SKB. We use SememeCell to incorporate this Dataset Method LSTM GRU BiLSTM BiGRU pseudo-SKB into the two base models as base- vanilla 80.66 82.00 81.30 81.61 lines, which have exactly the same numbers of +Pseudo 81.28 80.90 81.91 82.07 +HowNet 81.87 82.90 82.55 83.15 SNLI parameters as EDSKB-incorporated models. +Definition 81.62 82.80 81.10 83.22 • +Definition. EDSKB is obtained from dictio- +EDSKB 82.32 83.18 82.54 83.55 ∗ nary definitions by the transformation from a +EDSKB 81.78 82.10 82.11 82.35 sequence of words (definition) into several dis- vanilla 74.17 76.37 77.62 78.76 +Pseudo 73.96 75.44 76.16 78.20 crete semantic labels (sememes). We intend to +HowNet 76.47 78.57 77.66 76.25 CR compare the EDSKB-incorporated models and +Definition 76.29 78.20 77.19 77.77 models incorporated with the complete dictio- +EDSKB 77.51 79.68 78.95 78.88 ∗ nary definitions. Since SememeCell only takes +EDSKB 75.09 77.54 76.90 78.18 a vector (i.e., the sum of sememe embeddings) Table 4: Accuracy results of different models on the as input, we can leverage it to incorporate defi- test sets of SNLI and CR. nitions into RNNs by encoding definitions into vectors with a sentence encoder. Specifically, we choose the powerful pre-trained language embedding size, we choose the medium version of model BERT (Devlin et al., 2019) as the sen- BERT, which has 512-dimensional hidden vectors 6 tence encoder and use the hidden vector of the and 8 layers. As for evaluation metrics, we use [CLS] token as the definition embedding. The accuracy for both NLI and sentiment analysis. definition-incorporated RNN models are also Experimental Results Table4 shows the evalua- baselines. tion results on the test sets of SNLI and CR. We can Downstream Tasks and Datasets RNNs are ba- see that RNNs incorporated with dictionary-based sic sequence encoders and can be used in many SKB, especially the full version (+EDSKB), yield downstream NLP tasks. Following Qin et al. overall better results than vanilla RNNs, which (2020), we choose two representative tasks to eval- proves that the dictionary-based SKB can improve uate the sentence modeling ability of EDSKB- the sequence modeling ability of RNNs. Further- incorporated RNNs. more, the +EDSKB models outperform +Pseudo • Natural language inference (NLI), which is models that have the same number of parameters, aimed at determining whether a natural language +Definition models that have the same semantic in- hypothesis can be inferred from a premise. It is a formation source, and +HowNet models that incor- typical sentence pair classification task. We use porate another SKB. These results demonstrate the the SNLI dataset (Bowman et al., 2015) for eval- superiority of discrete sememes over definitions, uation. SNLI contains about 570, 000 English and the advantage of dictionary-based SKB over premise-hypothesis pairs, and each pair is manu- HowNet in enhancing RNNs. +Pseudo performs ally labeled one of three relation labels, namely slightly better than vanilla in some cases, which “entailment”, “contradiction” and “neutral”. is probably because +Pseudo utilizes the random meaningless labels as noises. The addition of noise • Sentiment analysis, which aims to recognize the has been proven a regularization method for mit- sentiment orientation of a sentence and is a typ- igating overfitting and improving performance in ical single sentence classification task. Follow- neural networks (Bishop, 1995). ing Qin et al.(2020), we use the CR dataset (Hu and Liu, 2004) for evaluation. It contains 5.3 Textual Adversarial Attacking about 8, 000 product reviews and each review is In this subsection, we investigate the effectiveness labeled with “positive” or “negative”. of EDSKB in textual adversarial attacking. Experimental Settings We use the official im- Adversarial attacking has attracted considerable plementation of SememeCell (Qin et al., 2020) and research attention recently, mainly because it can the default hyper-parameter settings and training reveal the vulnerability of neural network models methods, where the embedding size (for both word and help improve their robustness and interpretabil- and sememe embeddings) is 300 and hidden size is ity (Xu et al., 2020). Adversarial attacks use ad- 2, 048. In the baseline method +Definition, to keep 6https://github.com/google-research/ the definition vector size comparable with sememe bert versarial examples (Szegedy et al., 2014), which Victim Attack Method ASR %M %IGE PPL are maliciously crafted by perturbing the original +Synonym 79.0 10.45 7.59 593.09 model input, to fool the victim model. In textual +Definition 90.0 8.76 7.56 518.71 BiLSTM +HowNet 93.6 9.02 2.57 468.92 adversarial attacking, word-level attack methods, +EDSKB 26.5 8.27 3.77 538.46 mainly based on word substitution, are a kind of +EDSKB∗ 94.0 8.29 1.27 507.34 popular attack method and have demonstrated over- +Synonym 81.3 9.22 8.00 576.82 all better attack performance (Wang et al., 2019). +Definition 86.3 8.03 7.18 538.00 Zang et al.(2020) decompose the process of BERT +HowNet 91.2 8.25 2.08 503.06 +EDSKB 29.7 8.10 3.36 485.00 word-level attacks into two steps: (1) determin- +EDSKB∗ 93.3 7.66 1.07 544.51 ing the substitute set for each word in the orig- inal input via a word substitution strategy, e.g., Table 5: Adversarial attack results of different word synonym-based and word embedding-based sub- substitution strategies. ASR is short for attack success stitution strategies; and (2) searching the combi- rate. %M, %IGE and PPL denote word modification nations of each original word’s substitutes for ad- rate, increase rate of grammatical errors and perplexity, respectively. versarial examples that can successfully fool the victim model. They also propose an adversarial attack approach tution strategy. that employs a sememe-based word substitution In this task, the +Pseudo baseline in the previous strategy and achieves strong attack performance. section cannot work because it would regard ran- The sememe-based word substitution strategy es- dom words as substitutes of the target word. sentially regards a word w1 as the substitute of another word w2, if one sense of w1 has the same Victim Models and Datasets Following Zang sememes as one sense of w2, according to an SKB. et al.(2020), we choose BiLSTM and BERT, specif- We use this approach to conduct textual adversarial ically BERTBASE as the victim models we intend attacks and measure the attack performance. to attack. The evaluation task is sentiment analysis and the evaluation dataset is SST-2 (Socher et al., Baseline Methods In addition to the original 2013). SST-2 comprises about 10, 000 sentences sememe-based attack approach that uses HowNet in movie reviews and each sentence is labeled with as the SKB, we choose some other baseline meth- “positive” or “negative”. The accuracy results of ods for comparison. Notice that all these baseline BiLSTM and BERT on the test set of SST-2 are methods use the same approach to search for ad- 83.75 and 90.28. versarial examples (the aforementioned step 2) and differ in word substitution strategies (step 1) only. Experimental Settings We use the official im- • +Synonym, the attack method that uses plementation of the sememe-based attack approach synonym-based word substitution strategy. Fol- (Zang et al., 2020) and the default hyper-parameter lowing previous work (Ren et al., 2019), we use settings. WordNet as the thesaurus and the words in a Evaluation Metrics Following Zang et al. synset can be regarded as substitutes of each (2020), we use attack success rate to measure the ef- other. fectiveness of an attack method and three metrics to • +Definition, the attack method that uses a assess the quality of its adversarial examples. The definition-based word substitution strategy. In- three metrics are (1) word modification rate, the spired by word embedding-based word substi- percentage of words in the original input that are tution, we encode the definition of each sense perturbed; (2) increase rate of grammatical errors of words into a vector and define the similarity in adversarial examples compared with original in- between two words as the cosine similarity be- put, where LanguageTool grammar checker is used; tween their closest definition vectors. Then, a and (3) perplexity given by GPT-2 (Radford et al., certain number of words that are most similar 2019) that is used to measure the fluency of adver- to the target word are regarded as its substitutes. sarial examples. The lower the three metrics are, Specifically, we still use the medium-size BERT the better the quality of adversarial examples is. to encode definitions into 512-dimensional vec- tors. And the number of substitutes of each word Experimental Results According to Table5, we is the same as that in the sememe-based substi- find that the attack method based on EDSKB∗ Word SKB Sememes HowNet human, occupation, entertainment, compile, shows screenwriter EDSKB someone, write, play, film, television EDSKB∗ write, play, film, television HowNet InstitutePlace, ProperName, produce, software, LookFor, document, information, internet Sense 1: bird, make, high, small, short, sound EDSKB tweet Sense 2: service, message, network, short, send, use, social Sense 1: bird, sound EDSKB∗ Sense 2: message, send, use, network

Table 6: Two cases of sememe annotations in HowNet, EDSKB and EDSKB∗. not only achieves the highest attack success rates HowNet, e.g., message in EDSKB/EDSKB∗ can but also generates adversarial examples with over- better describe the core meaning of “tweet” than all higher quality. These results show that the document and information in HowNet. dictionary-based SKB EDSKB∗ can better capture From the two cases, we can see the advan- the semantic relations between words and find ap- tage of the dictionary-based SKBs over HowNet propriate substitutes for adversarial attacks. Attack in terms of sememe annotations. We hope that success rates of the EDSKB-based method are ex- the dictionary-based SKBs can be used to perfect tremely low. It is because EDSKB has too many HowNet by supplying more senses and annotating sememes per sense, which causes the found substi- more suitable sememes. tutes to be very few (EDSKB 1.6, EDSKB∗ 12.6 and HowNet 15.3 on average), according to the 7 Conclusion and Future Work sememe-based word substitution strategy that re- quires substitutes to have the same sememes. In this paper, we propose to utilize a dictionary to build an SKB for the first time, which can be imple- 6 Case Study on Sememe Annotations mented by a simple, quick and fully automatic pro- cess. We try utilizing existing dictionaries to build In this section, we give two cases of sememe anno- an English SKB and a French SKB, and demon- tations in EDSKB and EDSKB∗ as well as HowNet strate their effectiveness on multiple NLP tasks. in Table6. Extensive experimental results prove the reliability The first case is the word “screenwriter”. In and practicality of our idea about dictionary-based HowNet, this word has only one sense that is anno- SKB construction. tated by five sememes, as listed in the second row It is worth mentioning that although EDSKB of Table6. As for EDSKB and EDSKB ∗, accord- delivers better empirical results than HowNet, ing to Longman Dictionary of Contemporary En- HowNet has its unique advantages including better glish (LDOCE), this word also has only one sense interpretability and multilinguality. In the future, whose definition is “someone who writes plays for therefore, we will systematically compare the se- film or television”. EDSKB provides five sememes meme annotations in EDSKB and HowNet and try and one (someone) is filtered out in EDSKB∗. to use EDSKB to improve and expand HowNet. By comparison, we can find that sememes in ED- Besides, the hierarchical structures of sememes in SKB and EDSKB∗ can represent the meaning of HowNet are neglected in this paper. We will also the word more specifically, e.g, write and play, explore to extract sememes with hierarchy from while sememes in HowNet seem to express a more dictionary definitions. general meaning. The second case is about the word “tweet”. Acknowledgements HowNet only annotates one sense for this word, i.e., “to send a message on Twitter”. As for EDSKB and This work is supported by the National Key EDSKB∗, since LDOCE contains the basic mean- Research and Development Program of ing of this word, i.e., “to make the short high sound China (Grant No. 2020AAA0106502 and of a small bird”, the sememes including bird and No. 2020AAA0106501) and Beijing Academy sound are extracted to express this meaning. In of Artificial Intelligence (BAAI). We also thank addition, for the shared sense, sememes in ED- all the anonymous reviewers for their valuable SKB and EDSKB∗ are more succinct than those in comments and suggestions. Ethical Considerations Stephen Bullon. 2006. Longman Dictionary of Con- temporary English. Pearson Education Limited. In this paper, we use two copyrighted dictionaries, namely Longman Dictionary of Contemporary En- Kyunghyun Cho, Bart Van Merrienboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Holger glish and Le Petit Robert French Dictionary. We Schwenk, and Yoshua Bengio. 2014. Learning extract data from the electronic versions of the two phrase representations using rnn encoder-decoder dictionaries we bought for the research purpose for statistical machine translation. In Proceedings only. We will not release the dictionary data. In of EMNLP. addition, the datasets we use in downstream tasks Alexis Conneau, Ruty Rinott, Guillaume Lample, Ad- are all open and free (actually also widely used). ina Williams, Samuel Bowman, Holger Schwenk, The task we tackle is sememe knowledge base and Veselin Stoyanov. 2018. Xnli: Evaluating cross- construction, which is not a practical application lingual sentence representations. In Proceedings of EMNLP. and is only related to NLP research. Therefore, the datasets we build and the models we use would not Jacob Devlin, Ming-Wei Chang, Kenton Lee, and be misused by common people. Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- In addition, since we do not use very large mod- standing. In Proceedings of NAACL-HLT. els, the required energy in this work is very limited. Finally, we use no demographic or identity charac- Zhendong Dong and Qiang Dong. 2003. Hownet-a teristics. hybrid language and knowledge resource. In Pro- ceedings of International Conference on Natural Language Processing and Knowledge Engineering. IEEE. References Zhendong Dong and Qiang Dong. 2006. HowNet BT Sue Atkins and Michael Rundell. 2008. The Oxford and the Computation of Meaning (With CD-Rom). guide to practical lexicography. Oxford University World Scientific. Press. Jiaju Du, Fanchao Qi, and Maosong Sun. 2019. Using Dzmitry Bahdanau, Tom Bosc, Stanislaw Jastrzebski, bert for word sense disambiguation. arXiv preprint Edward Grefenstette, Pascal Vincent, and Yoshua arXiv:1909.08358. Bengio. 2017. Learning to compute word embed- dings on the fly. arXiv preprint arXiv:1706.00286. Xiangyu Duan, Jun Zhao, and Bo Xu. 2007. Word sense disambiguation through sememe labeling. In Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Proceedings of IJCAI. Christian Jauvin. 2003. A neural probabilistic lan- guage model. Journal of Machine Learning Re- Xianghua Fu, Guo Liu, Yanyan Guo, and Zhiqiang search, 3(Feb):1137–1155. Wang. 2013. Multi-aspect sentiment analysis for Chinese online social reviews based on topic mod- Chris M Bishop. 1995. Training with noise is equiva- eling and HowNet lexicon. Knowledge-Based Sys- lent to tikhonov regularization. Neural computation, tems, 37:186–195. 7(1):108–116. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Terra Blevins and Luke Zettlemoyer. 2020. Moving Tafjord, Pradeep Dasigi, Nelson F Liu, Matthew Pe- down the long tail of word sense disambiguation ters, Michael Schmitz, and Luke Zettlemoyer. 2018. with gloss-informed biencoders. In Proceedings of Allennlp: A deep semantic natural language process- ACL. ing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1–6. Leonard Bloomfield. 1926. A set of postulates for the science of language. Language, 2(3):153–164. Cliff Goddard and Anna Wierzbicka. 1994. Semantic and lexical universals: Theory and empirical find- Tom Bosc and Pascal Vincent. 2018. Auto-encoding ings, volume 25. John Benjamins Publishing. dictionary definitions into consistent word embed- dings. In Proceedings of EMNLP. Yihong Gu, Jun Yan, Hao Zhu, Zhiyuan Liu, Ruobing Xie, Maosong Sun, Fen Lin, and Leyu Lin. 2018. Samuel R Bowman, Gabor Angeli, Christopher Potts, Language modeling with sparse product of sememe and Christopher D Manning. 2015. A large anno- experts. In Proceedings of EMNLP. tated corpus for learning natural language inference. In Proceedings of EMNLP. Felix Hill, KyungHyun Cho, Anna Korhonen, and Yoshua Bengio. 2016. Learning to understand David Bullock. 2011. Nsm+ ldoce: A non-circular dic- phrases by embedding the dictionary. Transactions tionary of english. International Journal of Lexicog- of the Association for Computational Linguistics, raphy, 24(2):226–240. 4:17–30. Sepp Hochreiter and Jurgen Schmidhuber. 1997. Teng Long, Emmanuel Bengio, Ryan Lowe, Jackie Long short-term memory. Neural Computation, Chi Kit Cheung, and Doina Precup. 2017. World 9(8):1735–1780. knowledge for reading comprehension: Rare entity prediction with hierarchical lstms using external de- Bairu Hou, Fanchao Qi, Yuan Zang, Xurui Zhang, scriptions. In Proceedings of EMNLP. Zhiyuan Liu, and Maosong Sun. 2020. Try to sub- stitute: An unsupervised chinese word sense disam- Edward Loper and Steven Bird. 2002. Nltk: The natu- biguation method based on hownet. In Proceedings ral language toolkit. In Proceedings of the ACL-02 of COLING. Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Compu- Minqing Hu and Bing Liu. 2004. Mining and summa- tational Linguistics. rizing customer reviews. In Proceedings of KDD. Fuli Luo, Tianyu Liu, Zexue He, Qiaolin Xia, Zhi- Luyao Huang, Chi Sun, Xipeng Qiu, and Xuan-Jing fang Sui, and Baobao Chang. 2018a. Leveraging Huang. 2019. Glossbert: Bert for word sense disam- gloss knowledge in neural word sense disambigua- biguation with gloss knowledge. In Proceedings of tion by hierarchical co-attention. In Proceedings of EMNLP-IJCNLP. EMNLP.

Huiming Jin, Hao Zhu, Zhiyuan Liu, Ruobing Xie, Fuli Luo, Tianyu Liu, Qiaolin Xia, Baobao Chang, and Maosong Sun, Fen Lin, and Leyu Lin. 2018. In- Zhifang Sui. 2018b. Incorporating glosses into neu- corporating chinese characters of words for lexical ral word sense disambiguation. In Proceedings of sememe prediction. In Proceedings of ACL. ACL.

Phillip Keung, Yichao Lu, Gyorgy¨ Szarvas, and Mitchell Marcus, Beatrice Santorini, and Mary Ann Noah A Smith. 2020. The multilingual amazon re- Marcinkiewicz. 1993. Building a large annotated views corpus. In Proceedings EMNLP. corpus of english: The penn treebank. Computa- tional Linguistics, 19(2). Sandra Kubler, Ryan McDonald, and Joakim Nivre. Stephen Merity, Nitish Shirish Keskar, and Richard 2009. Dependency parsing. Synthesis lectures on Socher. 2018. Regularizing and optimizing lstm lan- human language technologies, 1(1):1–127. guage models. In Proceedings of ICLR.

Sawan Kumar, Sharmistha Jat, Karan Saxena, and Stephen Merity, Caiming Xiong, James Bradbury, and Partha Talukdar. 2019. Zero-shot word sense dis- Richard Socher. 2017. Pointer sentinel mixture mod- ambiguation using sense definition embeddings. In els. In Proceedings of ICLR. Proceedings of ACL. George Miller. 1998. WordNet: An electronic lexical Hang Le, Lo¨ıc Vial, Jibril Frej, Vincent Segonne, Max- database. MIT press. imin Coavoux, Benjamin Lecouteux, Alexandre Al- lauzen, Benoit Crabbe, Laurent Besacier, and Didier Paul Nation and Robert Waring. 2004. Vocabulary size, Schwab. 2020. Flaubert: Unsupervised language text coverage and word lists. model pre-training for french. In Proceedings of LREC. Roberto Navigli and Simone Paolo Ponzetto. 2012. Ba- belNet: The automatic construction, evaluation and Michael Lesk. 1986. Automatic sense disambiguation application of a wide-coverage multilingual seman- using machine readable dictionaries: how to tell a tic network. Artificial Intelligence, 193:217–250. pine cone from an ice cream cone. In Proceedings of the 5th annual international conference on Systems Yilin Niu, Ruobing Xie, Zhiyuan Liu, and Maosong documentation. Sun. 2017. Improved word representation learning with sememes. In Proceedings of ACL. Ziran Li, Ning Ding, Zhiyuan Liu, Haitao Zheng, and Ying Shen. 2019. Chinese relation extraction William O’Grady, Michael Dobrovolsky, and Francis with multi-grained information and external linguis- Katamba. 1997. Contemporary linguistics. tic knowledge. In Proceedings of ACL. Mohammad Taher Pilehvar. 2019. On the importance Qun Liu and Sujian Li. 2002. Word similarity com- of distinguishing word meaning representations: A puting based on HowNet. International Journal case study on reverse dictionary mapping. In Pro- of Computational Linguistics & Chinese Language ceedings of NAACL-HLT. Processing, 7(2):59–76. Fanchao Qi, Liang Chang, Maosong Sun, Sicong Yangguang Liu, Fanchao Qi, Zhiyuan Liu, and Ouyang, and Zhiyuan Liu. 2020a. Towards build- Maosong Sun. 2020. Research on consistency check ing a multilingual sememe knowledge base: Predict- of sememe annotations in hownet. Journal of Chi- ing sememes for babelnet synsets. In Proceedings nese Information Processing. of AAAI. Fanchao Qi, Junjie Huang, Chenghao Yang, Zhiyuan Maosong Sun and Xinxiong Chen. 2016. Embedding Liu, Xiao Chen, Qun Liu, and Maosong Sun. 2019a. for words and word senses based on human anno- Modeling semantic compositionality with sememe tated knowledge base: Use HowNet as a case study. knowledge. In Proceedings of ACL. Journal of Chinese information processing, 30(6).

Fanchao Qi, Yankai Lin, Maosong Sun, Hao Zhu, Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Ruobing Xie, and Zhiyuan Liu. 2018. Cross- Joan Bruna, Dumitru Erhan, Ian Goodfellow, and lingual lexical sememe prediction. In Proceedings Rob Fergus. 2014. Intriguing properties of neural of EMNLP. networks. In Proceedings of ICLR.

Fanchao Qi, Ruobing Xie, Yuan Zang, Zhiyuan Liu, Julien Tissier, Christophe Gravier, and Amaury and Maosong Sun. 2020b. Sememe knowledge com- Habrard. 2017. Dict2vec: Learning word embed- putation: a review of recent advances in application dings using lexical dictionaries. In Proceedings of and expansion of sememe knowledge bases. Fron- EMNLP. tiers of Computer Science. Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Qiang and Rob Fergus. 2013. Regularization of neural net- Dong, Maosong Sun, and Zhendong Dong. 2019b. works using dropconnect. In Proceedings of ICML. Openhownet: An open sememe-based lexical knowl- arXiv preprint arXiv:1901.09957 edge base. . Xiaosen Wang, Hao Jin, and Kun He. 2019. Natural Fanchao Qi, Yuan Yao, Sophia Xu, Zhiyuan Liu, and language adversarial attacks and defenses in word Maosong Sun. 2021. Turn the combination lock: level. arXiv preprint arXiv:1909.06723. Learnable textual backdoor attacks via word substi- tution. In Proceedings of ACL-IJCNLP. Michael Philip West. 1953. A general service list of En- glish words: with semantic frequencies and a supple- Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, mentary word-list for the writing of popular science and Christopher D Manning. 2020c. Stanza: A and technology. Longmans, Green. python natural language processing toolkit for many human languages. In Proceedings of ACL. Anna Wierzbicka. 1996. Semantics: Primes and uni- versals: Primes and universals. Oxford University Yujia Qin, Fanchao Qi, Sicong Ouyang, Zhiyuan Liu, Press, UK. Cheng Yang, Yasheng Wang, Qun Liu, and Maosong Sun. 2020. Improving sequence modeling ability of Adina Williams, Nikita Nangia, and Samuel Bowman. recurrent neural networks via sememes. IEEE/ACM 2018. A broad-coverage challenge corpus for sen- Transactions on Audio, Speech, and Language Pro- tence understanding through inference. In Proceed- cessing. ings of NAACL-HLT.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, and Dario Amodei, and Ilya Sutskever. 2019. Language Maosong Sun. 2016. Representation learning of models are unsupervised multitask learners. OpenAI knowledge graphs with entity descriptions. In Pro- Blog, 1(8). ceedings of AAAI.

Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Ruobing Xie, Xingchi Yuan, Zhiyuan Liu, and 2019. Generating natural language adversarial ex- Maosong Sun. 2017. Lexical sememe prediction via amples through probability weighted word saliency. word embeddings and matrix factorization. In Pro- In Proceedings of ACL. ceedings of IJCAI. Paul Robert, Alain Rey, and Josette Rey-Debove. 2015. Han Xu, Yao Ma, Hao-Chen Liu, Debayan Deb, Hui Dictionnaire Le Petit Robert 2016. Dictionnaires Le Liu, Ji-Liang Tang, and Anil K Jain. 2020. Adversar- Robert. ial attacks and defenses in images, graphs and text: Thijs Scheepers, Evangelos Kanoulas, and Efstratios A review. International Journal of Automation and Gavves. 2018. Improving word embedding compo- Computing, 17(2):151–178. sitionality using lexicographic definitions. In Pro- ceedings of WWW. Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020. Richard Socher, Alex Perelygin, Jean Wu, Jason Word-level textual adversarial attacking as combina- Chuang, Christopher D Manning, Andrew Ng, and torial optimization. In Proceedings of ACL. Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment tree- Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. bank. In Proceedings of EMNLP. 2014. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329. Jingguang Sun, Dongfeng Cai, Dexin Lv, and Yanju Dong. 2007. HowNet based Chinese question auto- Lei Zhang, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, matic classification. Journal of Chinese Information Qun Liu, and Maosong Sun. 2020. Multi-channel Processing, 21(1):90–95. reverse dictionary model. In Proceedings of AAAI. Yuntao Zhang, Ling Gong, and Yongcheng Wang. SKB #Word/Phrase #Sense #Sememe #AvgSem 2005. Chinese word sense disambiguation using FDSKB 55,836 113,722 2,919 4.32 HowNet. In Proceedings of International Confer- FDSKB∗ 55,836 113,722 2,919 1.97 ence on Natural Computation. Table 7: Statistics of FDSKB and its distilled version Huaping Zhong, Jianwen Zhang, Zhen Wang, Hai Wan, FDSKB∗. #AvgSem denotes the average sememe num- and Zheng Chen. 2015. Aligning knowledge and ber per sense. text embeddings by entity descriptions. In Proceed- ings of EMNLP. Dataset French News FR-Wikipedia Yan-Lan Zhu, Jin Min, Ya-qian Zhou, Xuan-jing Model Valid Test Valid Test Huang, and Li-De Wu. 2006. Semantic orientation computing based on HowNet. Journal of Chinese Tied LSTM 17.02 18.35 17.23 15.75 information processing, 20(1):14–20. +FDSKB 15.72 16.85 17.14 15.60 +FDSKB∗ 15.70 16.89 16.50 15.15 A Building Process of a French SKB AWD-LSTM 18.41 19.71 16.76 15.30 +FDSKB 15.41 16.47 15.45 15.72 In this section, we describe the building process +FDSKB∗ 14.03 15.14 15.90 14.10 of a dictionary-based French SKB that we call FD- SKB. We choose Le Petit Robert French Dictionary Table 8: Perplexity results on the validation and test 2016 edition (Robert et al., 2015), a very popular sets of French News and FR-Wikipedia. The boldfaced French dictionary, as the base dictionary. results show statistically significant improvement over the best results from baselines with p<0.1 given by t- Constructing the Sememe Set We first con- test, and the underlined results represent having no sig- struct a sememe set from the defining vocabulary of nificant difference.7 the dictionary. Similar to EDSKB, we remove the most frequent and infrequent words that appear in B.1 Intrinsic Evaluation definitions as well as some stop words, and obtain a sememe set comprising 2, 919 sememes (defining We still use CCSA (Liu et al., 2020) to measure words). the sememe annotation consistency. The MAP and F1 score for FDSKB are 83.47 and 80.51 respec- Extracting Sememes from Definitions We use tively, and those for FDSKB∗ are 90.03 and 90.01 Stanza (Qi et al., 2020c) to tokenize and lemmatize respectively. These results are comparable to those the definitions of all words in the dictionary and of EDSKB and can prove good sememe annotation extract the sememes of each sense of each word consistency of FDSKB. Besides, similar to ED- according to the sememe set. So far, we have ob- SKB, the distilled version FDSKB∗ delivers better tained the full version of FDSKB, whose statistics sememe annotation consistency than FDSKB be- are shown in Table7. cause it has fewer sememes per sense.

Distilling Sememes of Senses We adopt a simi- B.2 Extrinsic Evaluation lar way to EDSKB to distill the sememes of senses. We conduct extrinsic evaluation for FDSKB and Specifically, we use Stanza to conduct dependency FDSKB∗ on three tasks including language mod- parsing for every definition and obtain the impor- eling, natural language inference (NLI) and text tance score of each sememe. Then we empirically classification. remove the unimportant sememes according to the experimental results of downstream tasks. In this Language Modeling way, we obtain the distilled version of FDSKB Similar to EDSKB, we use SDLM (Gu et al., 2018) ∗ (FDSKB ), whose statistics are also in Table7. to incorporate FDSKB into Tied LSTM (Zaremba et al., 2014) and AWD-LSTM (Merity et al., 2018). B Evaluation of FDSKB The experimental settings are the same as those in In this section, similar to EDSKB, we conduct English experiments. both intrinsic and extrinsic evaluations for FDSKB We choose two evaluation datasets: (1) French 8 and FDSKB∗. Notice that since HowNet covers News , which comprises French news articles from only English and Chinese, there are no available 8https://webhose.io/free-datasets/ HowNet-based baseline methods for French. french-news-articles/ Dataset Method LSTM GRU BiLSTM BiGRU sentence pairs in the validation and test sets which vanilla 61.14 60.88 61.56 61.36 are manually translated from English. MARC +Pseudo 60.96 61.46 61.10 61.12 (Multilingual Amazon Reviews Corpus) is a large +Definition 61.64 61.10 61.80 61.86 XNLI +FDSKB 62.38 61.54 62.52 61.44 corpus of Amazon reviews in 6 languages. We use +FDSKB∗ 62.52 61.74 62.16 62.06 its French part for product category classification. vanilla 79.98 79.16 80.35 80.64 It has 40, 000 / 1, 323 / 1, 345 reviews in its training +Pseudo 80.35 79.38 79.16 80.53 / validation / test sets. +Definition 81.10 79.39 80.65 80.80 MARC +FDSKB 81.39 80.58 80.65 82.14 Table9 shows the accuracy results on the test +FDSKB∗ 81.43 81.10 80.98 81.84 sets of XNLI and MARC. The results are basically consistent with the experimental results in English Table 9: Accuracy results of different models on the datasets. The incorporation of the dictionary-based test sets of XNLI and MARC. SKB can improve the performance of RNN models on the two different tasks, which reflects that the popular news sites. It has 2, 131, 774 / 358, 972 / SKB has enhanced the sequence modeling ability 370, 059 tokens in its training / validation / test of RNNs. Moreover, the results also demonstrate sets. (2) FR-Wikipedia9, which is composed of the usefulness and effectiveness of our dictionary- French Wikipedia articles. The token numbers in based SKB and its building method. its training / validation / test sets are 3, 252, 094 / C Experiment Running Environment 520, 333 / 517, 669. The experimental results are given in Table8. For all the experiments, we use a server whose We observe that both FDSKB and FDSKB∗ bring major configurations are as follows: (1) CPU: In- decreases of perplexity, which demonstrates the tel(R) Xeon(R) E5-2680 v4 @ 2.40GHz, 56 cores; effectiveness of the French dictionary-based SKB (2) RAM: 125GB; (3) GPU: 8 Nvidia RTX2080 in language modeling and the practicality of our GPUs, 12GB memory. The operation system dictionary-based SKB building method. Notice that is Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-108- since the perplexity results of original Tied LSTM generic x86 64). We use PyTorch10 v1.5.0 and and AWD-LSTM on the two French datasets are Python v3.6.9 as the programming framework for quite good, the enhancement brought by FDSKB the experiments on neural network models. is comparatively less than that in English.

NLI and Text Classification Similar to EDSKB, we use SememeCell (Qin et al., 2020) to incorporate FDSKB into RNNs and mea- sure the improvement of sequence modeling ability on the tasks of NLI and text classification. The base models are still LSTM, GRU, BiLSTM and BiGRU. And the baseline methods are also +Pseudo and +Definition. Here we use FlauBERT (Le et al., 2020), a French pre-trained language model, to encode definitions into 768-dimensional vectors. The other experimental settings are the same as English. As for evaluation datasets, we use XNLI (Con- neau et al., 2018) and MARC (Keung et al., 2020) respectively. XNLI is a cross-lingual NLI dataset in 15 languages. It is based on another NLI dataset MNLI (Williams et al., 2018) and constructs its training set by machine translation, which has 361, 469 sentence pairs. It has 2, 500 and 5, 000

9http://redac.univ-tlse2.fr/corpora/ wikipedia_en.html 10https://pytorch.org/