A Survey on Neural Network Language Models Kun Jing∗ and Jungang Xu∗ School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing [email protected], [email protected] Abstract which are estimated by maximum likelihood estimation. Perplexity (PPL) [Jelinek et al., 1977], an information- As the core component of Natural Language Pro- theoretic metric that measures the quality of a probabilistic cessing (NLP) system, Language Model (LM) can model, is a way to evaluate LMs. Lower PPL indicates a bet- provide word representation and probability indi- ter model. Given a corpus containing N words and a language cation of word sequences. Neural Network Lan- model LM, the PPL of LM is guage Models (NNLMs) overcome the curse of di- − 1 PN log LM(w jw ···w ) mensionality and improve the performance of tra- 2 N t=1 2 t 1 t−1 : (3) ditional LMs. A survey on NNLMs is performed in this paper. The structure of classic NNLMs is de- It is noted that PPL is related to corpus. Two or more LMs scribed firstly, and then some major improvements can be compared on the same corpus in terms of PPL. are introduced and analyzed. We summarize and However, n-gram LMs have a significant drawback. The compare corpora and toolkits of NNLMs. Further- model would assign probabilities of 0 to the n-grams that do more, some research directions of NNLMs are dis- not appear in the training corpus, which does not match the cussed. actual situation. Smoothing techniques can solve this prob- lem. Its main idea is robbing the rich for the poor, i.e., reduc- ing the probability of events that appear in the training corpus 1 Introduction and assigning the probability to events that do not appear. LM is the basis of various NLP tasks. For example, in Ma- Although n-gram LMs with smoothing techniques work chine Translation tasks, LM is used to evaluate probabilities out, there still are other problems. A fundamental problem is of outputs of the system to improve fluency of the translation the curse of dimensionality, which limits modeling on larger in the target language. In Speech Recognition tasks, LM and corpora for a universal language model. It is particularly ob- acoustic model are combined to predict the next word. vious in the case when one wants to model the joint distri- Early NLP systems were based primarily on manually writ- bution in discrete space. For example, if one wants to model ten rules, which is time-consuming and laborious and cannot an n-gram LM with a vocabulary of size 10; 000, there are cover a variety of linguistic phenomena. In the 1980s, sta- potentially 10000n − 1 free parameters. tistical LMs were proposed, which assigns probabilities to a In order to solve this problem, Neural Network (NN) is in- sequence s of N words, i.e., troduced for language modeling in continuous space. NNs including Feedforward Neural Network (FFNN), Recurrent P (s) =P (w1w2 ··· wN ) Neural Network (RNN), can automatically learn features and =P (w )P (w jw ) ··· P (w jw w ··· w ); (1) 1 2 1 N 1 2 N−1 continuous representation. Therefore, NNs are expected to be arXiv:1906.03591v2 [cs.CL] 13 Jun 2019 where wi denotes i-th word in the sequence s. The probabil- applied to LMs, even other NLP tasks, to cover the discrete- ity of a word sequence can be broken into the product of the ness, combination, and sparsity of natural language. conditional probability of the next word given its predeces- The first FFNN Language Model (FFNNLM) presented by sors that are generally called a history of context or context. [Bengio et al., 2003] fights the curse of dimensionality by Considering that it is difficult to learn the extremely many learning a distributed representation for words, which rep- parameters of the above model, an approximate method is resents a word as a low dimensional vector, called embed- necessary. N-gram model is an approximation method, which ding. FFNNLM performs better than n-gram LM. Then, RNN was the most widely used and the state-of-the-art model be- Language Model (RNNLM) [Mikolov et al., 2010] also was fore NNLMs. A (k+1)-gram model is derived from the k- proposed. Since then, the NNLM has gradually become the order Markov assumption. This assumption illustrates that mainstream LM and has rapidly developed. Long Short-term the current state depends only on the previous k states, i.e., Memory RNN Language Model (LSTM-RNNLM) [Sunder- P (w jw ··· w ) ≈ P (w jw ··· w ); (2) meyer et al., 2012] was proposed for the difficulty of learning t 1 t−1 t t−k t−1 long-term dependence. Various improvements were proposed ∗K. Jing and J. Xu are corresponding authors. for reducing the cost of training and evaluation and PPL such P(wt|context) x(t) softmax U w(t) tanh W V U s(t) y(t) H C(wt−n+1) … C(wt−2) C(wt−1) delayed s(t-1) wt−n+1 wt−2 wt−1 Figure 1: The FFNNLM proposed by [Bengio et al., 2003]. In this model, in order to predict the conditional probability of the word wt, its previous n − 1 words are projected by the shared projection ma- Figure 2: The RNNLM proposed by [Mikolov et al., 2010; Mikolov jV |×m trix C 2 R into a continuous feature vector space according et al., 2011]. The RNN has an internal state that changes with the to their index in the vocabulary, where jV j is the size of the vocabu- input on each time step, taking into account all previous contexts. lary and m is the dimension of the feature vectors, i.e., the word wi The state st can be derived from the input word vector wt and the m is projected as the distributed feature vector C(wi) 2 R . Each row state st−1. of the projection matrix C is a feature vector of a word in the vocab- ulary. The input x of the FFNN is a concatenation of feature vectors of n − 1 words. This model is followed by Softmax output layer where H, U, and W are the weight matrixes that is for the to guarantee all the conditional probabilities of words positive and connections between the layers; d and b are the biases of the summing to one. The learning algorithm is the Stochastic Gradient hidden layer and the output layer. Descent (SGD) method using the backpropagation (BP) algorithm. FFNNLM implements modeling on continuous space by learning a distributed representation for each word. The word as hierarchical Softmax, caching, and so on. Recently, atten- representation is a by-product of LMs, which is used to im- tion mechanisms have been introduced to improve NNLMs, prove other NLP tasks. Based on FFNNLM, two word repre- which achieved significant performance improvements. sentation models, CBOW and Skip-gram , were proposed by In this paper, we concentrate on reviewing the methods and [Mikolov et al., 2013]. FFNNLM overcomes the curse of di- trends of NNLM. Classic NNLMs are described in Section 2. mensions by converting words into low-dimensional vectors. Different types of improvements are introduced and analyzed FFNNLM leads the trend of NNLM research. separately in Section 3. Corpora and toolkits are described in However, it still has a few drawbacks. The context size Section 4 and Section 5. Finally, conclusions are given, and specified before training is limited, which is quite different new research directions of NNLMs are discussed. from the fact that people can use lots of context informa- tion to make predictions. Words in a sequence are time- 2 Classic Neural Network Language Models related. FFNNLM does not use timing information for mod- eling. Moreover, fully connected NN needs to learn many 2.1 FFNN Language Models trainable parameters, even though these parameters are less [Xu and Rudnicky, 2000] tried to introduce NNs into LMs. than n-gram LM, which still is expensive and inefficient. Although their model performs better than the baseline n- gram LM, their model with poor generalization ability cannot 2.2 RNN Language Models capture context-dependent features due to no hidden layer. [Bengio et al., 2003] proposed the idea of using RNN for According to Formula 1, the goal of LMs is equiv- LMs. They claimed that introducing more structure and pa- alent to an evaluation of the conditional probability rameter sharing into NNs could capture longer contextual in- P (wkjw1 ··· wk−1). But the FFNNs cannot directly process formation. variable-length data and effectively represent the historical The first RNNLM was proposed by [Mikolov et al., 2010; context. Therefore, for sequence modeling tasks like LMs, Mikolov et al., 2011a]. As shown in Figure 2, at time step t, FFNNs have to use fixed-length inputs. Inspired by the n- the RNNLM can be described as: gram LMs (see Formula 2), FFNNLMs consider the previous T T T n − 1 words as the context for predicting the next word. xt = [wt ; st−1] ; [Bengio et al., 2003] proposed the architecture of the orig- st = f(Uxt + b); inal FFNNLM, as shown in Figure 1. This FFNNLM can be yt = g(V st + d); (5) expressed as: where U, W , V are weight matrixes; b, d are the biases of the y = b + W x + Utanh(d + Hx); (4) state layer and the output layer respectively; in [Mikolov et al., 2010; Mikolov et al., 2011a], f is the sigmoid function, 3 Improved Techniques and g is the Softmax function.