Neural Networks Language Models

Neural Networks Language Models

Neural Networks Language Models Philipp Koehn 1 October 2020 Philipp Koehn Machine Translation: Neural Networks 1 October 2020 N-Gram Backoff Language Model 1 • Previously, we approximated p(W ) = p(w1; w2; :::; wn) • ... by applying the chain rule X p(W ) = p(wijw1; :::; wi−1) i • ... and limiting the history (Markov order) p(wijw1; :::; wi−1) ' p(wijwi−4; wi−3; wi−2; wi−1) • Each p(wijwi−4; wi−3; wi−2; wi−1) may not have enough statistics to estimate ! we back off to p(wijwi−3; wi−2; wi−1), p(wijwi−2; wi−1), etc., all the way to p(wi) – exact details of backing off get complicated — ”interpolated Kneser-Ney” Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Refinements 2 • A whole family of back-off schemes • Skip-n gram models that may back off to p(wijwi−2) • Class-based models p(C(wi)jC(wi−4);C(wi−3);C(wi−2);C(wi−1)) ) We are wrestling here with – using as much relevant evidence as possible – pooling evidence between words Philipp Koehn Machine Translation: Neural Networks 1 October 2020 First Sketch 3 wi Softmax Output Word h FF Hidden Layer wi-4 wi-3 wi-2 wi-1 History Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Representing Words 4 • Words are represented with a one-hot vector, e.g., – dog = (0,0,0,0,1,0,0,0,0,....) – cat = (0,0,0,0,0,0,0,1,0,....) – eat = (0,1,0,0,0,0,0,0,0,....) • That’s a large vector! • Remedies – limit to, say, 20,000 most frequent words, rest are OTHER p – place words in n classes, so each word is represented by ∗ 1 class label ∗ 1 word in class label Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Word Classes for Two-Hot Representations 5 • WordNet classes • Brown clusters • Frequency binning – sort words by frequency – place them in order into classes – each class has same token count ! very frequent words have their own class ! rare words share class with many other words • Anything goes: assign words randomly to classes Philipp Koehn Machine Translation: Neural Networks 1 October 2020 6 word embeddings Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Add a Hidden Layer 7 wi Softmax Output Word h FF Hidden Layer Ew Embed Embed Embed Embed Embedding wi-4 wi-3 wi-2 wi-1 History • Map each word first into a lower-dimensional real-valued space • Shared weight matrix E Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Details (Bengio et al., 2003) 8 • Add direct connections from embedding layer to output layer • Activation functions – input!embedding: none – embedding!hidden: tanh – hidden!output: softmax • Training – loop through the entire corpus – update between predicted probabilities and 1-hot vector for output word Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Word Embeddings 9 Word Embedding C • By-product: embedding of word into continuous space • Similar contexts ! similar embedding • Recall: distributional semantics Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Word Embeddings 10 Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Word Embeddings 11 Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Are Word Embeddings Magic? 12 • Morphosyntactic regularities (Mikolov et al., 2013) – adjectives base form vs. comparative, e.g., good, better – nouns singular vs. plural, e.g., year, years – verbs present tense vs. past tense, e.g., see, saw • Semantic regularities – clothing is to shirt as dish is to bowl – evaluated on human judgment data of semantic similarities Philipp Koehn Machine Translation: Neural Networks 1 October 2020 13 recurrent neural networks Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Recurrent Neural Networks 14 Softmax Output Word tanh Hidden Layer 0 Embed Embedding w1 History • Start: predict second word from first • Mystery layer with nodes all with value 1 Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Recurrent Neural Networks 15 Softmax Softmax Output Word tanh copy tanh Hidden Layer 0 Embed Embed Embedding w1 w2 History Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Recurrent Neural Networks 16 Softmax Softmax Softmax Output Word tanh copy tanh copy tanh Hidden Layer 0 Embed Embed Embed Embedding w1 w2 w3 History Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Training 17 Cost yt Softmax Output Word ht 0 RNN Hidden Layer Ewt Embed Embedding wt w1 History • Process first training example • Update weights with back-propagation Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Training 18 Cost yt Softmax Output Word ht RNN RNN Hidden Layer Ewt Embed Embedding wt w2 History • Process second training example • Update weights with back-propagation • And so on... • But: no feedback to previous history Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Back-Propagation Through Time 19 Cost Cost Cost yt Softmax Softmax Softmax Output Word ht 0 RNN RNN RNN Hidden Layer Ewt Embed Embed Embed Embedding wt w1 w2 w3 History • After processing a few training examples, update through the unfolded recurrent neural network Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Back-Propagation Through Time 20 • Carry out back-propagation though time (BPTT) after each training example – 5 time steps seems to be sufficient – network learns to store information for more than 5 time steps • Or: update in mini-batches – process 10-20 training examples – update backwards through all examples – removes need for multiple steps for each training example Philipp Koehn Machine Translation: Neural Networks 1 October 2020 21 long short term memory Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Vanishing Gradients 22 • Error is propagated to previous steps • Updates consider – prediction at that time step – impact on future time steps • Vanishing gradient: propagated error disappears Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Recent vs. Early History 23 • Hidden layer plays double duty – memory of the network – continuous space representation used to predict output words • Sometimes only recent context important After much economic progress over the years, the country ! has • Sometimes much earlier context important The country which has made much economic progress over the years still ! has Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Long Short Term Memory (LSTM) 24 • Design quite elaborate, although not very complicated to use • Basic building block: LSTM cell – similar to a node in a hidden layer – but: has a explicit memory state • Output and memory state change depends on gates – input gate: how much new input changes memory state – forget gate: how much of prior memory state is retained – output gate: how strongly memory state is passed on to next layer. • Gates can be not just be open (1) and closed (0), but slightly ajar (e.g., 0.2) Philipp Koehn Machine Translation: Neural Networks 1 October 2020 LSTM Cell 25 LSTM Layer Time t-1 m ⊗ forget gate Preceding Layer gate output X i ⊕ ⊗ gate input Next Layer m ⊗ o h Y LSTM Layer Time t Philipp Koehn Machine Translation: Neural Networks 1 October 2020 LSTM Cell (Math) 26 • Memory and output values at time step t t t t−1 memory = gateinput × input + gateforget × memory t t output = gateoutput × memory • Hidden node value ht passed on to next layer applies activation function f ht = f(outputt) • Input computed as input to recurrent neural network node t t t – given node values for prior layer ~x = (x1; :::; xX) ~ t−1 t−1 t−1 – given values for hidden layer from previous time step h = (h1 ; :::; hH ) – input value is combination of matrix multiplication with weights wx and wh and activation function g X H ! t X x t X h t−1 input = g wi xi + wi hi i=1 i=1 Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Values for Gates 27 • Gates are very important • How do we compute their value? ! with a neural network layer! • For each gate a 2 (input; forget; output) – weight matrix W xa to consider node values in previous layer ~xt – weight matrix W ha to consider hidden layer ~ht−1 at previous time step t−1 – weight matrix W ma to consider memory at previous time step memory~ – activation function h X H H ! X xa t X ha t−1 X ma t−1 gatea = h wi xi + wi hi + wi memoryi i=1 i=1 i=1 Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Training 28 • LSTM are trained the same way as recurrent neural networks • Back-propagation through time • This looks all very complex, but: – all the operations are still based on ∗ matrix multiplications ∗ differentiable activation functions ! we can compute gradients for objective function with respect to all parameters ! we can compute update functions Philipp Koehn Machine Translation: Neural Networks 1 October 2020 What is the Point? 29 (from Tran, Bisazza, Monz, 2016) • Each node has memory memoryi independent from current output hi i i • Memory may be carried through unchanged (gateinput = 0, gatememory = 1) ) can remember important features over long time span (capture long distance dependencies) Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Visualizing Individual Cells 30 Karpathy et al. (2015): ”Visualizing and Understanding Recurrent Networks” Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Visualizing Individual Cells 31 Philipp Koehn Machine Translation: Neural Networks 1 October 2020 Gated Recurrent Unit (GRU) 32 GRU Layer Time t-1 h update gate reset gate ⊗ Next Layer Preceding Layer h Y X x ⊕ GRU Layer Time t Philipp

View Full Text

Details

  • File Type
    pdf
  • Upload Time
    -
  • Content Languages
    English
  • Upload User
    Anonymous/Not logged-in
  • File Pages
    38 Page
  • File Size
    -

Download

Channel Download Status
Express Download Enable

Copyright

We respect the copyrights and intellectual property rights of all users. All uploaded documents are either original works of the uploader or authorized works of the rightful owners.

  • Not to be reproduced or distributed without explicit permission.
  • Not used for commercial purposes outside of approved use cases.
  • Not used to infringe on the rights of the original creators.
  • If you believe any content infringes your copyright, please contact us immediately.

Support

For help with questions, suggestions, or problems, please contact us