Deep Learning for Natural Language Processing Develop Deep Learning Models for Natural Language in Python Jason Brownlee i Disclaimer The information contained within this eBook is strictly for educational purposes. If you wish to apply ideas contained in this eBook, you are taking full responsibility for your actions. The author has made every effort to ensure the accuracy of the information within this book was correct at time of publication. The author does not assume and hereby disclaims any liability to any party for any loss, damage, or disruption caused by errors or omissions, whether such errors or omissions result from accident, negligence, or any other cause. No part of this eBook may be reproduced or transmitted in any form or by any means, electronic or mechanical, recording or by any information storage and retrieval system, without written permission from the author. Acknowledgements Special thanks to my copy editor Sarah Martin and my technical editors Arun Koshy and Andrei Cheremskoy. Copyright Deep Learning for Natural Language Processing © Copyright 2017 Jason Brownlee. All Rights Reserved. Edition: v1.1 Contents Copyright i Contents ii Preface iii I Introductions iv Welcome v Who Is This Book For?...................................v About Your Outcomes................................... vi How to Read This Book.................................. vi About the Book Structure................................. vii About Python Code Examples............................... viii About Further Reading................................... viii About Getting Help.................................... ix Summary.......................................... ix II Foundations1 1 Natural Language Processing2 1.1 Natural Language...................................2 1.2 Challenge of Natural Language...........................3 1.3 From Linguistics to Natural Language Processing.................3 1.4 Natural Language Processing............................5 1.5 Further Reading....................................6 1.6 Summary.......................................7 2 Deep Learning8 2.1 Deep Learning is Large Neural Networks......................8 2.2 Deep Learning is Hierarchical Feature Learning.................. 11 2.3 Deep Learning as Scalable Learning Across Domains............... 12 2.4 Further Reading.................................... 13 2.5 Summary....................................... 14 ii CONTENTS iii 3 Promise of Deep Learning for Natural Language 16 3.1 Promise of Deep Learning.............................. 16 3.2 Promise of Drop-in Replacement Models...................... 17 3.3 Promise of New NLP Models............................. 17 3.4 Promise of Feature Learning............................. 18 3.5 Promise of Continued Improvement......................... 18 3.6 Promise of End-to-End Models........................... 19 3.7 Further Reading.................................... 20 3.8 Summary....................................... 20 4 How to Develop Deep Learning Models With Keras 21 4.1 Keras Model Life-Cycle................................ 21 4.2 Keras Functional Models............................... 26 4.3 Standard Network Models.............................. 27 4.4 Further Reading.................................... 32 4.5 Summary....................................... 33 III Data Preparation 34 5 How to Clean Text Manually and with NLTK 35 5.1 Tutorial Overview................................... 35 5.2 Metamorphosis by Franz Kafka........................... 36 5.3 Text Cleaning Is Task Specific............................ 36 5.4 Manual Tokenization................................. 37 5.5 Tokenization and Cleaning with NLTK....................... 41 5.6 Additional Text Cleaning Considerations...................... 46 5.7 Further Reading.................................... 46 5.8 Summary....................................... 47 6 How to Prepare Text Data with scikit-learn 48 6.1 The Bag-of-Words Model............................... 48 6.2 Word Counts with CountVectorizer ........................ 49 6.3 Word Frequencies with TfidfVectorizer ...................... 50 6.4 Hashing with HashingVectorizer .......................... 51 6.5 Further Reading.................................... 52 6.6 Summary....................................... 53 7 How to Prepare Text Data With Keras 54 7.1 Tutorial Overview................................... 54 7.2 Split Words with text to word sequence ..................... 54 7.3 Encoding with one hot ................................ 55 7.4 Hash Encoding with hashing trick ........................ 56 7.5 Tokenizer API.................................... 57 7.6 Further Reading.................................... 59 7.7 Summary....................................... 59 CONTENTS iv IV Bag-of-Words 61 8 The Bag-of-Words Model 62 8.1 Tutorial Overview................................... 62 8.2 The Problem with Text................................ 62 8.3 What is a Bag-of-Words?............................... 63 8.4 Example of the Bag-of-Words Model........................ 63 8.5 Managing Vocabulary................................. 65 8.6 Scoring Words..................................... 66 8.7 Limitations of Bag-of-Words............................. 67 8.8 Further Reading.................................... 67 8.9 Summary....................................... 68 9 How to Prepare Movie Review Data for Sentiment Analysis 69 9.1 Tutorial Overview................................... 69 9.2 Movie Review Dataset................................ 70 9.3 Load Text Data.................................... 71 9.4 Clean Text Data................................... 73 9.5 Develop Vocabulary.................................. 76 9.6 Save Prepared Data.................................. 80 9.7 Further Reading.................................... 83 9.8 Summary....................................... 84 10 Project: Develop a Neural Bag-of-Words Model for Sentiment Analysis 85 10.1 Tutorial Overview................................... 85 10.2 Movie Review Dataset................................ 86 10.3 Data Preparation................................... 86 10.4 Bag-of-Words Representation............................ 92 10.5 Sentiment Analysis Models.............................. 98 10.6 Comparing Word Scoring Methods......................... 103 10.7 Predicting Sentiment for New Reviews....................... 108 10.8 Extensions....................................... 112 10.9 Further Reading.................................... 112 10.10Summary....................................... 113 V Word Embeddings 114 11 The Word Embedding Model 115 11.1 Overview........................................ 115 11.2 What Are Word Embeddings?............................ 115 11.3 Word Embedding Algorithms............................ 116 11.4 Using Word Embeddings............................... 119 11.5 Further Reading.................................... 120 11.6 Summary....................................... 121 CONTENTS v 12 How to Develop Word Embeddings with Gensim 122 12.1 Tutorial Overview................................... 122 12.2 Word Embeddings.................................. 123 12.3 Gensim Python Library............................... 123 12.4 Develop Word2Vec Embedding........................... 123 12.5 Visualize Word Embedding.............................. 126 12.6 Load Google's Word2Vec Embedding........................ 128 12.7 Load Stanford's GloVe Embedding......................... 129 12.8 Further Reading.................................... 131 12.9 Summary....................................... 132 13 How to Learn and Load Word Embeddings in Keras 133 13.1 Tutorial Overview................................... 133 13.2 Word Embedding................................... 134 13.3 Keras Embedding Layer............................... 134 13.4 Example of Learning an Embedding......................... 135 13.5 Example of Using Pre-Trained GloVe Embedding................. 138 13.6 Tips for Cleaning Text for Word Embedding.................... 142 13.7 Further Reading.................................... 142 13.8 Summary....................................... 143 VI Text Classification 144 14 Neural Models for Document Classification 145 14.1 Overview........................................ 145 14.2 Word Embeddings + CNN = Text Classification.................. 146 14.3 Use a Single Layer CNN Architecture........................ 147 14.4 Dial in CNN Hyperparameters............................ 148 14.5 Consider Character-Level CNNs........................... 150 14.6 Consider Deeper CNNs for Classification...................... 151 14.7 Further Reading.................................... 152 14.8 Summary....................................... 152 15 Project: Develop an Embedding + CNN Model for Sentiment Analysis 153 15.1 Tutorial Overview................................... 153 15.2 Movie Review Dataset................................ 153 15.3 Data Preparation................................... 154 15.4 Train CNN With Embedding Layer......................... 160 15.5 Evaluate Model.................................... 167 15.6 Extensions....................................... 171 15.7 Further Reading.................................... 172 15.8 Summary....................................... 173 CONTENTS vi 16 Project: Develop an n-gram CNN Model for Sentiment Analysis 174 16.1 Tutorial Overview................................... 174 16.2 Movie Review Dataset................................ 174 16.3 Data Preparation..................................