Easy Data Augmentation Techniques for Boosting Performance on Text Classiﬁcation Tasks

EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classiﬁcation Tasks

Jason Wei1,2 Kai Zou3 1Protago Labs Research, Tysons Corner, Virginia, USA 2Department of Computer Science, Dartmouth College 3Department of Mathematics and Statistics, Georgetown University [email protected] [email protected]

Abstract Operation Sentence None A sad, superior human comedy played out We present EDA: easy data augmentation on the back roads of life. techniques for boosting performance on text SR A lamentable, superior human comedy played out on the backward road of life. classification tasks. EDA consists of four sim- RI A sad, superior human comedy played out ple but powerful operations: synonym replace- on funniness the back roads of life. ment, random insertion, random swap, and RS A sad, superior human comedy played out random deletion. On five text classification on roads back the of life. tasks, we show that EDA improves perfor- RD A sad, superior human out on the roads of life. mance for both convolutional and recurrent neural networks. EDA demonstrates particu- Table 1: Sentences generated using EDA. SR: synonym larly strong results for smaller datasets; on av- replacement. RI: random insertion. RS: random swap. erage, across five datasets, training with EDA RD: random deletion. while using only 50% of the available training set achieved the same accuracy as normal training with all available data. We also per- (Xie et al., 2017) and predictive language models formed extensive ablation studies and suggest for synonym replacement (Kobayashi, 2018). Al- parameters for practical use. though these techniques are valid, they are not of- 1 Introduction ten used in practice because they have a high cost of implementation relative to performance gain. Text classification is a fundamental task in natu- In this paper, we present a simple set of univer- ral language processing (NLP). Machine learning sal data augmentation techniques for NLP called and deep learning have achieved high accuracy on EDA (easy data augmentation). To the best of our tasks ranging from sentiment analysis (Tang et al., knowledge, we are the first to comprehensively 2015) to topic classification (Tong and Koller, explore text editing techniques for data augmen- 2002), but high performance often depends on the tation. We systematically evaluate EDA on five size and quality of training data, which is often te- benchmark classification tasks, showing that EDA dious to collect. Automatic data augmentation is provides substantial improvements on all five tasks commonly used in computer vision (Simard et al., and is particularly helpful for smaller datasets. 1998; Szegedy et al., 2014; Krizhevsky et al., Code is publicly available at http://github. 2017) and speech (Cui et al., 2015; Ko et al., 2015) com/jasonwei20/eda_nlp. and can help train more robust models, particularly when using smaller datasets. However, be- 2 EDA cause it is challenging to come up with generalized rules for language transformation, universal data Frustrated by the measly performance of text clas- augmentation techniques in NLP have not been sifiers trained on small datasets, we tested a num- thoroughly explored. ber of augmentation operations loosely inspired by Previous work has proposed some techniques those used in computer vision and found that they for data augmentation in NLP. One popular study helped train more robust models. Here, we present generated new data by translating sentences into the full details of EDA. For a given sentence in the French and back into English (Yu et al., 2018). training set, we randomly choose and perform one Other work has used data noising as smoothing of the following operations:

6382 Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6382–6388, Hong Kong, China, November 3–7, 2019. c 2019 Association for Computational Linguistics 1. Synonym Replacement (SR): Randomly 3.2 Text Classification Models choose n words from the sentence that are not We run experiments for two popular models in stop words. Replace each of these words with text classification. (1) Recurrent neural networks one of its synonyms chosen at random. (RNNs) are suitable for sequential data. We use a 2. Random Insertion (RI): Find a random syn- LSTM-RNN (Liu et al., 2016). (2) Convolutional onym of a random word in the sentence that is neural networks (CNNs) have also achieved high not a stop word. Insert that synonym into a ran- performance for text classification. We implement dom position in the sentence. Do this n times. them as described in (Kim, 2014). Details are in 3. Random Swap (RS): Randomly choose two Section 9.1 in Supplementary Materials. words in the sentence and swap their positions. 4 Results Do this n times. 4. Random Deletion (RD): Randomly remove In this section, we test EDA on five NLP tasks with each word in the sentence with probability p. CNNs and RNNs. For all experiments, we average results from five different random seeds.

Since long sentences have more words than short 4.1 EDA Makes Gains ones, they can absorb more noise while maintain- We run both CNN and RNN models with and ing their original class label. To compensate, we without EDA across all five datasets for varying vary the number of words changed, n, for SR, RI, training set sizes. Average performances (%) are and RS based on the sentence length l with the for- shown in Table2. Of note, average improve- mula n=α l, where α is a parameter that indicates ment was 0.8% for full datasets and 3.0% for the percent of the words in a sentence are changed N =500. (we use p=α for RD). Furthermore, for each orig- train inal sentence, we generate naug augmented sen- Training Set Size tences. Examples of augmented sentences are Model 500 2,000 5,000 full set shown in Table1. We note that synonym replace- RNN 75.3 83.7 86.1 87.4 ment has been used previously (Kolomiyets et al., +EDA 79.1 84.4 87.3 88.3 2011; Zhang et al., 2015; Wang and Yang, 2015), CNN 78.6 85.6 87.7 88.3 but to our knowledge, random insertions, swaps, +EDA 80.7 86.4 88.3 88.8 and deletions have not been extensively studied. Average 76.9 84.6 86.9 87.8 +EDA 79.9 85.4 87.8 88.6 3 Experimental Setup Table 2: Average performances (%) across five text We choose five benchmark text classification tasks classification tasks for models with and without EDA and two network architectures to evaluate EDA. on different training set sizes.

3.1 Benchmark Datasets 4.2 Training Set Sizing We conduct experiments on five benchmark text Overfitting tends to be more severe when training classification tasks: (1) SST-2: Stanford Senti- on smaller datasets. By conducting experiments ment Treebank (Socher et al., 2013), (2) CR: cus- using a restricted fraction of the available train- tomer reviews (Hu and Liu, 2004; Liu et al., 2015), ing data, we show that EDA has more significant (3) SUBJ: subjectivity/objectivity dataset (Pang improvements for smaller training sets. We run and Lee, 2004), (4) TREC: question type dataset both normal training and EDA training for the fol- (Li and Roth, 2002), and (5) PC: Pro-Con dataset lowing training set fractions (%): {1, 5, 10, 20, (Ganapathibhotla and Liu, 2008). Summary statis- 30, 40, 50, 60, 70, 80, 90, 100}. Figure1(a)- tics are shown in Table5 in Supplemental Mate- (e) shows performance with and without EDA for rials. Furthermore, we hypothesize that EDA is each dataset, and1(f) shows the averaged perfor- more helpful for smaller datasets, so we delegate mance across all datasets. The best average accu- the following sized datasets by selecting a random racy without augmentation, 88.3%, was achieved subset of the full training set with Ntrain={500, using 100% of the training data. Models trained 2,000, 5,000, all available data}. using EDA surpassed this number by achieving an

6383 SST-2 (N=7,447) CR (N=4,082) SUBJ (N=9,000) 1 1 1

0.8 0.8 0.8

0.6 Normal 0.6 Normal 0.6 Normal Accuracy EDA Accuracy EDA Accuracy EDA 0.4 0.4 0.4 0 20 40 60 80 100 0 20 40 60 80 100 0 20 40 60 80 100 Percent of Dataset (%) Percent of Dataset (%) Percent of Dataset (%) (a) (b) (c) TREC (N=5,452) PC (N=39,418) All Datasets 1 1 1

0.8 0.8 0.8

0.6 Normal 0.6 Normal 0.6 Normal Accuracy EDA Accuracy EDA EDA 0.4 0.4 0.4 0 20 40 60 80 100 0 20 40 60 80 100 Average Accuracy 0 20 40 60 80 100 Percent of Dataset (%) Percent of Dataset (%) Percent of Dataset (%) (d) (e) (f)

Figure 1: Performance on benchmark text classiﬁcation tasks with and without EDA, for various dataset sizes used for training. For reference, the dotted grey line indicates best performances from Kim (2014) for SST-2, CR, SUBJ, and TREC, and Ganapathibhotla (2008) for PC.

average accuracy of 88.6% while only using 50% Pro (original) of the available training data. Pro (EDA) Con (original) 4.3 Does EDA conserve true labels? Con (EDA) In data augmentation, input data is altered while class labels are maintained. If sentences are significantly changed, however, then original class labels may no longer be valid. We take a visualization approach to examine whether EDA operations significantly change the meanings of augmented sentences. First, we train an RNN on the pro-con classification task (PC) without aug- Figure 2: Latent space visualization of original and mentation. Then, we apply EDA to the test set augmented sentences in the Pro-Con dataset. Aug- by generating nine augmented sentences per orig- mented sentences (small triangles and circles) closely inal sentence. These are fed into the RNN along surround original sentences (big triangles and circles) with the original sentences, and we extract the out- of the same color, suggesting that augmented sentences maintianed their true class labels. puts from the last dense layer. We apply t-SNE (Van Der Maaten, 2014) to these vectors and plot their 2-D representations (Figure2). We found to explore the effects of each operation in EDA. that the resulting latent space representations for Synonym replacement has been previously used augmented sentences closely surrounded those of (Kolomiyets et al., 2011; Zhang et al., 2015; Wang the original sentences, which suggests that for the and Yang, 2015), but the other three EDA opera- most part, sentences augmented with EDA con- tions have not yet been explored. One could hy- served the labels of their original sentences. pothesize that the bulk of EDA’s performance gain is from synonym replacement, so we isolate each 4.4 Ablation Study: EDA Decomposed of the EDA operations to determine their indi- So far, we have seen encouraging empirical re- vidual ability to boost performance. For all four sults. In this section, we perform an ablation study operations, we ran models using a single oper-

6384 SR RI RS RD 3 3 3 3 2 2 2 2 N=500 1 1 1 1 N=2,000 0 0 0 0 −1 −1 −1 −1 N=5,000 −2 −2 −2 −2 Full Data −3 −3 −3 −3 0 0.1 0.2 0.3 0.4 0.5 0 0.1 0.2 0.3 0.4 0.5 0 0.1 0.2 0.3 0.4 0.5 0 0.1 0.2 0.3 0.4 0.5

Performance Gain (%) α parameter α parameter α parameter α parameter

Figure 3: Average performance gain of EDA operations over five text classification tasks for different training set sizes. The α parameter roughly means “percent of words in sentence changed by each augmentation.” SR: synonym replacement. RI: random insertion. RS: random swap. RD: random deletion. ation while varying the augmentation parameter ing sets, overfitting was more likely, so generat- α={0.05, 0.1, 0.2, 0.3, 0.4, 0.5} (Figure3). ing many augmented sentences yielded large per- It turns out that all four EDA operations con- formance boosts. For larger training sets, adding tribute to performance gain. For SR, improvement more than four augmented sentences per original was good for small α, but high α hurt perfor- sentence was unhelpful since models tend to gen- mance, likely because replacing too many words eralize properly when large quantities of real data in a sentence changed the identity of the sentence. are available. Based on these results, we recom- For RI, performance gains were more stable for mend usage parameters in Table3. different α values, possibly because the original words in the sentence and their relative order were Ntrain α naug maintained in this operation. RS yielded high per- 500 0.05 16 formance gains at α≤0.2, but declined at α≥0.3 2,000 0.05 8 since performing too many swaps is equivalent to 5,000 0.1 4 shuffling the entire order of the sentence. RD had More 0.1 4 the highest gains for low α but severely hurt per- Table 3: Recommended usage parameters. formance at high α, as sentences are likely un- intelligible if up to half the words are removed. Improvements were more substantial on smaller 5 Comparison with Related Work datasets for all operations, and α=0.1 appeared to be a “sweet spot” across the board. Related work is creative but often complex. Back- translation (Sennrich et al., 2016), translational 4.5 How much augmentation? data augmentation (Fadaee et al., 2017), and nois- The natural next step is to determine how the num- ing (Xie et al., 2017) have shown improvements in ber of generated augmented sentences per original BLEU measure for machine translation. For other sentence, naug, affects performance. In Figure4, tasks, previous approaches include task-specific we show average performances over all datasets heuristics (Kafle et al., 2017) and back-translation for naug={1, 2, 4, 8, 16, 32}. For smaller train- (Silfverberg et al., 2017; Yu et al., 2018). Regard- ing synonym replacement (SR), one study showed 3 a 1.4% F1-score boost for tweet classification by N=500 finding synonyms with k-nearest neighbors us- N=2,000 2 N=5,000 ing word embeddings (Wang and Yang, 2015). Full Data Another study found no improvement in tempo- 1 ral analysis when replacing headwords with synonyms (Kolomiyets et al., 2011), and mixed re-

Performance Gain (%) 0 0 2 4 8 16 32 sults were reported for using SR in character-level naug text classification (Zhang et al., 2015); however, Figure 4: Average performance gain of EDA across five neither work conducted extensive ablation studies. text classification tasks for various training set sizes. Most studies explore data augmentation as a naug is the number of generated augmented sentences complementary result for translation or in a task- per original sentence. specific context, so it is hard to directly compare

6385 EDA with previous literature. But there are two recent years, we suspect that researchers will soon studies similar to ours that evaluate augmentation find higher-performing augmentation techniques techniques on multiple datasets. Hu (2017) pro- that will also be easy to use. posed a generative model that combines a varia- Notably, much of the recent work in NLP fo- tional auto-encoder (VAE) and attribute discrim- cuses on making neural models larger or more inator to generate fake data, demonstrating a 3% complex. Our work, however, takes the opposite gain in accuracy on two datasets. Kobayashi approach. We introduce simple operations, the re- (2018) showed that replacing words with other sult of asking the fundamental question, how can words that were predicted from the sentence con- we generate sentences for augmentation without text using a bi-directional language model yielded changing their true labels? We do not expect EDA a 0.5% gain on five datasets. However, training to be the go-to augmentation method for NLP, ei- a variational auto-encoder or bidirectional LSTM ther now or in the future. Rather, we hope that our language model is a lot of work. EDA yields re- line of thought might inspire new approaches for sults on the same order of magnitude but is much universal or task-specific data augmentation. easier to use because it does not require training a Now, let’s note many of EDA’s limitations. language model and does not use external datasets. Foremost, performance gain can be marginal when In Table4, we show EDA’s ease of use compared data is sufficient; for our five classification tasks, with other techniques. the average performance gain for was less than 1% when training with full datasets. And while perfor- Technique (#datasets) LM Ex Dat mance gains seem clear for small datasets, EDA Trans. data aug.1 (1) yes yes might not yield substantial improvements when Back-translation2 (1) yes yes using pre-trained models. One study found that VAE + discrim.3 (2) yes yes EDA’s improvement was negligible when using Noising4 (1) yes no ULMFit (Shleifer, 2019), and we expect similar Back-translation5 (2) yes no results for ELMo (Peters et al., 2018) and BERT LM + SR6 (2) yes no (Devlin et al., 2018). Finally, although we evalu- Contextual aug.7 (5) yes no ate on five benchmark datasets, other studies on SR - kNN8 (1) no no data augmentation in NLP use different models EDA (5) no no and datasets, and so fair comparison with related work is highly non-trivial. Table 4: Related work in data augmentation. #datasets: number of datasets used for evaluation. Gain: reported 7 Conclusions performance gain on all evaluation datasets. LM: requires training a language model or deep learning. Ex We have shown that simple data augmentation op- 9 Dat: requires an external dataset. erations can boost performance on text classification tasks. Although improvement is at times marginal, EDA substantially boosts performance 6 Discussion and Limitations and reduces overfitting when training on smaller Our paper aimed to address the lack of standard- datasets. Continued work on this topic could ex- ized data augmentation in NLP (compared to vi- plore the theoretical underpinning of the EDA op- sion) by introducing a set of simple operations that erations. We hope that EDA’s simplicity makes a might serve as a baseline for future investigation. compelling case for further thought. With the rate that NLP research has progressed in 8 Acknowledgements 1(Fadaee et al., 2017) for translation 2(Yu et al., 2018) for comprehension We thank Chengyu Huang, Fei Xing, and Yifang 3(Hu et al., 2017) for text classification Wei for help with study design and paper revi- 4(Xie et al., 2017) for translation sions, and Chunxiao Zhou for insightful feedback. 5 (Sennrich et al., 2016) for translation Jason Wei thanks Eugene Santos for inspiration. 6(Kolomiyets et al., 2011) for temporal analysis 7(Kobayashi, 2018) for text classification 8(Wang and Yang, 2015) for tweet classification 9EDA does use a synonym dictionary, WordNet, but the cost of downloading it is far less than training a model on an external dataset, so we don’t count it as an “external dataset.”

6386 References Xin Li and Dan Roth. 2002. Learning question classi- fiers. In Proceedings of the 19th International Con- Xiaodong Cui, Vaibhava Goel, and Brian Kingsbury. ference on Computational Linguistics - Volume 1, 2015. Data augmentation for deep neural net- COLING ’02, pages 1–7, Stroudsburg, PA, USA. work acoustic modeling. IEEE/ACM Trans. Audio, Association for Computational Linguistics. Speech and Lang. Proc., 23(9):1469–1477. Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Recurrent neural network for text classification with Kristina Toutanova. 2018. BERT: pre-training of multi-task learning. In Proceedings of the Twenty- deep bidirectional transformers for language under- Fifth International Joint Conference on Artificial standing. CoRR, abs/1810.04805. Intelligence, IJCAI’16, pages 2873–2879. AAAI Marzieh Fadaee, Arianna Bisazza, and Christof Monz. Press. 2017. Data augmentation for low-resource neural Qian Liu, Zhiqiang Gao, Bing Liu, and Yuanlin Zhang. machine translation. In Proceedings of the 55th An- 2015. Automated rule selection for aspect extrac- nual Meeting of the Association for Computational tion in opinion mining. In Proceedings of the 24th Linguistics (Volume 2: Short Papers), pages 567– International Conference on Artificial Intelligence, 573. Association for Computational Linguistics. IJCAI’15, pages 1291–1297. AAAI Press. Murthy Ganapathibhotla and Bing Liu. 2008. Mining opinions in comparative sentences. In Proceedings George A. Miller. 1995. Wordnet: A lexical database of the 22Nd International Conference on Computa- for english. Commun. ACM, 38(11):39–41. tional Linguistics - Volume 1, COLING ’08, pages Bo Pang and Lillian Lee. 2004. A sentimental educa- 241–248, Stroudsburg, PA, USA. Association for tion: Sentiment analysis using subjectivity summa- Computational Linguistics. rization based on minimum cuts. In Proceedings of Minqing Hu and Bing Liu. 2004. Mining and summa- the 42Nd Annual Meeting on Association for Com- rizing customer reviews. In Proceedings of the Tenth putational Linguistics, ACL ’04, Stroudsburg, PA, ACM SIGKDD International Conference on Knowl- USA. Association for Computational Linguistics. edge Discovery and Data Mining , KDD ’04, pages Jeffrey Pennington, Richard Socher, and Christo- 168–177, New York, NY, USA. ACM. pher D. Manning. 2014. Glove: Global vectors for Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan word representation. In Empirical Methods in Nat- Salakhutdinov, and Eric P. Xing. 2017. Toward con- ural Language Processing (EMNLP), pages 1532– trolled generation of text. In ICML. 1543. Kushal Kafle, Mohammed Yousefhussien, and Christo- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt pher Kanan. 2017. Data augmentation for visual Gardner, Christopher Clark, Kenton Lee, and Luke question answering. In Proceedings of the 10th In- Zettlemoyer. 2018. Deep contextualized word rep- ternational Conference on Natural Language Gen- resentations. CoRR, abs/1802.05365. eration, pages 198–202. Association for Computa- David Rolnick, Andreas Veit, Serge J. Belongie, and tional Linguistics. Nir Shavit. 2017. Deep learning is robust to massive Yoon Kim. 2014. Convolutional neural networks for label noise. CoRR, abs/1705.10694. sentence classification. CoRR, abs/1408.5882. Rico Sennrich, Barry Haddow, and Alexandra Birch. Tom Ko, Vijayaditya Peddinti, Daniel Povey, and San- 2016. Improving neural machine translation mod- jeev Khudanpur. 2015. Audio augmentation for els with monolingual data. In Proceedings of the speech recognition. In INTERSPEECH. 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages Sosuke Kobayashi. 2018. Contextual augmentation: 86–96. Association for Computational Linguistics. Data augmentation by words with paradigmatic re- lations. In NAACL-HLT. Sam Shleifer. 2019. Low resource text classification with ulmfit and backtranslation. CoRR, Oleksandr Kolomiyets, Steven Bethard, and Marie- abs/1903.09244. Francine Moens. 2011. Model-portability experiments for textual temporal analysis. In Proceed- Miikka Silfverberg, Adam Wiemerslage, Ling Liu, and ings of the 49th Annual Meeting of the Association Lingshuang Jack Mao. 2017. Data augmentation for for Computational Linguistics: Human Language morphological reinflection. In Proceedings of the Technologies: Short Papers - Volume 2, HLT ’11, CoNLL SIGMORPHON 2017 Shared Task: Univer- pages 271–276, Stroudsburg, PA, USA. Association sal Morphological Reinflection, pages 90–99. Asso- for Computational Linguistics. ciation for Computational Linguistics. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hin- Patrice Simard, Yann LeCun, John S. Denker, and ton. 2017. Imagenet classification with deep convo- Bernard Victorri. 1998. Transformation invari- lutional neural networks. Commun. ACM, 60(6):84– ance in pattern recognition-tangent distance and tan- 90. gent propagation. In Neural Networks: Tricks of

6387 the Trade, This Book is an Outgrowth of a 1996 NIPS Workshop, pages 239–27, London, UK, UK. Springer-Verlag. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Ng, and Christopher Potts. 2013. Parsing With Composi- tional Vector Grammars. In EMNLP. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Ser- manet, Scott E. Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Ra- binovich. 2014. Going deeper with convolutions. CoRR, abs/1409.4842. Duyu Tang, Bing Qin, and Ting Liu. 2015. Document modeling with gated recurrent neural network for sentiment classification. pages 1422–1432. Simon Tong and Daphne Koller. 2002. Support vector machine active learning with applications to text classification. J. Mach. Learn. Res., 2:45–66. Laurens Van Der Maaten. 2014. Accelerating t-sne using tree-based algorithms. J. Mach. Learn. Res., 15(1):3221–3245. William Yang Wang and Diyi Yang. 2015. That’s so annoying!!!: A lexical and frame-semantic em- bedding based data augmentation approach to automatic categorization of annoying behaviors using #petpeeve tweets. In Proceedings of the 2015 Con- ference on Empirical Methods in Natural Language Processing, pages 2557–2563. Association for Com- putational Linguistics. Ziang Xie, Sida I. Wang, Jiwei Li, Daniel Levy, Aim- ing Nie, Dan Jurafsky, and Andrew Y. Ng. 2017. Data noising as smoothing in neural network language models. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. Qanet: Combining local convolution with global self-attention for reading comprehension. CoRR, abs/1804.09541. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proceedings of the 28th International Conference on Neural Information Processing Sys- tems - Volume 1, NIPS’15, pages 649–657, Cam- bridge, MA, USA. MIT Press.

6388