sign in sign up
<<
Home , Deep learning, GloVe (machine learning), Machine learning, Sentiment analysis, Word2vec, Word embedding

Improving the Accuracy of Pre-trained Embeddings for

Seyed Mahdi Rezaeinia a,b, Ali Ghodsi a, Rouhollah Rahmani b

a Department of and Actuarial Science, University of Waterloo, Waterloo, Canada b and Technology Department, University of Tehran, Tehran, Iran

Abstract Sentiment analysis is one of the well-known tasks and fast growing research areas in natural language processing (NLP) and text classifications. This technique has become an essential part of a wide range of applications including politics, business, advertising and . There are various techniques for sentiment analysis, but recently word embeddings methods have been widely used in sentiment classification tasks. and GloVe are currently among the most accurate and usable methods which can convert into meaningful vectors. However, these methods ignore sentiment information of texts and need a huge corpus of texts for training and generating exact vectors which are used as inputs of deep models. As a result, because of the small size of some corpuses, researcher often have to use pre-trained word embeddings which were trained on other large such as News with about 100 billion words. The increasing accuracy of pre-trained word embeddings has a great impact on sentiment analysis research. In this paper we propose a novel method, Improved Word Vectors (IWV), which increases the accuracy of pre-trained word embeddings in sentiment analysis. Our method is based on Part-of-Speech (POS) tagging techniques, lexicon-based approaches and Word2Vec/GloVe methods. We tested the accuracy of our method via different models and sentiment datasets. Our experiment results show that Improved Word Vectors (IWV) are very effective for sentiment analysis.

1. Introduction Sentiment analysis is a practical technique that allows businesses, researchers, governments, politicians and organizations to know about people’s sentiments, which play an important role in decision making processes. Word Embedding is one of the most useful deep learning methods used for constructing vector representations of words and documents. These methods have received a lot of attention in text and sentiment analysis because of their abilities to capture the syntactic and semantic relations among words. The two successful deep learning methods of word embeddings are Word2Vec [1,2] and Global Vectors (GloVe) [3]. Many researchers have used these two methods in their sentiment analysis research [4,5,6,7].

Although very effective, these methods have several limits and need to be improved. The Word2Vec and GloVe need very large corpuses for training and presenting an acceptable vector for each word [8,6]. For example, Google has used about 100 billion words for training Word2Vec and has re-released pre-trained word vectors with 300 . Because of the small size of some datasets, investigators have to use pre-trained word vectors such as Word2Vec and GloVe, which may not be the best fit for their [9,10,11,12,13,14,15]. Another problem is that the word vector calculations of the two methods that are used to represent a document do not consider the context of the document [16]. For example, the word vector for “beetle” as a car is equal to its word vector as an animal. In addition, both models ignore the relationships between terms that do not literally co-occur [16]. Also, Cerisara et al. [17] have found that the standard Word2Vec word embedding techniques don't bring valuable information for dialogue act recognition in three different languages. Another important problem of these word embedding techniques is ignoring the sentiment information of the given text [6,7,8]. The side effect of this problem is that those words with opposite polarity are mapped into close vectors and it is a disaster for sentiment analysis [4].

In this research, we propose a novel method to improve the accuracy of pre-trained Word2Vec/Glove vectors in sentiment analysis tasks. The proposed method was tested by different sentiment datasets and various deep learning models from published papers. The results show that the method increases the accuracy of pre-trained word embeddings vectors for sentiment analysis. The organization of this paper is as follows: Section 2 describes the related works and literature review for this research. Section 3 presents our proposed method and , and additionally describes the proposed deep learning model for testing the method. Section 4 reports our experiments, showing results along with evaluations and discussions. Section 5 is the conclusion and future works.

2. Related Works

2.1. Lexicon-based method

Sentiment classification techniques are mainly divided into lexicon-based methods and methods such as Deep Learning [18,19]. The lexicon-based sentiment analysis approach is typically based on lists of words and phrases with positive and negative connotations [20,21,22]. This approach needs a dictionary of negative and positive sentiment values assigned to words. These methods are simple, scalable, and computationally efficient. As a result, they are mostly used to solve general sentiment analysis problems [18]. However, lexicon-based methods depend on human effort in human-labeled documents [19] and sometimes suffer from low coverage [8]. Also, they depend on finding the sentiment lexicon which is applied to analysis the text [18].

Because of the accuracy improvement of text classification, the lexicon-based approaches have combined with machine learning methods recently. Several authors found that the machine learning methods were more accurate than the lexicon methods [19,23]. Mudinas et al. [24] increased the accuracy of sentiment analysis by combining lexicon-based and Support Vector Machine (SVM) methods. Zhang et al. [25] successfully combined lexicon-based approach and binary classifier for sentiment classification of data. Basari et al. [26] have combined the Particle Swarm Optimization (PSO) technique and SVM method for sentiment analysis of movie reviews. In all of these cases, the machine learning techniques improved the accuracy of text classifications.

2.2. Deep learning method

Recently, deep learning methods have played an increasing role in natural language processing. Most of the deep learning tasks in NLP has been oriented towards methods which using word vector representations [6]. Continuous vector representations of words algorithms such as Word2Vec and GloVe are deep learning techniques which can convert words into meaningful vectors. The vector representations of words are very useful in text classification, clustering and . Word embeddings techniques have some advantages compare to bag-of-words representation. For instance, words close in meaning are near together in the word embedding space. Also, word embeddings have lower dimensionality than the bag-of-words [2].

The accuracy of the Word2vec and Glove depends on text corpus size. Meaning, the accuracy increases with the growth of text corpus. Tang et al. [4] proposed learning continuous word representations for sentiment analysis on Twitter which is a large social networks dataset. Severyn and Moschitti [27] used Word2Vec method to learn the word embeddings on 50M tweets and applied generated pre-trained vectors as inputs of a deep learning model. Recently, Lauren et al. [28] have proposed a discriminant document embeddings method which has used skip-gram for generating word embeddings of clinical texts. Fu et al. [5] applied Word2Vec for word embeddings of English dataset and Chinese Wikipedia dataset. The word embeddings used as inputs of recursive for sentiment analysis approach. Ren et al. [7] proposed a new word representation method for Twitter sentiment classification. They used Word2Vec to generate word embeddings of some datasets in their method. Qin et al. [29] trained Word2Vec algorithm by English Wikipedia corpus which has 408 million words. They used those pre-trained vectors as inputs of convolutional neural networks for data-driven tasks. Nevertheless, as already mentioned, these word embedding algorithms need a huge corpus of texts for training [8] and most of them ignore the sentiment information of text [4,6]. Because of the limitations and restrictions in some corpuses, researchers prefer to use pre-trained word embeddings vectors as inputs of machine learning models. Kim [9] has used pre-trained Word2Vec vectors as inputs to convolutional neural networks and has increased the accuracy of sentiment classification. Also, Camacho-Collados et al. [11] used pre-trained Word2Vec vectors for the representation of . Zhang and Wallace [10] have applied pre-trained GloVe and Word2Vec vectors in their deep learning model and have improved the accuracy of sentence and sentiment classification. Caliskan et al. [12] used pre-trained GloVe word embeddings vectors for increasing the accuracy of their proposed method. Wang et al. [13] applied pre- trained GloVe vectors as inputs of attention-based LSTMs model for aspect-level sentiment analysis. Liu et al. [15] have used pre-trained Word2Vec as a word embedding representation for recommending Idioms in essay writing.

As a result, increasing the accuracy of pre-trained word embedding is very important and plays a vital role in sentiment classification methods. Zhang and Wallace [10] combined pre-trained Word2Vec and GloVe vectors in their deep learning model, but the accuracies were decreased. Some results are shown in table 1.

Table1: The results of the combination of pre-trained Word2Vec and Glove on three datasets [10]

Dataset Reduced accuracy rate (%) Movie reviews (MR) -0.22 % Stanford sentiment (SST) -1.1% TREC -0.17%

According to table1, combination of pre-trained Word2Vec and Glove decreased the accuracy of text and sentiment classification on some datasets. Also, Kamkarhaghighi and Makrehchi [16] have proposed an algorithm for increasing the accuracy of pre-trained Word2Vec and Glove. Their algorithm was tested on two datasets and the accuracy of Word2Vec was decreased on one dataset by the proposed algorithm. In the following section we present in detail our proposed model, its algorithm and the proposed deep learning model for checking our method.

3. Proposed method

In our proposed method, Improved Word Vector (IWV) we have increased the accuracy of word embedding vectors based on the combination of natural language processing techniques, lexicon-based approaches and Word2Vec/GloVe methods which have high accuracies. The main architecture of the proposed method has been shown in figure 1.

Figure1: The main architecture of the proposed method (Improved word vector)

3.1. Word2Vec and GloVe

Word2Vec and GloVe are two successful word embedding algorithms which have high accuracy. Word2Vec is based on continuous Bag-of-Words (CBOW) and Skip-gram architectures which can provide high quality word embedding vectors. CBOW predicts a word given its context and Skip-gram can predict the context given a word. The generated vectors of words which appear in common contexts in the corpus are located close to each other in the . GloVe word embedding is a global log-bilinear regression model and is based on co-occurrence and factorization of in order to get vectors. In this study, we used 300- word2vec embeddings trained on Google News and 300-dimension GloVe word embeddings.

3.2. POS2Vec (P2V)

Part-of-speech (POS) tagging is an important and fundamental step in Natural Language Processing which is the process of assigning to each word of a text the proper POS . The Part-of-speech gives large amount of information about a word and its neighbors, syntactic categories of words (nouns, verbs, adjectives, adverbs, etc.) and similarities and dissimilarities between them. We converted each generated POS tag to a constant vector and concatenated with Word2Vec/GloVe vectors. We assigned 50- dimensional vectors to each word. As a result, Word2Vec/Glove vectors will have syntactic information of words.

3.3. Lexicon2Vec (L2V)

The sentiment and lexicons are lists of phrases and words which have polarity scores and can be used to analyze texts. Each lexicon contains of words and their values which are the sentiment scores for that words. There are various sentiment and emotion lexicons that can be used, but choosing the proper lexicon is very important. We selected six lexicons as our resources and assigned 6-dimensional vectors to each word.

• National Research Council Canada (NRC) Emoticon Affirmative Context Lexicon and NRC Emoticon Negated Context Lexicon [30,31,32]. • NRC Hashtag Affirmative Context Sentiment Lexicon and NRC Hashtag Negated Context Sentiment Lexicon [30,31,32]. • NRC Emoticon Lexicon [30,31,32]. • NRC Hashtag Sentiment Lexicon [30,31,32]. • SemEval-2015 English Twitter Sentiment Lexicon [33,31]. • Amazon Laptop Sentiment Lexicon [34].

Finding the proper combination of lexicons is not easy and plays a vital role in sentiment analysis. We tested various lexicons and selected six of them as one of the best combination.

Algorithm 1 shows the process of our proposed method. It gets a sentence and returns improved word vectors of the sentence. In the first step, assign a constant vector to each POS tag. In the second step, each word vector of the input sentence is extracted from Word2Vec or GloVe datasets and if a word doesn’t exist in the datasets its vector will generate randomly. In the third step, POS tag of each word is determined and assign a constant vector to each one. In the next step, sentiment scores of each word are extracted from all lexicons and will normalize them. If a word doesn’t exist in any lexicons, its score will be zero. The generated vectors from each step will be concatenated with other vectors from previous steps. Algorithm 1 : Improved Word Vector (IWV) generation Inputs:

S = {W1, W2,……., Wn} , Input sentence S contains n words PT = {T1, T2,……, Tm}, All POS tags W2VD: Google Word2Vec Dataset GloVeD: Glove Dataset h: Number of lexicons LEX1, LEX2,……., LEXh All lexicons Output: IMV: Improved word vectors of sentence S

1. for j=1 to m do 2. VTj GenerateVector ( Tj ) 3. Tj < Tj , VTj > 4. end for

5. for each Wi in S do 6. If Wi exist in W2VD then extract VecWi 7. MVi VecWi 8. elseif Wi exist in GloVeD then extract VecWi 9. MVi VecWi 10. else 11. MVi RandomGenVec (VecWi) 12. endif

13. POS ExtractPOS ( Wi ) 14. for k=1 to m do 15. If POS=Tk then ADD VTk into MVi 16. end if 17. end for

18. for k=1 to h do 19. If Wi in LEXk then 20. Sik FindVector ( Wi ) 21. Sik Normalize ( Sik ) Between -0.995 and +0.995 22. else 23. Sik 0 24. end if 25. ADD Sik into MVi 26. end for 27. ADD MVi into IMV 28. Return IMV 29. end for

We proposed a deep learning model (we called it Model 1) for evaluating our generated vectors on well- known datasets. The model consists of three convolutional neural networks (CNN), a pooling, and a fully connected and the inputs of the model are the improved word vectors (IWV). The model is shown in figure 2.

Figure 2: The proposed model (Model1) for evaluating our proposed approach

In addition, for more confidence about the accuracy of our method, we checked the proposed method by other three deep learning models from published papers. The second deep learning model (Model2) was introduced by Kim [9]. The third model (Model3) was applied by Ouyang et al. [35] and the fourth one (Model4) was proposed by Deriu et al. [36].

4. Experiments

In this section, we describe the datasets and experimental evaluations to show the effectiveness of our proposed method.

4.1. Datasets

Datasets that were used in our research are as follows:

MR: Movie reviews dataset with equal positive and negative sentences and each review contains a sentence [37].

CR: Customer reviews of products classified into positive and negative reviews [22]. SST: The Stanford sentiment treebank introduced by Socher et al. [38] contains train/development/test texts. We used only train and test sentences and binary labels (positive, negative) for our model.

4.2. Results

We have tested our approach on four different deep learning models and various sentiment datasets which have different features. Our implementations were GPU-based and have trained our models on four GeForce GTX Titan X GPUs. We have used Tensorflow for implementing and training all the deep learning models in our research. All reports are based on the average of accuracies calculated over multiple runs of 10-fold cross-validation(CV), however SST has predefined train and test sets. In 10-fold CV, the dataset is randomly partitioned into 10 folds. Among them, nine folds are used for training and the remaining one fold is used for testing. We compared the results of the 356-dimension IWV with 300-dimension Word2Vec and 300-dimension GloVe.

As already mentioned before, six sentiment lexicons were used to extract and generate the lexicon vectors. We only used unigram scores for our research. The distribution of the lexicons used in our research is listed in table 2.

Lexicon Positive Negative Neutral Total Scores Ranges NRC Emoticon Affirmative Context Lexicon 28025 27121 0 55146 -5.844 to +4.495 and NRC Emoticon Negated Context Lexicon NRC Hashtag Affirmative Context Sentiment 19502 24447 0 43949 -10.025 to +10.661 Lexicon and NRC Hashtag Negated Context Sentiment Lexicon NRC Emoticon Lexicon 38312 24156 0 62468 -4.999 to +5.0 NRC Hashtag Sentiment Lexicon 32048 22081 0 54129 -6.925 to +7.526 SemEval-2015 English Twitter Sentiment 776 726 13 1515 -0.984 to +0.984 Lexicon Amazon Laptop Sentiment Lexicon 14651 11926 0 26577 -5.27 to +3.702

Table 2: Statistics of the lexicons which were used in the research

In our proposed deep learning model the filter size of CNN1, CNN2 and CNN3, respectively are 3,4 and 5 with 100 feature maps each. The flatten layer has 95 nodes and the of the layers is Rectified Linear Unit (ReLU). The results are shown in table 3.

Table 3: The accuracy (%) comparisons between our method (IWV) and other methods based on the deep learning model 1

Method Dim MR CR SST

Word2Vec 300 79.3 81.8 82.0

Glove 300 79.2 81.3 81.0

IWV 356 79.8 82.5 83.7

As shown in table3, the accuracy of our combined vector is higher than the existing pre-trained vectors on three sentiment datasets. In other word, the proposed method increased the accuracy of sentiment analysis in our proposed deep learning model. The results show that the accuracy of SST, CR and MR were increased by 1.7% ,0.7% and 0.5% respectively. Levy et al. [39] showed that Word2Vec performs better than GloVe in various tasks. Also, we found that the Word2Vec is generally more accurate than the GloVe, so all words are searched firstly on the Word2Vec dataset and then are searched on GloVe by the proposed algorithm.

In order to test the IWV more fully, we checked our approach by other three deep learning models on MR and CR datasets which are balanced and unbalanced. The results have shown in table 4.

Table 4: The accuracy (%) comparisons between our method and other methods based on models 2,3 and 4. Each cell reports the average accuracies calculated over multiple runs of 10-fold CV.

Model Dataset Word2Vec(300) GloVe(300) IWV(356) Model 2 MR 79.4 78.7 79.8 Kim [9] CR 82.8 82.9 83.2 Model 3 MR 78.0 77.3 78.2 Ouyang et al. [35] CR 80.0 79.5 81.1 Model 4 MR 79.4 78.4 79.6 Deriu et al. [36] CR 81.6 80.1 81.8

According to table 4, our proposed method is more accurate than other methods in models 2, 3 and 4. As a result, the IWV was compared to other methods six times and all accuracies were improved.

The MR dataset 80 79.5 79

78.5 78 77.5 Accuracy (%) Accuracy 77 Model 1 Model 2 Model 3 Model 4

Glove Word2Vec IWV

Figure 3: The accuracy (%) comparisons of three methods on four deep learning models for MR dataset

Figure 3 shows the accuracy of Glove, Word2Vec and IWV vectors on four deep learning models for MR which is a balanced dataset. It contains a total of 10662 reviews are divided into equal positive and negative reviews. As can be seen, IWV method has the highest accuracy and the Glove has the lowest accuracy among three methods. More generally, the IWV method has increased the accuracy of sentiment analysis in MR dataset between 0.2% and 0.5%.

The CR dataset 83.5 83 82.5

82 81.5 81 80.5 Accuracy (%) Accuracy 80 79.5 79 Model 1 Model 2 Model 3 Model 4

Glove Word2Vec IWV

Figure 4: The accuracy (%) comparisons of three methods on four deep learning models for CR dataset

Figure 4 indicates that the IWV method generally has performed better than other per-trained word embeddings for sentiment analysis of CR dataset. The CR is an unbalanced dataset which contains 2397 positive and 1406 negative reviews. As can be seen, pre-trained Word2vec embedding is almost more accurate than pre-trained Glove embedding, however it is reverse in the model 2. The IWV provides absolute accuracy improvements of 0.7%, 0.4%, 1.1% and 0.2% for model 1, model 2, model 3 and model 4, respectively.

5. Conclusion

In this paper, we proposed a new method to improve the accuracy of well-known pre-trained word embeddings for sentiment analysis. Our method has improved the accuracy of pre-trained word embeddings based on the combination of three approaches such as lexicon-based approach, POS tagging approach and Word2Vec/GloVe approach. In order to ensure about the accuracy of our proposed method, we have tested it nine times with different deep learning models and sentiment datasets. The experimental results indicated that our method has increased the accuracy of sentiment classification tasks in all models and datasets. Briefly, the main advantages of the proposed method are:

• Because of the accuracy of pre-trained Word2Vec/Glove, adding any vector to them decreased the accuracy according to previous researches, but our proposed method has increased the accuracy of pre-trained vectors in sentiment analysis for the first time.

• One of the best combination of lexicons was introduced in our research. This combination increased the accuracy of all tested deep learning models and datasets.

• Any improvements in pre-trained word embeddings/POS tagging/Lexicons in the future, will increase the accuracy of our method.

As a result, our proposed method can be the basis for all sentiment analysis techniques which are used deep learning approaches.

References

[1] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, J. Dean, Distributed representations of words and phrases and their compositionality, In: Proceedings of the 26th International Conference on Neural Information Processing Systems. 2 (2013) 3111-3119. [2] T. Mikolov, K. Chen, G. Corrado, J. Dean, Efficient estimation of word representations in vector space, ICLR Workshop. (2013). [3] J. Pennington, R. Socher, C. Manning, GloVe: Global Vectors for Word Representation, In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). (2014) 1532– 1543. [4] D. Tang, F. Wei, N. Yang, M. Zhou, T. Liu, b. Qin, Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification, In: Proceedings of the 52nd Annual Meeting of the Association for Computational . 1 (2014) 1555–1565. [5] X. Fu, W. Liu, Y. Xu, L. Cui, Combine HowNet lexicon to train phrase recursive autoencoder for sentence-level sentiment analysis, Neurocomputing. 241 (2017) 18-27. [6] O. Araque, I. Corcuera-Platas, J. Sánchez-Rada, A. Iglesias, Enhancing deep learning sentiment analysis with ensemble techniques in social applications, Expert Systems with Applications. 77 (2017) 236- 246. [7] Y. Ren, R. Wang, D. Ji, A topic-enhanced word embedding for Twitter sentiment classification, Information Sciences. 369 (2016) 188-198. [8] M. Giatsoglou, M. Vozalis, K. Diamantaras, A. Vakali, G. Sarigiannidis, K. Chatzisavvas, Sentiment analysis leveraging and word embeddings, Expert Systems with Applications. 69 (2017) 214-224. [9] Y. Kim, Convolutional Neural Networks for Sentence Classification, In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). (2014) 1746-1751. [10] Y. Zhang, B. Wallace, A of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification, arXiv, 1510.03820v4. (2015) [11] J. Camacho-Collados, M. Pilehvar, R. Navigli, Nasari: Integrating explicit knowledge and corpus statistics for a multilingual representation of concepts and entities, . 24 (2016) 36-64. [12] A. Caliskan, J.J. Bryson, A. Narayanan, Semantics derived automatically from language corpora contain human-like , Science. 356 (2017) 183-186. [13] Y. Wang, M. Huang, L. Zhao, X. Zhu, Attention-based LSTM for Aspect-level Sentiment Classification, In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. (2106) 606-615. [14] M. Iyyer, V. Manjunatha, J. Boyd-Graber, H. Daume, Deep Unordered Composition Rivals Syntactic Methods for Text Classification, In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. 1 (2015) 1681-1691. [15] Y. Liu, B. Liu, L. Shan, X. Wang, Modelling Context with Neural Networks for Recommending Idioms in Essay Writing, Neurocomputing. (2017) doi:10.1016/j.neucom.2017.11.005 [16] M. Kamkarhaghighi, M. Makrehchi, Content Word Embedding for document representation, Expert Systems with Applications. 90 (2017) 241-249. [17] C. Cerisara, P. Kral, L. Lenc, On the effects of using word2vec representations in neural networks for dialogue act recognition, Speech & Language. 47 (2018) 175-193.

[18] W. Medhat, H. Hassan, H. Korashy, Sentiment analysis algorithms and applications: A survey. Ain Shams Engineering Journal. 5 (2014) 1093-1113. [19] Z. Hailong, W. Gan, B. Jiang, Machine Learning and Lexicon based Methods for Sentiment Classification: A Survey. In: Proceedings of the 2014 11th Web and Application Conference. (2014) 262-265. [20] M. Taboada, J. Brooke, M. Tofiloski, K. Voll, M. Stede, Lexicon-based methods for sentiment analysis, Computational Linguistics. 37 (2011) 267-307. [21] X. Ding, B. Liu, P. Yu, A holistic lexicon-based approach to opinion mining, In: Proceedings of the 2008 international conference on web search and . (2008) 231-240. [22] M. Hu, B. Liu, Mining and summarizing customer reviews. In: Proceedings of the tenth ACM SIGKDD international conference on Knowledge and data mining. (2004) 168-177. [23] K. Ravi, V. Ravi, A survey on opinion mining and sentiment analysis: Tasks, approaches and applications, Knowledge-Based Systems. 89 (2015) 14-46. [24] A. Mudinas, D. Zhang, M. Levene, Combining Lexicon and Learning based Approaches for - Level Sentiment Analysis, In: Proceedings of the First International Workshop on Issues of Sentiment Discovery and Opinion Mining. 5 (2012). [25] L. Zhang, R. Ghosh, M. Dekhil, M. Hsu, B. Liu, Combining Lexicon-based and Learning-based Methods for Twitter Sentiment Analysis, Hewlett -Packard Development Company. (2011)

[26] A.S.H. Basari, B. Hussin, I.G.P. Ananta, J. Zeniarja, Opinion mining of movie review using hybrid method of support vector machine and particle swarm optimization, Proc. Eng. 53 (2013) 453-462. [27] A. Severyn, A. Moschitti, Twitter Sentiment Analysis with Deep Convolutional Neural Networks, In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. (2015) 959-962. [28] P. Lauren, G. Qu, F. Zhang, A. Lendasse, Discriminant document embeddings with an for classifying clinical narratives, Neurocomputing. (2017) 1-10. [29] P. Qin, W. Xu, J. Guo, An empirical convolutional neural network approach for semantic relation classification, Neurocomputing.190 (2016) 1-9. [30] S. Mohammad, S. Kiritchenko, X. Zhu, NRC-Canada: Building the State-of-the-Art in Sentiment Analysis of Tweets, In: Proceedings of the seventh International Workshop on Semantic Evaluation Exercises. (2013) 321-327. [31] S. Kiritchenko, X. Zhu, S. Mohammad, Sentiment Analysis of Short Informal Texts, Journal of Artificial Intelligence Research. 50 (2014) 723-762. [32] X. Zhu, S. Kiritchenko, S. Mohammad, NRC-Canada-2014: Recent Improvements in Sentiment Analysis of Tweets, In: Proceedings of the eight international workshop on Semantic Evaluation Exercises. (2014) [33] S. Rosenthal, P. Nakov, S. Kiritchenko, S. Mohammad, A. Ritter, V. Stoyanov, SemEval-2015 Task 10: Sentiment Analysis in Twitter, In: Proceedings of the ninth international workshop on Semantic Evaluation Exercises. (2015) 451-463. [34] S. Kiritchenko, X. Zhu, C. Cherry, S. Mohammad, NRC-Canada-2014: Detecting Aspects and Sentiment in Customer Reviews. In: Proceedings of the eight international workshop on Semantic Evaluation Exercises. (2014)

[35] X. Ouyang, P. Zhou, C. Li, L. Liu, Sentiment Analysis Using Convolutional Neural Network, IEEE International Conference on Computer and Information Technology; Ubiquitous and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing (CIT/IUCC/DASC/PICOM). (2015) [36] J. Deriu, A. Lucchi, V. Luca, T. Hofmann, M. Jaggi, Leveraging Large Amounts of Weakly Supervised Data for Multi-Language Sentiment Classification, International Conference Committee (IW3C2). (2017) 1045-1052.

[37] B. Pang, L. Lee, Seeing stars: Exploiting class relationships for sentiment with respect to rating scales, In: Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. (2005) 115-124. [38] R. Socher, A. Perelygin, J.Wu, J. Chuang, C. Manning, A. Ng, C. Potts, Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank, In: Proceedings of EMNLP. (2013) [39] O. Levy, Y. Goldberg, I. Dagan, Improving distributional similarity with lessons learned from word embeddings, Transactions of the Association for Computational Linguistics. 3 (2015) 211-225.