A MACHINE LEARNING APPROACH TO SENTIMENT ANALYSIS AND STANCE DETECTION FOR POLITICAL TWEETS EXPLORING THE INFLUENCE OF IRONY ON THE PREDICTABILITY OF SENTIMENT AND STANCE

Aantal woorden: 14658

Lot De Kimpe Studentennummer: 01404202 Promotor: Prof. Dr. Els Lefever

Masterproef voorgelegd voor het behalen van de graad Master in de Meertalige Communicatie Academiejaar: 2017 - 2018

Verklaring i.v.m. auteursrecht De auteur en de promotor(en) geven de toelating deze studie als geheel voor consultatie beschikbaar te stellen voor persoonlijk gebruik. Elk ander gebruik valt onder de beperkingen van het auteursrecht, in het bijzonder met betrekking tot de verplichting de bron uitdrukkelijk te vermelden bij het aanhalen van gegevens uit deze studie.

Het auteursrecht betreffende de gegevens vermeld in deze studie berust bij de promotor(en). Het auteursrecht beperkt zich tot de wijze waarop de auteur de problematiek van het onderwerp heeft benaderd en neergeschreven. De auteur respecteert daarbij het oorspronkelijke auteursrecht van de individueel geciteerde studies en eventueel bijhorende documentatie, zoals tabellen en figuren.

Page 3 of 76

Abstract With the emergence of Web 2.0, easy-accessible microblogging platforms such as Facebook and Twitter have allowed users to easily share their opinions online. Sentiment analysis and stance detection, allow a business, organization or political party to gather all these viewpoints and to find out which sentiment (positive, negative or neutral) a piece of text contains to optimize their products and services. Despite the fast developments in this field of study, challenges for the automatic prediction of sentiment and stance labels are still present (Pang et al., 2008; Kumar & Sebastian, 2012; Mandya et al., 2016). In this research, it was explored how well a machine learning system performs for sentiment analysis and stance detection on an English Twitter corpus of 482 political tweets with #Brexit. The manually annotated labels were compared to the predictions of a machine learning system, considering the possible impact of irony on the performance of our system. The results show that the system performs fairly well on sentiment analysis (accuracy of 0,55) and stance detection (accuracy of 0,61). It remains, however, unclear to which extent irony affects the quality of the automatic predictions. Further research could specifically focus on the comparison between irony detection and sentiment analysis or stance detection. (204)

Page 4 of 76

Acknowledgements First of all, I would like to express my gratitude towards my sister, Lies De Kimpe. As from the very first letter of my bachelor’s paper, she provided me with professional feedback and encouraging pep talks. And even now, up until the very last letter of my master’s thesis, she has always been by my side for support. Without her eye for detail and willingness to answer every little question, this paper would have not reached the quality it has today.

Secondly, many thanks go to my supervisor, Els Lefever, for her help, patience and support during the last two years. She has always given me useful advice and motivating compliments, which encouraged me to complete my thesis successfully. Furthermore, I also wish to thank her for being such an approachable and kind mentor.

Thirdly, I would like to thank my friends Julie Carton, Lien De Wulf, Bo Van Eetvelde and the entire group of KLJ people, who were my towers of strength in stressful moments. They were wonderful in offering my daily dose of distraction in solitary times behind my desk. Special thanks go to Hanne Christiaens, who was willing to share her recognizable experiences as a master’s student at VTC with me.

And last but definitely not least, I wish to thank my parents from the bottom of my heart for allowing me to pursue any possible dream and keeping their endless faith in me.

Page 5 of 76

Table of contents

List of tables and figures ...... 8 1. INTRODUCTION ...... 9 2. LITERATURE STUDY ...... 11 2.1 Sentiment analysis ...... 11 2.1.1 Terminology ...... 12 2.2 Approaches to SA ...... 13 2.2.1 Lexicon-based approach to SA ...... 14 2.2.2 Supervised machine learning approach to SA ...... 15 2.3 ABSA: Aspect Based Sentiment Analysis ...... 17 2.4 Sentiment analysis for political tweets ...... 17 2.4.1 Twitter ...... 18 2.5 Stance detection ...... 19 2.6 Irony detection ...... 20 2.6.1 What is irony? ...... 20 2.6.2 Difficulties and challenges ...... 21 3. RESEARCH DESIGN ...... 22 3.1 Research questions and hypotheses ...... 22 3.2 Methodology ...... 23 3.2.1 Data collection ...... 23 3.1.1 Annotation ...... 24 3.1.2 Experimental approach ...... 25 4. RESULTS ...... 26 4.1 Results manual annotation ...... 26 4.1.1 Sentiment and topics...... 26 4.1.2 Stance and irony ...... 28 4.2 Results machine learning system ...... 30 4.2.1 Sentiment and topics...... 30 4.2.2 Stance and irony ...... 31 4.3 Analysis ...... 33 4.3.1 Sentiment analysis: tenfold cross-validation scheme ...... 33 4.3.2 Sentiment analysis: general overview ...... 34 4.3.2.1 Impact of irony on the prediction of sentiment ...... 34

Page 6 of 76

4.3.2.2 Error Analysis ...... 34 4.3.3 Stance detection: tenfold cross-validation scheme ...... 39 4.3.4 Stance detection: general overview ...... 40 4.3.2.1 Impact of irony on the prediction of stance ...... 41 4.3.2.2 Error analysis ...... 42 4.3.5 Comparison sentiment analysis and stance detection ...... 43 5. CONCLUSION ...... 46 6. LIMITATIONS AND FURTHER RESEARCH ...... 49 APPENDIX 1 ...... 54 APPENDIX 2 ...... 57

Page 7 of 76

List of tables and figures Table 1 - Sentiment per topic (manual annotation) ...... 28 Table 2 - Manual annotation sentiment, stance and irony ...... 30 Table 3 - Presence of irony per topic ...... 30 Table 4 - Sentiment per topic (machine learning approach) ...... 31 Table 5 - Machine learning annotation sentiment, stance and irony ...... 33 Table 6 - Tenfold cross-validation scheme for sentiment analysis ...... 34 Table 7 - Comparison manual and automatic sentiment analysis + irony presence ...... 35 Table 8 - Precision, recall, F1-score and accuracy of sentiment labels ...... 35 Table 9 - Tenfold cross-validation scheme for stance detection ...... 40 Table 10 - Comparison manual and automatic stance detection + irony presence ...... 40 Table 11 - Precision, recall, F1-score and accuracy of stance labels ...... 41 Table 12 - Comparison results sentiment analysis and stance detection ...... 44 Table 13 - Precision, recall, F1-score and accuracy of sentiment and stance labels ...... 44 Table 14 - Accordance of manually assigned sentiment labels with their stance labels ...... 45 Table 15 - Accordance of automatically assigned sentiment labels with their stance labels ...... 45

Figure 1 - Manual annotation sentiment ...... 26 Figure 2 - Topics in tweets ...... 27 Figure 3 - Manual annotation stance ...... 29 Figure 4 - Machine learning annotation sentiment ...... 31 Figure 5 - Machine learning annotation stance ...... 32

Page 8 of 76

1. INTRODUCTION

With the rise of Web 2.0, microblogging websites have increasingly become a valuable platform for people to express their opinion and sentiment on a certain topic. Blogs, forums and social media platforms allow users to easily add blogposts, reviews, reactions and ratings to share their point of view on the internet. Since online opinionated texts offer a wide range of easy accessible data, businesses, services and political parties are not bound to conduct surveys or carry out opinion polls (Lui, 2012) to gather the public’s sentiment anymore. Nowadays, organisations have enough feedback at their disposal to examine how people’s views differ on a certain product, service or policy.

The system used to extract and analyse the public’s opinion is called sentiment analysis (SA). Opinion mining or sentiment analysis is defined as “the computational study of opinions, sentiments and emotions expressed in text” (Kumar & Sebastian, 2012, p.2). A product or service can be adapted based on the outcome of the sentiment analysis, regarding whether the general opinion of a certain aspect is neutral, positive or negative. This is beneficial for the quality of the product or service and therefore the consumer’s satisfaction. Sentiment analysis can be applied for various purposes: to detect trends on the social media of a business, to discover what the underlying reason for the success or failure of a certain product is, or to predict the outcome of a referendum or elections. Whereas SA helps to determine the speaker’s sentiment in a piece of text, stance detection (SD) aims at extracting the author’s opinion towards a certain target or entity. SD thus encloses “the task of automatically determining from text whether the author of the text is in favour or, against, or neutral towards a proposition or target” (Mohammad & Kiritchenko et al., 2016, p.31). It can be used to reveal weak spots or positive aspects of a target. A target may be a product, an aspect of a certain service, a person, a political point of view, an organisation, a policy, et cetera.

Of all social platforms that offer a wide range of accessible opinions and sentiment, Twitter is notably interesting for both researchers and marketers who can use tweets to easily collect the opinion of a large audience. This social network is particularly instrumental for politicians and political parties to evaluate their position within the political landscape. Tumasjan et al. (2010) indicate that Twitter is used frequently to speculate about politics, since messages mentioning a party tend to reflect the outcome of, for instance, an election. Consequently, they conclude that political tweets plausibly give an indication of how the current offline political landscape is divided.

Page 9 of 76

Besides the advantages of Twitter as an easy accessible source of user-generated data, the platform has its own specific characteristics. For the very reason that the data is user-generated, Twitter messages or tweets will often contain language phenomena that are typical for online messaging (e.g. flooding, abbreviations, emoji’s). Furthermore, as users are limited to write a message in under 280 characters they are obliged to phrase their messages creatively. As a result, irony, sarcasm, metaphors or other figurative use of language frequently appear in tweets, which is challenging to detect due to its creative character. Regardless of the challenging nature of figurative language such as irony, Reyes et al. (2012) constructed an irony detection model specifically for short online texts. This model provided valuable insights into figurative language use on Twitter and tasks such as sentiment analysis. Moreover, Van Hee (2017) explored the automatic detection of irony on social media and found that a machine learning approach can lead to good performance in combination with a varied set of information sources. In the future, this irony detection system could, however, be further optimised. In the framework of SemEval-2015 Ghosh et al. (2015) explored the determination of sentiment in tweets containing irony, sarcasm or metaphors. For this purpose, they measured the polarity of tweets that use creative and figurative language. Their system appeared to be useful for further research.

In this study, we want to make a contribution to the existing findings on automatic sentiment analysis on Twitter. It will be further explored how well a machine learning system performs for sentiment analysis and stance detection on a Twitter corpus of political tweets. Additionally, we will attempt to provide more insight into the impact of ironic language in tweets on the predictability of sentiment and stance labels. The manual annotation of sentiment in a Twitter corpus containing 482 tweets with the hashtag Brexit (#Brexit) will be compared with the predicted sentiment labels of a supervised machine learning system for sentiment analysis. Apart from sentiment labels, the tweets will also be provided with a stance label to explore whether a machine learning system is able to detect (implicit) stance. On the basis of the collected and analysed data and experimental results, we will try to provide an answer to following questions: Can we automatically predict sentiment with the help of sentiment analysis? Which impact does irony have on the predictability of sentiment? Can a machine learning system detect implicitly expressed stance? In further passages of this study, we will give an in-depth overview of the current state-of-the-art of sentiment analysis, including ABSA, stance detection and irony detection.

Page 10 of 76

2. LITERATURE STUDY

2.1 Sentiment analysis

People’s opinions on and reviews about a certain event, product or service can have a big influence on future customers. The latter often turn to blogs, review platforms and social media to gather information and advice to form their own opinions. Therefore, sentiment analysis (SA) is an important field of study. By performing SA, it can be determined whether a piece of text is neutral, positive or negative. To gain insight into for instance sales figures or election results it is beneficial for companies and organisations to keep track of their online reputation. Preferably, SA is performed by an automatic system, since manual annotation would appear to be a time-consuming, intensive task.

The term sentiment analysis appeared for the first time in 2003 in a study by Nasukawa and Yi (2003). In the same year, Dave et al. (2003) mentioned opinion mining in their work. Nowadays, both terms are used interchangeably to denote the same field of study. More specifically, SA is considered to be “the field of study that analyses people’s opinions, sentiments, evaluations, appraisals, attitudes and emoticons towards entities such as products, services, organizations, individuals, issues, events, topics, and their attributes” (Liu, 2012, p.7). As from the year 2003, this field of study has been well-studied and is still constantly developing.

Nowadays, a range of applications are available to systematically organize opinions, reviews, ideas on entities and events as well as products. Some applications, for example, arrange reviews and ratings according to sentiment, and other recommendation systems are able to only recommend products or advertisements with positive sentiment (Pang et al, 2008, p.8). Additionally, SA may also be valuable for governments to detect possible opposed or negative voices. Especially during times of elections, the latest technologies can assist in keeping an overview of the various points of view on politicians, parties, bills, et cetera. Besides numerous possibilities for commercial applications, several problems still rise. These need solutions and therefore make SA a relevant field of study to explore thoroughly and in further detail.

In this study, an in-depth overview will be given of the SA-related terminology in section 2.1.1 and the two approaches to SA in chapter 2.2. In chapter 2.3, Aspect Based Sentiment Analysis (ABSA) will be explained, whereas in chapter 2.4, the particular nature of SA for political tweets will be discussed. Lastly, the aspects, difficulties and challenges of irony detection will be discovered in chapter 2.5.

Page 11 of 76

2.1.1 Terminology

As SA has increasingly been discussed over the last decennia, it is important to keep a clear overview of which terms are being used in the field of study of Natural Language Processing (NLP). In literature the terms “opinion mining, review mining and appraisal attraction” (Kumar & Sebastian, 2012, p.2) are used interchangeably with the concept of SA. According to Liu (2012), the terms mentioned all differ from each other on a certain level. They can, however, all be classified under the umbrella term of SA.

In the field of SA, it is essential to make a distinction between subjectivity and subjectivity analysis on the one hand, and sentiment and sentiment analysis on the other hand. Pang et al. (2008) believe subjectivity comprises everything that contains a personal opinion, such as evaluation, emotions and speculations. Subjectivity analysis is used to distinguish facts (objective sentences) from opinions and sentiment (subjective sentences). Furthermore, subjectivity and sentiment do not have the exact same meaning. Sentiment implies the expression of an opinion, which often reveals the speaker’s attitude towards a certain topic. Not only subjective sentences contain sentiment. The following sentence “The battery of my Bluetooth speaker only lasts for 8 hours”, is for example an objective sentence with negative sentiment. The usage only reveals a sense of disappointment by the speaker. By means of automatic SA, polarity, opinions, emotions and other subjective information can be automatically detected and analysed (Desmet et al., 2014). It is often used to determine which sentiment (positive, negative, neutral) belongs to a piece of text.

Moreover, two kinds of opinions can be distinguished, namely regular opinions and comparative opinions (Liu, p.12). A regular opinion expresses sentiment on a specific feature or aspect of a product, for example “The battery of my Bluetooth speaker lasts for a very long time.”. A comparative opinion, however, compares different products or services based on certain aspects they (don’t) have in common, for instance “My JBL speaker has a further Bluetooth ranger than my old Philips speaker.”. Specific words that regularly express the same kind of sentiment can be used to easily recognize opinions. These words are defined as sentiment words or opinion words (Liu, 2012, p.12). Some opinion words are generally positively used such as ‘beautiful, nice, lovely’ and others usually have a negative connotation, such as ‘ugly, stupid, boring’. In addition, there are also sayings or proverbs who are systematically classified under a certain polarity label. ‘To have a finger in every pie’ for example, is generally used negatively.

Page 12 of 76

There are different ways to categorize sentiment. For one, SA often makes use of a binary classification and only assigns the labels ‘positive’ and ‘negative’. This is also defined as sentiment polarity classification (Pang et al, 2008, p17). In some cases, the degree of positivity and negativity can be more thorough and complex. Therefore, Boertjes (2011) also defines a non-binary classification system. When following this system, sentiment can then for example vary on a scale going from ‘extremely dissatisfied’ to ‘extremely satisfied’. This is also called multi-class classification (Kumar & Sebastian, 2012, p.4), which implies SA with more than two sentiment categories. In the latter, the star system is also often applied, in which one star expresses dissatisfaction and five stars satisfaction. Cambria et al. (2013, p.16) also distinguish two common sentiment analysis tasks: polarity classification (supra) and agreement detection. On the one hand, they add other examples to the binary polarity classification, such as ‘thumbs up’ and ‘thumbs down’ or ‘like’ and ‘dislike’. On the other hand, they introduce agreement detection as another example of a binary classification system. This task helps to determine whether two texts hold the same opinion and should receive the same or opposite sentiment labels.

2.2 Approaches to SA

Sentiment analysis is an interdisciplinary field of study which focusses on web mining (extracting and analysing information on the web) as well as natural language processing (NLP or computational linguistics). Kumar and Sebastian (2012) describe four levels at which sentiment can be determined: feature level, word level, sentence level and document level. Feature based SA on entity and aspect level (Liu, 2012, p.11) focusses on the opinion on different aspects or features instead of the sentiment of entire paragraphs, sentences or phrases. The sentence “My new smartphone has a great camera, but the sound quality is horrible.”, for instance, cannot be labelled as entirely positive or negative because it reports on two features of the same product. The evaluation of the sound quality of this new smartphone is negative, but the camera is evaluated positively. Feature based SA is often seen as the most challenging level of SA since two or more aspects can be covered within one sentence. On word level, the polarity label is predicted for each separate word as each word is then considered as a separate unit, which can hold different sentiment (Kolkur et al, 2015, p.2). According to Kumar and Sebastian (2012), adjectives are the parts of speech that contain the most explicit sentiment. On sentence level, a polarity label is given to each separate sentence. A neutral label usually equals the fact that the sentence does not hold a personal opinion or that the sentence is both positive

Page 13 of 76

as negative which neutralizes the sentiment. For SA on document level, entire documents (e.g., product reviews) are labelled positively or negatively as a whole. This level of SA is not applicable when the document covers multiple products or aspects.

Systems for automatic SA rely on one of two valid approaches for the prediction of a sentiment label: either a lexicon-based approach or machine learning approach is applied.

2.2.1 Lexicon-based approach to SA

The lexicon-based approach to SA generally makes use of a “dictionary of opinion words to identify and determine sentiment orientation (positive, negative or neutral)” (Zhang et al, 2011, p.2). These opinion words are words that each have their own sentiment label, based on the positive or negative connotation they commonly have. The dictionary, which is also called a sentiment lexicon or opinion lexicon (Liu, 2012, p.12), is usually completed with synonyms and antonyms of the opinion words.

However, Zhang et al. (2011) indicate that the lexicon-based approach would lead to a low recall problem. Due to the specific nature of online language, it is impossible to add every existing opinion word to the sentiment lexicon. In other words, some words will explicitly express sentiment, but will not be picked up using the lexicon-based approach. For example, in the following sentence ‘I haaaate going the dentist.’, the negative expression ‘haaaate’ will most likely not be detected. These expressions change continuously and, therefore, adding them to the opinion lexicon would appear to be an endless and time-consuming task.

Moreover, Kumar et al. (2012) explain that the lexicon-based approach does not work on domain specific level. More specifically, this means that a certain word can have a positive or negative label depending on the context or domain. The word ‘unpredictable’, for instance, can be positive when it is used in a film review, but negative when it is used to evaluate the steering behaviour of a brand new car. So far, lexicon-based methods are not yet able to interpret the sentiment or meaning of a word depending on the situation. According to Khan et al. (2015), it would be too labour intensive and time-consuming to manually label each opinion word per specific context (e.g. film or car review).

Page 14 of 76

2.2.2 Supervised machine learning approach to SA

Machine learning allows computers to automatically carry out analysis tasks by means of self- learning algorithms (such as Naïve Bayes and Support Vector Machines). As Zhang et al. (2011) explain, these algorithms or sentiment classifiers are trained using features such as unigrams or bigrams (sequences of one or two words, respectively). Machine learning systems can then deduce rules or patterns based on manually annotated data to predict polarity labels of unseen data. The model is thus built based on labelled pieces of texts (sentences, paragraphs, documents), called training data.

During this training phase, the data is processed in the form of structured information or “features” extracted from the text. On the basis of all this information, called “feature vectors”, self-learning systems can then predict which combination of information results in which polarity label. This approach is therefore referred to as “supervised learning, because the classifier is given direction in terms of which are good or bad examples of the class.” (Taboada, 2016, p.6).

To set up a statistic machine learning system, data has to be pre-processed in advance on different levels. Large pieces of text are split up in separate sentences (sentence splitting) and these are then again broken down into words or tokens (tokenization). Another pre-processing step can be Part-of-Speech tagging (PoS-tagging) which attributes the morphosyntactic category to the corresponding word (adjectives, adverbs, nouns, verbs). To analyse various word forms as a single item, lemmatisation can be used to group together inflected forms of a word. The infinitives of verbs will then, for example, be recognized in their conjugated forms.

After the pre-processing phase, features (namely lexical features and syntactic features) can be extracted from a certain text. This process is called feature extraction. Lexical features, on the one hand, such as tokens or lemmas (E.g., the word ‘worse’ is derived from ‘bad’), can offer insight in which words occur in a text. Syntactic features, on the other hand, give grammatical information about the text, such as the syntactic categories of words (e.g. bad is an adjective).

2.2.3 Classification: Support Vector Machines

Shoeb and Ahmed (2017) state that data classification aims to classify data into categories in the most efficient and productive way. The goal is then to predict the correct category for unseen data. One regularly used supervised machine learning algorithm is a Support Vector Machine (SVM) which can be applied for both classification and regression. According to Pang and Lee

Page 15 of 76

(2002), SVMs have been rather effective in comparison to other traditional text categorization algorithms, showing better results than the Naïve Bayes method for example. By means of a number of features, the classifier categorizes objects into one of two classes, represented by a vector. In the context of Twitter, each feature could represent a single word found in a tweet. Another application of SVMs could be to classify a set of documents into two sentiment groups: positive or negative documents. The classification process would then be based on other documents which have already received a positive or negative label. The goals of a SVM is to train a system that classifies new unseen objects into a certain category. Apart from the application of SVMs in machine learning approaches to sentiment analysis, an SVM is also used for text classification tasks such as detecting spam. For this, the classification would rely on a large corpus of e-mails or other documents which have already manually been marked as spam or non-spam. Moreover, SVMs are used for the recognition of images, where the algorithm attempts to recognize aspects or colours of an image.

More formally12, SVMs attempt to find a hyperplane that can divide a dataset into two classes. A hyperplane is a line that linearly separates and classifies a set of data. The data points (or feature vectors) that are the nearest to this hyperplane are called support vectors. These points

1 http://blog.aylien.com/support-vector-machines-for-dummies-a-simple 2 https://www.quantstart.com/articles/Support-Vector-Machines-A-Guide-for-Beginners Page 16 of 76

are the hardest to classify. However, the further away data points are positioned from the hyperplane, the more reliable the classification of these points are.

2.3 ABSA: Aspect Based Sentiment Analysis

Whereas regular SA determines whether a piece of text is positive, negative or neutral, ABSA or Aspect Based Sentiment Analysis tries to trace back the target of an opinion. According to Declercq et al. (2017), this comes down to a very fine-grained approach to SA. ABSA systems attempt to detect all expressions of sentiment within a piece of text. On sentence level, this means certain entities can be identified and then be paired up with the corresponding attribute. In a review on the newest Iphone, the entity types ‘battery’ or ‘camera’ can for example be linked to the right attribute label such as ‘price’ or ‘quality’ (Pontiki & Galanis et al., 2016, p.20). The system then attempts to detect the main attributes (features) of the entity to make an estimate of the average sentiment of a certain text per aspect. In other words, an overview is given of the positivity or negativity of opinions for each single aspect mentioned (Pavlopoulos, 2014, p.2).

Pavlopoulus (2014) distinguishes three subtasks of Aspect Based Sentiment Analysis: aspect term extraction, aspect term aggregation and aspect term polarity estimation. First of all, aspect term extraction helps to detect words or phrases that indicate a certain aspect of the entity that is being discussed (e.g. battery, camera). After this first step, the extracted words or phrases are called aspect terms. Secondly, the next subtask is described as aspect term aggregation, which means the system clusters aspect terms which are quite similar (such as ‘camera’ and ‘video camera’). Lastly, during the aspect term polarity estimation, the system evaluates all aspect terms and estimates the average sentiment of every aspect term or cluster of aspect terms.

2.4 Sentiment analysis for political tweets

With the help of SA, companies and services can gain insight in the perception and reception of their product or service. It is a source of valuable customer feedback that can help companies to fine-tune or make adaptions to their product by taking the results of the SA into account. Moreover, social organizations might be interested to know people’s opinions on current controversial or social debates. Intuitively, the domain where opinions differ regularly is politics. As reported by Pak and Paroubek (2010), it may be profitable for political parties to gain a perception of whether people support their party programme or not. Whether it involves a new policy or upcoming elections, social media such as Facebook or Twitter are constantly

Page 17 of 76

booming with posts containing political views. Tumasjan et al. (2010) point out that, specifically in the weeks leading up to a political event (such as elections), political themes are clearly on many users’ minds. Even politicians themselves attempt to reach the electorate and mobilize possible supporters by communicating via Twitter. In chapter 2.4.1 we will give an overview of the particular characteristics of Twitter and why this could be an interesting social platform to perform SA on.

2.4.1 Twitter

Social networks offer a wide range of accessible opinions and sentiment. Khan et al. (2015) indicate that Twitter3 in particular has a great amount of data at their disposal. On this social platform, users can share their opinions with the rest of the tweeting community in the form of tweets (posts on Twitter) consisting of maximum 280 characters. Each day an estimated 60 million tweets are sent into the world. Twitter is therefore an easy-accessible and informative platform for both researchers and marketers to collect sentiment of a large audience. According to Liu (2002), tweets are easier to analyse thanks to their length, when compared to reviews for example, because tweeters attempt to come to the point in a concise answer. Pak and Paroubek (2010) also indicate Twitter users come from different social groups with varying interests and backgrounds. Even though American users are prevailing, Kulshrestha et al. (2012) indicate that the twitter audience is represented by users from all over the world (231 countries). Therefore, it is possible to collect data and build a corpus in different languages.

Besides all the advantages of using Twitter data, the special characteristics of Twitter can cause some specific problems. The language use is often informal and can contain typical abbreviations or words which are only used in online messaging (eg., ‘lmao’, ‘lovvve’). In that regard, an automatic SA system probably will not recognize any opinion words or positive sentiment in a sentence such as “I loooove McDonald’s new hamburger!”, because ‘loooove’ is not picked up as the word ‘love’. Furthermore, tweets with the hashtag ‘not’ (#not) usually hold an ironic or sarcastic message, as can be seen in the following sentence “I love it when my train is delayed #not”. The hashtag makes the previous statement invalid, but when it is not picked up by the SA system, the tweet will receive a positive label instead of the correct negative label. From time to time, Twitter users tend to self-annotate their own usage of irony,

3 www.Twitter.com

Page 18 of 76

as is confirmed by Reyes et al. (2012). They then add the hashtag ‘irony’ (#irony) to point out their ironic use of language.

In addition, emoticons or emoji’s that express a certain emotion can also give an entirely different meaning to a tweet, such as in the sentence:

“Luckily, my train has arrived right on time as usual ☹”.

On first glance, the tweet seems to contain positive sentiment, however, the sad and unsatisfied smiley indicates the statement is written in a negative tone. Moreover, retweets (the sharing of someone’s tweet on one’s own profile), replies to other tweets (marked with an ‘@’-sign followed by a username) and adding pictures or links can pose a challenge. Retweets and replies can be misinterpreted when the tweets are read separately, because then there is a risk that valuable context is lost.

Even though the phenomena mentioned above are challenging, there have already been several attempts to tackle such problems. A study by Van Hee et al. (2014), for example, has shown that feature extraction (as discussed in chapter 2.2.2) using a machine learning approach performs better than SA systems using lexicon-based approaches. The results reveal that after the extraction of specific Twitter features, the SA system performed very well (with an F1-score of 86,28) on a Twitter corpus. After programming rules for flooding, for example, the word ‘loooove’ could be picked up in the previously mentioned sentence “I loooove McDonald’s new hamburger!”.

2.5 Stance detection

Whereas SA aims at detecting the sentiment of an opinion in a piece of text, stance detection (SD) is used to pick up whether someone is for or against the subject (target) being debated. This target may be an organisation, product, service, person or policy. To decide whether an author is for or against an issue, it is important to follow the reasoning rather closely. Mandya et al. (2016) indicate the problem that posts containing rebuttal arguments are not clear enough to be classified as ‘for’ or ‘against’ the main issue being debated. Posts are most often independent or non-dialogic and thus all features for classification have to be derived from the post itself. To facilitate stance classification, Mandya et al. (2016) state that topic-stance features or topic terms can be automatically extracted. For the topic ‘gun control’ the terms would include for instance ‘firearm’, ‘rifle’ or ‘license’. Each topic term is then associated with the author’s stance towards that topic. In the sentence “Firearms are nothing but trouble.”, for

Page 19 of 76

example, ‘firearms’ would be associated with the topic ‘gun control’. Then, the stance towards ‘gun control’ would be negative.

The author’s stance or outlook towards the target is in favour if we can deduce from a piece of text that he or she supports the target. Mohammad et al. (2016) observe different expressions of favourability, such as simply supporting the target, repeating the positive stance of someone else or opposing someone who is opposed to the target. Opposite expressions generally result in negative stance and disapproval of the target. If there is no evidence found whether someone is for or against a certain target, this does not necessarily mean the author is neutral. It can also be the case that the stance from this piece of text simply could not be detected.

2.6 Irony detection

Barbieri and Saggion (2014) argue that computational creativity or the creative use of language has been one of the most challenging topics of Artificial Intelligence (AI) and NLP nowadays. Even though irony has received little attention in computational linguistics, it is considered to be a vital and relevant aspect in fields of study such as SA. Therefore, irony detection has become an increasingly discussed task. Irony detection is the task of automatically classifying pieces of text into the classes ‘ironic’ or ‘non-ironic’. According to Reyes et al. (2012) the automatic detection of irony could be relevant in various research areas, such as electronic commerce, product tracking and online marketing. Van Hee (2017) completes the list with other fields of study such as language psychology, sociolinguistics and cyberbullying detection.

For SA, the presence of irony can affect the outcome drastically. Classic sentiment analysis tools are generally not sensitive to the use of irony. They will therefore perform less accurate when applied to ironic utterances. To successfully detect irony, it is important to firstly define the concept and possible subcategories of irony. Only then, the specific forms and characteristics of irony that are susceptible to computational analysis can be identified. For that reason, we will elaborate on what is understood as ironic in chapter 2.5.1. Then, secondly, the difficulties and challenges of irony detection will be covered in chapter 2.5.2.

2.6.1 What is irony?

Irony is a creative use of language which is omnipresent in human interaction. Van Hee et al. (2015) define irony as “an evaluative expression whose polarity (i.e., positive, negative) is changed between the literal and the intended evaluation, resulting in an incongruence between the literal evaluation and its content.”. Since the language use is figurative, ironic pieces of text

Page 20 of 76

should not be interpreted literally. A more complex approach is thus required to also detect the correct context or make associations with common knowledge.

As irony is a playful way to express oneself it comes in many different varieties. Kreuz and Roberts (1993) already made a distinction between three varieties of irony: Socratic or dramatic irony, irony of fate and verbal irony. Firstly, Socratic irony or dramatic irony describes the tension between the hearer’s knowledge and what the hearer pretends to know. Here, ignorance is sometimes feigned in order to reveal the errors in someone’s viewpoint or argument. An example of Socratic irony could be a situation in which a parent is aware that his/her child has come home after curfew. Instead of confronting the child with the facts, the parent will ask a series of seemingly innocent questions that will eventually result in a confession. Secondly, irony of fate, is explained as an incongruence between two situations. It is also referred to as situational irony as the situations that are being discussed fail to meet some expectations. An example of this could be a no-dog sign in an animal shelter. Lastly, if someone uses verbal irony, the speaker intentionally implies the opposite of what he or she believes. Reyes et al. (2012) only make a distinction between the two broad categories of verbal irony and situation irony. Karoui et al. (2017), however, retain eight different categories. The first category covers analogies, metaphors and comparisons which aligns two things with contrasting or different concepts or domains. Secondly, the category of hyperboles and exaggeration enlarges a situation to lay emphasis on a point. Thirdly, Karoui et al. (2017) also distinguish euphemisms, which are phrases or words that help to soften reality. The fourth category contains rhetorical questions and the fifth context shifts. The latter covers an abrupt change of the topic or tone of the conversation. False assertions, the sixth category, are declarations that conflict with common sense. Then, whereas a false assertion is implicit, an oxymoron or paradox (the seventh category) explicitly expresses the contradiction. In the last category, all other expressions containing situational irony are covered.

Furthermore, other figurative uses of language such as sarcasm need to be distinguished. Even though there is an overlap in usage, they differ in “usage, tone and obviousness” (Reyes et al., p.260, 2013). Sarcasm, for example, has a higher level of aggressiveness to it. According to Van Hee (2017), it is often used with the intention to hurt a target directly and intentionally. In comparison with irony, the intensity is greater due to the combination of ridicule and negativity. Irony is considered to be more so subtle and therefore more sophisticated.

2.6.2 Difficulties and challenges

Page 21 of 76

Since irony is already such a complex concept on its own, Reyes et al. (2013) indicate that it would be unrealistic to set hopes on a single computational silver bullet for irony. More specific, the lack of facial expression and vocal intonation in ironic tweets makes it a challenging task to automatically detect irony. Considering that irony touches on almost every aspect of language, a multidimensional approach for detecting irony in Twitter is desirable.

Furthermore, as already mentioned earlier (cf. chapter 2.4.1), micro-bloggers in Twitter regularly add the hashtag irony (#irony) to indicate their use of it. Some speakers are aware their use of language is ironic, however, Wang (2013) found Twitter users make no distinction between irony and sarcasm. Reyes et al. (2013) affirm that users who add #irony to their tweet merely have a diffuse and vague idea of what it is understood as an ironic text. Furthermore, Van Hee (2017) found that one in five tweets carrying the hashtag were not ironic. As a result, it can be concluded that manual annotations are of help training automatic irony detection systems.

Current machine learning approaches to irony detection, for example the system of Van Hee (2017), show that ironic tweets that hold a polarity contrast are more likely to be identified than other types of irony. Yet, what remains a challenge is the detection of implicit sentiment (E.g., situations that have a specific positive or negative connotation, such as ‘going to the dentist’ or ‘hearing your train is delayed’). Experiments in this research, however, reveal “that analysing tweets about a concept or situation appears to be a viable method to determine implicit sentiment related to that concept or situation.” (Van Hee, p. 124, 2017).

3. RESEARCH DESIGN

In this study, we attempt to explore how well a machine learning system performs for SA on Twitter. The following chapters will offer insight into the data collection and annotation of the corpus. We will then discuss and analyse the results in chapter 4.

3.1 Research questions and hypotheses

This research aims at answering the question of how well a machine learning approach to SA performs on a Twitter corpus of political tweets. Additionally, we will attempt to provide more insight into whether the presence of ironic language in tweets influences the predictability of tweets. To formulate a well-founded answer to these questions, we will firstly try to answer following sub questions: Can we automatically predict sentiment with the help of sentiment

Page 22 of 76

analysis? Which impact does irony have on the predictability of sentiment and stance? Can a machine learning system detect implicitly expressed stance?

Based on previous studies (cf. chapter 3) we can formulate the following hypotheses: We expect our machine learning system to deliver reliable results for SA on a political Twitter corpus. Presumably, they will not yet be able to detect and interpret ironic language use correctly in most cases.

3.2 Methodology

To answer our research question(s), a Twitter corpus of 482 tweets was built and the tweets were manually annotated with the labels positive, negative or neutral. The same tweets were then annotated by a machine learning system for SA. The results of both the manual labelling and automatic labelling will be compared and analysed in the results section (cf. chapters 4.1, 4.2, 4.3). In this section, we will discuss in detail how our data were collected and annotated.

3.2.1 Data collection

As one of the first steps in the data collection process, we decided on which topic we would collect tweets. We chose to gather English tweets with the hashtag Brexit (#Brexit) of the 24th of June 2016. Since the Brexit referendum was held the day before, on the 23rd of June, we believed the day after the outcome would provide us with divided opinions. The referendum decided on whether the UK would leave the European Union or not. 51,9% of the voters appeared to be in favour of leaving the EU and won, whereas an almost equally large group (48,1%) was against the Brexit and lost. Ever since the announcement of the Brexit referendum, it has been a highly discussed topic within the European political landscape.

We included every tweet of the 24th of June 2016 with #Brexit in chronological order, but decided to ignore tweets that appeared twice or more. Furthermore, we decided to discard tweets with double opinions. An example of such a tweet would be “Scary stuff! Still #brexit is best!!”. In the first part of the tweet, ‘scary stuff’ could be interpreted as negative, however the second sentence clearly contains positive sentiment. The same conclusion can be drawn from the following tweet “Britons will enjoy their victory today. But tomorrow, the hangover will be fierce #Brexit #UKReferendum”. The first sentence is positive, whereas the second part warns

Page 23 of 76

for probable negative consequences of the Brexit. Since these contradictory sentiments could easily confuse the classifier of the machine learning system, we decided to exclude tweets with double opinions from the corpus. Our aim was to collect circa 500 tweets and originally our corpus consisted of 512 tweets with #Brexit. After removing tweets that appeared twice or more, or tweets with double opinions, we eventually retained 482 tweets.

3.1.1 Annotation

For the manual annotation of the Twitter corpus we focussed on four categories: sentiment, topic/aspect, stance and irony. The purpose of these four categories was to gain insight into the nature of the tweet, considering various approaches and classifications. In the first category, sentiment, we looked at the tweet as a whole and decided on whether the content was positive, negative or neutral, regardless what the actual subject of the tweet was. If the opinion expressed in the tweet was indefinable, we attached a neutral label to the piece of text.

Furthermore, we manually classified all tweets into various topics or aspects, the second category. Eventually we narrowed 72 topics (cf. Appendix 1) down to eight categories of topics: Brexit, celebrities/politicians, economy, EU, Scottish referendum, Trump, USA and other. We decided to focus on a small number of topics that serve as an umbrella term for several aspects, to keep an convenient overview of the themes discussed. Under the topic ‘celebrities/politicians’ for example, we classified all tweets that mentioned a specific name of a public or political figure (e.g., Lindsey Lohan, Boris Johnson, Nigel Farrage). The topic ‘other’ served as an undefined category to classify numerous dissimilar subjects (e.g., personal information, jokes).

The annotation of the third category concentrated on the stance expressed by the author towards the Brexit. As already mentioned in chapter 2.4.2, SA aims at detecting the sentiment of an opinion in a piece of text, whereas stance detection is used to pick up whether someone is for or against the subject (in this case: the Brexit) being debated.

Lastly, to explore which impact irony has on the predictability of sentiment, we added a fourth category in the annotation process. Based on the eight different categories of irony Karoui et al. (2017) distinguished (analogies, metaphors and comparisons; hyperboles and exaggerations;

Page 24 of 76

euphemisms; rhetorical questions; context shifts; false assertions; oxymorons or paradoxes; situational irony), we marked tweets with ironic messages.

3.1.2 Experimental approach

For the experimental component of this thesis, a machine learning model was created to label both the sentiment and stance of each tweet automatically. In total, two experiments were thus carried out: one to predict sentiment and one to predict stance. Firstly, the Twitter corpus was split up in, on the one hand, the separate tweets and, on the other hand, their manually annotated stance and sentiment label. The tweets were separated from the labels by tab.

Secondly, all tweets were tokenised with LeTs, a multilingual linguistic pre-processing toolkit developed by Van de Kauter et al (2013). This toolkit includes Part-of-Speech taggers, lemmatizers and named entity recognizers. For this study, all tokens (words, punctuation, numbers, symbols) were separated from each other with the pre-processing tool.

Thirdly, several features were extracted: unigrams, bigrams, trigrams, character n-gram features with a range of 3-4 tokens, the number of flooded tokens, the number of flooded punctuation tokens, the number of capitalized tokens and a sentiment lexicon look-up. The lexicon used is called AFINN4, which is a list of English words for valence with an integer between minus five (negative) and plus five (positive). With this sentiment lexicon, the number of positive, negative and neutral tokens as well as the overall value of one tweet can be extracted.

Lastly, the Support Vector Machine was run and tested on a tenfold cross-validation scheme, which means that 90% of the Twitter corpus was used as a train fold and 10% as a test fold. This process was repeated ten times: each time with another 10% as test corpus up until the moment that the entire corpus has served as test fold. The machine learning experiments were carried out with LIBSVM5, which is an integrated software for support vector classification, regression and distribution estimation. For this study, the linear kernel was used.

4 http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6010 5 https://www.csie.ntu.edu.tw/~cjlin/libsvm/

Page 25 of 76

4. RESULTS

In this chapter, we will discuss both the results of the SA in the manually annotated Twitter corpus (cf. chapter 4.1) and the outcome of the automatically annotated Twitter corpus (cf. chapter 4.2). Then, the two result sections will be compared and thoroughly analysed (cf. chapter 4.3).

4.1 Results manual annotation

4.1.1 Sentiment and topics

The Twitter corpus consisted of 482 tweets with the hashtag Brexit (#Brexit). The manual annotation resulted in 84 tweets with a positive sentiment label (17%), 254 tweets with a negative sentiment label (53%) and 144 (30%) with a neutral sentiment label.

FIGURE 1 - MANUAL ANNOTATION SENTIMENT (N=482)

Positive (84) Negative (254) Neutral (144)

17% 30%

53%

Figure 1 - Manual annotation sentiment

As already mentioned (cf. chapter 3.2.2), we made a distinction between 8 categories of topics (cf. Figure 2). The greater part of the tweets (57%) addressed the topic of the Brexit itself. This could be explained by the specific hashtag in every tweet (#Brexit). Then, several users also mentioned the consequences for the economy after the Brexit outcome (7%). To nearly the same extent, users tweeted about celebrities or politicians, using the hashtag Brexit (6%). Only a negligible number of tweets (2 out of 482 tweets) considered a Scottish referendum, however, the USA (4%) and Trump (4%) were discussed more frequently. Both the USA and Trump were equally discussed within the context of the Brexit, mostly as a political point of

Page 26 of 76

comparison. American elections were then to be held in November 2017, which was a highly debated topic at that time as well. An example of this would be: “Britons voted to strengthen their borders. Will you do the same in November? #Brexit”. Besides discussing the referendum or the outcome of the referendum, many users gave personal information or any other remarks in their tweets. This resulted in a fair number of tweets commenting upon other topics or considering personal information (19%). Following tweet is an example of such a tweet: “A very good cereal served at my amazing property in Turnberry! #Brexit of Champions, just like me! Enjoy!”.

FIGURE 2 - TOPICS IN TWEETS (N= 482) Brexit (273) Celebrities/politicians (29) Economy (34) EU (13) Scottish referendum (2) Trump (21) USA (21) Other (89)

19%

4%

4% 0% 3% 57%

7%

6%

Figure 2 - Topics in tweets

By comparing the previous results (cf. Figure 1 and Figure 2) and joining them together (cf. Table 1), it is noticeable that, with the exception of the category ‘other’, negative labels are predominant in nearly every topic. Especially when discussing the Brexit and its outcome itself, the tweets are notably negative. In the category ‘other’, there are 34 negative and 37 neutral tweets. An explanation for this could be the presence of personal information, which tends to be either critical or merely narrative. An example of the latter could be the following tweet: “I know it is not good for me, but, on days when Britain chooses to #Brexit, I like to drink a Coke and eat a cookie. #EURefResults”. In this tweet, the actual subject is not the Brexit but the author’s diet. The sentiment is negative even though the tweet considers ‘Coke and cookies’. Besides personal information, jokes are also omnipresent in the category ‘other’, as can be observed in the following tweet: “Is this going to affect my chances of getting into Hogwarts?

Page 27 of 76

#Brexit”. Here, the sentiment is neutral, because the author does not express itself negatively towards the Brexit, but simply jokes around.

TOPIC NEGATIVE NEUTRAL POSITIVE Brexit (273) 153 (56%) 74 (27%) 46 (17%) Other (89) 34 (38%) 37 (42%) 18 (20%) Economy (34) 17 (50%) 11 (32%) 6 (18%) Celebrities (29) 19 (66%) 8 (28%) 1 (34%) USA (21) 7 (33%) 7 (33%) 7 (33%) Trump (21) 17 (81%) 1 (5%) 3 (14%) EU (13) 7 (54%) 4 (31%) 2 (15%) Scottish referendum 0 (0%) 2 (100%) 0 (0%) (2) TOTAL 254 144 84 Table 1 - Sentiment per topic (manual annotation) 4.1.2 Stance and irony To explore whether the author is for or against the subject of the Brexit, we also labelled the stance of a tweet as positive, negative or neutral. In Figure 3, an overview is given of the stance expressed in our Twitter corpus. The majority of the authors (288 tweets) expresses itself negatively towards the Brexit (60%), whereas 26% has a rather neutral stance (124 tweets). Moreover, 14% takes a positive stance on the referendum (70 tweets).

In some tweets, the stance is rather implicit, for instance in the following tweet: “So, that was the dress rehearsal. Now that you Leavers have seen the effects of your vote, would you like to try that again? #Brexit”. Even though the author does not explicitly express itself for or against the Brexit, it is clear that he/she mocks with the leave-voters and is against Britain leaving the EU. Tweets can also contain implicit positive stance, such as in the following tweet: “Leftists are freaking out over the #Brexit. Why, because the people finally rejected tyranny? Just proves: liberals = tyrants. #tcot”. The author is fairly negative towards the people who voted to remain part of the EU and is therefore a supporter of the Brexit, which results in a positive stance label.

Page 28 of 76

FIGURE 3 - MANUAL ANNOTATION STANCE (N=482)

Positive (70) Negative (288) Neutral (124)

14% 26%

60%

Figure 3 - Manual annotation stance If we compare the results of the SA with those of our stance detection (cf. Table 2), we find that 218 tweets (out of 482) hold negative sentiment as well as negative stance. Additionally, 43 tweets contain positive sentiment and stance, whereas 89 tweets have neutral sentiment and stance. In total, 350 tweets (73%) have the same sentiment and stance label, which means 132 tweets show differences in the annotation of sentiment and stance. An example of the latter could be the following tweet: “The only good thing to come out of the #Brexit is the dearth of insults being hurled at @realDonaldTrump by the lovely people of Scotland”. Here the sentiment of the tweet is positive, because the author explains a positive result of the Brexit. The stance expressed, however, is negative, for the reason that the author does not see any other positive consequences of the Brexit apart from new insults about Trump.

It is noticeable, that besides the relatively frequent use of irony in tweets (9%) with negative sentiment and stance, especially tweets with neutral sentiment holding either negative or neutral stance also contain ironic language (4%). In chapter 4.3, we will further explore whether a machine learning system for sentiment analysis and stance detection will be influenced by the usage of irony. As can be drawn from the overview in Table 3, irony is particularly used in tweets specifically considering the economic consequences of the Brexit, followed by tweets about Trump and the Brexit itself.

Page 29 of 76

TOTAL IRONIC IRONY SENTIMENT STANCE PERCENTAGE TWEETS TWEETS PERCENTAGE NEGATIVE NEGATIVE 218 45% 47 22% NEUTRAL 19 4% 3 16% POSITIVE 10 2% 0 0% NEUTRAL NEGATIVE 45 9% 19 42% NEUTRAL 89 18% 19 21% POSITIVE 10 2% 0 0% POSITIVE NEGATIVE 25 5% 8 32% NEUTRAL 16 3% 4 25% POSITIVE 43 9% 2 5% = 482 = 103 Table 2 - Manual annotation sentiment, stance and irony

4.2 Results machine learning system

TOPIC NUMBER OF TWEETS IRONY PERCENTAGE CONTAINING IRONY PER TOPIC Brexit (274) 60 22% Celebrities/politicians (29) 6 21% Scottish referendum (2) 0 0% Economy (34) 9 26% EU (14) 2 14% other (94) 19 20% Trump (21) 5 24% USA (22) 2 9% = 103 The second step in the Table 3 - Presence of irony per topic experimental part of this study consisted of the automatic annotation of the 482 political tweets with the hashtag Brexit. In this section, we will discuss the results of both the sentiment and stance labels. In chapter 4.3, we will compare the results of the manual with those of the machine learning approach.

4.2.1 Sentiment and topics

As can be drawn from the pie chart below (cf. Figure 5), the greater part (69%) of the Twitter corpus consists, according to the SA-tool, of negative tweets. The following tweet, for example, was picked up by the system and labelled as negative: “Still so sad about #Brexit. What is this dark, absurd future being carved out for the world?”. Here, the sentiment words ‘sad’ and ‘dark’ were presumably a deciding factor. 117 tweets or 24% of the corpus received a neutral label and a small 7% of the tweets was labelled as positive. An example of the latter could be: “Learning some great new swears, thanks Scotland! #Brexit”. In this tweet, the author is delighted to learn new insulting language phenomena from Scotland.

Page 30 of 76

FIGURE 4 - MACHINE LEARNING ANNOTATION SENTIMENT (N=482)

Positive (34) Negative (331) Neutral (117)

7% 24%

69%

Figure 4 - Machine learning annotation sentiment

As can be drawn from Table 4, the distribution of sentiment in each topic is not equally divided: in every topic category, the negative labels are predominant, followed by considerably less neutral and even fewer positive tweets. This can also be concluded from Table 1. Only in the category ‘USA’, there is one more positive tweet than the neutral ones and in the category ‘Scottish referendum’, there are merely 2 tweets, which are both neutral. The majority of the Twitter corpus consists of negative tweets on the Brexit referendum itself. An example of this could be: “Never underestimate the power of stupid people in large groups! #Brexit #jokeofthecentury”. In this tweet, the author utters itself negatively towards the Leave-voters of the Brexit. The topic on ‘voters’ was classified under ‘Brexit’, as can be seen in Appendix 1. TOPIC NEGATIVE NEUTRAL POSITIVE Brexit (273) 190 (70%) 64 (23%) 19 (7%) Other (89) 53 (60%) 30 (34%) 6 (6%) Economy (34) 23 (68%) 9 (26%) 2 (6%) Celebrities (29) 23 (80%) 4 (14%) 2 (7%) USA (21) 14 (67%) 3 (14%) 4 (19%) Trump (21) 17 (81%) 3 (14%) 1 (5%) EU (13) 11 (85%) 2 (15%) 0 (0%) Scottish referendum 0 (0%) 2 (100%) 0 (0%) (2) TOTAL 331 117 34 Table 4 - Sentiment per topic (machine learning approach) 4.2.2 Stance and irony

Considering the results of the stance detection returned by our classifier, it can be said that the distribution of negative, neutral and positive stance is fairly similar compared to the distribution

Page 31 of 76

of sentiment (cf. Figure 5). However, the percentage of tweets with negative stance (79%) is considerably higher than tweets with negative sentiment (69%). In Table 5, we can see that, according to the automatic system for SA, 302 tweets contain both negative sentiment and stance, which counts for 63% of all tweets. Of all these tweets, 13% percent contains ironic language. With regard to the neutral labels, it can be drawn from Table 5 that 10% of the tweets contain a neutral sentiment label as well as a neutral stance label. In 12% of all cases, tweets with neutral sentiment received a negative stance label. Furthermore, it is remarkable that in total there are very little entirely positive tweets (2%).

FIGURE 5 - MACHINE LEARNING ANNOTATION STANCE (N=482) Positive (22) Negative (381) Neutral (79)

5% 16%

79%

Figure 5 - Machine learning annotation stance Concerning the percentage of ironic tweets, it can be stated that 32% of the tweets that received a positive sentiment label and a negative stance label from the machine learning system contain ironic language. Moreover, it is notable that a quarter of the tweets that contain positive sentiment and negative stance are ironic. In absolute numbers, the category with both negative sentiment and negative stance holds the highest number of ironic tweets, namely 64. In section 4.3.2.2, we will compare the accordance of sentiment with stance to the influence of irony in further detail.

Page 32 of 76

SENTIMENT STANCE TOTAL PERCENTAGE IRONIC IRONY TWEETS TWEETS PERCENTAGE NEGATIVE NEGATIVE 302 63% 64 21% NEUTRAL 22 5% 4 18% POSITIVE 7 1% 1 14% NEUTRAL NEGATIVE 60 12% 15 25% NEUTRAL 50 10% 9 18% POSITIVE 7 1% 2 29% POSITIVE NEGATIVE 19 4% 6 32% NEUTRAL 7 1% 1 14% POSITIVE 8 2% 1 13% = 482 = 103 Table 5 - Machine learning annotation sentiment, stance and irony 4.3 Analysis

In this chapter, we will compare and interpret the results of both the manual annotation and the labelling of the machine learning system. We will verify how accurate the machine learning system performs for sentiment analysis and stance detection on a Twitter corpus with political tweets. Furthermore, we will analyse whether or not the presence of ironic language affects the predictability of sentiment or stance.

4.3.1 Sentiment analysis: tenfold cross-validation scheme

In the methodology section (cf. 3.2) of this study, it was already explained that our machine learning system was run and tested on a tenfold cross-validation scheme6. This means that 90% of the Twitter corpus was used as a train fold and 10% as a test fold. This process was repeated 10 times to check the quality of the system. The tenfold cross-validation approach is found to be reliable, for the reason that the entire corpus is used for both training and validation. Moreover, each set is used for validation exactly once. In Table 10, an overview is given of the results of our tenfold cross-validation scheme. In chapter 4.3.2, we will discuss the global results of dataset as a whole.

It is noticeable that the scores can vary from fold to fold, considering that the cross-validation process uses different train and test data in every fold. In Table 8, the ten accuracy scores are similar to one another, whereas precision and recall scores differ notably. For example, the precision scores of the positive labels in the various folds vary from zero to 0.67. The same goes for the positive recall score: in Fold 01, for instance, the score is zero, whereas a score of

6 https://www.openml.org/a/estimation-procedures/1

Page 33 of 76

0.43 is given Fold 08. The F1 scores of the negative labels are all closely situated next to one another, with only one peak in Fold 06 of 0.8. All negative F1 scores are well above 0.6, which makes negative sentiment the best predicted label. The precision and recall of neutral labels tend to lie far apart from each other in the different folds. In Fold 02, for instance, the neutral precision score is 0,27, when in Fold 03 the score is remarkably higher (0.65).

Sentiment pos neg neutr pos neg neutr pos neg neutr accuracy

prec prec prec recall recall recall F1 F1 F1 Fold 00 0.20 0.61 0.27 0.11 0.73 0.25 0.14 0.67 0.26 0.49 Fold 01 0 0.58 0.70 0 0.84 0.50 0 0.69 0.58 0.58 Fold 02 0.50 0.60 0.27 0.14 0.75 0.23 0.22 0.67 0.25 0.52 Fold 03 0 0.54 0.65 0 0.78 0.52 0 0.64 0.58 0.54 Fold 04 0.67 0.56 0.38 0.18 0.71 0.42 0.29 0.63 0.40 0.52 Fold 05 0 0.58 0.50 0 0.73 0.33 0 0.64 0.40 0.50 Fold 06 0 0.75 0.40 0 0.86 0.67 0 0.80 0.50 0.63 Fold 07 0 0.62 0.33 0 0.78 0.25 0 0.69 0.29 0.52 Fold 08 0.43 0.70 0.55 0.43 0.78 0.43 0.43 0.74 0.48 0.63 Fold 09 0.75 0.50 0.57 0.30 0.87 0.22 0.43 0.63 0.32 0.53 Table 6 - Tenfold cross-validation scheme for sentiment analysis

4.3.2 Sentiment analysis: general overview

In the first column of Table 7, an overview is given of the number of the manually given labels per sentiment. The second column shows in yellow how many correct labels the machine learning system assigned to the tweets. Besides the number of these true positives, an overview is given of all false positives per sentiment category. The third column presents the number of ironic tweets per automatically predicted label to provide insight into the effect of irony on the predictability of sentiment

Page 34 of 76

MANUALLY TWEETS THAT ARE NUMBER OF IRONY ANNOTATED AUTOMATICALLY IRONIC TWEETS PERCENTAGE SENTIMENT PREDICTED AS (103) NEGATIVE 254 NEGATIVE 199 (78%) 39 20% NEUTRAL 40 (16%) 8 20% POSITIVE 15 (6%) 4 27% NEUTRAL 144 NEGATIVE 81 (56%) 21 26% NEUTRAL 54 (38%) 16 30% POSITIVE 9 (6%) 1 11% POSITIVE 84 NEGATIVE 51 (61%) 9 18% NEUTRAL 23 (27%) 2 9% POSITIVE 10 (12%) 3 30% Table 7 - Comparison manual and automatic sentiment analysis + irony presence

To interpret the results presented in Table 7, we calculated precision, recall, and F1-score as well as the accuracy of our machine learning system. Firstly, precision is the number of true positives divided by the total number of true positives and false positives returned by the classifier and is used to find out how “correct” the predictions per label are. Secondly, the recall score reveals how many sentiment labels are predicted by dividing true positives by the total number of true positives and false negatives returned by the machine learning system (per label). Thirdly, the F1-score is the average of both precision and recall and shows how well the system performs per sentiment category. The F1 or F-score is the result of the following 2 . (precision.recall) division: 푓 = . Lastly, the accuracy of the test set was calculated as a whole, precision+recall by dividing the number of correctly predicted labels by the total number of instances.

SENTIMENT Negative Neutral Positive 199 54 10 Precision 푝 = 푝 = 푝 = (199 + 81 + 51) (40 + 54 + 23) (15 + 9 + 10) 푝 = 0.6 푝 = 0.46 푝 = 0.29

199 54 10 Recall 푟 = 푟 = 푟 = 254 144 84

푟 = 0.78 푟 = 0.38 푟 = 0.12

F1-score f = 0.69 f = 0.42 f = 0.21

199 + 54 + 10 Accuracy 푎 = 482 푎 = 0.55

Table 8 - Precision, recall, F1-score and accuracy of sentiment labels

Page 35 of 76

As can be drawn from Table 8, the accuracy of the machine learning system results in a score of 0.55. This means that the performance of our SA system can be evaluated as moderate, because the score is slightly above 0.5. The high precision score of the negative sentiment shows that the system returned more true positives than false positives. Moreover, the high recall score indicates that the SA system returned most of the relevant results. Yet, there is still room for improvement, especially when looking at the precision and recall scores of the neutral and positive sentiment labels. For example, the recall score of the positive labels shows that the machine learning failed to return many of the positive tweets.

If we compare Table 8 with Table 6, we notice that our machine learning system for SA is rather biased towards negative tweets. In other words, our classifier tends to overgeneralize negative labels and attributes them falsely in 34% of the cases. Neutral labels are assigned falsely for 13% of the tweets and positive labels are only ascribed wrongly for 5% of the cases. This might explain the high precision, recall and F1 scores within the negative category and the low results for both positive and neutral labels. As a result of this difference between results for the negative sentiment category on the one hand, and those for the positive and neutral sentiment category on the other hand, the accuracy scores are only slightly above average.

From the third column in Table 7, it can be concluded that 45 of all 215 false positive cases contained ironic language. In other words, irony was used in 21 percent of all falsely labelled tweets. This could be an explanation for many of the errors, meaning that irony influences the predictability of sentiment in political tweets. In chapter 4.3.1.3 we will discuss the various errors in greater detail.

4.3.2.1 Impact of irony on the prediction of sentiment In Table 7, an overview is given of how many tweets contain irony to analyse the possible effect of creative language on the predictability of sentiment. In the fourth column, the irony percentage in each sentiment category is presented. It can be noted, that in the negative manually annotated category the irony percentages all lie between 20 and 27 percent. As a result, it is difficult to conclude, whether the false positives (negative tweets that received a neutral or positive label) are influenced by the presence of the ironic language or not, without looking at the content of the tweets itself. For instance, in 20% of all negative tweets that received a positive label, 27% contained irony. The same goes for the neutral and positive sentiment category, where the true positives itself (e.g. neutral tweets that received a neutral

Page 36 of 76

label by our machine learning system) appear to contain the highest percentages of ironic tweets. In total, if the amount of ironic false positives is divided by the total number of false positives, it appears that 21% of the falsely labelled tweets by the machine learning system contained irony. If we follow the same procedure for all true positives, it shows that 22% of the correctly automatically labelled tweets also contained irony.

Considering the content of all falsely labelled tweets, it can, however, be concluded that irony does have a certain effect on the machine learning system. In the following negative tweets, for instance, the classifier ascribed a positive or neutral label due to irony:

(1) “Thanks British #brexit twats, I'm feeling poorer today.”

(2) “The next James Bond will just be him spending 2 hours in passport control De Gaulle #Brexit #JamesBond”

(3) “Sebastian was right, can I become a mermaid now pls #Brexit”

(4) "They took back their country and that’s a great thing," Trump said of #Brexit, while in Scotland IN SCOTLAND!!!!!

In tweet (1), our machine learning for SA does not detect the ironic expression of gratitude and the negative connotation attributing ‘feeling poor’. In other words, the implicit sentiment in ‘feeling poor’ was not recognized. Moreover, the negative and insulting word ‘twats’ was not picked up. Then, in tweet (2) the classifier decided on a neutral label, regardless of the fact that spending 2 hours in passport control is a rather unpleasant activity. In the third tweet (3), a hyperbole is used to express the author’s disbelief, with regard to the leave-voters in the Brexit referendum. Here, some specific background knowledge is needed to understand the reference made to the Little Mermaid. There, the character Sebastian attempts to warn mermaid Ariel for the foolishness of human beings, by saying ‘The human world is a mess’. In (3), the author confirms the truth in Sebastian’s reasoning. The system, however, fails to interpret the context and irony in the tweet and assigns a neutral label. Lastly, tweet (4) is an example of situational irony. The fact that the American president Trump expresses himself in favour of Britain ‘taking their country back’ while being in Scotland is ironic, in the sense that Scotland itself has been struggling in their search for independency for years. Here again, the system lacks the right

Page 37 of 76

contextual information to interpret the tweet correctly. Furthermore, the repetition of ‘in Scotland’ in capital letters, followed by a punctuation flooding was not detected.

In chapter 4.3.4.1 we will go further into the possible effect of irony on our stance classifier.

4.3.2.2 Error analysis Besides the effect of irony possibly causing the machine learning system to make errors, we will consider another factor that may influence the automatic attribution of a wrong sentiment label in this section. In some tweets, the machine learning system appears to have a notable lack of common of specific knowledge. Therefore, following tweets were misinterpreted and therefore falsely labelled:

(5) #Brexit, Monty Python & Silly Walks.

(6) Wales should have been more careful what it wished for. It's going to be given it. #Brexit

(7) Proud to be #Brexit! Proud to stand alone in a world where most are too scared to be alone, to have their own opinion. Proud to me! #UKref

In tweet (5), the classifier attributed a negative label, whereas it was originally granted a neutral label. Presumably, the word ‘silly’ was picked up as a negative word. The tweet, however, refers to the popular Monty Python sketch ‘Ministry of Silly Walks’, firstly broadcast in 1970, which does not hold a negative connotation. The sixth tweet (6), contains an allusion to the common expression ‘be careful what you wish for’. According to the Merriam-Webster7 dictionary, it is usually used ‘to tell people to think before they say that they want something and to suggest that they may not actually want it’. Nonetheless, this negative tweet received a neutral label. The last example (7), being alone is valued as a positive situation. The author explains that he/she sees being alone as having an individual opinion, which is something to be proud of. Therefore, the tweet is positive, but the classifier hands out a negative label because it sees being alone as something negative.

7 https://www.merriam-webster.com/ Page 38 of 76

4.3.3 Stance detection: tenfold cross-validation scheme

To gain more detailed information on the accuracy of the stance labels returned by the classifier, the tenfold cross-validation approach was once again applied on the Twitter corpus. In comparison with the accuracy scores of the SA, it can be concluded that, in general, the classifier returned better accuracy results for stance detection than for SA. This will be further discussed in chapter 4.3.4. Furthermore, it is remarkable that in, for instance, Fold 05 the accuracy is relatively low (0.45) when compared to the high score of 0.73 in Fold 09. Mainly, these differences can be declared by focussing on the positive precision and recall scores. Remarkably, the positive precision scores vary between zero and one. An explanation for this phenomenon could be the fact that in a tenfold cross-validation scheme, only 10 percent of all data is used as test data in each fold. Considering that only 10 positive stance labels were predicted correctly, it is comprehensible that in some folds none of the corresponding tweets appeared. This then results in a score of zero for both precision and recall.

In general, the positive recall scores are all fairly low, varying between zero and 0.22. In comparison, however, the precision scores are higher than the recall scores in the positive stance category. The higher precision score indicates a low false positive rate, but it can be concluded from the low recall that, in general, very few results were predicted. As regards the neutral accuracy, the precision scores are again rather varied. They run from a very low 0.22 in Fold 02 to a fairly high score of 0.67 in Fold 03. The neutral recall scores do not reach above average, with the exception of Fold 08. Moreover, the negative recall scores are all fairly continuous and all lie between 0.75 and an almost perfect score of 0.93. It is noteworthy that the negative precision scores are, for example in Fold 03, rather average, whereas Fold 02 shows a peak of 0.76.

Page 39 of 76

Stance pos neg neutr pos neg neutr pos neg neutr accuracy

prec prec prec recall recall recall F1 F1 F1 Fold 00 0.50 0.64 0.42 0.13 0.75 0.45 0.20 0.69 0.43 0.57 Fold 01 1 0.59 0.63 0.13 0.92 0.33 0.22 0.72 0.43 0.60 Fold 02 0.50 0.76 0.22 0.14 0.82 0.29 0.22 0.79 0.25 0.65 Fold 03 0 0.47 0.67 0 0.9 0.29 0 0.62 0.40 0.48 Fold 04 1 0.67 0.38 0.14 0.84 0.30 0.25 0.74 0.33 0.63 Fold 05 0.25 0.45 0.67 0.14 0.81 0.20 0.18 0.58 0.31 0.46 Fold 06 0 0.7 0.43 0 0.88 0.27 0 0.78 0.33 0.65 Fold 07 0.50 0.68 0.25 0.17 0.81 0.2 0.25 0.74 0.22 0.60 Fold 08 1 0.70 0.67 0.22 0.93 0.55 0.37 0.80 0.60 0.71 Fold 09 0.50 0.77 0.33 0.40 0.90 0.13 0.44 0.83 0.18 0.73 Table 9 - Tenfold cross-validation scheme for stance detection

4.3.4 Stance detection: general overview

As already explained in chapter 4.3.2, an overview (cf. Table 7) was made of both the manually distributed labels, with in the second column the corresponding labels extracted by the machine learning system. In the third column, the number of ironic tweets was listed per label to analyse whether ironic language influences the predictability of stance or not. Furthermore, precision, recall, F1 score and accuracy were once again calculated to provide more insight into the quality of the stance labels returned by the system. Table 10 and 11 show results of the entire data set, after 10-fold cross validation.

MANUALLY TWEETS THAT ARE NUMBER OF IRONY ANNOTATED AUTOMATICALLY IRONIC TWEETS PERCENTAGE STANCE PREDICTED AS (103) NEGATIVE 288 NEGATIVE 246 (85%) 63 26% NEUTRAL 33 (12%) 8 24% POSITIVE 9 (3%) 3 33% NEUTRAL 124 NEGATIVE 84 (68%) 19 23% NEUTRAL 37 (30%) 6 16% POSITIVE 3 (2%) 1 33% POSITIVE 70 NEGATIVE 51 (73%) 3 6% NEUTRAL 9 (13%) 0 0% POSITIVE 10 (14%) 0 0% Table 10 - Comparison manual and automatic stance detection + irony presence

Page 40 of 76

As can be concluded from Table 10 and 11, the most accurate predictions were made for the negative stance category. Both precision and recall show reasonably good results with correspondingly a score of 0.65 and 0.85, which results in an F1-score of 0.75. It is remarkable that the scores in the neutral and positive category are again noticeably lower, with F1-scores of 0.39 and 0.30. In total, the machine learning system reaches a fairly high accuracy score of 0.61. Furthermore, it is remarkable that the precision score of the positive category is nearly three times higher than the recall score. This indicates that the machine learning system predicts on the one hand very few positive stance labels, but that, on the other hand, these labels are in many cases correct.

STANCE Negative Neutral Positive 246 37 10 Precision 푝 = 푝 = 푝 = (246 + 84 + 51) (33 + 37 + 9) (9 + 3 + 10) 푝 = 0.65 푝 = 0.47 푝 = 0.45

246 37 10 Recall 푟 = 푟 = 푟 = 288 124 70

푟 = 0.85 푟 = 0.30 푟 = 0.14

F1-score f = 0.75 f = 0.39 f = 0.30

246 + 37 + 10 Accuracy 푎 = 482 푎 = 0,61

Table 11 - Precision, recall, F1-score and accuracy of stance labels

4.3.4.1 Impact of irony on the prediction of stance To verify the impact of ironic language on the predictability of stance, Table 10 shows the absolute numbers and percentages of irony in the Twitter corpus. In total 103 of 482 tweets were manually labelled as ironic. The use of irony is fairly equally distributed between the negative and neutral stance categories. It is, however, noticeable that the irony use in tweets with positive stance is rather rare. In 26% and 16% of the corresponding negative and neutral categories of true positives, the tweets contain irony. In addition, it can be stated that the irony percentage in the false positives is just as high (26%) for tweets with negative stance and somewhat higher (23%) for tweets with neutral stance.

Page 41 of 76

To interpret the meaning of these irony percentages, a closer look to the content of the falsely labelled tweets is required. By considering merely the ironic tweets that were predicted falsely, the following examples were retained.

(8) Can we solve #Brexit issues using the old "reset" method? Switch off, leave Europe for 10 secs then plug ourselves bck in? Oh wait... oops?

(9) So, that was the dress rehearsal. Now that you Leavers have seen the effects of your vote, would you like to try that again? #Brexit

(10) Churchill said, "Heroes fight like Greeks". Like a Greek i have to say that "Heroes, vote like British!" #Brexit

In tweet (8) the author raises several rhetorical questions, a form of irony as defined by Karoui et al. (2017). The classifier fails to pick up on this and returns a neutral stance label, whereas the tweet was manually labelled as negative. The same goes for tweet (9), where the machine learning system does not detect the ironic use of language and therefore decides to label the stance as positive. The author, however, is against the outcome of the Brexit and jokingly proposes to organize another referendum now the leave voters understand the negative consequences of leaving the European Union. As a result, the tweet was manually annotated with a negative stance label. In the last example (10), the author uses a metaphor and analogy to express his positive attitude towards the Brexit. The system could not interpret the complex nature of this tweet and assigned a negative label to it. Apart from tweet (8), (9) and (10), clear examples of how irony influenced the detection of stance were rare.

4.3.4.2 Error analysis As already established in chapter 4.3.2.2, it can be assumed that other factors influence stance detection apart from irony presence. In this section, several examples of errors will be listed and discussed.

(11) "protection!? Protection of what!? #TzeGermans!?" #Brexit #funfacts

(12) Proud of #Britain and @David_Cameron for doing the right thing today. #Brexit #EUref #independence #freedom

Page 42 of 76

(13) Britons voted to strengthen their borders. Will you do the same in November? #Brexit

(14) Your country is truly inspiring thank you #Brexit

(15) Man, that was so cool. #Brexit

(16) The #Brexit was necessary. The EU was turning into that evil corporation from the Aliens franchise.

(17) On my room balcony posing with the freshly brexited EU flag. Giving it some company. #iifa2016 #iifadiaries #brexit

The author’s stance in tweet (11) is negative. The content refers to one of the main arguments of the Leave-voters in the Brexit referendum: leaving the EU is a form of protectionism and equals ‘putting the UK first’. The author of tweet (11) uses a reference from the British movie Snatch to point out that there is nothing to protect the UK from. The machine learning system is not aware of this specific movie reference and judges that the stance is neutral. In tweets (12), (13), (14) and (15) the trend that the classifier tends to overgenerate negative labels – just as specified in the analysis of the SA results (cf. 4.3.2) – can be observed. The four tweets contain positive stance and positive sentiment words. Nonetheless, the machine learning system decides on a negative label. In the last example (16), the author appears to be in favour of the Brexit outcome and against the EU as an institution. Therefore, the author’s stance towards the Brexit is positive. It can be presumed that the classifier interpreted the stance as negative, as a consequence of the use of ‘evil’. In the last tweet (17) with positive stance, there are no clear sentiment words present, which results in a false attribution of a neutral stance label.

4.3.5 Comparison sentiment analysis and stance detection

In chapters 4.3.1, 4.3.2, 4.3.3 and 4.3.4 we analysed the results returned by the machine learning system. We separately analysed the results of the sentiment analysis and stance detection, by observing precision, recall, F1 and accuracy scores. In this section, we will focus on differences

Page 43 of 76

in performance and accordance between the prediction of sentiment and stance. Furthermore, we will again consider the possible impact of irony use.

4.3.5.1 Comparison performance In Table 12, all results of both the sentiment analysis and stance detection were listed. The first and third column show the total number of tweets with negative, neutral or positive sentiment/stance. The second and fourth column give an overview of how the machine learning system attributed sentiment and stance labels. In addition, to compare the performance of our system in terms of sentiment analysis and stance detection, Table 13 shows all precision, recall,

F1 and accuracy scores.

MANUALLY TWEETS THAT ARE MANUALLY TWEETS THAT ARE ANNOTATED AUTOMATICALLY ANNOTATED AUTOMATICALLY SENTIMENT PREDICTED AS STANCE PREDICTED AS NEGATIVE 254 NEGATIVE 199 NEGATIVE 288 NEGATIVE 246 NEUTRAL 40 NEUTRAL 33 POSITIVE 15 POSITIVE 9 NEUTRAL 144 NEGATIVE 81 NEUTRAL 124 NEGATIVE 84 NEUTRAL 54 NEUTRAL 37 POSITIVE 9 POSITIVE 3 POSITIVE 84 NEGATIVE 51 POSITIVE 70 NEGATIVE 51 NEUTRAL 23 NEUTRAL 9 POSITIVE 10 POSITIVE 10 Table 12 - Comparison results sentiment analysis and stance detection

SENTIMENT Negative Neutral Positive STANCE Negative Neutral Positive

Precision 푝 = 0.6 푝 = 0.46 푝 = 0.29 Precision 푝 = 0.65 푝 = 0.47 푝 = 0.45

Recall 푟 = 0.78 푟 = 0.38 푟 = 0.12 Recall 푟 = 0.85 푟 = 0.30 푟 = 0.14

F1-score f = 0.69 f = 0.42 f = 0.21 F1-score f = 0.75 f = 0.39 f = 0.30

Accuracy 푎 = 0.55 Accuracy 푎 = 0.61

Table 13 - Precision, recall, F1-score and accuracy of sentiment and stance labels As can be drawn from Table 12 and 13, the classifier shows very similar results for the prediction of both sentiment and stance. The negative and neutral precision scores are very similar to one another. The positive precision score, however, is remarkably higher for the prediction of stance labels (0.45) than for the prediction of sentiment labels (0.29). Overall, it

Page 44 of 76

is remarkable that the negative labels are predicted best, followed by the neutral category and then the positive category. A possible explanation for this could be the smaller number of neutral and positive tweets in our corpus, due to which the machine learning system had more had more training examples to be trained on for the negative tweets.

In comparison, our classifier performs slightly better on stance detection, considering the accuracy score of 0.55 for sentiment analysis and 0.61 for stance detection.

4.3.5.2 Comparison accordance and the influence of irony In this section, we will compare the accordance of the manually assigned sentiment labels with their stance labels (cf. Table 14) to the accordance of the automatically assigned sentiment labels with their stance labels (cf. Table 15). In the fourth and fifth column, the number of ironic tweets per sentiment/stance category was listed with their percentage.

SENTIMENT STANCE PREDICTIONS ACCORDANCE IN NUMBER OF IRONY PREDICTIONS (manual annotation) PERCENTAGES IRONIC IN (manual annotation) TWEETS PERCENTAGES NEGATIVE 254 NEGATIVE 218 86% 47 22% NEUTRAL 19 7% 3 16% POSITIVE 17 7% 1 6% NEUTRAL 144 NEGATIVE 45 31% 19 42% NEUTRAL 89 62% 19 21% POSITIVE 10 7% 0 0% POSITIVE 84 NEGATIVE 25 30% 8 32% NEUTRAL 16 19% 4 25% POSITIVE 43 51% 2 5% Table 14 - Accordance of manually assigned sentiment labels with their stance labels

SENTIMENT STANCE PREDICTIONS ACCORDANCE IN NUMBER IRONY PREDICTIONS (classifier) PERCENTAGES OF IRONIC IN (classifier) TWEETS PERCENTAGES NEGATIVE 331 NEGATIVE 302 91% 64 21% NEUTRAL 22 7% 4 18% POSITIVE 7 2% 1 14% NEUTRAL 117 NEGATIVE 60 51% 15 25% NEUTRAL 50 43% 9 18% POSITIVE 7 6% 2 29% POSITIVE 34 NEGATIVE 19 56% 6 32% NEUTRAL 7 20% 1 14% POSITIVE 8 34% 1 13% Table 15 - Accordance of automatically assigned sentiment labels with their stance labels From Table 14, it can be concluded that 86% of all tweets with negative sentiment also hold negative stance. Then, for tweets with neutral sentiment, the correspondence with neutral stance is somewhat lower, with a percentage of 62. Moreover, in 51% of all positive tweets the author’s stance towards the Brexit is also positive. It is remarkable that negative tweets rarely hold

Page 45 of 76

neutral or positive stance, whereas differences between sentiment and stance are more frequent in the neutral and positive category. A possible explanation for this could be the presence of irony, especially for tweets with neutral/positive sentiment and negative stance. This means that, at first glance, a certain tweet seems positive/neutral, however, after the interpretation of irony use, it becomes clear that the author’s stance is actually negative. An example of this could be: “If I had a time machine, I'd happily take a molesting from Farage, just to Yewtree the cunt out of any political standing. #Brexit”. The sentiment in this tweet is positive because of his enthusiasm expressed in the word ‘happily’. However, the author presumably does not literally mean he would violently attack Nigel Farage, if he had a time machine. The author is, in other words, ironically discussing his negative attitude/stance towards the politician.

In addition, Table 15 shows similar trends in the results of the machine learning system. Yet, the results returned by the classifier do differ in terms of accordance in percentages, with the exception of the negative category. In our gold standard, the majority of all tweets with neutral sentiment hold neutral stance, whereas the machine learning system returns a higher percentage of neutral tweets holding negative stance. The same goes for the largest manually annotated category of positive tweets with positive stance, where the classifier finds more positive tweets holding negative stance. Concerning the effect of irony in Table 15, it can be observed that the highest irony percentages pop up in the neutral and positive tweets with negative sentiment. Therefore, this could mean that our machine is able to interpret irony correctly to some extent. However, from chapters 4.3.2.1 and 4.3.4.1 we learnt that in some cases the system fails to label sentiment or stance correctly as a result of ironic language use.

5. CONCLUSION

With the emergence of Web 2.0 at the beginning of the 21st century, easy-accessible microblogging platforms such as Facebook and Twitter have become omnipresent within the digital landscape. Blogs, forums and social media websites allow users to easily share their point of view, by means of blogposts, reviews, reactions and ratings. Due to the high amount of feedback or criticism on, for instance, products, services or political ideas there was a call for an automatic system that could gather a whole range of opinions on a certain topic. With the help of SA, a business or organization can find out which sentiment (positive, negative or neutral) a piece of text contains. As discussed by Pak and Paroubek (2009), political parties and

Page 46 of 76

politicians can gain a perception of how people view their programmes or how people see them within the political landscape. In addition, SD aims at discovering the point of view expressed by the author towards the subject being discussed. Nowadays, organisations consequently have enough feedback at their disposal to examine how people’s views differ on a certain product, service or policy. As a result, it is easier to detect the reason behind, for example, the success of a certain campaign or the low sales figures of a certain product.

In this study, we attempted to contribute to the existing findings on automatic sentiment analysis and stance detection on Twitter. Furthermore, we aimed to explore whether ironic language influences the performance of machine learning systems. For these purposes, we built a Twitter corpus consisting of 482 political tweets with the hashtag Brexit. The manually annotated corpus was then compared to both the predicted sentiment and stance labels. Based on the collected and analysed data and experimental results, we attempted to draw conclusions concerning the predictability of sentiment and stance, as well as the impact of irony on the performance of automatic systems.

Firstly, we observed that our Twitter corpus consisted mostly of negative sentiment (53%), followed by tweets with neutral (30%) and positive sentiment (17%). Similarly, the machine learning system also mostly predicted tweets with negative sentiment (69%), then tweets with positive sentiment (24%) and lastly positive tweets (7%). The topic that was discussed the most was the Brexit itself, presumably because all tweets already contained “#Brexit”. Furthermore, negative labels were predominant in nearly every topic (such as ‘Brexit’, ‘USA’, ‘Trump’, ‘other’, …), which was a logical consequence, considering the high percentages of negative tweets. We noticed that our machine learning system for sentiment analysis was rather biased towards negative tweets, meaning that the classifier tended to overgenerate negative labels. It was observed that negative labels were attributed falsely in 34% of the cases, whereas neutral and positive labels were only assigned falsely in correspondingly 13 and 5 percent of the cases.

This could be a possible explanation for the high precision, recall and F1-scores within the negative category and the rather low results in both the positive and neutral category. Overall, the system scored 0.55 on accuracy which is, considering the relatively small amount of tweets in our corpus, a fairly good score. What remains a challenge is the detection of implicit sentiment (e.g. ‘feeling poor’ is a negative feeling), as was already stated by Van Hee (2017).

Page 47 of 76

Secondly, the distribution of the stance labels was relatively similar to the percentages of negative, neutral and positive sentiment labels. The gold standard consisted mostly of tweets with negative stance (60%), followed by tweets holding neutral (26%) and positive (5%) stance. Our machine learning system predicted that 79% of all tweets contained negative stance, 16% neutral stance and 5% positive stance. Regardless of the lower precision and recall scores in the neutral and positive stance categories, our system scored 0.61 on accuracy, which is slightly better than the score for sentiment analysis. It was noticeable that the system again tended to overgenerate negative stance labels, which would explain the rising of the percentage of negative labels predicted by the machine learning system with nearly a fifth. Furthermore, it became clear in the analysis of the content of the tweets that the system failed to interpret specific references to, for instance, movies correctly.

Regarding the accuracy results of our sentiment analysis (0.55) as well as stance detection (0.61), our first hypothesis can be confirmed: the machine learning system delivers fairly reliable results on a political Twitter corpus.

Thirdly, we considered the impact of the 103 ironic tweets in our corpus on the predictability of both sentiment and stance. It appeared that true positives as well as false positives contained similar irony percentages. As a result, it was rather hard to interpret whether irony influenced the outcome of the sentiment analysis or stance detection, without evaluating the content of the falsely predicted labels. Considering the content of all falsely labelled tweets, it could, however, be concluded that in some cases irony was interpreted literally. In other words, irony did have a certain negative effect on the performance of the machine learning system, but not in all cases. As a result, our second hypothesis, which said that our machine learning system would not be able to detect and interpret ironic language use correctly in most cases, cannot be confirmed entirely.

In conclusion, it can be stated that a machine learning approach for sentiment analysis and stance detection is already seemingly reliable. However, there is still room for improvement, considering both accuracy scores are under 65%. Furthermore, we need to tackle the challenge and indistinctness concerning ironic language in tweets. In chapter 6, suggestions on further research will be formulated.

Page 48 of 76

6. LIMITATIONS AND FURTHER RESEARCH

Overall, we found that our system for sentiment analysis and stance detection scores fairly well, however, there is still room for improvement. Therefore, we will discuss the limitations we encountered during our research and we will provide some suggestions for further research in this section.

We chose to build a corpus consisting of more or less 500 political tweets. Due to practical limitations, the corpus was limited in size compared to other studies using Twitter corpora (cf. Van Hee, 2017). This resulted in less reliable scores and percentages to interpret or to draw conclusions on. Moreover, we had no insight into which sentiment words were or were not detected by the machine learning system. Further research could, however, provide us with more insight for the error analysis. Furthermore, it would be interesting to compare a machine learning approach to a lexicon-based approach, to explore which approach generates the best results.

In further research on sentiment analysis and stance detection on a Twitter corpus, it would also be interesting to further explore the consequences of irony presence in political tweets. The reason for this is twofold. Firstly, it remained relatively unclear in this research to which extent irony influences the predictability of sentiment and/or stance. Secondly, the combination of sentiment/stance on Twitter and irony has been rarely been investigated, which offers many possibilities for improvement. Moreover, it would also be useful to perform irony detection on the Twitter corpus itself, as was done by Van Hee (2017). In this study, we were not able to automatically extract irony labels and thus, we could not analyse the effect of ironic language thoroughly. For further research, it would be interesting to compare the results of, for example, stance detection as well as irony detection.

Page 49 of 76

List of references Bambrick, N. (24.06.2016). Support Vector Machines for dummies; A Simple Explanation.

Retrieved February 20, 2018 from http://blog.aylien.com

Barbieri, F., & Saggion, H. (2014). Modelling Irony in Twitter. In EACL (pp. 56–64).

Boertjes, E., Jamakovic A. & Raaijmakers S. (2011). Analyse van sociale media. STAtOR,

November, 3-4, 12, (pp. 41-47).

Cambria, E., Schuller, B., Xia, Y., & Havasi, C. (2013). New avenues in opinion mining and

sentiment analysis. IEEE Intelligent Systems, 28(2), 15–21.

Chang, C., Lin, C. (2016). LIBSVM: A Library for Support Vector Machines. Retrieved February

20, 2018 from https://www.csie.ntu.edu.tw/~cjlin/libsvm/

Dave, K., Lawrence S. & Pennock D.M. (2003). Mining the peanut gallery: Opinion extraction and

semantic classification of product reviews. In Proceedings of the 12th international conference

on World Wide Web(WWW), (pp. 519–528).

De Clercq, O., Lefever, E., Jacobs, G., Carpels, T., & Hoste, V. (2017). Towards an integrated

pipeline for aspect-based sentiment analysis in various domains. Proceedings of the 8th

Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media

Analysis (pp. 136–142). Presented at the EMNLP2017, Association for Computational

Linguistics.

Desmet, B., De Clercq, O., Van de Kauter, M., Schulz, S., Van Hee, C., & Hoste, V. (2014).

Taaltechnologie 2.0: sentimentanalyse en normalisatie. In Beschouwingen uit een talenhuis :

opstellen over onderwijs en onderzoek in de vakgroep Vertalen, Tolken en Communicatie

aangeboden aan Rita Godyns, (pp. 157–161).

Ghosh, A., Li, G., Veale, T., Rosso, P., Shutova, E., Barnden, J., & Reyes, A. (2015). Semeval-2015

task 11: Sentiment analysis of figurative language in twitter. In Proceedings of the 9th

International Workshop on Semantic Evaluation (SemEval 2015) (pp. 470–478).

Page 50 of 76

Karoui, J., Farah, B., Moriceau, V., Patti, V., Bosco, C., & Aussenac-Gilles, N. (2017). Exploring

the impact of pragmatic phenomena on irony detection in tweets: A multilingual corpus study.

In Proceedings of the 15th Conference of the European Chapter of the Association for

Computational Linguistics: Volume 1, Long Papers (Vol. 1, pp. 262–272).

Khan, A. Z., Atique, M., & Thakare, V. M. (2015). Combining lexicon-based and learning-based

methods for Twitter sentiment analysis. International Journal of Electronics, Communication

and Soft Computing Science & Engineering (IJECSCSE), 89.

Kolkur, S., Dantal, G., & Mahe, R. (2015). Study of different levels for sentiment analysis.

International Journal of Current Engineering and Technology, 5(2), 768Ű770.

Kreuz, R. J., & Roberts, R. M. (1993). On satire and parody: The importance of being

ironic. Metaphor and Symbol, 8(2), 97-109.

Kulshrestha, J., Kooti, F., Nikravesh, A., & Gummadi, P. K. (2012, June). Geographic Dissection of

the Twitter Network. In ICWSM.

Kumar, A., & Sebastian, T. M. (2012). Sentiment analysis on twitter. IJCSI International Journal of

Computer Science Issues, 9(3), 372–378.

Liu, B. (2012). Sentiment Analysis and Opinion Mining. Synthesis lectures on human language

technologies, 5(1), 1-167.

Mandya, A., Siddharthan, A., & Wyner, A. (2016). Scrutable feature sets for stance classification.

ACL 2016, 60.

Mohammad, S., Kiritchenko, S., Sobhani, P., Zhu, X.-D., & Cherry, C. (2016). SemEval-2016 Task

6: Detecting Stance in Tweets. In SemEval@ NAACL-HLT (pp. 31–41). Retrieved from

http://www.aclweb.org/anthology/S/S16/S16-1003.pdf

Nasukawa, T., & Yi, J. (2003). Sentiment analysis: Capturing favorability using natural language

processing. In Proceedings of the 2nd international conference on Knowledge capture (pp. 70-

77). ACM.

Page 51 of 76

Nielsen, F. (2011). IMM Publications: AFINN. Retrieved February 20, 2018 from

http://www2.imm.dtu.dk/

Pak, A., & Paroubek, P. (2010). Twitter as a corpus for sentiment analysis and opinion mining. In

LREc (Vol. 10).

Pang, B., & Lee, L. (2008). Opinion mining and sentiment analysis. Foundations and Trends. In

Information Retrieval, 2(1–2), 1-135

Pang, B., Lee, L., & Vaithyanathan, S. (2002). Thumbs Up?: Sentiment Classification Using

Machine Learning Techniques. In Proceedings of the ACL-02 Conference on Empirical

Methods in Natural Language Processing - Volume 10 (pp. 79–86). Stroudsburg, PA, USA:

Association for Computational Linguistics. doi: 10.3115/1118693.1118704

Pavlopoulos, I. (2014). Aspect based sentiment analysis. Athens University of Economics and

Business.

Pontiki, M., Galanis, D., Papageorgiou, H., Androutsopoulos, I., Manandhar, S., AL-Smadi, M., …

others. (2016). SemEval-2016 task 5: Aspect based sentiment analysis. In ProWorkshop on

Semantic Evaluation (SemEval-2016) (pp. 19–30). Association for Computational Linguistics.

Retrieved from https://biblio.ugent.be/publication/8131987

QuantStart Team. (12.09.2014). Support Vector Machines: A Guide for Beginners. Retrieved

February 20, 2018 from https://www.quantstart.com/

Reyes, A., Rosso, P., & Buscaldi, D. (2012). From humor recognition to irony detection: The

figurative language of social media. Data & Knowledge Engineering, 74, 1–12.

Reyes, A., Rosso, P., & Veale, T. (2013). A multidimensional approach for detecting irony in twitter.

Language Resources and Evaluation, 47(1), 239–268.

Shoeb, M., & Ahmed, J. (2017). Sentiment Analysis and Classification of Tweets Using Data

Mining. Work, 4(12).

Page 52 of 76

Taboada, M. (2016). Sentiment Analysis: An Overview from Linguistics. Annual Review of

Linguistics, 2. doi:10.1146-011415-040518

Tumasjan, A., Sprenger, T. O., Sandner, P. G., & Welpe, I. M. (2010). Predicting elections with

twitter: What 140 characters reveal about political sentiment. Icwsm, 10(1), 178–185.

Van de Kauter, M. (2013). LeTs preprocess: the multilingual LT3 linguistic preprocessing toolkit.

Van Hee, C. (2017). Can machines sense irony? : exploring automatic irony detection on social

media. Ghent University. Faculty of Arts and Philosophy, Ghent, Belgium.

Van Hee, C., Lefever, E., & Hoste, V. (2015). LT3: sentiment analysis of figurative tweets: piece of

cake #NotReally. In SemEval : 9th International Workshop on Semantic Evaluations at Naacl

2015, Proceedings (pp. 684–688). Association for Computational Linguistics. Retrieved from

http://hdl.handle.net/1854/LU-6827146

Van Hee, C., Van de Kauter, M., De Clercq, O., Lefever, E., & Hoste, V. (2014). LT3: sentiment

classification in user-generated content using a rich feature set. In Proceedings of the 8th

International Workshop on Semantic Evaluation (pp. 406–410). Association for Computational

Linguistics. Retrieved from http://hdl.handle.net/1854/LU-5684543

Wang, P.-Y. A. (2013). # Irony or# Sarcasm—A Quantitative and Qualitative Study Based on

Twitter. In Proceedings of the 27th Pacific Asia Conference on Language, Information, and

Computation (PACLIC 27) (pp. 349–356).

Zhang, L., Ghosh, R., Dekhil, M., Hsu, M., & Lius, B. (2011). Combining Lexicon-based and

Learning-based Methods for Twitter Sentiment Analysis. In Hewlett-Packard

Labs Technical Report HPL-2011-89.

Page 53 of 76

APPENDIX 1

Brexit - Brexit - Consequence Brexit - Ignorance about Brexit - Joke on Brexit - Link to article on Brexit - Link to video on Brexit - Second referendum - Voters Celebrities - Angela Merkel - Boris Johnson - Boris Johnson + link to article - Celebrities - Celebrities & link to article on Brexit - Celebrities (Sarah Palin) - David Cameron - Hilary Clinton - Lindsey Lohan - Nicola Sturgeon - Obama - Thatcher Economy - Economy - Economy - Voters & economy EU - EU - EU & Brexit - EU & NATO - EU & UN Other - Article - British housing - Citizenship - British politics - Consequences & voters - Democracy - Education

Page 54 of 76

- Football - German Foreign Office - Greece - Housing - Immigration - International politics - Japan - Joke - Leftists - Link to an article - Monarchy - Other - Personal information - Petition - Press - Racism - Refugees - Science - Scotland and Ireland - Sore losers - Texas - UN - US housing - Work opportunities Scottish referendum - Economy & Scottish referendum - Scottish referendum - Scottish referendum + link to article Trump - Brexit & Donald Trump - Brexit & Trump - Donald Trump - Donald Trump + Texas - Insults about Donald Trump - Trump - USA & Trump USA - America - American elections

Page 55 of 76

- Brexit & US - US - USA

Page 56 of 76

APPENDIX 2

stan ce tow senti Tweet ards Senti Stanc ment the ment e topic/aspec Bre iro predic predic t xit ny tions tions $2.7 TRILLION lost on global markets after negat @RupertMurdoch has his #Brexit dreams come true. People nega ive will die from impacts of such losses. economy tive 2 2 @sorrelita "protection!? Protection of what!? neutr nega #TzeGermans!?" #Brexit #funfacts al other tive 2 3 Final #Brexit tally is in: 48% Sense and Sensibility, 52% neutr nega Pride and Prejudice. al other tive 2 2 Do we even care? – #Brexit aftermath negat http://snowcalmth.online/dowecare/ Overview: - Blogposts nega ive delayed. - The Forgotten #Youth - Do we know the #EU? other tive 3 3 neutr nega #Brexit, Monty Python & Silly Walks. @NewYorker al other tive 2 3 Proud of #Britain and @David_Cameron for doing the positi posit right thing today. #Brexit #EUref #independence #freedom ve Brexit ive 2 2 @stuartpstevens @AshleyRParker I didn't know about negat neut #brexit either just some gun control stunt ive Brexit ral 2 2 negat posit "I told y'all to vote REMAIN tho" #Brexit ive Brexit ive 3 3 "The World Turned Upside Down." Not necessarily the negat nega song I wanted to be humming this morning. #Brexit ive Brexit tive 1 1 They want to put all of US to the back of the bus to their negat posit global masters. We say...piss off! #Brexit #Trump2016 ive USA ive 2 1 negat nega said of #Brexit, while in Scotland IN SCOTLAND!!!!! ive Trump tive x 1 2 Why are #liberal #democrats all over TV crying about negat #Brexit and the markets? Didnt you people want the big banks nega ive to lose money and power? economy tive x 2 2 Boris Johnson goes from court jester to crown prince after neutr celebrities/p neut #Brexit win http://bloom.bg/28RCZLw al oliticians ral 3 3 Love being in Sweden, Bruce in every bar #priceyoupay positi neut #sweden #Brexit ve other ral 2 2 Breaking!!! #UK votes to ease epcot!! #BrexitVote neutr neut Brexit #Brexit #BrexitHumor #tcot #RedNationRising al ral 1 3 Lindsay Lohan fumes over #Brexit, Elizabeth Hurley neutr celebrities/p neut

sleeps soundly al oliticians ral 2 2 @torontodan As the final lines of the old Presbyterian negat joke go: "Lord, Lord, we didna ken". "Weel, Weel, ye ken nega ive noo".#Brexit Brexit tive x 2 2 Why #Brexit is terrible for UK science, in one map negat http://www.economist.com/news/britain/21699504-most- nega ive scientists-want-stay-eu-european-experiment … Brexit tive 2 2 .@piersmorgan - I always thought #Brexit was that good negat dump you take after morning coffee. TOTALLY confused. Pls nega ive call me. Brexit tive x 2 2 Ahead on New York Tonight: Leaders react to #Brexit. neutr @POTUS designates @TheStonewallNYC as nat'l monument, neut al and a preview of #NYCPrideMarch Brexit ral 2 2 Calls for a second Scottish independence referendum neutr Scottish nega grow louder after #Brexit http://econ.st/28XxXhn al referendum tive 3 3 The nightmare of the #EU & #UN Elites is for these two negat nations to rise up and say ENOUGH! #Brexit #America nega ive #Britain EU tive 2 2 neutr neut I leave Twitter for a week and #Brexit happens. al Brexit ral 3 3

Page 57 of 76

You know what's "bizarre"? Media folks who see the negat nega #Brexit vote & take virtually no relevant lesson from it. ive Brexit tive 2 2 Watch Christiane Amanpour Get ANGRY That Britain negat nega ALLOWED The People To Vote On #Brexit ive Brexit tive 2 2 .@LizClaman: “The losses on paper are now tallying negat nega $900 billion on the U.S. stock market.” #Greta #Brexit ive economy tive 2 2 negat nega No: History will show that #brexit Is good for nobody ive Brexit tive 3 3 Take Note! Here in the States this would be the equivelant negat nega of “I thought it wud be funny to vote for Trump" #Brexit ive USA tive 2 2 Shall we start preparing EU shores for an influx of British negat nega refugees? #brexit ive Brexit tive x 3 2 I'm going to go with #brexit damage control for $1000 positi nega Alex! ve economy tive 2 2 SG #NATO Sees Unifying Role as #UK #Brexit Shakes neutr #Allies:#Britain’s vote to leave #EU leaves Europe more nega al fragmented. EU tive 2 2 Lindsay Lohan passionately expressed her stance against neutr celebrities/p neut #Brexit in now-deleted tweets http://peoplem.ag/ullfemm al oliticians ral 1 3 Celebrating #Brexit with the British Players in beautiful positi posit Kensington, MD. ve Brexit ive 3 2 Odd how some on the right are in despair over #Brexit. negat nega Socialist globalization is no way to run a planet, people. ive Brexit tive 2 2 Poor folks voting for #Brexit is the equivalent of a Turkey negat voting for Thanksgiving. White Nationalists can never be nega ive accused of rationality. Brexit tive x 2 2 Oh, God, they're giving the keys for the Tridents to BoJo negat celebrities/p nega the Clown... #Brexit ive oliticians tive 2 2 Britons voted to strengthen their borders. Will you do the positi posit same in November? #Brexit ve USA ive 3 2 Wales should have been more careful what it wished for. negat nega It's going to be given it. #Brexit ive Brexit tive 3 2 And suddenly the birds are singing.....still glued to the TV positi posit though #Brexit ve Brexit ive 3 2 The worst has yet to come #sarahpalin #brexit negat nega Brexit #exitstupidity ive tive 2 2 #Blairites using #Brexit to (yet again) try & unseat negat celebrities/p neut #Corbyn proof of their lack of allegiance to Labour. ive oliticians ral 2 2 Hey United Kingdom imma let you finish but America positi neut had one of the greatest #Brexit's of ALL TIME ve USA ral x 1 3 Good branding: #Brexit, how "attractive" is the name! + positi Good hype: Thanks to #SocialMedia. + Intense emotion: Fear posit ve = Results = #Marketing101 Brexit ive x 2 2 Really interesting piece on the #Brexit where the fragility positi neut of masculinity surfaces again. ve Brexit ral 2 2 .@jasonrileywsj: EU needed Britain more than Britain neutr needed the EU -OTR #greta #Brexit #PoliticalPanel posit al @FoxNews EU ive 2 2 negat nega Don’t think #brexit is a big deal? Here’s a chart. ive Brexit tive 2 3 Everyone should be worried about a Trump #Drumpf negat presidency! Now #Brexit! World heading in an interesting nega ive direction Trump tive 2 2 #Brexit threatens damage to U.S.-UK ties, could negat nega embolden Russia's Putin/@mattspetalnick @yarabayoumy ive Brexit tive 2 2 #Academics fear new #Brexit – a brain exit – after #referendum vote negat http://www.independent.co.uk/news/science/brain-drain-brexit- ive universities-science-academics-referendum-eu- nega a7100266.html … #EURefResults Brexit tive 3 3 The #Brexit was necessary. The EU was turning into that positi posit evil corporation from the Aliens franchise. ve EU ive 2 2 GBP weakens, world markets fall, housebuilders shares negat drop 25%. Scottish referendum 2.0? Youth vote ignored. nega ive Cameron resigns. #Brexit #Day1 Brexit tive 2 2

Page 58 of 76

The @realDonaldTrump right again. #Brexit #VoteLeave positi posit #UKIP #UK #MakeAmericaGreatAgain #TrumpTrain ve Trump ive 3 1 With all the hysteria and over the top hyperbole over neutr Brexit, maybe we should all just take a deep breath and relax neut al for a few days. #Brexit other ral 3 2 Lonng #Brexit day done, let's get back to #IREvFRA!! positi posit #Remain #EURO2016 ve other ive 2 2 Brexit vote: Anger in the bedroom, joy on the streets neutr neut http://cnn.it/28V6bCD #Brexit #seeEUlater al Brexit ral x 2 3 Trump/Sanders #Brexit philosophy: Seize corps overseas neutr ops & ban foreign sales. HUGE govt! @SMShow neut al @SueinRockville USA ral 2 2 Can Nicola Sturgeon get a Faroes-style opt out from neutr #Brexit? And might that persuade Scots we r ready 4 neut al #indyref2? Brexit ral x 3 2 On my room balcony posing with the freshly brexited EU positi posit flag. Giving it some company. #iifa2016 #iifadiaries #brexit ve Brexit ive 2 3 Do those who claim it is "stupid" to propose Esperanto as negat official for EU after #Brexit even know anything about the neut ive language? other ral x 2 2 Still so sad about #Brexit. What is this dark, absurd future negat nega being carved out for the world? ive Brexit tive 2 2 If I had a time machine, I'd happily take a molesting from negat Farage, just to Yewtree the cunt out of any political standing. nega ive #Brexit Brexit tive x 1 2 @charlescwcooke and I disagree on #Brexit. But I love negat nega his British understatement. ive Brexit tive 1 2 positi posit your country is truly inspiring thank you #Brexit ve Brexit ive 2 2 A lot of the #Glastonbury audience are wearing either negat #Hibs tops, #sunglasses or #SantaHats - I am confused in this nega ive post-#brexit world. other tive 2 2 Briton on FB claims his country was "raped" - you loose a negat posit Brexit referendum and your country was raped? #brexit #democracy ive ive 2 2 Great Britain secedes from the European Union. Millions neutr nega EU in the EU now looking for real jobs. #Brexit al tive 2 2 Massive props to David Cameron for giving Poms chance positi to decide own fate, honourably stepping aside when result celebrities/p neut ve against him. #brexit oliticians ral 2 2 #Brexit no, Corbyn has a great deal to answer for as do negat many others, its not just about the Tories however unpopular celebrities/p nega ive that view is. oliticians tive 2 2 The Working Classes will be the first to be shunted! So negat short sighted! #Brexit #shouldhavegonetospecsavers nega ive #universityofJeremyKyle Brexit tive 2 2 business: Boris Johnson goes from court jester to crown neutr celebrities/p neut prince after #Brexit win http://bloom.bg/28RCZLw al oliticians ral 3 3 I never thought of Britain as being European anyway. negat posit #Brexit ive EU ive 2 2 Instead of posting hilarious gifs maybe the #remain side negat should start thinking about the future of this country outside of nega ive the EU #Brexit Brexit tive 2 2 "World's 400 Richest Lose $127 Billion" It's happening! neutr #Brexit #LeaveWins #FirstHubrisThenNemesis neut al http://bloom.bg/28TnoIm economy ral 3 2 The Atlantic Daily: The Great British Break- neutr Off #brexit http://brexitwhatnext.com/2016/06/emthe- neut al atlanticem-daily-the-great-british-break-off/ … Brexit ral x 3 3 #Trump reaction to #Brexit: 1) See, they like my wall negat nega plan too, and 2) I'll make $$ off the tanking pound. #prick ive Trump tive 2 2 Only 72% voter turnout to a decision that literally negat nega changed the entire economy. #WakeUpPeople #Brexit ive economy tive 2 2 Do you think Boris Johnson will become the next Prime neutr Minister of Great Britain? celebrities/p neut al http://americansdecide.com/topic/do-you-think-boris-johnson- oliticians ral 2 2

Page 59 of 76

will-become-the-next-prime-minister-of-great-britain/ … #Brexit Is this going to effect my chances of getting into neutr nega Hogwarts? #Brexit al other tive x 2 2 This is one factor of the #Brexit vote. Lot more negat nega complications than this. ive Brexit tive 2 2 positi nega ' See EU Later 'was my best headline on #Brexit ve other tive 2 3 #Calls For #Texas #Independence #Surge In #Wake Of neutr #Brexit #Vote - http://www.angrysummit.com/calls-for-texas- neut al independence-surge-in-wake-of-brexit-vote … other ral 3 3 Even his wife doesn't believe him...just look at her negat celebrities/p nega expression. This man is a pariah #DavidCameron #Brexit ive oliticians tive 2 2 Assume everything said by politicians lawyers & negat nega corporate moguls is a lie until U see credible proof. #Brexit ive Brexit tive 2 2 Not liking this because it's exactly what I feared #Brexit negat nega lot have led everyone into #neverneverland OMG ive Brexit tive x 2 2 Interesting how we act on uncertainty when things are negat under control. Panic creates crisis.. instead of keeping a cool nega ive head #Brexit aftermath Brexit tive 2 2 The UK mucked up big time and the UKIP still wants negat access to the EU market? Lmao what a joke #Brexit economy nega ive #EURefResults #UKreferendum tive 2 2 positi posit Man, that was so cool. #Brexit Brexit ve ive 2 2 Putting himself 1st, Trump says #Brexit will help HIS resort: “When the down pound goes down, more people are negat coming to Turnberry, frankly" - Why…because now they have ive nega to swim the #EnglishChannel to invade Britain? Trump tive x 2 2 Sooo like everyone today, I logged in to check my 401k neutr because #Brexit...expecting it to be but is..anyone want to neut al explain? economy ral 2 2 negat nega Ya fucked up #Brexit ive Brexit tive 2 2 HAHAHA! Everyone's trying to move here (Canada), positi nega now! #Brexit #BrexitOrNot #BelieveItOrNot ve Brexit tive 2 2 neutr neut This. #Brexit al Brexit ral 2 2 Well #Brexit happened. Here's a look at the possible impacts on UK #RealEstate. neutr http://www.forbes.com/sites/carlapassino/2016/06/24/will-the- al uks-real-estate-sector-survive-brexit/#1427abb22120 … neut #pound Brexit ral 3 2 #Brexit should be a wake-up call to US #liberals: don’t assume Drumpf will lose http://www.vox.com/2016/6/24/12023816/brexit-donald- negat trump- Trump ive winning?utm_campaign=vox&utm_content=article%3Afixed &utm_medium=social&utm_source=twitter … via nega @voxdotcom #Worried tive 2 2 Hozier calls #Brexit a "massive betrayal": "My heart negat celebrities/p breaks" http://blbrd.cm/iAA1ss pic.twit... neut ive oliticians http://bit.ly/296vVcl #ShowTime ral 3 3 What does #Brexit mean to the San Francisco housing neutr market? Read more from @PacUnion Chief Economist Selma Brexit neut al Hepp: ral 2 2 Proud to be #Brexit! Proud to stand alone in a world positi where most are too scared to be alone, to have their own Brexit posit ve opinion. Proud to me! #Ukref ive 2 2 Repeated refrain re #Brexit -Elections have consequences negat -remember that in November folks #NeverTrump USA nega ive #Election2016 tive 2 2 If you care about our future join 450,000 people negat petitioning parliament for a 2nd referendum Brexit nega ive http://www.independent.co.uk/news/uk/brexit-petition-for- tive 2 2

Page 60 of 76

second-eu-referendum-so-popular-the-government-sites- crashing-a7099996.html# … #Brexit "Drunk Shakespeare, probably the only proper activity negat nega Brexit after #Brexit https://www.instagram.com/p/BHDnj20jXxO/ ive tive x 2 2 Can anyone tell me what's the status of eu member state neutr citizens now residing in the Uk? Illegal? Visa? #Brexit Brexit nega al #BrexitVote #Lexit #Leave tive x 3 2 This #Brexit thing has me really worked up. It doesn't negat nega Brexit bode well for the U.S. staying in the EU. ive tive 2 2 After #Brexit, another EU is possible with UK, Norway neutr nega Brexit and Switzerland. al tive x 3 3 I think is the natural progression of human society to negat become more integrated as time advances. #Brexit is the old Brexit nega ive world fighting back. tive 2 2 Why the surprise over #Brexit? This is the same country neutr that threw Churchill out of office after he pulled their nuts out Brexit nega al of the fire in WW2 tive 2 3 Secede and keep seceding, don't stop until you get to the negat nega Brexit individual! #Brexit ive tive x 2 2 If u are rich & white- yup! Here is #Brexit promise negat nega Brexit walked back hours after count ive tive 2 2 Brits will be poorer because #brexit but don't have to negat weigh bananas with #metric system. A wise people, indeed. Brexit nega ive @ilduce2016 @billmaher tive 2 2 #Brexit: Do you suspect that the new negotiated treaties negat will replicate membership in the European Union? Brexit nega ive #worldismovingtoofastdepartment tive 2 2 Traditionalist Catholic blog: The Filioque Clause. neutr neut other http://www.stuart-filioque.blogspot.com #PopeFrancis #Brexit al ral 2 2 This is not like Idiocracy. In Idiocracy, once they found a negat nega Brexit smart person, they made him fix their problems. #Brexit ive tive x 2 2 Nice attempt at making shit up! #Brexit #abc7chicago negat nega Brexit #iteam #millennials ?? ive tive 3 2 negat nega We won't win Eurovision for 69 years #EUref #Brexit Brexit ive tive x 2 2 Beneath the cross of Jesus, His family is my own. #Brexit neutr neut other #EUref al ral 2 2 neutr nega Britons seek to 'move to Canada' after #Brexit vote Brexit al tive 2 2 A very good cereal served at my amazing property in positi posit other Turnberry! #Brexit of Champions, just like me! Enjoy! ve ive 2 2 I'd like bier, croissant and wusrt please. What's the tinned negat nega Brexit stuff? spam? #Brexit #EURefResults #WhatHaveWeDone ive tive 3 2 I've never seen Americans talk about Britain on my negat nega Brexit Twitter feed before. And they're all taking the piss #Brexit ive tive 2 2 I did speak out on the positives of a sensible #Brexit based on negat nega Brexit democratic process. But this is just populist bollocks. ive tive 2 2 Attention fellow Scots!!! It's ok!! I've had an idea!! We negat can build a wall... #Brexit #remain other nega ive #MakeDonaldDrumpfAgain tive 2 2 I dare not go to YouTube for #Brexit videos. Can you negat nega Brexit imagine the dross? ive tive 2 2 Learning some great new swears, thanks Scotland! positi nega other #Brexit ve tive 1 2 #BrExit is exercise of the most important check on elite negat mismanagement - the peoples' power to vote for HORRIBLE Brexit nega ive IDEAS. tive 2 2 #Regrexit: Speculation grows on the options for #Brexit actually NOT happening. negat Brexit https://waitingfortax.com/2016/06/24/when-i-say-no-i-mean- ive neut maybe/ … ral 2 2 The U.K. could really use a more robust system of negat posit Brexit excheques & balances #Brexit #BrexitVote ive ive 3 3 #Brexit Is Sending Markets Diving. #Twitter Could Be negat nega economy Making It Worse http://dlvr.it/Lf6Zcl #Wired ive tive 2 3

Page 61 of 76

Brought some humor to the situation @camilluddington positi nega other #Brexit ve tive 2 2 positi posit I've changed my mind. #Brexit is good. Brexit ve ive 3 3 Brought some humor to the situation #BrexitVote #Brexit positi nega other #BrexitHumor #tcot #RedNationRising ve tive 3 2 neutr neut Okay can someone explain #brexit to me Brexit al ral 2 3 The French now want to move the Calais 'jungle' migrant neutr nega Brexit camp to British soil after #BREXIT al tive 2 3 negat nega Patsy & Edina totally voted Remain. #Brexit Brexit x ive tive 3 1 Mass referendums at their best Brits don't know what they negat voted for #Brexit #EU #EuropeanUnion Brexit nega ive #DavidCameronResigns tive 2 2 #Brexit: the day rational choice theory blew up into negat nega Brexit thousand pieces ive tive 3 2 Churchill said, "Heroes fight like Greeks". Like a Greek i positi posit Brexit have to say that "Heroes, vote like British!" #Brexit ve ive 2 2 negat nega Can we have a redo? Where is the reset button? #Brexit Brexit ive tive 2 2 #Brexit – The New Modern-Nationalism is #Global #Governance - http://www.angrysummit.com/brexit-the-new- neutr Brexit modern-nationalism-is-global-governance … al neut #ModernNationalism ral 2 3 WaPo: #Brexit vote sends a message to politicians neutr nega Brexit everywhere: It can happen here al tive 2 2 Now keep the promise of £350m a week for our #NHS - Sign the petition: #EuRef #Leave #Brexit neutr https://you.38degrees.org.uk/petitions/invest-ps350-million- Brexit al saved-from-eu-in-nhs-by-2018?bucket=fb&source=twitter- posit share-button … via @38_degrees ive 3 1 A big day with #Brexit, but I made history too when I negat bent over to tie my shoe at the gym and a guy rushed over to other nega ive ask if I was OK. #only42! tive x 1 2 negat nega The pound goes down and so do stocks #Brexit economy ive tive 3 3 Last chance to vote: Is #Brexit good for neutr @realDonaldTrump? Tweet YES OR NO using #greta USA neut al @FoxNews ral 2 2 Leftists are freaking out over the #Brexit. Why, because negat the people finally rejected tyranny? Just proves: liberals = Brexit posit ive tyrants. #tcot ive 2 2 neutr nega #Brexit #Leave Please, welcome a #Britishrefugee Brexit al tive 2 2 positi posit Congrats to the UK for #Brexit Brexit ve ive 2 2 neutr neut Last in first Out #brexit #uk #eu #eng Brexit al ral 3 3 @JaneNormanInt Need to make my order now when it is neutr still possible before #Brexit. Any plans to move your office to other neut al #EU? #onlineordering ral 3 2 ISIS takes credit for every terrible thing that happens on negat earth, but even they're saying today "don't hang that #Brexit Brexit nega ive crap on us!" tive x 2 2 negat nega #Brexit Well, that required an active stupidity that rivals Brexit ive tive x 2 2 #Brexit’s uncertainty in Europe will ripple back to Central negat nega other Texas http://atxne.ws/28T4wvL ive tive 3 3 "Migration isn't the underlying cause of the thrust towards negat nega Brexit #Brexit. Austerity is." - @yanisvaroufakis ive tive 2 2 UK want to leave the republic...aaahm eu? First thing in neutr my mind is a clone army #StarWars #Brexit #justkidding #sad Brexit neut al but that's #democracy ral x 3 1 #Brexit: Up until midnight last night #voteremain was neutr neut Brexit leading on social media: http://brnw.ch/28Tnrr1 al ral 2 2

Page 62 of 76

Beginning of the end for the European Union: Best précis negat nega Brexit of impact of #Brexit I've read to date. #corpgov #strategy ive tive 2 2 Hillary Clinton urges 'experienced leadership' after neutr #Brexit from #EU http://goo.gl/fb/0eksvp #europe Brexit neut al #europeanunion ral 3 2 #Brexit could break up #EU and #NATO, prevent World negat nega Brexit War III: Paul Craig Roberts http://goo.gl/fb/Ihi2EQ #europe ive tive 3 2 Britain: Let's grab a pint. EU: No thanks. I don't drink negat nega Brexit x during the day. #Brexit ive tive 2 2 negat nega #Brexit, the political equivalent of cat videos Brexit ive tive x 2 2 JPMorgan staff memo from Jamie Dimon, others about neutr neut Brexit Brexit referendum vote #Brexit #JPMorgan #UKref al ral 2 2 Scotland, Wales, & London voted to #Remain, everyone positi else voted for #Brexit. #Texas wants to #Secede. Can we trade Brexit posit ve Texas for those first 3? ive x 3 2 sooooooo i still have over £40 that i never exchanged for positi neut other dollars. i feel less guilty about that now. #Brexit ve ral 2 3 Hillary Clinton urges 'experienced leadership' after neutr #Brexit from #EU http://goo.gl/fb/aoXo3G #europe Brexit neut al #europeanunion ral 3 2 #Brexit could break up #EU and #NATO, prevent World neutr War III: Paul Craig Roberts Brexit neut al http://goo.gl/fb/qkMCWm #europe ral 3 2 MSM treats #Brexit as Europe's demise.History might say positi it set in motion needed revisions of Social Contract&economic Brexit posit ve machinery of Europe ive 3 2 President Vladimir Putin says #Russia has 'never neutr neut Brexit interfered' in #Brexit http://goo.gl/fb/Dq5PGX #eu #europe al ral 3 3 Top Google search in Britain AFTER the #Brexit vote negat was "What is the EU?" After the vote? #Feckin morons need a other nega ive monarch. tive 3 2 Also strenghtened by the renovated and expanded flexible neutr neut Brexit credit line with the IMF #Brexit #Mexico #PressRelease al ral 1 2 President Vladimir Putin says #Russia has 'never neutr neut Brexit interfered' in #Brexit http://goo.gl/fb/tJXPni #eu #europe al ral 3 3 Boris Johnson goes from court jester to crown prince after neutr celebrities/p neut #Brexit win http://bloom.bg/28RCZLw al oliticians ral 3 3 I h8 the phrase "take back our country," whether it's used negat 4 the US or the UK bc it's fear-mongering by spreading hate 4 Brexit nega ive foreigners #Brexit tive 2 2 My vote was to be free of unelected EU commissioners positi passing laws that our country has no say in. Glad to be out, Brexit posit ve they need us more. #Brexit ive 2 2 @truthout It hasn't even been 24 hours and you're judging neutr posit Brexit the outcome? Britain hasn't even officially the left EU yet. al ive 2 2 Hillary Clinton represents the crony capitalist kleptocracy negat celebrities/p nega the author identifies as responsible for #BREXIT. DOA. ive oliticians tive 2 2 The latest The Sciarra Stefano Daily! http://paper.li/Colonnasciarra/1334314750?edition_id=998632 neutr other 10-3a67-11e6-92d0-0cc47a0d1605 … Thanks to @rpinci al neut @VentagliP @Surfiniae #brexit #business ral 3 3 True elites want to run their own lives and own countries positi neut Brexit not be told by central government what to do. #Brexit ve ral 2 1 After failure to get into #NSG, MODI JI should start neutr pushing to get a entry into #EU, Britain's vacant place awaits Brexit neut al for India. #Brexit ral 2 2 The idea that UKIP will now disband, its mission negat accomplished, is delusional. What emboldened reactionary Brexit nega ive ever gave up their fight? #Brexit tive 2 2 “I still have my ice cream. How can #Brexit be a big positi neut Brexit deal.” - @yogurtearl, while eating ice cream. ve ral x 2 2 Sacramento: Tune into @kcranews at 5PM to catch our neutr very own Theo Slater provide our campaign's reaction to Brexit neut al #Brexit #BrexitVote. ral 1 2

Page 63 of 76

http://Brexit101.com is for sale neutr http://ht.ly/LS98301BKIq #Brexit #BrexitVote #UK #Trump other neut al #Cameron #Britain #EU #Greece #FX #Zika #Grexit ral 2 2 #Brexit Get out of the stockmarket now! Go see your negat nega economy local ResiShare agent http://bit.ly/28IlT3x ive tive 2 2 ICYMI- Dow plunges 611 Points, British PM David neutr Cameron to resign, England to free from EU... #brexit lots Brexit neut al going on http://fb.me/7RgbzHxU0 ral 2 2 Amazing, Blair involved in illegal war killing 100s of negat celebrities/p thousands no 1 tried 2 remove him,Corbyn in job a wet week, nega ive oliticians coup against him #brexit tive x 2 2 So does this mean we get a football team in England? neutr posit other #Brexit #Nfl al ive 3 3 I just read that France is planning to send the thousands of neutr neut Brexit refugees in Calais towards UK. #brexit #immigration al ral 2 2 What the hell did you do uk? #brexit #banksy negat #yearofthemonkey #day137 #stupid #uk #eu #exit… Brexit nega ive https://www.instagram.com/p/BHDnfFsj8DM/ tive 2 2 #Brexit is the warning to the EU leadership after Greek negat referendum. They did not get it last year I hope they do now! other posit ive @EU_Commission ive 2 2 The frustrated Cold War warriors seeing #Brexit through negat the prism of their paranoia about Vladimir Putin appear to be Brexit nega ive revving their engines. tive 2 2 @ABCNews24 LOL give me a 'Dog's Brexit' anyday over negat an AbbottTurnbull government mate! #brexit was Rich v's Brexit nega ive Poor, Far Right v's Centre Left tive 1 1 Local Leave supporters 'pleased' & 'happy' with #Brexit positi referendum result: http://chattelevision.ca/__news/local-leave- Brexit posit ve supporters-pleased-and-happy-with-eu-referendum-result/ … ive 3 3 Oh no #Brexit , how could you do this to large investors! negat Today was mildly annoying, caused by "uncertainty". economy nega ive #traderphobic #EUref #UK tive 2 2 Now keep the promise of £350m a week for our #NHS - Sign the petition: #EuRef #Leave #Brexit neutr https://you.38degrees.org.uk/petitions/invest-ps350-million- Brexit al saved-from-eu-in-nhs-by-2018?bucket=fb&source=twitter- posit share-button … via @38_degrees ive 3 1 #Brexit sounds like a breakfast cereal #comedy neutr neut Brexit https://vine.co/v/5u7ntpa5B3U al ral 3 2 MUST-READ #Brexit commentary: "But first, we will neutr neut Brexit have to think, probably more deeply than ever." al ral 2 2 I wanna find my fellow Jubilee line riders who took down neutr neut the drunk Welsh #brexit supporter. al other ral 2 2 Crude oil prices slammed after Britain votes to leave EU negat http://klou.tt/104hlokjipjzm #brexit #oilprice #oilandgas economy nega ive #petroleum tive 3 2 positi nega Wonderfully succinct statement #Brexit #wtfbritain Brexit ve tive x 2 2 You know, seriously, it's vapid & arrogant comments negat nega Brexit from talking blow-dry heads like this that CAUSED #brexit. :/ ive tive 2 2 #brexit today's removal of Paul Day's St Pancras "Meeting neutr neut other Place" lovers statue. Things are happening fast. al ral 2 3 Watching Underworld at #Glastonbury2016 tonight in the negat aftermath of #Brexit, I miss the 90's more than ever. Such Brexit nega ive innocent times. Such hope. tive 2 2 The club regret to confirm that the transfer of negat @Ibra_official has collapsed due to the fall in value of the other nega ive pound as a result of #Brexit. tive 2 2 We owe so much to Nigel Farage for his unwavering positi commitment to the #Brexit cause. What an historic day for Brexit posit ve Britain #IndependenceDay ive 2 2 Saying "everyone" online talking about #Brexit is as negat smart as their dog inherently is discrediting those who are other nega ive talking tive x 2 2

Page 64 of 76

@realDonaldTrump so busy promoting his biz, his initial negat nega Trump reaction to #Brexit was, "there's nothing to talk about." ive tive 2 2 Honestly most elites seem to think everyone needs a negat daddy mommy bureaucrat to manage their lives for them. More Brexit nega ive effete than elite. #Brexit tive 2 2 New pictures of referendum result emerge... #brexit neutr neut Brexit @Oldfirmfacts1 al ral 2 2 This prerequisite of being a sociopath to have power is negat nega Brexit getting a bit tedious now. #Brexit ive tive 2 2 .@SketchesbyBoze Perhaps this whole mess could've negat nega Brexit been avoided if #Brexit was named Brexity McBrextface ive tive x 3 3 END OF THE EU? #Germany warns FIVE more neutr countries could leave Europe after #Brexit | World | News | Brexit neut al Daily Empress ral 3 3 negat nega I regret voting for #Brexit under these false pretenses. Brexit ive tive 2 2 Britain will be better off just like a spun out Corp from a positi posit Brexit conglomerate, give 'er a day or two ;) #Brexit ve ive 2 2 there's no perfect society but Britain voted for worst out negat nega Brexit of two. However 48% of UK ppl ain't happy with it #Brexit ive tive 2 2 I heard this great line on the @marklevinshow "when negat immigrants don't assimilate it's called colonisation." Brexit nega ive #immigration #Brexit #vidcon2016 tive 2 2 So, appears I successfully avoided making premature, not positi informed enough direct correlation tween #brexit & U.S. 2016. other nega ve Mission Accomplished tive 2 2 positi posit With revolution life is so much better #Brexit Brexit ve ive 3 2 The UK is like a kid that's been threatening to run away negat nega Brexit from home, finally do & then immediately regret it #Brexit ive tive 2 2 Yep, him too. Saw a woman on #newsnight crying with positi nega Brexit joy because she thought #Brexit had saved the NHS. ve tive 2 2 A day full of sad news. #Brexit then @Yellowcard negat announce their end. Last tour tickets go on sale on payday, other nega ive seems like fate. Have to go! tive 2 2 Hey older generations in the UK. Thanks for putting that negat final nail in the coffin for the rest of us...You CUNTS. Brexit nega ive #BrexitVote #Brexit tive 1 3 The #majority imposed their will on a very significant negat nega Brexit minority in the #Brexit referendum ive tive 2 2 holy shit Nigel Farage has an Alan Partridge voice you neutr celebrities/p nega should have picked up on this British people! #brexit al oliticians tive x 2 2 @realDonaldTrump You had NO FUCKING IDEA what negat #Brexit or #BrexitVote was 2 days ago. LMAO Trump nega ive duhhhhhhhhhhhhhhhhh tive 2 2 Well well well, an appropriate song for the #Brexit result! positi nega Brexit Sex Pistols Anarchy in The UK ve tive x 1 2 negat nega Sorry to see this happening... #Brexit Brexit ive tive 3 3 Is this #Brexit going to mess with my UPC/VirginMedia negat nega because my internet has been shit all day ive other tive 3 2 negat nega Dominar Farage strikes again... #farscape #Brexit #EUref Brexit ive tive 3 3 #Brexit #GOT #GameofThrones The 'Brexit' referendum and 'Game of Thrones' aren't all that different neutr other http://mashable.com/2016/06/23/brexit-game-of- al nega thrones/#MQSrpW_B805A … via @mashable tive 2 2 Mark it down, folks who are against #Brexit & calling the negat #Leave folks racist, bigots, & xenophobes don't have a clue Brexit posit ive about the real issues ive 2 2 neutr neut Buy Pounds .... Save Euros ... #Brexit economy al ral 3 2 By George, he gets it! Britain voted for #Brexit because it positi nega Brexit wants to be #Canada http://pllqt.it/Tn6552 ve tive x 2 2

Page 65 of 76

IMO David Cameron & Jeremy Corbyn should've put negat politics & campaigned together on the platform of staying in Brexit nega ive the EU... #Brexit #BrexitVote tive 2 2 Dow jones took a MASSIVE blow today due to #Brexit. negat nega economy This is a sneak peak at what would happen under Trump. ive tive 3 2 Victoria Nuland, Assistant Secretary of State, supporting neutr neut Brexit #Brexit in 2014, before it was mainstream. al ral 2 2 #Brexit has led to some great memes #BrexitVote positi neut Brexit #politics ve ral 3 3 This Dude Who Thought His #Brexit Vote Wouldn't negat Matter Is A Valuable Lesson For All Of Us other nega ive https://www.mhb.io/e/1c6ei/cp tive 2 2 #Brexit is really complicated. I think the UK will be better positi posit Brexit off in the long run, but I'm not certain. ve ive 2 2 Hillary picked to be on the wrong side of history for the negat celebrities/p nega 951st time. @realDonaldTrump #Brexit ive oliticians tive 2 3 Right, off to bed. Pretty tired so it'll be nice to lay my positi head down on my soon to be de-regulated pillow other posit ve @iamjohnoliver #Brexit ive 1 2 Going in on @bondibeachradio in a couple of minutes neutr neut other with #brexit bangers. Some geezer classics, some euro power. al ral 3 2 Did #vapers sway the #brexit vote? Perhaps everyone has negat nega Brexit to deal with #Article50 because of #Artilce20 of the #TPD ive tive 1 2 #Brexit example of democracy in action. You must suck positi posit Brexit and shut up ve ive 2 2 neutr neut #Brexit today, #Libexit in a week. #auspol #ausvotes other al ral 3 3 From Rule Britannia to Cool Britannia to Fool Britannia negat nega Brexit #brexit ive tive 1 1 negat nega Sigh. You had one job; #Britain #brexit Brexit ive tive 2 2 Surprised #Sutton was one of few #London boroughs to negat vote #Leave as was a @LibDems seat until 2015 #Brexit Brexit neut ive @scullyp ral 1 2 The #EU is far more about the New World Order of One negat World Government than it is about a Global economy. #NWO posit ive #Brexit @morningmika @brithume EU ive 2 2 neutr neut #Brexit #history - future history exam question(s) other al ral 2 2 Super serious situation, but I chuckled. #Brexit Peace to positi nega Brexit all OUR UK family! #Repost @tcb… http://itsOURshow.net ve tive 2 2 TheEconomist: Calls for a second Scottish independence neutr Scottish neut referendum grow louder after #Brexit … al referendum ral 3 2 Before and after, my grass lost the referendum too negat nega Brexit #grassxit #brexit , see what's left? ive tive 2 2 Damn you #Brexit supporters. I lost almost $10K in my negat retirement today because of you and I'm from the U.S. I hope Brexit nega ive you feel a similar pain! tive 2 2 Does spellcheck have any part in Western Civilization? neutr neut other #Brexit al ral 2 2 Nice work @CommBank leaving customers stranded negat without access to their OWN money in UK cos it costs YOU other nega ive $$ #Brexit tive 1 1 Age old struggle between freedom vs. security. When positi security just wasn't all that secure the people chose freedom. Brexit posit ve #Brexit ive 2 2 #Brexit generational gap is unbelievable - negat neut Brexit http://www.cbc.ca/1.3650826 ive ral 2 2 #Brexit: The Consequences Of Xenophobic Nationalism https://theobamadiary.com/2016/06/24/brexit-the- negat Brexit consequences-of-xenophobic-nationalism/ … via ive nega @TheObamaDiary tive 2 3 Right behind you Marsha and Robert!! positi posit USA #MakeAmericaGreatAgain #Brexit ve ive 3 1

Page 66 of 76

Haven't seen much excitement over the #Brexit vote. negat Mostly a lot of despair and anger. I don't know if I should feel Brexit neut ive sorry for the UK. ral 2 2 @EricIdle #Brexit voters reminds me of the Crimson negat Permanent Assurance pirates. Have they interrupted our lives Brexit nega ive and ruined the world? tive 2 2 END OF THE EU? Germany warns FIVE more countries neutr neut other could leave Europe after #Brexit al ral 3 3 The #brexit discussion on Hardball dismissing Bernie and negat both D & R voters for not embracing the "way we do things". USA neut ive Still don't get it! ral 2 2 Education matters, sure, but it's also a privilege. Scary negat thing is that it's been becoming more and more of one. So ... nega ive your move? #brexit other tive 2 2 How do you two on 'team globalist' feel about being on negat neut Brexit the wrong side of history today? #Brexit #IndependenceDay ive ral 2 1 negat nega #Brexit Britain's biggest mistake since the 1773 Tea Act Brexit ive tive 2 2 I take this as Target forewarning us of the disastrous negat nega Brexit #Brexit fallout. Just the title, not the hackneyed plot. ive tive 2 2 Look out for Ed Sheerans Brexit cash in single "You need neutr me, I don't need EU" #edsheeran #Brexit #BrexitVote Brexit neut al #EUreferendum #EUref #jokes ral x 2 2 Up until this weekend I thought #Brexit was some sort of neutr neut Brexit cracker for tea. al ral x 2 2 don't even use #Brexit you were for Remain your negat posit Brexit continual lying will not get you anything traitor! ive ive 2 2 Jermaine Pennant asks the most important question of the neutr day http://dailym.ai/28RBRZW via @MailOnline #brexit other neut al #hesaidwhat? ral 2 2 If the EU savings aren't going to the NHS as promised, neutr nega Brexit where pray tell are they going? #Brexit al tive 2 2 #IVotedLeave #England #brexit I'm not British but i hope positi posit Brexit the best for England ve ive 2 2 Being united w other countries is something so important negat - it provides a sense of safety & solidarity. I am so, so sorry Brexit nega ive Britain. #Brexit tive 2 2 Of all UK tweets responding to #Trump stupidity, this is positi my fave: "Scotland hates both #Brexit and you, you mangled, Trump nega ve apricot, hellbeast." tive 2 2 i miss original recipe steven colbert. hed be killing #brexit negat watching him on the late show feels like watching jordan play other nega ive baseball tive 2 2 neutr neut Stay tuned for some amazing Crop Circles. #brexit other al ral 1 3 Disaster Ahead! Trump on Brexit: America is next negat neut Trump http://cnn.it/28VPZjy #Brexit #Election2016 ive ral 3 2 neutr neut #Brexit Won! What does it mean to #domain investors? economy al ral 2 2 Or is it the fear of Changing the Paradigm or is it the Fear neutr that the Paradigm will NEVER change? #BrexitVote Brexit neut al #BREAKING #Brexit ral 2 3 EU can’t go on forever without earning consent of the negat governed: @InklessPW http://on.thestar.com/28WI4Cr #euref Brexit posit ive #Brexit #democracy #consent ive 2 2 Hats off to the German Foreign Office! #BrexitVote positi neut #Brexit #EURef ve other ral 3 2 #Brexit & #Czechout & #Finish, oh my! Denmark too. Is neutr neut Brexit #EUXit next? #MAGA #Trump2016 al ral x 3 2 #Brexit #DavidCameron I wonder will Cameron be see negat as. #nevillechamberlain figure "Europe in our time" figure. Brexit nega ive Disaster #PM tive 2 2 Just kidding One of the key reasons you have to be careful negat nega Brexit when trusting populist propaganda. #brexit ive tive 2 2

Page 67 of 76

LOL. I heard someone say, "make England great again." negat That sounds awfully familiar... #Brexit USA nega ive #MakeAmericaGreatAgain tive 3 3 1/ A possible historical parallel to London post #brexit is neutr neut Brexit Montreal al ral 2 2 At what point does #Leave #Brexit get called out for flat negat out lying in campaign commercials? Nope. NHS won't get that Brexit nega ive money. #UKreferendum tive 3 2 British Lose Right to Claim That Americans Are Dumber http://www.newyorker.com/humor/borowitz-report/british- neutr Brexit lose-right-to-claim-that-americans-are-dumber … via al nega @BorowitzReport #Brexit tive x 3 2 "#Brexit is not gonna play well in Washington." negat nega Brexit #BBCWorld ive tive 2 2 Should we start calling this what it is - the rise of fascism negat nega Brexit in #Britain #brexit #EURefResults ive tive 2 2 After #Brexit? Departugal. Italeave. Fruckoff. Czechout. neutr neut Brexit x Oustria. Finish. Slovlong. Latervia. Byegium. = Germlonely al ral 3 3 Separatists in Scotland and terrorists in Northern Ireland negat are cynically calling for new referenda to achieve their raisons other neut ive d'être. #Brexit ral 2 2 Don't like bureaucracy but a lot of #Brexit/Trump votes negat are based on fear and hate. That's what I'm against more than Trump nega ive anything. Depressing. tive 2 2 Just something to lighten the mood a little bit. positi @naomivowles you can never go wrong w/ Spice Girls #Brexit other nega ve http://www.youtube.com/watch?v=SoxxHeBJmz8&sns=tw … tive 3 2 BostonGlobe: After #Brexit, European property investors positi neut economy may see better value — and stability — in Mass. … ve ral 3 2 #Brexit would have been defeated if #EU had accepted negat NO to #treaties,listened to ppl & reformed.predict EU nega ive #dominoeffect tive 2 2 People "going with thier gut" instead of their brain has negat nega Brexit never lead to anything other than rasict dumbassery. #Brexit ive tive 2 2 positi posit I wanted #Brexit hugely. Brexit ve ive 3 3 @MSNBC @amjoyshow Lindsey Lohan tweeting about negat #Brexit is no more “Breaking News” than me tweeting about celebrities/p neut ive it! oliticians ral 2 3 @realDonaldTrump Florida is in Scotland? Or is Scotland negat nega Trump in Florida? I'm so confused. #brexit #yousirareanidiot ive tive 2 2 Sweet, I just found £8 down the back of the sofa. Thanks positi neut other #Brexit ve ral x 2 2 How can we criticize the #brexit when our president flies negat nega Trump on a plane named after a shoe ive tive x 2 2 Serious question: does anyone have a time machine? negat nega other #Brexit ive tive x 2 3 BostonGlobe: #Brexit was spawned by tensions over negat nega Brexit globalization, President Barack Obama says … ive tive 2 2 #Brexit is all shits & giggles to me until I check my BT negat nega Brexit European funds ive tive 3 2 lol. Looking forward to the day the english come crawling positi nega Brexit back. #Brexit #Leave #brexitfail ve tive x 3 2 Almost incredible that none of the #Eurocrats seem to negat have had a contingency plan for #brexit, but should have Brexit nega ive expected it. tive 2 2 I use to think I was proud to be British but not proud of negat nega Brexit Britain, but now, im not so sure about the second part #Brexit ive tive 2 1 if someone had said that today’s economic situation negat would be the immediate outcome of #brexit how many would economy nega ive have believed? (vs more FUD) tive x 2 2 Out of the #Brexit ashes I feel sorry for the Japanese. The negat Japan economy desperately needs a lower Yen but the Brexit neut ive computer says "No" economy ral x 2 2

Page 68 of 76

This is a pretty perfect response. @dominicnahr #brexit positi #montypython other neut ve https://www.instagram.com/p/BHDnVGoA3Ve/ ral 3 2 Finally logged off from work. Thanks for the overtime positi neut cash #Brexit ve other ral 2 2 negat nega There is some satire in this #brexit other ive tive 3 3 neutr neut Rise of #Bitcoin and #Gold as #Brexit Turns into Reality economy al ral 2 2 neutr posit Israel and America now should leave the UN #Brexit other al ive 3 2 It can't be "take back control" for England and "lose all negat control" for Scotland & NI. #UnitedIreland #Indy2 #IndyScot Brexit nega ive #Brexit tive 2 2 considering the breakdown of voting in #Brexit I propose neutr a new country form and its name should be Scotirelondon Brexit nega al #scotirelondon tive x 2 2 negat nega Have the four horseman appeared yet? #Brexit ive Brexit tive x 2 2 When we leave EU we won't be protected by the Privacy negat Shield agreement with the US. Goodbye privacy, thanks Brexit nega ive #Brexit. tive 1 2 I think if #Texit happens it will start a domino effect just neutr nega other like #Brexit did al tive 3 2 The only good thing to come out of the #Brexit is the positi dearth of insults being hurled at @realDonaldTrump by the nega ve lovely people of Scotland Trump tive x 2 2 #Canada pls keep it together so we have somewhere to go negat nega if Drumpf becomes POTUS #Brexit :( ive Trump tive 3 2 negat nega Come to think of it, #Brexit does have a smoky taste to it ive Brexit tive 2 2 $2.7 TRILLION lost on global markets after negat @RupertMurdoch has his #Brexit dreams come true. People nega ive will die from impacts of such losses. economy tive 2 2 Stop being afraid of what could go wrong and start being positi posit excited about what could go right . #Brexit #VotedLeave ve Brexit ive 3 2 "Democracy is the theory that the common people know neutr what they want, and deserve to get it good and hard." -H.L. posit al Mencken #Brexit #BrexitVote other ive 2 2 Thought of #TREXIT tantalizing, not happening. After negat #Brexit the Scots hate him, where shall The Donald go? nega ive Instead #ImWithHer Trump tive 2 2 Despite the alerts, did not open Robinhood once today. positi nega #Brexit #dontpanic #letthatpoundcomedown #ineverbeentogb ve other tive 2 2 ..And the Oscar as worst world leader ever goes to.. negat celebrities/p nega #Cameron #Brexit #ByeByeUKEP #UKreferendum ive oliticians tive 2 2 U2 said it best "And I wait without EU / With or without neutr neut EU" #Brexit al Brexit ral x 3 2 My point is the racists that have always been there now negat nega seem to think it is acceptable to be openly racist since #Brexit ive other tive 2 2 Sloooow down #MariaBartiromo...it's not the end of the positi posit world yet. #Brexit #greta ve Brexit ive 2 2 #Brexit' to be followed by Grexit. Departugal. Italeave. neutr Fruckoff. Czechout. Oustria. Finish. Slovakout. Latervia. neut al Byegium. Brexit ral x 3 3 isn't that exactly what voters HATE. Pos afraid for negat THEIR political futures..the country's be damned, be it #brexit nega ive or NRA Brexit tive 2 2 The first task of the new #ToryLeadership will be how to neutr nega diminish #UKIP #bestofenemies #Brexit al Brexit tive 3 3 I bet right now almost everybody in Florida is looking at negat this whole #Brexit thing and thinking, "Aw, man, did I just shit nega ive my pants again?" Brexit tive x 2 2 Every time someone says #brexit our cats lift up head neutr with big eyes as if it means #biscuit ... at this rate they'll never neut al learn English. other ral x 3 3

Page 69 of 76

#Brexit was just a massive hoax so we could have a negat nega global meme competition. ive Brexit tive x 2 2 Does this mean no more Tim Tams? #Referendumb negat nega #Brexit ive other tive x 3 3 Brits were tired of hearing from "outside experts" #brexit negat celebrities/p nega #lohanknew ive oliticians tive x 2 2 I bet that's not the first time Obama's been mentioned in neutr celebrities/p neut the same sentence with an "eight ball." #Brexit al oliticians ral x 2 2 I know it is not good for me, but, on days when Britain negat chooses to #Brexit, I like to drink a Coke and eat a cookie. nega ive #EURefResults other tive 2 2 On some level UK after #Brexit must feel a little like US negat after GW Bush reelection. Mainly in how the rest of the world nega ive is like "Seriously?" Brexit tive x 2 2 No one saw #Brexit coming. No one will see FBI negat indictment recommendations of Hillary coming. 2016 is a wild nega ive year. Be ready. USA tive 2 2 The average life experience of those whining about negat posit #Brexit seems to be about 12 years. ive Brexit ive x 2 2 #OutMeansOut - Let's leave this mess before it all negat posit collapses all around us. #LeaveEU #Brexit ive Brexit ive 3 3 As someone who works closely with highly skilled positi migrants, I can say I value your work and you are an asset to nega ve this country #brexit other tive 3 2 Did that 1% growth from the $50billion business tax cut neutr after 10 years include modelling in a #brexit scenario? neut al #ausvotes economy ral x 2 2 Sebastian was right, can I become a mermaid now pls negat nega #Brexit #EURefResults ive Brexit tive x 3 2 If some of you people had your way, the USA would've positi never existed. Would've been happy paying taxes to the British nega ve for a lifetime. #Brexit Brexit tive 2 2 #HiddleSwift or #Brexit, don't make me choose! neutr (Meanwhile, in America...) #TGIF and the markets are closed, nega al whew! other tive x 2 2 Don't brex my heart, Never leave me again #Brexit neutr neut #BrexitVote #ToniBraxton #Nailedit #Youaresonotfunny al Brexit ral x 2 2 When you see that Trump endorsed UK leaving EU, you negat nega can realise how stupid that idea it is. #Brexit ive Trump tive 2 2 negat nega its begun, The Begining of the END #Brexit #Damn ive Brexit tive 2 2 Now keep the promise of £350m a week for our #NHS - Sign the petition: #EuRef #Leave #Brexit neutr https://you.38degrees.org.uk/petitions/invest-ps350-million- al saved-from-eu-in-nhs-by-2018?bucket=blast&source=twitter- posit share-button … via @38_degrees Brexit ive 3 1 #Mckinney #FreddieGray #GeneleLaird #Brexit negat #immigration The news has been exhausting these past few nega ive days other tive 2 2 All I have to say about #Brexit #love #ignorance #hate neutr #nofear #peace #tolerance… nega al https://www.instagram.com/p/BHDnRjwA-_k/ Brexit tive 1 3 So Black & brown ppl should stay in their "native" negat countries? I could be ok w/ that, white ppl, if you also stay in nega ive "yours." #Brexit #Trump Trump tive 2 2 Churchill must be smiling down on #obama as Briton positi follows her history & refuses to surrender, once again #brexit posit ve #tcot #p2 #gop #dem Brexit ive 2 2 #Cameron & #Osborne tried to call the bluff of Tory negat nega troublemakers for party reasons -it's backfired badly #brexit ive Brexit tive 2 2 Brexit big blow to UK science, say top British scientists- https://www.theguardian.com/science/2016/jun/24/brexit-big- negat blow-to-uk-science-say-top-british-scientists?CMP=twt_a- ive nega science_b-gdnscience … #Brexit #Science Brexit tive 2 2 This outcome was set from the stary, calling it #Brexit neutr nega instead of Bremain. al Brexit tive 2 2

Page 70 of 76

#Brexit affirms what's innate about bureaucracy - in spite negat of intentions it grows, empowers elites, squelches public and nega ive self determination Brexit tive 3 3 I guess all those UK Citizens of every color voting for negat nega #Brexit were racists too? ive other tive x 2 2 "I'm divided." British expat in Co talks about uncertainty negat neut in the UK after #Brexit. Her story @6. ive Brexit ral 2 2 #BREXIT good timing!!! EU nazi caliphate waiting just positi posit on other side of the channel. @FoxNews ve Brexit ive 3 2 David #Cameron, you are so outta here! #Brexit #EU negat nega #EUreferendum #notmyvote #Johnson ive Brexit tive x 2 2 We may end up w/ @BorisJohnson and negat @realDonaldTrump WOW!!! FDR and Churchill r checking celebrities/p nega ive w/ God about reincarnation #Brexit #insanity oliticians tive x 2 3 Relatedly I’m going to compile a reading list for myself neutr on possible EU reforms bc clearly that was a huge neut al (understandable) #Brexit factor. other ral 2 2 neutr nega So...... the Queen kills everybody now, right? #Brexit al Brexit tive x 2 2 positi posit #Brexit well wishes from the neighbours! ve Brexit ive 2 2 BRB. Reading all about #Brexit and possibly planning a neutr neut trip to England ASAP before things get too crazy. al other ral 3 3 I bet a lot of UK businesses are feeling pretty bummed negat about their .eu domain extensions... #brexit #EURefResults nega ive #EUref #firstworldproblems other tive x 2 2 This bozo found a legal loophole... #Brexit but with the negat nega "benefit" of unfettered immigration into the UK... ive Brexit tive 3 2 Nice to see how it takes #brexit to make @guardiannews negat see the private sector as more than just an iniquitous force that nega ive needs to be taxed Brexit tive 2 2 Looks like the Coudenhove-Kalergi plan and his book negat Praktischer Idealismus for the European Union are failing nega ive thanks to Britain. #Brexit other tive 2 2 Whatif?@bernardchickey #Brexit #RemainINEU UK had neutr neut heard non political/self-interest view, just facts - same result? al Brexit ral x 2 2 neutr neut Day 1 of #brexit Still in the EU. al Brexit ral x 3 2 Following that course, disintegration of social bonds and negat solidarity is the result. That's where we are where we are. nega ive #Brexit Brexit tive 2 2 neutr neut Will England remain at Epcot Center? #Brexit al Brexit ral 2 2 I am starting to learn #French and using more of my negat nega #german ... #Brexit #tragic ive other tive 3 3 @David_Cameron 's pull out game is really weak. You're negat supposed to pullout BEFORE you mess up "her" life. RIP nega ive Britain. #BrexitVote #Brexit Brexit tive 2 2 So when do we start sending ration packs to the UK. neutr nega #Brexit al Brexit tive x 1 1 Soooo what if Trump bought Texas and renamed it neutr nega Trumpxas. Then we can vote for a #Trexit... #Brexit al Trump tive x 2 2 I just woke up and there is a severed horse head next to negat me... What does this mean? Did I piss off the Mafia? #help nega ive #Brexit #Mafia #horse other tive x 2 2 Two #Brexit fears are hard to reconcile 1. Scotland, N negat Ireland leaving the UK to join EU while 2. EU countries like nega ive France move toward exit. Brexit tive 1 2 His cluelessness of #Brexit neg impact on Sctland as he negat nega talked is n xmple of infantile diplomacy. ive Trump tive 2 2 What can the EU do that it never did before against ISIS, negat #Brexit changed nothing UK doing all heavy lifting and paying posit ive EU invites them in EU ive 2 2 I just hope my 401(k) didn't shit the bed today over negat nega #Brexit. It probably did tho. ive other tive 2 2

Page 71 of 76

“Anyone else think #brexit sounds like a super healthy neutr digestive type of biscuit? 'Ooeck owsabout you pass me the... nega al http://fb.me/5jalEFbdB Brexit tive x 2 2 France is now the 5th largest economy in the world, took positi nega just 5hrs, thanks to #Brexit ve economy tive x 1 2 I want #AMERICA back!! #MakeAmericaGreatAgain neutr posit #MakeAmericaSafeAgain #Brexit al USA ive 1 1 .@lsarsour #Brexit is, or should be, a first step to local positi autonomy and bio-regional governance, without which posit ve #sustainability is impossible Brexit ive 3 3 Seems like my boys spent most of their time at HSS for negat the last week or so. Wall Street was like being at a wake today. neut ive #Brexit economy ral x 2 2 If Trump and the French national front celebrate the brexit negat celebrities/p nega vote.... You know you've fucked up royally.... #Brexit ive oliticians tive 2 2 The latest The Makam News! neutr http://paper.li/igbariam/1347306962?edition_id=3e09c910- neut al 3a67-11e6-b556-0cc47a0d15fd … #nbadraft #brexit other ral 3 3 We'll find out the effect of #Brexit on the performance of neutr British national teams very soon. #EURO2016, #England, neut al #Wales, #Ireland, #NIR other ral 3 2 Why are people encouraging the dismantling of the EU? negat nega #Brexit ive Brexit tive 2 2 #greta who would you rather making the deals with neutr #Brexit @HillaryClinton or @realDonaldTrump this kind of neut al skews the election USA ral 2 2 If no fed rate hike now, mtg. rates should stay around negat historic lows for a while. How long though and is that really nega ive beneficial? #Brexit Brexit tive x 3 2 Just watched Jim Cramer on this morning's neutr @TheTodayShow telling an astonished Matt Lauer that the nega al economic impact of #Brexit is not so bad. economy tive 2 2 Americans have no respect for Obama why would the negat celebrities/p nega Britts #Brexit #greta #Trump2016 ive oliticians tive 2 2 Just for clarification since I've gotten 10+ messages on negat nega this: No I don't think #Brexit was solely due on racism, clearly. ive Brexit tive 2 2 Colonises half the World and complains about negat nega #immigrants. #Brexit ive other tive x 3 2 Seriously, this is so retarded my head hurts. #brexit got negat posit the most votes on a referendum. You lost. Cry elsewhere ive Brexit ive 2 2 Ahh @JoyAnnReid brings up the “Anti Expert” element negat of #Brexit. I look around & see debate against science & nega ive experts in my own state often Brexit tive 2 2 The fascist, one-world tyrant-lovers cuss a lot when they negat venture outside of their safe place to complain about #texit or posit ive #brexit. Haha Brexit ive 2 2 We should have right to retain our EU citizenship even neutr nega after #Brexit. al Brexit tive 2 2 Brits who voted to leave in #Brexit, to be safer, apparently negat nega never heard of #DivideAndConquer ive Brexit tive 2 2 #Brexit I'm American, but I am also (by decision) Greek, negat French, Italian, Spanish, Dutch, and Belgian. But I will never neut ive be an #Englishwoman. other ral 1 2 American Media narrative bout #Brexit focuses on negat THEIR stupidity while reportin Lohan tweet, views of nega ive presumptive nominee Trump #GlassHouses USA tive 2 2 Utility stocks, along with Treasury bonds, serve as safe neutr haven during tumultuous times. @SmithRebecca neut al http://ow.ly/ldWK301CoUC #brexit economy ral 3 3 This is Great Britain not a 3rd world country. The elite negat selling today will be buying next week, part of their scare nega ive tactics. #Brexit economy tive 2 2 Can we solve #Brexit issues using the old "reset" method? neutr Switch off, leave Europe for 10 secs then plug ourselves bck nega al in? Oh wait... oops? Brexit tive x 3 3

Page 72 of 76

Shot Across #liberal Bow: Brits decided to ‘Make neutr England Great Again' →NEXT-UP #MakeAmericaGreatAgain nega al #Brexit #NEVERhillary @LouDobbs @greta USA tive 2 3 Hoping for Steve Harvey to come out and say the neutr #EUreferendum result was wrong lol. #Brexit #EURefResults nega al #NotMyVote Brexit tive x 2 2 Am I reading that right? Is the fact that Lindsay Lohan negat tweeted about #brexit "Breaking News." Or did I have a celebrities/p neut ive stroke? #msnbc oliticians ral 2 3 Salute to British state and people to sticking to will of negat nega majority and democracy, come what may #Brexit #Uk ive Brexit tive 2 2 I gotta be honest, it's bit of a relief to get confirmation that positi nega America isn't the only country with stupid citizens. #Brexit ve USA tive 2 2 neutr neut Last in first Out! #brexit al Brexit ral 3 3 Hey, United Kingdom, Imma let u finish, but USA had positi neut the greatest #Brexit of ALL TIME! ve USA ral 1 3 oh boy, #texas is inspired to secede again based on negat #Brexit. Not only is it illegal, but the real question is who nega ive really cares for Texas? other tive 2 2 #NeverTrump goes to Scotland and talks about negat sprinkler.Missed the whole point! #Brexit was crucial and nega ive warranted some explanation from trump Trump tive 2 2 negat nega Thanks British #brexit twats, I'm feeling poorer today ive economy tive x 1 2 Asked my wife if she heard about #Brexit and she said no. negat Started to explain and she doesn’t even know about the EU. I neut ive quit. Brexit ral 3 3 #Brexit is crushing victory 4 ppl against the positi posit establishment. Get used 2 it. #EngalndLiberation #LeaveWins ve Brexit ive 2 2 @StephenNolan @bbc5live this Gerry guy is the negat embodiment of smug, arrogant BBC leftwing metropolitan posit ive self-proclaimed elites. Happy 4 #Brexit Brexit ive 2 2 An electoral college system sounds like a good idea right neutr nega abut now dunnit? #Brexit al other tive x 2 2 We have an extraordinarily high level of international positi posit reserves 177 billion dollars #Brexit #Mexico #PressRelease ve economy ive 3 2 #Brexit is similar to union-busting but not a union of dock negat workers, it's a union of bankers, technocrats, oligarchs, fascists, nega ive etc. Brexit tive 2 2 Looks like the EU has about 1 GB more free space now. neutr neut #Brexit al EU ral x 3 2 Let love win! Let the religion of peace engulf your once positi posit proud nation. Celebrate their tolerance! #onpoli #Brexit ve Brexit ive 1 1 @Lizabs68 By time #Brexit is complete, a majority of negat those who voted & are still alive will be #Remain supporters. nega ive #BrexitMustBeStopped Brexit tive 2 2 Talk of a 2nd referendum if EU gives us a new deal which neutr will appease the people voting #Leave Hopefully that comes to nega al pass #EUref #Brexit Brexit tive 2 2 This is shocking. “What is the EU”? asks citizens of the negat nega UK, AFTER #Brexit. ive EU tive 2 2 Help my 'Get the fuck out of England' fund with this #EURefResults design. neutr http://www.redbubble.com/people/lunarblaze/works/22256458 al nega -dont-blame-me-eu … #EUreferendum #BrexitVote #Brexit other tive x 3 3 Bed after a long two days. Three events today with one neutr topic #Brexit. People are uncertain but rational about what nega al happens next. Brexit tive 2 2 Nothing can save this day, but I guess we will always negat nega remember it! #brexit ive Brexit tive 2 2 Guess I won’t have to deal with anymore Brits on holidy positi neut who never tip #cheerio #brexit ve other ral x 2 2 Hey. England. If you wanna be a wanker go ahead… But negat how about cutting Scotland and Ireland loose before ya drag nega ive them down too! #Brexit Brexit tive 2 2

Page 73 of 76

What would it be called if the US decided to do a #Brexit? neutr neut U Sexit? al USA ral x 2 2 #HistoryNotes #Brexit illustrates economic point no neutr school teaches: Money is Imaginary – Economies are Ideas – nega al they are what you make them economy tive x 2 2 There’s a vacancy, we already have a foot in the door via positi Eurovision Australia might ass well have a crack! Let’s join nega ve the EU! #Brexit EU tive x 2 2 Increase Google searches in that region of the wold asking neutr neut “what is #Brexit?” and “what is the UK?” al Brexit ral 2 2 I'll bet all these Hollywood stars so knowledgable about negat #Brexit are the same ones who were primate experts just a few celebrities/p nega ive weeks ago. #clueless oliticians tive x 2 2 #Brexit Angela Merkel's country faces having to pay an negat extra £2.44billion a year to the annual EU budget once Britain celebrities/p nega ive has left. oliticians tive 2 1 DID YOU HEAR? The UK #Brexit took their country positi back. Time for s to take USA back! Donate to Trump posit ve Campaign now at USA ive 2 2 Both #Brexit and #Trump are disasters. No thanks. I'll negat celebrities/p nega vote for @HillaryClinton and other sane people. ive oliticians tive 2 2 Will #Brexit disintegrate the EU? The EU needs a new negat democratic constitution or it will disintegrate! Join #DiEM25 nega ive community, save EU! EU tive 3 2 To understand #brexit, the immigration issue etc...is to neutr understand y they are leaving home in the first place. Who is neut al shaping these events? Brexit ral x 2 2 The British call their Independence #Brexit... maybe we neutr posit should call ours #Mexit Vote #Trump2016 al USA ive 2 1 One positive from #Brexit is that it has shown which areas positi nega of the UK need better funding for education. ve Brexit tive 1 2 Congratulations on your #Brexit from the globalists, hope positi posit to do the same in Nov #Trump2016 #MakeAmericaGreatAgain ve USA ive 1 1 The EU's 'techno party' is hollowing out democracy http://openermedia.blogspot.com/2015/05/the-eus-techno- negat party-is-hollowing-out.html … #Brexit LoiTravail David ive nega Cameron Scotland other tive 2 2 If Sarah "Dumbass" Palin is excited about your decision.. negat celebrities/p nega you just made a HUGE global boner of a mistake. #Brexit ive oliticians tive 2 2 My thoughts and prayers go out to all the people affected neutr nega by this #Brexit fiasco al Brexit tive 2 2 I'd be more upset about #brexit if it didn't sound so much neutr nega like #breakfast . Instead, I'm just hungry. al other tive x 2 3 So, that was the dress rehearsal. Now that you Leavers have negat seen the effects of your vote, would you like to try that again? nega ive #Brexit Brexit tive x 2 1 Above anything else, I genuinely feel sad for our country negat nega today. #EUref #Brexit #notmyvote ive Brexit tive 2 2 negat nega I never thought I would live to see it broken up. #Brexit ive Brexit tive 2 2 IMHO #Brexit fallout is totally overblown. People are negat making money off the rampant panic & uncertainty. Folks need nega ive to calm the fuck down. Brexit tive 2 2 neutr nega So long….and thanks for all the bypasses. #Brexit al Brexit tive 1 2 neutr neut Something is changing...... #EURefResults #Brexit al Brexit ral 2 2 negat nega It’s failing #Brexit ive Brexit tive 3 2 Maybe he thought Brexit was the guy's name? Man Who negat Voted For #Brexit Is 'A Bit Shocked' His Vote Counted, nega ive 'Worried' Brexit tive x 2 2 #Brexit is the logical result of Thatcher's 'Big Bang.' negat State-sanctioned inequality + the creation of ultra-capital & nega ive #financialization. Brexit tive 2 2

Page 74 of 76

Lotsa peeps loving my fiction on #Brexit -> #Trump -> neutr nega World War 3 via Scots, Texit, Huxit, Grexit, Bletchit, Putin al other tive 2 2 #Brexit processing takes years.dont see why #US #Stock positi neut to be affected now?instead it is a good hands to buy in! ve economy ral 2 2 #Brexit sounds like a planet in the #StarWars EU which neutr nega makes this a very confusing day. al Brexit tive x 3 2 Don't know a lot about #Brexit, but i do enjoy watching positi the Super Pundit Class hyperventilating and clutching their nega ve pearls. economy tive x 2 2 Yes, stock markets will plummet. $$ interests only care neutr about making more $$ for the sake of itself. Common sense. nega al #brexit #financialization economy tive 2 2 I served in the UK at RAF Bentwaters/Woodbridge & positi could not be prouder of them taking their country back! posit ve #Brexit Brexit ive 2 2 Never underestimate the power of stupid people in large negat nega groups! #Brexit #jokeofthecentury ive Brexit tive x 2 2 #UK left the European Union...I guess it's time to find negat work elsewhere =_= Thanks #Brexit nega ive https://youtu.be/I17j7vzFnN0 via @YouTube Brexit tive x 2 2 The next James Bond will just be him spending 2 hours in negat nega passport control De Gaulle #Brexit #JamesBond ive other tive x 3 2 neutr neut Will #brexit hurt English Football League? al other ral 3 3 I need the #Brexit jokes to stop until I can refill my neutr nega prescription al other tive x 2 2 #BrexitAdam didn't think his vote would count? now you negat know folks, your vote ALWAYS counts! #Brexit #BrexitVote nega ive #BrexitOrNot #Election2016 Brexit tive 2 2 Can we all just fast forward to 2017 instead? #Brexit negat nega #DonaldTrump #RefugeeCrisis ive Brexit tive x 2 2 Thank you, @NicolaSturgeon, for demonstrating what positi true, compassionate leadership looks like in the face of celebrities/p nega ve adversity. #Brexit oliticians tive 1 2 if #Brexit was Rigged Find out who Invested in Gold neutr Profited just like insider Trading Before 911 on thr Airline neut al Stocks SECURITY FIRM COMEX economy ral 3 2 #BREXIT – Americans! Watch & learn from a VOTE negat based on revenge and xenophobia. If you vote unhinged, there nega ive are consequences! #NeverTrump Brexit tive 2 2 positi posit #brexit is so catchy I love it ve Brexit ive 2 2

Page 75 of 76

Page 76 of 76