arXiv:1811.00127v1 [cs.CL] 31 Oct 2018 oni ne otat2017-02429 contract under Council sentiments. and topics of various to matching related and sets term vocabulary, used lists frequency commonly producing of data; occur- in counting terms of than form, rences basic more most nothing its entails In SMM domain. SMM the within 2016 2002 Sridhar ag oyo cetfi ieaue n con- of and Tracking subfields and The literature, Detection Topic scientific interest. of commercial siderable has body with media) domain large application social a established in an discussions become online of ing Monitoring Media Social Introduction 1 ∗ iiaiyo h id“o iia ssource is similar source “how semantic A kind of the kind of a as similarity seen be set- can agenda ting and alignment outcomes. issue that electoral argue to We and used choice been voter have explain that science political from ship namely monitoring, for media venue social another investigate we paper, this In regarding about, X talked being is “what monitoring. as, questions such answers media methods common social these are Essentially, for analysis used topic methods and Sentiment iindo lcseicissues. con- bloc-specific on parties, politi- ditioned political and between media method similarity aligned this cally the test predic- measuring then for We by similarity of embeddings. notion tive new intro- a and similarity, duce conditioned measuring of towards kind progress that in We work present techniques. embedding word/document sue hsrsac a upre yteSeihResearch Swedish the by supported was research This r ohsinictpc pwe entirely spawned topics scientific both are ) ; ,ad“htd epefe,regarding feel, people do “what and ”, agadLee and Pang oiseowner issue to [email protected] , X and 2015 ,ada uhcnb esrdusing measured be can such as and ”, gnasetting agenda esrn su wesi sn odEmbeddings Word using Ownership Issue Measuring and ) mr uaGyllensten Cuba Amaru , Abstract etmn Analysis Sentiment 2008 P hntligaotis- about talking when , IEAI RISE ; hc r concepts are which , Liu SM ..monitor- i.e. (SMM; , ( la tal. et Allan 2012 su owner- issue ; oz tal. et Pozzi ( Turney , X 1998 ”. ; , , ehiust rdc h ucm felections, of ( outcome success varying the monitoring with media predict social to attempting of techniques forms studies social various numerous in use are write to they There what in media. manifested opin- as how elec- citizens’ ions of of apparent indication context an the also providing in for is tions used of be It understanding may analysis an such audience. gain target to their may analysis companies such service use and product re- ap- how is market parent It for departments. communications insight and useful search provide may it?” tions about feel they inotoe ( outcomes tion ownership sue aeue h ocpsof concepts the used scientists have Political SMM inadequate. be standard may opinion, techniques re- mass the questions measuring and to certain election lated for US that recent demonstrate most Brexit the techniques forecast measuring to opinion standard of quacy are learning. machine Latent supervised on as based analysis (such sentiment and detection Allocation) Dirichlet topic probabilistic forms of various use approaches sophisticated More atclriseadcs hi alt o htis- that for ( ballots owner their sue of cast and proponent issue party particular credible vot- a most that the states identify voting ers of theory ownership issue al. et Kiousis 2011 osntsfc omaueteocrec fcer- of occurrence the measure to suffice not it does manner; satisfactory a in to concepts unable these measure agenda are techniques public monitoring media the social in influencing issues in ( of role media’s importance the the to refers setting cob n Reynolds and Mccombs otntby h eeteape fteinade- the of examples recent the notably, Most h eta usin M ek oanswer to seeks SMM questions central The [email protected] wa ouestl about?” talk users do “what ; eo tal. et Ceron ansSahlgren Magnus ´ lne n Meguid B´elanger and , 2015 oepanvtrcoc n elec- and choice voter explain to lu e n aiSagarzazu naki Kl¨uver and IEAI RISE , 2015 ; Stubager nwr oteeques- these to Answers . emnhmadSmeaton and Bermingham ). , 2002 gnasetting agenda , 2018 .Nt htcurrent that Note ). , 2008 .I hr,the short, In ). ∗ and .Agenda ). hwdo “how , and 2016 is- ; , tain keywords, since most parties tend to use the sented by vectors whose elements encode (some same vocabulary to discuss issues, and sentiment function of) co-occurrences with other words. analysis does not touch upon the issue ownership Contemporary research on distributional seman- and agenda setting questions. What is needed for tics (Sahlgren, 2006; Bullinaria and Levy, 2007; measuring issue ownership and agenda setting is Turney and Pantel, 2010; Pennington et al., 2014) a way to measure language use, i.e. when talking still use largely the same mathematical machin- about an issue, to which extent does the language ery as the vector space model, and cosine sim- used align with issue owner A vs. issue owner B. ilarity is still the preferred similarity metric due We argue that issue alignment can be seen to its simplicity and use of length normaliza- as a kind of semantic source similarity of the tion. Even neural language models, which kind “how similar is source A to issue owner P, originate from the neural network community, when talking about issue X”, and as such can be employ cosine similarity to quantify similarity measured using word/document embedding tech- between learned representations (Mikolov et al., niques. To measure that kind of conditioned sim- 2013; Bojanowski et al., 2017). ilarity we introduce a new notion of similarity for Word embeddings, as these techniques are predictive word embeddings. This method enables nowadays referred to, have been used ex- us to manipulate the similarity measure by weight- tensively in SMM, both for topic detection ing the set of entities we account for in the pre- (Sridhar, 2015) and for sentiment analysis dictive scoring function. The proposed method is (Severyn and Moschitti, 2015). To the best applied to measure similarity between party pro- of our knowledge, only one previous study grams and various subsets of online text sources, (Dahlberg and Sahlgren, 2014) has used word em- conditioned on bloc specific issues. The results beddings to analyze issue ownership. However, indicate that this conditioning disentangles simi- that study relied on simple nearest neighbor anal- larity. We can, for example, observe that while the ysis using cosine similarity to study language use Left Party representation is, overall, similar to that in the Swedish blogosphere. of nativist media, it differs significantly on nativist We believe that prediction-based word embed- issue, while this effect is not seen to the same ex- dings such as Word2Vec are amenable to another tent on more mainstream left wing or right wing notion of similarity, which we call predictive sim- media. ilarity.

2 Vector Similarity 2.1 Predictive Similarity Given a function f : A × B → R, we define the Vector similarity has been a foundational concept predictive similarity of two items x,y ∈ A as the in natural language processing ever sine the intro- correlation of f(x, b), and f(y, b), where b is a duction of the vector space model for information random variable of type B: retrieval by Salton (1971). In this model, queries and document are represented as vectors in term space, and similarity is expressed using cosine cov (f (x, b) ,f (y, b)) similarity. The main reason for using cosine sim- psim(x,y)= var (f (x, b)) var (f (y, b)) ilarity in the vector space model is that it normal- (1) izes for vector length; the fact that a document (or At a very generalp level, prediction based word query) contains a certain word is more important embeddings such as Word2Vec or FastText con- than how many times it occurs in the document. sists of a scoring function s : C × T → R with an The vector space model was the main source objective function taking the following form: of inspiration for early work on vector se- mantics, such as Latent Semantic Analysis (Deerwester et al., 1990; Landauer and Dumais, l(s(c, t)) + l(−s(n,t)) (2) 1997) and the works on word space models t×C∈D c∈C n∈N  X X Xt,c by Sch¨utze (1992, 1993). These works contin-   ued to embrace cosine similarity as the simi- where l is the logistic loss function l(x) = log(1+ larity metric of choice, since length normaliza- e−x) and s being the model-specific scoring func- tion is equally desired when words are repre- tion that relates to the probability of observing the orange “juice”, and “county”, respectively) accounts for paint juice county half the weight in var(b), with all other words in 1 deep-red cranberry siskiyou the vocabulary having equal weights. 2 fuschia lime calaveras Predictive similarity can easily be extended 3 lime-green caraway ventura to similar models, and for the purpose of this 4 hand-woven fanta osceola paper in particular, we extend it to Doc2Vec 5 blue clove yolo (Le and Mikolov, 2014), a model where the notion 3 6 yellow zests mendocino of context is enriched by the source of the utter- 7 ocher coconut bernardino ance. The scoring function s then takes the fol- T 8 linoleum peppercorns okanogan lowing form: s(t, c, d) = t (c + d), with d being 9 duck-egg lemons okfuskee a vector representation of the source in question. 10 rust-colored peach tuolumne We argue that by using conditioned predictive similarity on document embeddings we can an- Table 1: Examples of predictive similarity neigh- swer questions such as: “how similar is The BBC borhoods of “orange” conditioned on “paint”, to The Daily Mail, when talking about Climate “juice”, and “county”, respectively. 2 Change”. The end goal is to measure aggregate similarity in specific issues: “when talking about health policy, to which extent does the general lan- target t in the context c. For the Skipgram vari- guage use align with Source A, Source B, Source ant of Word2Vec, this function s is simply the dot C, et.c.”. product between a vector representation of the tar- get word t, and a vector representation of the con- 3 Experiments text word c. To answer the language similarity question posed The predictive similarity has several interpreta- by issue ownership we measure aggregate predic- tions for the Skipgram model, but the simplest one tive similarity between party platforms and var- is the one where we let f = s, i.e. we say that the ious subsets of online text data, conditioned on similarity of two words x and y is the correlation words pertaining to left wing issues, right wing is- between the scores they assign to target words b, sues, nativist issues, and general political topics. i.e. corr(s(x, b),s(y, b)). Since s is linear, this correlation takes a fairly simple form: 1 We built Doc2Vec embeddings (Le and Mikolov, 2014) on Swedish online data from 2018 crawled by Trendiction and cov(s(x, b),f(s, b)) manually scraped party platforms from the eight T T parties in parliament and Feministiskt Initiativ = E x b − xT b y b − yT b (Feminist Initiative).4 Doc2Vec requires us to h T  T Ti = E x b − b y b − b define a notion of source. For the data crawled by Trendiction, we take the source to be the domain Th T i (3) = x E b − b

b − b y

name of the document, e.g. www.wikipedia.se, whereas for the manually scraped party platforms, xT varhb y

i = ( ) we assign it the appropriate party identifier. The T x var(b)y model was trained using the Gensim package psim(x,y)= xT var(b)xyT var(b)y (Reh˚uˇrekˇ and Sojka, 2010) with embedding dimension 100 and a context window of size 8. p We argue that we can get a a notion of condi- In collaboration with the Political Science de- tioned similarity by estimating a weighted correla- partment at Gothenburg University we also ex- tion, where the weighting acts as the conditioning. tracted keywords for each party from their party Table 1 shows a small example where we platform. We use these party specific keywords as queried the neighborhood of the word “orange”, a crude proxy for issues: we let left wing issues be conditioned such that a single word (“paint”, 3By source we can mean a paragraph, document, or in our 1It might be interesting to note that this coincides with case: domain name from which the utterance originates. cosine similarity if var(b) is a scalar multiple of the iden- 4A complete list of parties, their abbreviations, their En- tity, i.e. if there is no correlation between dimensions and all glish translations, and bloc affiliation can be found in Table dimensions have the same variance. 2. Abbr. Name Translation Word count Bloc V V¨ansterpartiet The Left Party 15,383 S Socialdemokraterna The Social Democrats 27,899 Left MP Milj¨opartiet The Green Party 19,471 C Centern The Centre Party 68,136 L Liberalerna The Liberals 64,276 Right KD Kristdemokraterna The Christian Democrats 16,494 M Moderaterna The Moderates 12,807 SD Sverigedemokraterna The Swedish Democrats 3,430 N/A (Nativist) FI Feministiskt Initiativ Feminist Initiative 84,424 N/A

Table 2: Party abbreviations, names, translated names, word count, and bloc allegiance. defined by the union of left bloc party keywords, 4 Discussion right wing issues be defined by right bloc party keywords, and nativist issues be defined by the keywords of Sverigedemokraterna (The Swedish As can be seen in Table 3, there is a marked dif- Democrats), we also let the union of all keywords ference when conditioning on issues versus using be representative for general political discourse. regular document — i.e. cosine — similarity. Fur- The parties’ bloc alignment and the size of the data thermore, we observe that conditioned similarity used to generate representations for them can be seems to align left wing media with left wing par- seen in Table 2. ties, nativist media with the Swedish Democrats, We let the conditioned predictive similarity be- but not align right wing media with right wing par- tween sources two x and y be defined by the fol- ties. This effect can be made more apparent by lowing equation (Equation 4), i.e. a weighted vari- grouping the parties into blocs and fitting a simple ant of equation 3, where only words among the additive model for the similarities along all dimen- given issues keywords are accounted for, as de- sions (i.e. Media, Issues, and Bloc), as a way to scribed by Equation 5. normalize for general Media, Issue, and Bloc sim- ilarity. The results of this normalization, i.e. the residuals, can be observed in Table 4. From this one can see a small trend where left wing media xT var(t; w)y psim(x,y)= is similar to left wing parties, nativist media being xT var(t; w)xyT var(t; w)y similar to the Swedish Democrats, and both left (4) wing media and right wing media being dissimilar p to the Swedish Democrats. 1,t ∈ Issue keywords Furthermore, we see a strong dissimilarity be- wt = (5) (0,t 6∈ Issue keywords tween nativist media and all parties regarding na- tivist issues. This is particularly true for parties Above, x and y are document vectors and promoting liberal immigration policy: The Left Party, The Social Democrats, The Green Party, var(t; wt) is the weighted covariance matrix of the target word vectors. This is the equivalent of let- The Centre Party, and The Moderates are all cur- ting s(d, c, t) = dT t, i.e. the case we ignore the rently or historically promoting liberal immigra- effect of context words. tion policy at odds with nativist sentiment. Table 3 (next side) shows the average predictive A shortcoming of the method used here is the similarity between the political party platforms rather limited amount of party specific data: the and various online data sources, conditioned on quality and the quantity of the text data used varies left wing party issues, right wing party issues, na- drastically between parties, as can be seen in Ta- tivist party issues, and general political discourse. ble 2. Using, for example, parliamentary debates, Average cosine similarity between the sources and opinion pieces, and other official party communi- parties is also shown as a comparison. cation might improve data coverage. V SMP C LKD M SD FI Media Issues Left wing 0.43 0.35 0.25 0.20 0.36 0.35 0.45 0.47 0.36 Right wing 0.44 0.38 0.36 0.34 0.41 0.36 0.45 0.45 0.32 Left wing Nativist 0.43 0.40 0.42 0.36 0.42 0.39 0.42 0.45 0.37 All 0.42 0.35 0.31 0.28 0.38 0.36 0.42 0.44 0.36 Cos 0.50 0.48 0.48 0.46 0.51 0.47 0.53 0.49 0.44 Left wing 0.25 0.24 0.31 0.25 0.31 0.27 0.35 0.34 0.16 Right wing 0.28 0.31 0.32 0.32 0.34 0.28 0.36 0.36 0.19 Right wing Nativist 0.29 0.32 0.36 0.34 0.38 0.34 0.36 0.36 0.21 All 0.26 0.27 0.31 0.30 0.34 0.30 0.35 0.34 0.18 Cos 0.44 0.45 0.44 0.47 0.51 0.47 0.51 0.45 0.41 Left wing 0.36 0.17 0.04 0.05 0.30 0.31 0.34 0.48 0.32 Right wing 0.28 0.09 0.08 0.17 0.30 0.32 0.30 0.39 0.23 Nativist Nativist 0.05 -0.11 0.02 0.01 0.17 0.16 0.03 0.21 0.08 All 0.28 0.08 0.06 0.10 0.28 0.31 0.27 0.39 0.29 Cos 0.51 0.45 0.47 0.45 0.56 0.53 0.56 0.61 0.53 Left wing 0.32 0.26 0.25 0.21 0.33 0.30 0.38 0.40 0.25 Right wing 0.33 0.30 0.30 0.31 0.36 0.32 0.38 0.40 0.24 All News Nativist 0.30 0.28 0.33 0.30 0.36 0.33 0.33 0.36 0.24 All 0.32 0.27 0.27 0.26 0.34 0.32 0.36 0.38 0.25 Cos 0.47 0.46 0.46 0.47 0.52 0.48 0.52 0.48 0.44 Left wing 0.07 0.11 0.18 0.06 0.06 0.08 0.09 0.12 0.18 Right wing 0.20 0.28 0.31 0.22 0.18 0.14 0.20 0.17 0.26 Social Nativist 0.12 0.18 0.19 0.08 0.09 0.07 0.12 0.22 0.21 All 0.13 0.18 0.20 0.14 0.11 0.10 0.13 0.16 0.23 Cos 0.42 0.42 0.42 0.40 0.39 0.41 0.42 0.45 0.39

Table 3: Average predictive similarity (and cosine similarity) between political parties and various subsets of the online sources. Bloc Left Nativist Right • Dala-Demokraten Media Issues • Folkbladet Left wing -0.02 -0.02 -0.06 • ETC Left wing Nativist 0.09 -0.00 0.03 Right wing 0.02 -0.05 -0.02 • Arbetaren Left wing 0.02 0.18 0.04 • Flamman Nativist Nativist -0.15 -0.06 -0.08 • Bang Right wing -0.03 0.08 0.05 Left wing -0.02 -0.06 -0.02 • Offensiv Right wing Nativist 0.07 -0.02 0.07 • Prolet¨aren Right wing 0.01 -0.06 -0.01 6.2 Right wing news sources Table 4: Grouped and normalized predictive simi- • Dagens Industri larity. • Dalabygden • Hallands Nyheter 5 Conclusion • Axess • Svensk Tidskrift In this paper we have introduced some very pre- • Hemmets V¨an liminary results on how to measure similarities in • Dagens Nyheter language use, conditioned on discourse, e.g. “how • G¨oteborgs-Posten similar is The BBC to The Daily Mail, when talk- • Helsingborgs Dagblad ing about Climate Change”. The end goal is to • Nerikes Allehanda measure aggregate similarity in specific issues, an- • Sydsvenskan swering questions such as “when talking about • Upsala Nya Tidning health policy, to which extent does the general lan- • Expressen guage use align with Source A, Source B, etc.”, and • Svenska Dagbladet use such an aggregate measure to study issue own- • Sm˚alandsposten ership at scale. • Norrbottens Kuriren We believe that issue ownership and agenda set- ting can be explored through the lens of language 6.3 Nativist news sources use and similarity, but deem it necessary to con- • Nordfront dition similarity to the specific issue at hand. The • Samh¨allsnytt reason for this is the need to distinguish between • Fria Tider level of engagement in an issue and agreement in • Nya Tider an issue: two sources that talk a lot about an issue • Samtiden — e.g. health insurance — but in very different ways should not be considered similar. Dually, if a source very rarely talks about an issue, but References consistently does so in a way that is very similar James Allan, Jaime Carbonell, George Doddington, to the way some political party talks about it, we Jonathan Yamron, Yiming Yang, et al. 1998. Topic consider it reasonable to believe that that source’s detection and tracking pilot study: Final report. opinion aligns with the political party in question In Proceedings of the DARPA broadcast news on that specific issue. transcription and understanding workshop, volume 1998, pages 194–218. Citeseer. While we have not found a satisfactory, direct, evaluation of this task, we do believe that the ex- Eric´ B´elanger and Bonnie M. Meguid. 2008. Is- amples we put forward show some face validity sue salience, issue ownership, and issue-based vote of the proposed method at measuring ideological choice. Electoral Studies, 27(3):477 – 491. alignment. Adam Bermingham and Alan Smeaton. 2011. On us- ing twitter to monitor political sentiment and predict 6 Appendix election results. In Proceedings of the Workshop on Sentiment Analysis where AI meets Psychology 6.1 Left wing news sources (SAAIP 2011), pages 2–10.

• Aftonbladet Piotr Bojanowski, Edouard Grave, Armand Joulin, and • Arbetarbladet Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Associa- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- tion for Computational Linguistics, 5:135–146. rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their composition- John A. Bullinaria and Joseph P. Levy. 2007. Ex- ality. In C. J. C. Burges, L. Bottou, M. Welling, tracting semantic representations from word co- Z. Ghahramani, and K. Q. Weinberger, editors, Ad- occurrence statistics: A computational study. Be- vances in Neural Information Processing Systems havior Research Methods, 39(3):510–526. 26, pages 3111–3119. Curran Associates, Inc.

Andrea Ceron, Luigi Curini, and Stefano M Iacus. Bo Pang and Lillian Lee. 2008. Opinion mining and 2015. Using sentiment analysis to monitor elec- sentiment analysis. Foundations and Trends in In- toral campaigns: Method mattersevidence from the formation Retrieval, 2(1-2):1–135. united states and italy. Social Science Computer Re- view, 33(1):3–20. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, representation. In EMNLP, volume 14, pages 1532– Thorsten Brants, Phillipp Koehn, and Tony Robin- 1543. son. 2013. One billion word benchmark for measur- ing progress in statistical language modeling. arXiv Federico Alberto Pozzi, Elisabetta Fersini, Enza preprint arXiv:1312.3005. Messina, and Bing Liu. 2016. Sentiment Analysis in Social Networks Stefan Dahlberg and Magnus Sahlgren. 2014. Issue , 1st edition. Morgan Kaufmann framing and language use in the swedish blogo- Publishers Inc., San Francisco, CA, USA. sphere: Changing notions of the outsider concept. ˇ In Bertie Kaal, Isa Maks, and Annemarie van El- Radim Reh˚uˇrek and Petr Sojka. 2010. Software frinkhof, editors, From Text to Political Positions: Framework for Topic Modelling with Large Text Analysis across Disciplines, pages 71–92. John Corpora. In Proceedings of the LREC 2010 Benjamins. Workshop on New Challenges for NLP Frame- works, pages 45–50, Valletta, Malta. ELRA. Scott Deerwester, Susan T. Dumais, George W. Fur- http://is.muni.cz/publication/884893/en. nas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Jour- Magnus Sahlgren. 2006. The Word-space model. nal of the American Society for Information Science, Ph.D. thesis, University of Stockholm (Sweden). 41(6):391–407. Gerard Salton. 1971. The SMART Retrieval System— Spiro Kiousis, Jesper Str¨omb¨ack, and Michael McDe- Experiments in Automatic Document Processing. vitt. 2015. Influence of issue decision salience on Prentice-Hall, Inc., Upper Saddle River, NJ, USA. vote choice: Linking agenda setting, priming, and issue ownership. International Journal of Commu- Hinrich Sch¨utze. 1992. Dimensions of meaning. In nication, 9(0). Proceedings of the 1992 ACM/IEEE Conference on Supercomputing, Supercomputing ’92, pages 787– Heike Kl¨uver and I˜naki Sagarzazu. 2016. Setting the 796, Los Alamitos, CA, USA. IEEE Computer So- agenda or responding to voters? political parties, ciety Press. voters and issue attention. West European Politics, 39(2):380–398. Hinrich Sch¨utze. 1993. Word space. In Advances in Neural Information Processing Systems 5, pages Thomas K Landauer and Susan T Dumais. 1997. A 895–902. Morgan Kaufmann. solution to plato’s problem: The latent semantic analysis theory of acquisition, induction, and rep- resentation of knowledge. Psychological review, Aliaksei Severyn and Alessandro Moschitti. 2015. 104(2):211–240. Twitter sentiment analysis with deep convolutional neural networks. In Proceedings of the 38th Inter- Quoc Le and Tomas Mikolov. 2014. Distributed rep- national ACM SIGIR Conference on Research and resentations of sentences and documents. In Inter- Development in Information Retrieval, pages 959– national Conference on Machine Learning, pages 962. ACM. 1188–1196. Vivek Kumar Rangarajan Sridhar. 2015. Unsupervised Bing Liu. 2012. Sentiment Analysis and Opinion Min- topic modeling for short texts using distributed rep- ing. Morgan & Claypool Publishers. resentations of words. In Proceedings of the 1st workshop on vector space modeling for natural lan- Maxwell Mccombs and Amy Reynolds. 2002. News guage processing, pages 192–200. influence on our pictures of the world. In Jen- nings Bryant and Dolf Zillmann, editors, Media Ef- Rune Stubager. 2018. What is issue ownership and fects. Advances in Theory and Research, pages 1– how should we measure it? Political Behavior, 18. Lawrence Erlbaum Associates. 40(2):345–370. Peter D. Turney. 2002. Thumbs up or thumbs down?: Semantic orientation applied to unsupervised classi- fication of reviews. In Proceedings of the 40th An- nual Meeting on Association for Computational Lin- guistics, ACL ’02, pages 417–424, Stroudsburg, PA, USA. Association for Computational Linguistics. Peter D. Turney and Patrick Pantel. 2010. From fre- quency to meaning: Vector space models of se- mantics. Journal of Artificial Intelligence Research, 37(1):141–188.