model

Vector space model or term vector model is an alge- braic model for representing text documents (and any ob- jects, in general) as vectors of identifiers, such as, for ex- ample, index terms. It is used in information filtering, , indexing and relevancy rankings. Its first use was in the SMART Information Retrieval Sys- tem.

1 Definitions

Documents and queries are represented as vectors.

dj = (w1,j , w2,j, . . . , wt,j) q = (w1,q, w2,q, . . . , wn,q) Each dimension corresponds to a separate term. If a term occurs in the document, its value in the vector is non-

zero. Several different ways of computing these values, (q in the figure) vectors, ∥d2∥ is the norm of vector d2, also known as (term) weights, have been developed. One and ∥q∥ is the norm of vector q. The norm of a vector is of the best known schemes is tf-idf weighting (see the calculated as such: example below). v The definition of term depends on the application. Typi- u cally terms are single words, keywords, or longer phrases. u∑n ∥ ∥ t 2 If words are chosen to be the terms, the dimensionality of q = qi the vector is the number of words in the vocabulary (the i=1 number of distinct words occurring in the corpus). As all vectors under consideration by this model are ele- Vector operations can be used to compare documents mentwise nonnegative, a cosine value of zero means that with queries. the query and document vector are orthogonal and have no match (i.e. the query term does not exist in the docu- ment being considered). See for further 2 Applications information.

Relevance rankings of documents in a keyword search can be calculated, using the assumptions of document 3 Example: tf-idf weights similarities theory, by comparing the deviation of an- gles between each document vector and the original query In the classic vector space model proposed by Salton, vector where the query is represented as the same kind of Wong and Yang [1] the term-specific weights in the docu- vector as the documents. ment vectors are products of local and global parameters. In practice, it is easier to calculate the cosine of the angle The model is known as term frequency-inverse document between the vectors, instead of the angle itself: frequency model. The weight vector for document d is T vd = [w1,d, w2,d, . . . , wN,d] , where

d2 · q cos θ = ∥ ∥ ∥ ∥ |D| d2 q w = tf · log t,d t,d |{d′ ∈ D | t ∈ d′}| Where d2 · q is the intersection (i.e. the dot product) of the document (d2 in the figure to the right) and the query and

1 2 7 SOFTWARE THAT IMPLEMENTS THE VECTOR SPACE MODEL

• tft,d is term frequency of term t in document d (a 1. Long documents are poorly represented because local parameter) they have poor similarity values (a small scalar prod- uct and a large dimensionality) • |D| log |{d′∈D | t∈d′}| is inverse document frequency (a global parameter). |D| is the total number of doc- 2. Search keywords must precisely match document uments in the document set; |{d′ ∈ D | t ∈ d′}| is terms; word substrings might result in a "false posi- the number of documents containing the term t. tive match”

3. Semantic sensitivity; documents with similar con- Using the cosine the similarity between document dj and text but different term vocabulary won't be associ- query q can be calculated as: ated, resulting in a "false negative match”.

∑ 4. The order in which the terms appear in the document · N w w dj q √ i=1 √i,j i,q is lost in the vector space representation. sim(dj, q) = = ∑ ∑ ∥dj∥ ∥q∥ N 2 N 2 i=1 wi,j i=1 wi,q 5. Theoretically assumed terms are statistically inde- pendent.

4 Advantages 6. Weighting is intuitive but not very formal.

The vector space model has the following advantages over Many of these difficulties can, however, be overcome the Standard Boolean model: by the integration of various tools, including mathemati- cal techniques such as singular value decomposition and 1. Simple model based on linear algebra lexical databases such as WordNet.

2. Term weights not binary

3. Allows computing a continuous degree of similarity 6 Models based on and extending between queries and documents the vector space model

4. Allows ranking documents according to their possi- Models based on and extending the vector space model ble relevance include: 5. Allows partial matching • Generalized vector space model Most of these advantages are a consequence of the dif- • ference in the density of the document collection repre- sentation between Boolean and tf-idf approaches. When • Term Discrimination using Boolean weights, any document lies in a vertex in a n-dimensional hypercube. Therefore, the possible docu- • n Rocchio Classification ment representations are 2√and the maximum Euclidean distance between pairs is n . As documents are added • Random Indexing to the document collection, the region defined by the hypercube’s vertexes become more populated and hence denser. Unlike Boolean, when a document is added using tf-idf weights, the idfs of the terms in the new document 7 Software that implements the decrease while that of the remaining terms increase. In vector space model average, as documents are added, the region where docu- ments lie expands regulating the density of the entire col- lection representation. This behavior models the original The following software packages may be of interest to motivation of Salton and his colleagues that a document those wishing to experiment with vector models and im- collection represented in a low density region could yield plement search services based upon them. better retrieval results. 7.1 Free open source software

5 Limitations • Apache Lucene. Apache Lucene is a high- performance, full-featured text search engine library The vector space model has the following limitations: written entirely in Java. 3

• is a Python+NumPy framework for Vector Space modelling. It contains incremental (memory- efficient) algorithms for Tf–idf, Latent Semantic In- dexing, Random Projections and Latent Dirichlet Allocation.

• Weka. Weka is a popular data mining package for Java including WordVectors and Bag Of Words models.

8 Further reading

• G. Salton, A. Wong, and C. S. Yang (1975), "A Vec- tor Space Model for Automatic Indexing,” Commu- nications of the ACM, vol. 18, nr. 11, pages 613– 620. (Article in which a vector space model was pre- sented) • David Dubin (2004), The Most Influential Paper Gerard Salton Never Wrote (Explains the history of the Vector Space Model and the non-existence of a frequently cited publication) • Description of the vector space model

• Description of the classic vector space model by Dr E. Garcia

• Relationship of vector space search to the “k- Nearest Neighbor” search

9 See also

• Bag-of-words model

• Compound term processing • Conceptual space

• Eigenvalues and eigenvectors • Inverted index

• Nearest neighbor search • Sparse distributed memory

• w-shingling

10 References

[1] G. Salton , A. Wong , C. S. Yang, A vector space model for automatic indexing, Communications of the ACM, v.18 n.11, p.613-620, Nov. 1975 4 11 TEXT AND IMAGE SOURCES, CONTRIBUTORS, AND LICENSES

11 Text and image sources, contributors, and licenses

11.1 Text

• Vector space model Source: https://en.wikipedia.org/wiki/Vector_space_model?oldid=744792705 Contributors: Michael Hardy, Kku, Dcljr, Stan Shebs, Jitse Niesen, Gdm, Beland, Thorwald, Rama, Mykhal, ESkog, Aaronbrick, .:Ajvol:., Jonsafari, Gary, Dominik Kuropka~enwiki, Bjh~enwiki, Ruud Koot, GregorB, Qwertyus, LanguageMan, Rjwilmsi, Gmelli, YurikBot, Conscious, Fmccown, Mike Dillon, SmackBot, MalafayaBot, Morecore~enwiki, JohnWhitlock, Stiang, Hankat, Padvi~enwiki, Thijs!bot, Oliver202, Remaire, AnAj, SamatJain, Jonemerson, Destynova, Ezani, Unkx80, Cometstyles, Dominich01, VolkovBot, Amroamroamro, Philip Trueman, Informa- tionSpace, Synthebot, Luc.denys, Disooqi, Maxalbanese, Dspattison, UKoch, PixelBot, Sir Tobek, Dwiddows, XLinkBot, Addbot, Favo- nian, Vuongvina, LatentDrK, Hyju, Suffusion of Yellow, TigerHokieFan, Riclas, Boraas, ZéroBot, Donner60, Tbear1234, PenelopeKit, Justincheng12345-bot, Lxcythian, Biogeographist, SergioJimenez, Alenrooni, Vítor and Anonymous: 54

11.2 Images

• File:Vector_space_model.jpg Source: https://upload.wikimedia.org/wikipedia/commons/f/ff/Vector_space_model.jpg License: CC BY 3.0 Contributors: Own work Original artist: Riclas

11.3 Content license

• Creative Commons Attribution-Share Alike 3.0