Indexing in Database Systems

Databases as a tool for the content specialist

Applying databases to indexing in the information specialists’ world

Johan van Wyk (M.Bibl.; BA Hons History, THED)

When you look into the literature on indexing in databases, you are confronted with terms such as BTree, TFSG parsing, clustered indexes, filtered" indices etc. Those are all techniques and tools used to achieve retrieval of information in database systems. But this is not at all our concern here today. Lets just say that we accept this as a given (like we do with most technology we don’t understand!) As information professionals in the information industry, what is our concern with database systems?

Database systems can be arranged on a continuum between DBMS’s and full text systems. On the one extreme are the DBMS’s: very structured, good sorting and report facility. But awful with textual information. It was never built to do that. Examples: DBMS’s : Oracle, SQL On the other extreme are the full text systems. The were specifically built for textual documents. Examples of those systems are BRS Search, Brainware and you could include Internet search engines such as Alta Vista. The distinguishing factors are: DBMS’s:  Structure  Report facility to manipulate output by manipulating the output by sorting the structure elements  Field lengths often limited  Searchable fields are limited  Complicated search

Full text systems:  The full document is the unit of information  Every word is searchable  No formal structure

Then we find a third category in the industry: text retrieval systems. Text retrieval systems were developed to get the best of both worlds. They were developed to have the structured elements of DBMS’s and the ability to handle full text. Thus we find that text retrieval systems have the following characteristics or features:  Structure facilitated by means of fields  Report facility  All fields of variable length  Indexing all fields, full text

Interestingly, though, Text retrieval systems were developed before Full text systems. Some of the reasons were computing power, retrieval techniques and (to us “obviously”) the needs of information specialists. One is always amazed at the searching functionality of systems like IBM’s “STAIRS” (1972) and online systems used by BRS, Medline and Dialog in the same period.

To be able to retrieve information in these systems programming techniques such as BTree etc. were developed. These techniques must be very successful today – volume of data is not a problem anymore. Today, as you all know, we search the internet and are amazed at the amount of information retrieved. And then we are confused and irritated by the overdose. And that is exactly what the core function of this profession is: to make sense of the masses of information. The user is not interested in being the world champion in the number of items

1 retrieved. The question is whether his need is being addressed: the retrieval of relevant information is what matters.

The IT sector has now reached a point where the indexing masses of data is “solved”. What more can you want when you have full text indexing? Why is the user now confused or irritated? The reason is that the information need was not met: finding relevant information.

Relevance is a very difficult concept to tie down. When testing information retrieval performance, this is the crucial variable to define, because it could skew your research totally. The measuring of retrieval performance is done using two parallel measures, both relying on the judgement of relevance:  Recall (The ability to retrieve relevant information)  Precision (the ability to withhold non-relevant information from the information retrieved) The technology used for full text indexing greatly enhances the recall performance or ability, to the detriment of precision performance. Then information retrieval research started focussing on techniques focussed on manipulating the output. Here we find techniques such as “relevance ranking” widely used such as in internet search engines. These techniques are quite successful since the computing power became available.

In these text retrieval systems we work with words, word stems or phrases. Almost all of the relevance ranking techniques are based upon the work of Karen Spark-Jones in Cambridge (UK). Spark-Jones brought us what became known as the term frequency–inverse document frequency theory. The tf–idf weight is a statistical measure used to evaluate how important a word is within a document and then within a collection of information items. The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in a collection. An unpublished study at Syracuse university (USA) in 1984 tested 29 variations of the tf–idf weighting scheme to determine the difference between these variations and came to the conclusion that there was no significant difference.

Another technique used for retrieval is the “fuzzy set theory” or “fuzzy logic”. This technique attempts to retrieve information without being bound to specific words or spelling. The results here are also greatly enhancing recall performance as opposed to precision.

But all of these techniques are on the output side. In the indexing profession we are on the input side. We would like to see that the entries created for retrieval improves the ability to retrieve relevant information.

Words and textual “strings” (multi-word sequences or phrases) are used index and search these systems. Words are obviously just representatives of concepts. The user actually needs information about a concept. This is exactly what information professional tries to achieve, namely to identify what the information item is “about”. Hutchins (1978) coined the phrase “aboutness” to describe this phenomenon. This is exactly where the indexer performs his skill: to create retrieval elements for optimising relevance. It is all about the process of identifying and creating index entries to represent “aboutness” of a concept.

One of the areas investigated since the 1970’s was the application of linguistic theory. The main problem here was that linguistic theory only gave answers to the function of words up to the unit of a sentence. Very little applicable theory was available for working with whole documents or eve collections of textual items.

Today we want to look at what Database systems can offer us on the input side.

The database systems gives us the following features: (specifically text retrieval systems)

 The ability to house and work with a collection of information items

2  The structure for organising or categorising meta-data of a collection  The ability to retrieve full text – every word  The ability to retrieve items specific to individual meta data categories  Searching functionality : Boolean, word stem- and proximity searching  Word and string indexes related to the collection as a whole or within a field  Batch modification of indexes  Some even gives us a thesaurus capability

Because text retrieval database systems build indexes, we can use these to identify more successful index terms. Here I would suggest using the elements of the term frequency– inverse document frequency theory: the word frequencies. This theory acknowledges the distinctive roles of words in a document, in a collection and the documents related to a word. Automatic indexing theory identified it as:  Within document frequency (WDF)  Collection frequency (CF)  Document frequency (DF)

WDF gives us the frequency of a word in a document. Too low and too high frequencies gives us non-significant words. We need to look at the medium to higher frequencies. The focus here is the significance and role of a term within a document.

CF gives you the frequency in a collection. Again, the medium to frequencies are significant. The focus here is the significance and role of a term within a document collection.

DF shows the number of documents linked to a word. Again, the medium to frequencies are significant. The focus here is the significance of a document in a collection with regards to a specific term or concept.

Text retrieval systems, because they build indexes, can give us these word frequencies. Through the knowledge of word frequency theory one can assess the significance of terms. Text retrieval systems also allows us to incorporate full text into one of the fields alongside the meta data.

Indexers tend to focus on index terms related to a specific item of information, relating to the “within document frequency” . One can therefore say that this is the element best catered for namely the role and significance of a term within an information item.

The other two elements should be a concern.

The significance in a collection is often taken care of by:  Using an indexer that “knows the subject area”  The indexer’s knowledge of the collection  The indexer’s knowledge of the users, company or environment. All three of these are done in a sub-conscious manner and often not really taken care of.

The third element, namely the role or significance of a document in a collection with regards to a specific term, is often not even considered. An analysis of the subject coverage of the documentation used in a country wide HSRC study in 1988 gave alarming results. This is not apparent looking at the study’s reports.

We can use text retrieval database systems to determine better index terms in the following way:

With single documents one can do an analysis of the term frequencies within a document. Most text retrieval systems can load a singe document full text into a record in the database, generating the word frequencies in the index. Stop word lists can successfully be applied.

3 With collections of documents the text retrieval database systems can be used to:  Determine collection frequencies of terms in a collection before indexing a new document within or across fields  Determining document frequencies of terms (the number of documents relating to the index term) in a collection when indexing a document.

The structure of database systems can successfully be used to:  Assigning index terms to a specific meta-data element, increasing the specific role of a term  Creating database fields that would enhance the role of index terms. An example here is where subject field is divided into broad terms for a controlled vocabulary and a separate field for free text indexing, and even having a separate field for names of persons, places and companies

Most Text retrieval systems has the ability to do batch changes in indexes. When Northern Province became Limpopo – how many of the information services made that change?

As mentioned, some text retrieval database systems has a thesaurus feature. Some are linked to an existing thesaurus, some allow the user or indexer to build a thesaurus. Building your own thesaurus can be of great value in narrower subject fields. This could be a very valuable tool fir the indexer to use on the input side.

Lastly we can use Text retrieval systems to test the effectiveness of our indexing, using the performance measures of recall and precision. Information services should, on a regular basis, reflect on the effectiveness of indexes of a collection by analysing and rectifying problems on a regular basis.

Database systems, and specifically Text retrieval databse systems have developed to a standard where they are very efficient with a whole collection of features relevant to the input soide of indexing. It is therefore a powerful tool in the hands of the indexer.

4 Bibliography

Bertino, E., B. C. Ooi, R. Sacks-Davis, K-L Tan, J. Zobel, B. Shidlovsky, and B. Catania. 1997. Indexing Techniques for Advanced Database Systems. Kluwer Academic Publishers

Elmasri R. and S. Navathe. 2000 Fundamentals of database systems. Addison-Wesley.

Graf P., 1996 Term Indexing. Springer.

Hutchins, W. J. The concept of 'aboutness' in subject indexing. Aslib Proceedings 30 (5), May 1978, p.172-181.

Manning, Christopher D. ; Raghavan, Prabhakar and Schütze, Hinrich Introduction to Information retrieval, 2008 Cambridge University Press. http://informationretrieval.org/ http://nlp.stanford.edu/IR-book/html/htmledition/irbook.html

Ramesh et al. ,2001 In: http://www.cs.toronto.edu/~mcosmin/publications/thesis/node61.html

Salton, G. M.J. McGill 1983. Introduction to modern information retrieval. McGraw Hill.