DIGITAL NOTES ON INFORMATION RETRIEVAL SYSTEMS (R17A1209) B.TECH IV YEAR - I SEM (2020-2021) DEPARTMENT OF INFORMATION TECHNOLOGY MALLA REDDY COLLEGE OF ENGINEERING & TECHNOLOGY (Autonomous Institution – UGC, Govt. of India) (Affiliated to JNTUH, Hyderabad, Approved by AICTE - Accredited by NBA & NAAC – ‘A’ Grade - ISO 9001:2015 Certified) Maisammaguda, Dhulapally (Post Via. Hakimpet), Secunderabad – 500100, Telangana State, INDIA. MRCET-IT Page 1 MALLA REDDY COLLEGE OF ENGINEERING & TECHNOLOGY DEPARTMENT OF INFORMATION TECHNOLOGY IV Year B.Tech IT –I Sem L T /P/D C 3 -/-/- 3 (R17A1209)INFORMATION RETRIEVAL SYSTEMS (Core Elective IV) OBJECTIVES Study fundamentals of DBMS, Data warehouse and Digital libraries Learn various preprocessing techniques and indexing approaches in text mining Know various clustering approaches and study different similarity measures Study various search techniques in information retrieval systems Know different cognitive approaches used in text retrieval systems and evaluation approaches Study retrieval in multimedia systems and know various evaluation measures Know about query languages and online IRsystem UNIT-I Introduction: Definition, Objectives, Functional Overview, Relationship to DBMS, Digital libraries and Data Warehouses. Information Retrieval System Capabilities: Search, Browse, Miscellaneous UNIT-II Cataloging and Indexing: Objectives, Indexing Process, Automatic Indexing, Information Extraction. Data Structures: Introduction, Stemming Algorithms, Inverted file structures, N-gram data structure, PAT data structure, Signature file structure, Hypertext data structure. UNIT-III Automatic Indexing: Classes of automatic indexing, Statistical indexing, Natural language, Concept indexing, Hypertext linkages Document and Term Clustering: Introduction, Thesaurus generation, Item clustering, Hierarchy of clusters. UNIT-IV User Search Techniques: Search statements and binding, Similarity measures and ranking, Relevance feedback, Selective dissemination of information search, weighted searches of Boolean systems, Searching the Internet and hypertext. Information Visualization: Introduction, Cognition and perception, Information visualization technologies. UNIT-V Text Search Algorithms: Introduction, Software text search algorithms, Hardware text search systems. Information System Evaluation: Introduction, Measures used in system evaluation, Measurement example – TREC results. TEXTBOOK: 1. Information Storage and Retrieval Systems: Theory and Implementation by Gerald J. Kowalski, Mark T. Maybury , Second Edition, Kluwer Academic Publishers. MRCET-IT Page 2 REFERENCES: 1. Frakes, W.B., Ricardo Baeza-Yates: Information Retrieval Data Structures and Algorithms, Prentice Hall, 1992. 2. Modern Information Retrival By Yates Pearson Education. 3. Information Storage & Retieval By Robert Korfhage – John Wiley & Sons. OUTCOMES: Upon completion of the course, the students are expected to: 1. Recognize the Boolean Model, Vector Space Model, and Probabilistic Model. 2. Understand retrieval utilities. 3. Understand different formatting tags 4. Understand cross-language information retrieval 5. Understand the clustering techniques 6. Determine the efficiency. MRCET-IT Page 3 MALLA REDDY COLLEGE OF ENGINEERING & TECHNOLOGY DEPARTMENT OF INFORMATION TECHNOLOGY INDEX S. No. Topic Page Unit no. 1 I Introduction 5 - 12 2 I Information Retrieval System Capabilities 12 -24 3 II Cataloging and Indexing 24-29 4 II Data Structures 30-41 5 III Automatic Indexing 42-45 6 III Document and Term Clustering 46-50 7 IV Text Search Algorithms 51-58 8 IV Information System Evaluation 58-66 9 V Text Search Algorithms 67-79 10 V Information System Evaluation 79-84 MRCET-IT Page 4 UNIT-I Introduction: Definition, Objectives, Functional Overview, Relationship to DBMS, Digital libraries and Data Warehouses. Information Retrieval System Capabilities: Search, Browse, Miscellaneous Introduction There is a potential for confusion in the understanding of the differences between Database Management Systems (DBMS) and Information Retrieval Systems. It is easy to confuse the software that optimizes functional support of each type of system with actual information or structured data that is being stored and manipulated. The importance of the differences lies in the inability of a database management system to provide the functions needed to process “information.” The opposite, an information system containing structured data, also suffers major functional deficiencies. Definition of Information Retrieval System An Information Retrieval System is a system that is capable of storage, retrieval, and maintenance of information6. Information in this context can be composed of text (including numeric and date data), images, audio, video and other multi-media objects. Techniques are beginning to emerge to search these other media types (e.g., EXCALIBUR’s Visual RetrievalWare, VIRAGE video indexer). The term “item” is used to represent the smallest complete unit that is processed and manipulated by the system. The definition of item varies by how a specific source treats information. A complete document, such as a book, newspaper or magazine could be an item. For example a video news program could be considered an item. It is composed of text in the form of closed captioning, audio text provided by the speakers, and the video images being displayed. An Information Retrieval System consists of a software program that facilitates a user in finding the information the user needs. The system may use standard computer hardware or specialized hardware to support the search sub function and to convert non-textual sources to a searchable media (e.g., transcription of audio to text). Thus search composition, search execution, and reading non-relevant items are all MRCET-IT Page 5 aspects of information retrieval overhead. With the advent of inexpensive powerful personnel computer processing systems and high speed, large capacity secondary storage products, it has become commercially feasible to provide large textual information databases for the average user. The introduction and exponential growth of the Internet along with its initial WAIS (Wide Area Information Servers) capability and more recently advanced search servers (e.g., INFOSEEK, EXCITE) has provided a new avenue for access to terabytes of information (over 800 million indexable pages -Lawrence-99.) The algorithms and techniques to optimize the processing and access of large quantities of textual data were once the sole domain of segments of the Government, a few industries, and academics. Images across the Internet are searchable from many web sites such as WEBSEEK, DITTO.COM, ALTAVISTA/IMAGES. News organizations such as the BBC are processing the audio news they have produced and are making historical audio news searchable via the audio transcribed versions of the news. Major video organizations such as Disney are using video indexing to assist in finding specific images in their previously produced videos to use in future videos or incorporate in advertising. Objectives of Information Retrieval Systems The general objective of an Information Retrieval System is to minimize the overhead of a user locating needed information. Overhead can be expressed as the time a user spends in all of the steps leading to reading an item containing the needed information (e.g., query generation, query execution, scanning results of query to select items to read, reading non-relevant items). In information retrieval the term “relevant” item is used to represent an item containing the needed information. From a user’s perspective “relevant” and “needed” are synonymous. The two major measures commonly associated with information systems are 1) Precision MRCET-IT Page 6 2) Recall When a user decides to issue a search looking for information on a topic, the total database is logically divided into four segments shown in Figure 1.1. Relevant items are those documents that contain information that helps the searcher in answering his question. Non-relevant items are those items that do not provide any directly useful information. There are two possibilities with respect to each item: it can be retrieved or not retrieved by the user’s query. Precision and recall are defined as: Precision is directly affected by retrieval of non-relevant items and drops to a number close to zero. Recall is not effected by retrieval of non-relevant items and thus remains at 100 per cent once achieved. Information Retrieval Systems such as RetrievalWare, TOPIC, AltaVista, Infoseek and INQUERY that the idea of accepting natural language queries is becoming a standard system feature. This allows users to state in natural language what they are interested in finding. But the completeness of the user specification is limited by the user’s willingness to construct long natural language queries. Most users on the Internet enter one or two search terms. Functional Overview A total Information Storage and Retrieval System is composed of four major functional processes: 1) Item Normalization MRCET-IT Page 7 2) Selective Dissemination of Information (i.e., “Mail”) 3) Archival Document Database Search, and an Index 4) Database Search along with the Automatic File Build process that supports Index Files. 1) Item Normalization: The first step in any integrated system is to normalize the incoming items to a standard format. Item normalization provides logical restructuring of the item. Additional operations during item normalization are needed