Information Retrieval, Part 1
Total Page:16
File Type:pdf, Size:1020Kb
Information Retrieval Deepak Kumar Information Retrieval Searching within a document collection for a particular information need. • Traditional vs. web IR • • Query Search Engines… Altavista Entireweb Leapfish Stumpdedia Ask Excite Lycos Teoma Baidu Faroo Monster Crawler WebCrawler Bing Info.com Naver Yahoo! Search Blekko Gigablast Omgili Yandex ChaCha Google Dmoz Dogpile Go Scrub The Web Daum Hakia Spezify DuckDuckGo HotBot Stinky Teddy Matching & Ranking matched pages ranked pages 1. 2. query 3. muddy waters matching ranking “hits” Index Inverted Index • A mapping from content (words) to location. • Example: the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat Inverted Index the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat a3 cat 1 3 dog 2 3 mat 1 2 on 1 2 sat 1 3 stood 2 3 the 1 2 3 while 3 Inverted Index the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat a3 cat 1 3 dog 2 3 mat 1 2 Every word in every on 1 2 web page is indexed! sat 1 3 stood 2 3 the 1 2 3 while 3 Searching the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat a3 cat 1 3 query dog 2 3 mat 1 2 cat on 1 2 sat 1 3 stood 2 3 the 1 2 3 while 3 Searching the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat a3 cat 1 3 query dog 2 3 mat 1 2 cat on 1 2 sat 1 3 stood 2 3 the 1 2 3 while 3 Searching the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat hits a3 cat 1 3 the cat sat on dog 2 3 1 query the mat mat 1 2 cat on 1 2 the cat stood 3 sat 1 3 while a dog sat stood 2 3 the 1 2 3 while 3 Searching the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat hits a3 cat 1 3 the dog stood on 2 query dog 2 3 the mat mat 1 2 dog on 1 2 the cat stood 3 sat 1 3 while a dog sat stood 2 3 the 1 2 3 while 3 Searching the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat a3 cat 1 3 query dog 2 3 mat 1 2 cat dog on 1 2 sat 1 3 stood 2 3 the 1 2 3 while 3 Searching the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat a3 cat 1 3 query dog 2 3 mat 1 2 cat dog on 1 2 sat 1 3 stood 2 3 the 1 2 3 while 3 Searching the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat hits a3 cat 1 3 the cat stood 3 query dog 2 3 while a dog sat mat 1 2 cat dog on 1 2 sat 1 3 stood 2 3 the 1 2 3 while 3 Searching the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat a3 cat 1 3 query dog 2 3 mat 1 2 cat the sat on 1 2 ??? sat 1 3 stood 2 3 the 1 2 3 while 3 Phrase Queries the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat hits a3 cat 1 3 the cat sat on dog 2 3 1 query the mat mat 1 2 “cat sat” on 1 2 the cat stood 3 sat 1 3 while a dog sat stood 2 3 the 1 2 3 while 3 Phrase Queries the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat hits a3 cat 1 3 the cat sat on dog 2 3 1 query the mat mat 1 2 “cat sat” on 1 2 the cat stood 3 sat 1 3 while a dog sat stood 2 3 the 1 2 3 while 3 How to tell if two words occur next to each other? Phrase Queries the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat hits a3 cat 1 3 the cat sat on dog 2 3 1 query the mat mat 1 2 “cat sat” on 1 2 the cat stood 3 sat 1 3 while a dog sat stood 2 3 the 1 2 3 while 3 How to tell if two words occur next to each other? EFFICIENTLY??? Inverted Index with Location the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat a3‐5 cat 1‐2 3‐2 dog 2‐2 3‐6 mat 1‐6 2‐6 on 1‐4 2‐4 sat 1‐3 3‐7 stood 2‐3 3‐3 the 1‐1 1‐5 2‐1 2‐5 3‐1 while 3‐4 Inverted Index with Location the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat a3‐5 cat 1‐2 3‐2 query dog 2‐2 3‐6 mat 1‐6 2‐6 “cat sat” on 1‐4 2‐4 sat 1‐3 3‐7 stood 2‐3 3‐3 the 1‐1 1‐5 2‐1 2‐5 3‐1 while 3‐4 Inverted Index with Location the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat a3‐5 cat 1‐2 3‐2 1‐2, 3‐2 query dog 2‐2 3‐6 mat 1‐6 2‐6 “cat sat” on 1‐4 2‐4 sat 1‐3 3‐7 1‐3, 3‐7 stood 2‐3 3‐3 the 1‐1 1‐5 2‐1 2‐5 3‐1 while 3‐4 Inverted Index with Location the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat a3‐5 cat 1‐2 3‐2 1‐2, 3‐2 query dog 2‐2 3‐6 mat 1‐6 2‐6 “cat sat” on 1‐4 2‐4 sat 1‐3 3‐7 1‐3, 3‐7 stood 2‐3 3‐3 the 1‐1 1‐5 2‐1 2‐5 3‐1 while 3‐4 Inverted Index with Location the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat hits a3‐5 cat 1‐2 3‐2 the cat sat on dog 2‐2 3‐6 1‐2 1 query the mat mat 1‐6 2‐6 “cat sat” on 1‐4 2‐4 sat 1‐3 3‐7 1‐3 stood 2‐3 3‐3 the 1‐1 1‐5 2‐1 2‐5 3‐1 while 3‐4 NEAR* Queries the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat hits a3‐5 cat 1‐2 3‐2 3‐2 the cat stood dog 2‐2 3‐6 3 query 3‐6 while a dog sat mat 1‐6 2‐6 cat NEAR dog on 1‐4 2‐4 sat 1‐3 3‐7 stood 2‐3 3‐3 the 1‐1 1‐5 2‐1 2‐5 3‐1 while 3‐4 *NEAR: distance <= 5 NEAR* Queries the cat sat on the dog stood on the cat stood 1 23 the mat the mat while a dog sat hits a3‐5 cat 1‐2 3‐2 3‐2 the cat stood dog 2‐2 3‐6 3 query 3‐6 while a dog sat mat 1‐6 2‐6 cat NEAR dog on 1‐4 2‐4 sat 1‐3 3‐7 stood 2‐3 3‐3 the 1‐1 1‐5 2‐1 2‐5 3‐1 while 3‐4 Useful in ranking! *NEAR: distance <= 5 Matching & Ranking matched pages ranked pages 1. 2. query 3. muddy waters matching ranking “hits” Ranking & Relevance 12By far the most common Our cause was not cause of malaria is helped by the poor being bitten by an health of the troops, infected mosquito, but many of whom were there are also other suffering from malaria ways to contract the and other tropical disease. diseases. Ranking & Relevance 12By far the most common Our cause was not cause of malaria is helped by the poor being bitten by an health of the troops, infected mosquito, but many of whom were there are also other suffering from malaria ways to contract the and other tropical disease. diseases. also 1‐19 … cause 1‐6 2‐2 … malaria 1‐8 2‐19 … whom 2‐15 Ranking & Relevance 12By far the most common Our cause was not cause of malaria is helped by the poor being bitten by an health of the troops, infected mosquito, but many of whom were there are also other suffering from malaria ways to contract the and other tropical disease.