Extracting Unstructured Data from Template Generated Web Documents Ling Ma Nazli Goharian Abdur Chowdhury Information Retrieval Laboratory Information Retrieval Laboratory America Online Inc. Computer Science Department Computer Science Department
[email protected] Illinois Institute of Technology Illinois Institute of Technology
[email protected] [email protected] ABSTRACT section of a page contributes to the same problem. A problem that We propose a novel approach that identifies web page templates is not well studied is the negative effect of such noise data on the and extracts the unstructured data. Extracting only the body of the result of the user queries. Removing this data improves the page and eliminating the template increases the retrieval effectiveness of search by reducing the irrelevant results. precision for the queries that generate irrelevant results. We Furthermore, we argue that the irrelevant results, even covering a believe that by reducing the number of irrelevant results; the small fraction of retrieved results, have the restaurant-effect, users are encouraged to go back to a given site to search. Our namely users are ten times less likely to return or use the search experimental results on several different web sites and on the service after a bad experience. This is of more importance, whole cnnfn collection demonstrate the feasibility of our approach. considering the fact that an average of 26.8% of each page is formatting data and advertisement. Table 1 shows the ratio of the formatting data to the real data in the eight web sites we examined. Categories and Subject Descriptors H.3.3 [Information Storage and Retrieval]: Search Process Table 1: Template Ratio in Web Collections Web page Collection Template Ratio General Terms Cnnfn.com 34.1% Design, Experimentation Netflix.com 17.8% Keywords NBA.com 18.1% Automatic template removal, text extraction, information retrieval, Amazon.com 19.3% Retrieval Accuracy Ebay.com 22.5% Disneylandsource.com 54.1% 1.