Provided by the author(s) and NUI Galway in accordance with publisher policies. Please cite the published version when available. Title Fast and Scalable Pattern Mining for Media-Type Focused Crawling Author(s) Umbrich, Jürgen; Karnstedt, Marcel; Harth, Andreas Publication Date 2009 Jürgen Umbrich, Marcel Karnstedt, Andreas Harth "Fast and Publication Scalable Pattern Mining for Media-Type Focused Crawling", Information KDML 2009: Knowledge Discovery, Data Mining, and Machine Learning, in conjunction with LWA 2009, 2009. Item record http://hdl.handle.net/10379/1121 Downloaded 2021-09-27T17:53:57Z Some rights reserved. For more information, please see the item record link above. Fast and Scalable Pattern Mining for Media-Type Focused Crawling∗ [experience paper] Jurgen¨ Umbrich and Marcel Karnstedt and Andreas Harthy Digital Enterprise Research Institute (DERI) National University of Ireland, Galway, Ireland fi
[email protected] Abstract 1999]) wants to infer the topic of a target page before de- voting bandwidth to download it. Further, a page’s content Search engines targeting content other than hy- may be hidden in images. pertext documents require a crawler that discov- ers resources identifying files of certain media types. Na¨ıve crawling approaches do not guaran- A crawler for media type targeted search engines is fo- tee a sufficient supply of new URIs (Uniform Re- cused on the document formats (such as audio and video) source Identifiers) to visit; effective and scalable instead of the topic covered by the documents. For a scal- mechanisms for discovering and crawling tar- able media type focused crawler it is absolutely essential geted resources are needed.