Web Crawler Architecture Central Database for Coordinating the Crawl
Total Page：16
File Type：pdf, Size：1020Kb
Web Crawler Architecture central database for coordinating the crawl. Heydon and Najork described Mercator [8,9], a distributed and MARC NAJORK extensible web crawler that was to become the Microsoft Research, Mountain View, CA, USA blueprint for a number of other crawlers. Other distributed crawling systems described in the literature Synonyms include PolyBot [11], UbiCrawler [1], C-proc [4] and Web crawler; Robot; Spider Dominos [7]. Definition Foundations A web crawler is a program that, given one or more Conceptually, the algorithm executed by a web crawler seed URLs, downloads the web pages associated with is extremely simple: select a URL from a set of these URLs, extracts any hyperlinks contained in them, candidates, download the associated web pages, extract and recursively continues to download the web pages the URLs (hyperlinks) contained therein, and add those identified by these hyperlinks. Web crawlers are an URLs that have not been encountered before to the important component of web search engines, where candidate set. Indeed, it is quite possible to implement they are used to collect the corpus of web pages a simple functioning web crawler in a few lines of a indexed by the search engine. Moreover, they are used high-level scripting language such as Perl. in many other applications that process large numbers However, building a web-scale web crawler of web pages, such as web data mining, comparison imposes major engineering challenges, all of which are shopping engines, and so on. Despite their conceptual ultimately related to scale. In order to maintain a search simplicity, implementing high-performance web engine corpus of say, ten billion web pages, in a crawlers poses major engineering challenges due to the reasonable state of freshness, say with pages being scale of the web. In order to crawl a substantial fraction refreshed every 4 weeks on average, the crawler must of the “surface web” in a reasonable amount of time, download over 4,000 pages/second. In order to achieve web crawlers must download thousands of pages per this, the crawler must be distributed over multiple second, and are typically distributed over tens or computers, and each crawling machine must pursue hundreds of computers. Their two main data structures multiple downloads in parallel. But if a distributed and – the “frontier” set of yet-to-be-crawled URLs and the highly parallel web crawler were to issue many set of discovered URLs – typically do not fit into main concurrent requests to a single web server, it would in memory, so efficient disk-based representations need to all likelihood overload and crash that web server. be used. Finally, the need to be “polite” to content Therefore, web crawlers need to implement politeness providers and not to overload any particular web policies that rate-limit the amount of traffic directed to server, and a desire to prioritize the crawl towards any particular web server (possibly informed by that high-quality pages and to maintain corpus freshness server’s observed responsiveness). There are many impose additional engineering challenges. possible politeness policies; one that is particularly easy to implement is to disallow concurrent requests to Historical Background the same web server; a slightly more sophisticated Web crawlers are almost as old as the web itself. In the policy would be to wait for time proportional to the last spring of 1993, just months after the release of NCSA download time before contacting a given web server Mosaic, Matthew Gray [6] wrote the first web crawler, again. the World Wide Web Wanderer, which was used from In some web crawler designs (e.g., the original 1993 to 1996 to compile statistics about the growth of Google crawler [2] and PolyBot [11]), the page the web. A year later, David Eichmann [5] wrote the downloading processes are distributed, while the major first research paper containing a short description of a data structures – the set of discovered URLs and the set web crawler, the RBSE spider. Burner provided the of URLs that have to be downloaded – are maintained first detailed description of the architecture of a web by a single machine. This design is conceptually crawler, namely the original Internet Archive crawler simple, but it does not scale indefinitely; eventually the [3]. Brin and Page’s seminal paper on the (early) central data structures become a bottleneck. The architecture of the Google search engine contained a alternative is to partition the major data structures over brief description of the Google crawler, which used a the crawling machines. Ideally, this should be done in distributed system of page-fetching processes and a such a way as to minimize communication between the crawlers. One way to achieve this is to assign URLs to based heap would be far too expensive, since adding crawling machines based on their host name. and removing a URL would require multiple seek Partitioning URLs by host name means that the crawl operations. The Mercator design uses a frontier data scheduling decisions entailed by the politeness policies structure that has two stages: a front-end that supports can be made locally, without any communication with prioritization of individual URLs and a back-end that peer nodes. Moreover, since most hyperlinks refer to enforces politeness policies; both the front-end and the pages on the same web server, the majority of links back-end are composed of a number of parallel FIFO extracted from downloaded web pages is tested against queues [9]. If the URL space is partitioned according to and added to local data structures, not communicated to host names among the web crawlers, the frontier data peer crawlers. Mercator and C-proc adopted this design structure is partitioned along the same lines. [9,4]. In the simplest case, the frontier data structure is Once a hyperlink has been extracted from a web just a collection of URLs. However, in many settings it page, the crawler needs to test whether this URL has is desirable to attach some attributes to each URL, such been encountered before, in order to avoid adding as the time when it was discovered, or (in the scenario multiple instances of the same URL to its set of of continuous crawling) the time of last download and pending URLs. This requires a data structure that a checksum or sketch of the document. Such historical supports set membership test, such as a hash table. information makes it easy to determine whether the Care should be taken that the hash function used is document has changed in a meaningful way, and to collision-resistant, and that the hash values are large adjust its crawl priority. enough (maintaining a set of n URLs requires hash In general, URLs should be crawled in such a way 2 values with log2n bits each). If RAM is not an issue, as to maximize the utility of the crawled corpus. the table can be maintained in memory (and Factors that influence the utility are the aggregate occasionally persisted to disk for fault tolerance); quality of the pages, the demand for certain pages and otherwise a disk-based implementation must be used. topics, and the freshness of the individual pages. All Implementing fast disk-based set membership tests is these factors should be considered when deciding on extremely hard, due to the physical limitations of hard the crawl priority of a page: a high-quality, highly- drives (a single seek operation takes on the order of 10 demanded and fast-changing page (such as the front ms). For a disk-based design that leverages locality page of an online newspaper) should be recrawled properties in the stream of discovered URLs as well as frequently, while high-quality but slow-changing and the domain-specific properties of web crawling, see fast-changing but low-quality pages should receive a [9]. If the URL space is partitioned according to host lower priority. The priority of newly discovered pages names among the web crawlers, the set data structure is cannot be based on historical information about the partitioned in the same way, with each web crawling page itself, but it is possible to make educated guesses machine maintaining only the portion of the set based on per-site statistics. Page quality is hard to containing its hosts. Consequently, an extracted URL quantify; popular proxies include link-based measures that is not maintained by the crawler that extracted it such as PageRank and behavioral measures such as must be sent to the peer crawler responsible for it. page or site visits (obtained from web beacons or Once it has been determined that a URL has not toolbar data). been previously discovered, it is added to the frontier In addition to these major data structures, most set containing the URLs that have yet to be web-scale web crawlers also maintain some auxiliary downloaded. The frontier set is generally too large to data structures, such as caches for DNS lookup results. be maintained in main memory (given that the average Again, these data structures may be partitioned across URL is about 100 characters long and the crawling the crawling machines. system might maintain a frontier of ten billion URLs). The frontier could be implemented by a simple disk- Key Applications based FIFO queue, but such a design would make it Web crawlers are a key component of web search hard to enforce the politeness policies, and also to engines, where they are used to collect the pages that prioritize certain URLs (say URLs referring to fast- are to be indexed. Crawlers have many applications changing news web sites) over other URLs. URL beyond general search, for example in web data mining prioritization could be achieved by using a priority (e.g., Attributor, a service that mines the web for queue implemented as a heap data structure, but a disk- copyright violations, or ShopWiki, a price comparison 7.