DOCSLIB.ORG

Application of ARIMA(1,1,0) Model for Predicting Time Delay of Search Engine Crawlers

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Application of ARIMA(1,1,0) Model for Predicting Time Delay of Search Engine Crawlers 26 Informatica Economică vol. 17, no. 4/2013 Application of ARIMA(1,1,0) Model for Predicting Time Delay of Search Engine Crawlers Jeeva JOSE1, P. Sojan LAL2 1 Department of Computer Applications, BPC College, Piravom, Kerala, India 2 School of Computer Sciences, Mahatma Gandhi University, Kottayam, Kerala, India [email protected], [email protected] World Wide Web is growing at a tremendous rate in terms of the number of visitors and num- ber of web pages. Search engine crawlers are highly automated programs that periodically visit the web and index web pages. The behavior of search engines could be used in analyzing server load, quality of search engines, dynamics of search engine crawlers, ethics of search engines etc. The more the number of visits of a crawler to a web site, the more it contributes to the workload. The time delay between two consecutive visits of a crawler determines the dynamicity of the crawlers. The ARIMA(1,1,0) Model in time series analysis works well with the forecasting of the time delay between the visits of search crawlers at web sites. We con- sidered 5 search engine crawlers, all of which could be modeled using ARIMA(1,1,0).The re- sults of this study is useful in analyzing the server load. Keywords: ARIMA, Search Engine Crawler, Web logs, Time delay, Prediction Introduction before it crawls the web pages. The crawlers 1 Crawlers also known as ‘bots’, ‘robots’ or which access this file first and proceeds to ‘spiders’ are highly automated programs crawling are known as ethical crawlers and which are seldom regulated manually[1][2]. other crawlers who do not access this file are Crawlers form the basic building blocks of called unethical crawlers. The robots.txt file search engines which periodically visit the contains the information about which pages web sites, identify new web sites, update the are allowed for crawling and which all fold- new information and index the web pages in ers and pages are denied access. Certain pag- search engine archives. The log files generat- es and folders are denied access because they ed at web sites play a vital role in analyzing contain sensitive information which is not in- user as well as the behavior of the crawlers. tended to be publically available. There may Most of the works in web usage mining or be situations where two or more versions of a web log mining is related to user behavior as page will be available one as html and other they have application in target advertising, one as pdf. The crawlers can be made do online sales and marketing, market basket avoid crawling the pdf version to avoid re- analysis, personalization etc. There is open dundant crawling. Also certain files like Ja- source software available like Google Ana- vaScript, images, style sheets etc. can be lytics which measures the number of visitors, avoided for saving the time and bandwidth. duration of the visits, the demographic from There are two ways to do this. First one is which the visitor comes etc. But it cannot with the help of robots meta tag and the other identify search engine visits because Google one is with the help of robots.txt file. The ro- Analytics track users with the help of JavaS- bots.txt file contains the list of all user agents cript and search engine crawlers do not ena- and the folders or pages which are disallowed ble the JavaScript embedded in web pages [30]. The structure of a robots.txt file is fol- when the crawlers visit the web sites [3]. lows. The search engine crawlers initially access User-agent: the robots.txt file which specifies the Robot Disallow: Exclusion Protocol. Robots.txt is a text file “User-agent:” is the search engine crawler kept at the root of the web site directory. The and “Disallow:” lists the files and directories crawlers are supposed to access this file first to be excluded from indexing. In addition to DOI: 10.12948/issn14531305/17.4.2013.03 Informatica Economică vol. 17, no. 4/2013 27 “User-agent:” and “Disallow:” entries, com- ther increase, when there are a number of ment lines are included by putting the # sign mobile crawlers from different search en- at the beginning of the line. For example all gines. user agents are disallowed from accessing the all these mobile crawlers will stay in the folder /a.# All user agents are disallowed to memory of the remote system and will see the /a directory. consume lot of memory that could have User-agent: * otherwise been used for some other use- Disallow: /a/ ful purposes; The crawlers which initially access the ro- it can also happen that the remote system bots.txt and then the subsequent files or fold- may not allow the mobile crawlers to re- ers are known as ethical crawlers whereas side permanently in its memory due to others are known as unethical crawlers. Some security reasons; crawlers like “Googlebot”, “Yahoo! Slurp” in case a page changes very quickly then and “MSNbot” cache the robots.txt file for a the mobile crawler immediately accesses web site and hence during the modification the changed page and sends it to the of robots.txt file, these robots may disobey search engine to maintain up-to-date in- the rules. Roughly, a crawler starts off with dex. This will result in wastage of net- the URL for an initial page p0. It retrieves p0, work bandwidth and CPU cycles etc [30]. extracts any URLs in it, and adds them to a Recently web crawlers are used for focused queue of URLs to be scanned. Then the crawling, shopbot implementation and value crawler gets URLs from the queue (in some added services on the web. As a result more order), and repeats the process. Every page active robots are crawling on the web and that is scanned is given to a client that saves many more are expected to follow which will the pages, creates an index for the pages, or increase the search engine traffic and web summarizes or analyzes the content of the server activity [4]. The Auto Regressive In- pages [26]. Certain crawlers avoid too much tegrated Moving Average (ARIMA) Model load on a server by crawling the server at a was used to predict the time delay between low speed during peak hours of the day and two consecutive visits of a search engine at a high speed during late night and early crawler. We used the differenced first-order morning [2]. A crawler for a large search en- autoregressive model, ARIMA(1,1,0) for gine has to address two issues. First, it has to forecasting the time delay between two con- have a good crawling strategy, i.e., a strategy secutive visits of search engine crawlers. for deciding which pages to download next. Second, it needs to have a highly optimized 2 Background Literature system architecture that can download a large There are several works that mentions about number of pages per second while being ro- the search engine crawler behavior. A fore- bust against crashes, manageable, and con- casting model is proposed for the number of siderate of resources and web servers [24]. pages crawled by search engine crawlers at a There are two important aspects in designing web site [3]. Sun et al has conducted a large efficient web spiders, i.e. crawling strategy scale study of robots.txt [2]. A characteriza- and crawling performance. Crawling strate- tion study and metrics of search engine gy deals with the way the spider decides to crawlers is done to analyze the qualitative what pages should be downloaded next. Gen- features, periodicity of visits and the perva- erally, the web spider cannot download all siveness of visits to a web site [4]. The work- pages on the web due to the limitation of its ing of a search engine crawler is explained in resources compared to the size of the web [5]. Neilsen NetRatings is one of the leading [28]. internet and digital media audience infor- The mobile crawlers that always stay in the mation and analysis services. NetRatings memory of the remote system occupy a con- have provided a study on the usage statistics siderable portion of it. This problem will fur- of search engines in United States [6]. Com- DOI: 10.12948/issn14531305/17.4.2013.03 28 Informatica Economică vol. 17, no. 4/2013 mercial search engines play a lead role in the traffic generated at web sites is contribut- World Wide Web information dissemination ed by search engine crawlers [13]. The ad- and access. The evidence and possible causes vantages of preprocessing are: of search engine bias is also studied [7]. An the storage space is reduced as only the empirical pilot study is done to see the rela- data relevant to web mining is stored; tionship between JavaScript usage and web the user visits and image files are re- site usage. The intention was to establish moved so that the precision of web min- whether JavaScript based hyperlinks attract ing is improved. or repel crawlers resulting in an increase or The web logs are unstructured and unformat- decrease in web site visibility [8]. The ethics ted raw source of data. Unsuccessful status of search engine crawlers is identified using codes and entries pertaining to irrelevant data quantitative models [9]. Analysis of the tem- like JavaScript, images, stylesheets etc. in- poral behavior of search engine crawlers at cluding user information are removed. The web sites is also done [10]. There is a signifi- most widely used log file formats are Com- cant difference in the time delay between and mon Log File Format and Extended Log File among various search engine crawlers at web Format.
Load more

Recommended publications
  • Study of Web Crawler and Its Different Types
    IOSR Journal of Computer Engineering (IOSR-JCE) e-ISSN: 2278-0661, p- ISSN: 2278-8727Volume 16, Issue 1, Ver. VI (Feb. 2014), PP 01-05 www.iosrjournals.org Study of Web Crawler and its Different Types Trupti V. Udapure1, Ravindra D. Kale2, Rajesh C. Dharmik3 1M.E. (Wireless Communication and Computing) student, CSE Department, G.H. Raisoni Institute of Engineering and Technology for Women, Nagpur, India 2Asst Prof., CSE Department, G.H. Raisoni Institute of Engineering and Technology for Women, Nagpur, India, 3Asso.Prof. & Head, IT Department, Yeshwantrao Chavan College of Engineering, Nagpur, India, Abstract : Due to the current size of the Web and its dynamic nature, building an efficient search mechanism is very important. A vast number of web pages are continually being added every day, and information is constantly changing. Search engines are used to extract valuable Information from the internet. Web crawlers are the principal part of search engine, is a computer program or software that browses the World Wide Web in a methodical, automated manner or in an orderly fashion. It is an essential method for collecting data on, and keeping in touch with the rapidly increasing Internet. This Paper briefly reviews the concepts of web crawler, its architecture and its various types. Keyword: Crawling techniques, Web Crawler, Search engine, WWW I. Introduction In modern life use of internet is growing in rapid way. The World Wide Web provides a vast source of information of almost all type. Now a day’s people use search engines every now and then, large volumes of data can be explored easily through search engines, to extract valuable information from web.
    [Show full text]
  • Distributed Web Crawling Using Network Coordinates
    Distributed Web Crawling Using Network Coordinates Barnaby Malet Department of Computing Imperial College London [email protected] Supervisor: Peter Pietzuch Second Marker: Emil Lupu June 16, 2009 2 Abstract In this report we will outline the relevant background research, the design, the implementation and the evaluation of a distributed web crawler. Our system is innovative in that it assigns Euclidean coordinates to crawlers and web servers such that the distances in the space give an accurate prediction of download times. We will demonstrate that our method gives the crawler the ability to adapt and compensate for changes in the underlying network topology, and in doing so can achieve significant decreases in download times when compared with other approaches. 3 4 Acknowledgements Firstly, I would like to thank Peter Pietzuch for the help that he has given me throughout the course of the project as well as showing me support when things did not go to plan. Secondly, I would like to thank Johnathan Ledlie for helping me with some aspects of the implemen- tation involving the Pyxida library. I would also like to thank the PlanetLab support team for giving me extensive help in dealing with complaints from web masters. Finally, I would like to thank Emil Lupu for providing me with feedback about my Outsourcing Report. 5 6 Contents 1 Introduction 9 2 Background 13 2.1 Web Crawling . 13 2.1.1 Web Crawler Architecture . 14 2.1.2 Issues in Web Crawling . 16 2.1.3 Discussion . 17 2.2 Crawler Assignment Strategies . 17 2.2.1 Hash Based .
    [Show full text]
  • Panacea D4.1
    SEVENTH FRAMEWORK PROGRAMME THEME 3 Information and communication Technologies PANACEA Project Grant Agreement no.: 248064 Platform for Automatic, Normalized Annotation and Cost-Effective Acquisition of Language Resources for Human Language Technologies D4.1 Technologies and tools for corpus creation, normalization and annotation Dissemination Level: Public Delivery Date: July 16, 2010 Status – Version: Final Author(s) and Affiliation: Prokopis Prokopidis, Vassilis Papavassiliou (ILSP), Pavel Pecina (DCU), Laura Rimel, Thierry Poibeau (UCAM), Roberto Bartolini, Tommaso Caselli, Francesca Frontini (ILC- CNR), Vera Aleksic, Gregor Thurmair (Linguatec), Marc Poch Riera, Núria Bel (UPF), Olivier Hamon (ELDA) Technologies and tools for corpus creation, normalization and annotation Table of contents 1 Introduction ........................................................................................................................... 3 2 Terminology .......................................................................................................................... 3 3 Corpus Acquisition Component ............................................................................................ 3 3.1 Task description ........................................................................................................... 4 3.2 State of the art .............................................................................................................. 4 3.3 Existing tools ..............................................................................................................
    [Show full text]
  • Report for Portal Specific
    Report for Portal specific Time range: 2013/01/01 00:00:07 - 2013/03/31 23:59:59 Generated on Fri Apr 12, 2013 - 18:27:59 General Statistics Summary Summary Hits Total Hits 1,416,097 Average Hits per Day 15,734 Average Hits per Visitor 6.05 Cached Requests 8,154 Failed Requests 70,823 Page Views Total Page Views 1,217,537 Average Page Views per Day 13,528 Average Page Views per Visitor 5.21 Visitors Total Visitors 233,895 Average Visitors per Day 2,598 Total Unique IPs 49,753 Bandwidth Total Bandwidth 77.49 GB Average Bandwidth per Day 881.68 MB Average Bandwidth per Hit 57.38 KB Average Bandwidth per Visitor 347.40 KB 1 Activity Statistics Daily Daily Visitors 4,000 3,500 3,000 2,500 2,000 Visitors 1,500 1,000 500 0 2013/01/01 2013/01/15 2013/02/01 2013/02/15 2013/03/01 2013/03/15 Date Daily Hits 40,000 35,000 30,000 25,000 Hits 20,000 15,000 10,000 5,000 0 2013/01/01 2013/01/15 2013/02/01 2013/02/15 2013/03/01 2013/03/15 Date 2 Daily Bandwidth 1,300,000 1,200,000 1,100,000 1,000,000 900,000 800,000 700,000 600,000 500,000 Bandwidth (KB) 400,000 300,000 200,000 100,000 0 2013/01/01 2013/01/15 2013/02/01 2013/02/15 2013/03/01 2013/03/15 Date Daily Activity Date Hits Page Views Visitors Average Visit Length Bandwidth (KB) Sun 2013/02/10 11,783 10,245 2,280 13:01 648,207 Mon 2013/02/11 16,454 14,146 2,484 10:05 906,702 Tue 2013/02/12 19,572 17,089 3,062 07:47 926,190 Wed 2013/02/13 14,554 12,402 2,824 06:09 958,951 Thu 2013/02/14 12,577 10,666 2,690 05:03 821,129 Fri 2013/02/15 15,806 12,697 2,868 07:02 1,208,095 Sat 2013/02/16 16,811 14,939
    [Show full text]
  • Scalability and Efficiency Challenges in Large-Scale Web Search
    5/1/14 Scalability and Efficiency Challenges in Large-Scale Web Search Engines " Ricardo Baeza-Yates" B. Barla Cambazoglu! Yahoo Labs" Barcelona, Spain" Disclaimer Dis •# This talk presents the opinions of the authors. It does not necessarily reflect the views of Yahoo Inc. or any other entity." •# Algorithms, techniques, features, etc. mentioned here might or might not be in use by Yahoo or any other company." •# Some non-technical material (e.g., images) provided in this presentation were taken from the Web." 1 5/1/14 Yahoo Labs Barcelona •# Research topics" •# Web retrieval" –# web data mining" –# distributed web retrieval" –# semantic search" –# scalability and efficiency" –# social media" –# opinion/sentiment retrieval" –# web retrieval" –# personalization" Outline of the Tutorial •# Background (35 minutes)" •# Main sections" –# web crawling (75 minutes + 5 minutes Q/A)" –# indexing (75 minutes + 5 minutes Q/A)" –# query processing (90 minutes + 5 minutes Q/A)" –# caching (40 minutes + 5 minutes Q/A)" •# Concluding remarks (10 minutes)" •# Questions and open discussion (15 minutes)" 2 5/1/14 Structure of Main Sections •# Definitions" •# Metrics" •# Issues and techniques" –# single computer" –# cluster of computers" –# multiple search sites" •# Research problems" Background 3 5/1/14 Brief History of Search Engines •# Past" -# Before browsers" -# Gopher" -# Before the bubble" -# Altavista" -# Lycos" -# Infoseek" -# Excite" -# HotBot" •# Current" •# Future" -# After the bubble" •# Global" •# Facebook ?" -# Yahoo" •# Google, Bing" •# …"
    [Show full text]
  • Towards a Distributed Web Search Engine
    Towards a Distributed Web Search Engine Ricardo Baeza-Yates Yahoo! Research Barcelona, Spain Joint work with Barla Cambazoglu, Aristides Gionis, Flavio Junqueira, Mauricio Marín, Vanessa Murdock (Yahoo! Research) and many other people Web Search Web Context 4 Web Search • This is one of the most complex data engineering challenges today: – Distributed in nature – Large volume of data – Highly concurrent service – Users expect very good & fast answers • Current solution: Replicated centralized system 5 WR Logical Architecture Web Crawlers 6 A Typical Web Search Engine • Caching – result cache – posting list cache – document cache • Replication – multiple clusters – improve throughput • Parallel query processing – partitioned index • document-based • term-based – Online query processing Search Engine Architectures • Architectures differ in – number of data centers – assignment of users to data centers – assignment of index to data centers System Size • 20 billion Web pages implies at least 100Tb of text • The index in RAM implies at least a cluster of 10,000 PCs • Assume we can answer 1,000 queries/sec • 350 million queries a day imply 4,000 queries/sec • Decide that the peak load plus a fault tolerance margin is 3 • This implies a replication factor of 12 giving 120,000 PCs • Total deployment cost of over 100 million US$ plus maintenance cost • In 201x, being conservative, we would need over 1 million computers! 10 Questions • Should we use a centralized system? • Can we have a (cheaper) distributed search system in spite of network latency?
    [Show full text]
  • Distributed Web Crawlers Using Hadoop
    International Journal of Applied Engineering Research ISSN 0973-4562 Volume 12, Number 24 (2017) pp. 15187-15195 © Research India Publications. http://www.ripublication.com Distributed Web Crawlers using Hadoop Pratiba D Assistant Professor, Department of Computer Science and Engineering, R V College of Engineering, R V Vidyanikethan Post, Mysuru Road Bengaluru, Karnataka, India. Orcid Id: 0000-0001-9123-8687 Shobha G Professor, Department of Computer Science and Engineering, R V College of Engineering, R V Vidyanikethan Post, Mysuru Road Bengaluru, Karnataka, India. LalithKumar H Student, Department of Computer Science and Engineering, R V College of Engineering, R V Vidyanikethan Post, Mysuru Road Bengaluru, Karnataka, India. Samrudh J Student, Department of Information Science and Engineering, R V College of Engineering, R V Vidyanikethan Post, Mysuru Road Bengaluru, Karnataka, India. Abstract With the increasing size of web content, it is very difficult to index the entire WWW. So, efforts must be put to limit the Web Crawler is a software, which crawls through the WWW amount of data indexed and at the same time maximize the to build database for a search engine. In recent years, web coverage. To minimize the amount of data indexed, Natural crawling has started facing many challenges. Firstly, the web Language Processing techniques need to be applied to filter pages are highly unstructured which makes it difficult to and summarize the web page and store only the essential maintain a generic schema for storage. Secondly, the WWW details. By doing so, a search can yield the most accurate and is too huge and it is impossible to index it as it is.
    [Show full text]
  • Design and Implementation of Distributed Web Crawler for Drug Website Search Using Hefty Based Enhanced Bandwidth Algorithms
    Turkish Journal of Computer and Mathematics Education Vol.12 No.9 (2021), 123-129 Research Article Design and Implementation of Distributed Web Crawler for Drug Website Search using Hefty based Enhanced Bandwidth Algorithms a b c Saran Raj S , Aarif Ahamed S , and R. Rajmohan a,b Vel Tech Rangarajan Dr Sagunthala R & D Institute of Science and Technology cIFET College of Engineering, Villupuram, Tamil Nadu, India Article History: Received: 10 January 2021; Revised: 12 February 2021; Accepted: 27 March 2021; Published online: 20 April 2021 Abstract: The development of an expert system based search tool needs a superficial structure to satisfy the requirements of current web scale. Search engines and the internet crawler are used to mine the required information from the web and surf the internet in an efficient fashion. Distributed crawler is one of the types of a web crawler, which is a dispersed computation method. In this paper, we design and implement the concept of Efficient Distributed Web Crawler using enhanced bandwidth and hefty algorithms. Mostly Web Crawler doesn’t have any distributed cluster performance system and any implemented algorithm. In this paper, a novel Hefty Algorithm and enhanced bandwidth algorithm are combined for a better-distributed crawling system. The hefty algorithm is implemented to provide strong and efficient surfing results while applying on the drug web search. We also concentrate on the efficiency of the proposed distributed web crawler by implementing Enhanced Bandwidth algorithm. Keywords: Distributed crawler, Page surfs, Bandwidth ___________________________________________________________________________ 1. Introduction Internet Page swarmer is a meta- hunt engine, which combines the top search results from the represented search engines.
    [Show full text]
  • A Practical Geographically Distributed Web Crawler
    UniCrawl: A Practical Geographically Distributed Web Crawler Do Le Quoc, Christof Fetzer Pierre Sutra, Valerio Schiavoni, Systems Engineering Group Etienne´ Riviere,` Pascal Felber Dresden University of Technology, Germany University of Neuchatel,ˆ Switzerland Abstract—As the wealth of information available on the web can be repeated until a given depth, and pages are periodically keeps growing, being able to harvest massive amounts of data has re-fetched to discover new pages and to detect updated content. become a major challenge. Web crawlers are the core components to retrieve such vast collections of publicly available data. The Due to the size of the web, it is mandatory to make the key limiting factor of any crawler architecture is however its crawling process parallel [23] on a large number of machines large infrastructure cost. To reduce this cost, and in particular to achieve a reasonable collection time. This requirement the high upfront investments, we present in this paper a geo- implies provisioning large computing infrastructures. Existing distributed crawler solution, UniCrawl. UniCrawl orchestrates commercial crawlers, such as Google or Bing rely on big data several geographically distributed sites. Each site operates an centers. However, this approach imposes heavy requirements, independent crawler and relies on well-established techniques notably on the cost of the network infrastructure. Furthermore, for fetching and parsing the content of the web. UniCrawl splits the high upfront investment necessary to set up appropriate the crawled domain space across the sites and federates their storage and computing resources, while minimizing thee inter-site data centers can only be made by few large Internet companies, communication cost.
    [Show full text]
  • A Focused Web Crawler Driven by Self-Optimizing Classifiers
    A Focused Web Crawler Driven by Self-Optimizing Classifiers Master’s Thesis Dominik Sobania Original title: A Focused Web Crawler Driven by Self-Optimizing Classifiers German title: Fokussiertes Webcrawling mit selbstorganisierenden Klassifizierern Master’s Thesis Submitted by Dominik Sobania Submission date: 09/25/2015 Supervisor: Prof. Dr. Chris Biemann Coordinator: Steffen Remus TU Darmstadt Department of Computer Science, Language Technology Group Erklärung Hiermit versichere ich, die vorliegende Master’s Thesis ohne Hilfe Dritter und nur mit den angegebe- nen Quellen und Hilfsmitteln angefertigt zu haben. Alle Stellen, die aus den Quellen entnommen wur- den, sind als solche kenntlich gemacht worden. Diese Arbeit hat in dieser oder ähnlicher Form noch keiner Prüfungsbehörde vorgelegen. Die schriftliche Fassung stimmt mit der elektronischen Fassung überein. Darmstadt, den 25. September 2015 Dominik Sobania i Zusammenfassung Üblicherweise laden Web-Crawler alle Dokumente herunter, die von einer begrenzten Menge Seed- URLs aus erreichbar sind. Für die Generierung von Corpora ist eine Breitensuche allerdings nicht effizient, da uns hier nur ein bestimmtes Thema interessiert. Ein fokussierter Crawler besucht verlinkte Dokumente, die von einer Entscheidungsfunktion ausgewählt wurden, mit höherer Priorität. Dieser Ansatz fokussiert sich selbst auf das gesuchte Thema. In dieser Masterarbeit beschreiben wir einen Ansatz für fokussiertes Crawling, der als erster Schritt für die Generierung von Corpora genutzt werden kann. Basierend auf einem kleinen Satz an Textdoku- menten, die das gesuchte Thema definieren, erstellt eine Pipeline, bestehend aus mehreren Klassifizier- ern, die Trainingsdaten für einen Hyperlink-Klassifizierer – die Entscheidungsfunktion des fokussierten Crawlers. Für die Optimierung der Klassifizierer benutzen wir einen evolutionären Algorithmus für die Feature Subset Selection. Die Chromosomen des evolutionären Algorithmus basieren auf einer serial- isierbaren Baumstruktur.
    [Show full text]
  • UCYMICRA: Distributed Indexing of the Web Using Migrating Crawlers
    UCYMICRA: Distributed Indexing of the Web Using Migrating Crawlers Odysseas Papapetrou, Stavros Papastavrou, George Samaras Computer Science Department, University of Cyprus, 75 Kallipoleos Str., P.O.Box 20537 {cs98po1, stavrosp, cssamara}@cs.ucy.ac.cy Abstract. Due to the tremendous increase rate and the high change frequency of Web documents, maintaining an up-to-date index for searching purposes (search engines) is becoming a challenge. The traditional crawling methods are no longer able to catch up with the constantly updating and growing Web. Real- izing the problem, in this paper we suggest an alternative distributed crawling method with the use of mobile agents. Our goal is a scalable crawling scheme that minimizes network utilization, keeps up with document changes, employs time realization, and is easily upgradeable. 1 Introduction Indexing the Web has become a challenge due to the Web’s growing and dynamic na- ture. A study released in late 2000 reveals that the static and publicly available Web (also mentioned as surface web) exceeds 2.5 billion documents while the deep Web (dynamically generated documents, intranet pages, web-connected databases etc) is almost three orders of magnitude larger [20]. Another study shows that the Web is growing and changing rapidly [17, 19], while no search engine succeeds coverage of more than 16% of the estimated Web size [19]. Web crawling (or traditional crawling) has been the dominant practice for Web in- dexing by popular search engines and research organizations since 1993, but despite the vast computational and network resources thrown into it, traditional crawling is no longer able to catch up with the dynamic Web.
    [Show full text]
  • DATA20021 Information Retrieval Lecture 5: Web Crawling Simon J
    DATA20021 University of Helsinki, Department of Computer Science Information Retrieval Lecture 5: Web Crawling Simon J. Puglisi [email protected] Spring 2020 Today’s lecture… 16.1: Introduction to Indexing - Boolean Retrieval model - Inverted Indexes 21.1: Index Compression - unary, gamma, variable-byte coding - (Partitioned) Elias-Fano coding (used by Google, facebook) 23.1: Index Construction - preprocessing documents prior to search - building the index efficiently 28.1: Web Crawling - getting documents off the web at scale - architecture of a large scale web search engine 30.1: Query Processing - scoring and ranking search results - Vector-Space model • Web search engines create web repositories – They cache the Web on their local machines • Web repositories provide fast access to copies of the pages on the Web, allowing faster indexing and better search quality • A search engine aims to minimize the potential differences between it’s local repo and the Web – Coverage and freshness allow better quality answers – Very challenging due to fast and continuous evolution of the web: huge changes in pages and content every second • The Web repository maintains only the most recently crawled versions of web pages – Raw HTML, but compressed, on a filesystem (not a DBMS) – Also a catalog containing location on disk, size, timestamp • Mechanisms for both bulk and random access to stored pages are provided – Bulk access is used, e.g., by the indexing system – Random access for, e.g., query-biased snippet generation <Some slides on query
    [Show full text]