DOCSLIB.ORG

Diapositivo 1

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Diapositivo 1 Instituto Politécnico de Tomar Introduction to Information Retrieval Data Acquisition Ricardo Campos Lic ITM – Técnicas Avançadas de Programação Abrantes, Portugal, 2019 This presentation was developed by Ricardo Campos, Professor of ICT of the Polytechnic Institute of Tomar and researcher of LIAAD - INESC TEC. Part of the slides used in this presentation were adapted from presentations found in internet and from reference bibliography: Web crawling and system administration What is Information Retrieval? Please refer to the following when using this presentation: Campos, Ricardo. (2019). A .ppt version of this presentation can be provided upon request by sending an email to [[email protected]] What is Information Retrieval? AGENDA What is this talk about? Data Acquisition APIs Web Scraping 1 2 3 Web Crawling Web Dynamics Web Archives 4 5 6 What is Information Retrieval? Example scenario: Your small business’s website has a form used to sign clients up for appointments. You want to give your clients the ability to automatically create a Google calendar event with the details for that appointment. An application program interface (API) is a set of routines, protocols, API Use: The idea is to have your website’s server send an API and tools for building software request directly to Google’s. Your server would then receive applications. Basically, an API Google’s response, process it, and send back relevant specifies how software information to the browser, such as a confirmation message to components should interact the user. What is the difference between an API and You’ve probably heard of any other remote server? companies packaging APIs as To render the whole web page, your browser expects a response products. For example, Weather in HTML, which contains presentational code, while Google Underground sells access to its Calendar’s API call would just return the data — likely in a format like JSON. If your website’s server is making the API request, then weather data API. your website’s server is the client (similar to your browser being the client when you use it to Whatnavigate is Information to a Retrieval? website). To summarize, when a company offers an API to their customers, it just means that they’ve built a set of dedicated URLs that return pure data responses — meaning the responses An application program interface won’t contain the kind of presentational overhead that you (API) is a set of routines, protocols, would expect in a graphical user interface like a website. and tools for building software applications. Basically, an API Can you make these requests with your specifies how software components should interact browser? Often, yes. Since the actual HTTP transmission happens in text, your browser will always do the best it can to display the You’ve probably heard of response. companies packaging APIs as products. For example, Weather For example, you can access GitHub’s API directly with your Underground sells access to its browser without even needing an access token. Here’s the JSON weather data API. response you get when you visit a GitHub user’s API route in your browser (https://api.github.com/users/rcampos): What is Information Retrieval? https://youtu.be/s7wmiS2mSXY What is Information Retrieval? Data acquisition can be done by means of a browser, a crawler or a web scraper What is Information Retrieval? The following slides (entitled Arquivo.pt) were kindly provided by Fernando Melo (Research & Development of Arquivo.pt - the Portuguese web-archive) and by Daniel Bicho (Web crawling and system administration of Arquivo.pt - the Portuguese web-archive) What is Information Retrieval? arquivo.pt/api What is Information Retrieval? URL List all preserved versions of URL: http://www.ipt.pt arquivo.pt/api https://arquivo.pt/textsearch?versionHistory=http://www.ipt.pt Default Parameters: • Offset =0 (first result) • maxItems =50 (number of results) What is Information Retrieval? What is Information Retrieval? URL List all preserved versions of URL: http://www.ipt.pt arquivo.pt/api https://arquivo.pt/textsearch?versionHistory=http://www.ipt.pt &offset=100 Parameters: • offset =100 (start in result 100) • maxItems =50 (number of results) What is Information Retrieval? URL List all preserved versions of URL: http://www.ipt.pt arquivo.pt/api https://arquivo.pt/textsearch?versionHistory=http://www.ipt.pt &maxItems=1000 Parameters: • maxItems =1000 (number of results) What is Information Retrieval? URL List all preserved versions of URL: http://www.ipt.pt arquivo.pt/api https://arquivo.pt/textsearch?versionHistory=http://www.ipt.pt &maxItems=1000&from=2010&to=2016 Parameters: • maxItems =1000 (number of results) • From and to: date between 2010 and 2016 What is Information Retrieval? URL List all preserved versions of URL: http://www.ipt.pt arquivo.pt/api https://arquivo.pt/textsearch?versionHistory=http://www.ipt.pt &from=20010224174130&to=20150424181416 Parameters: • From and to: date between 24/02/2001 and 24/04/2015 What is Information Retrieval? URL List all preserved versions of URL: http://www.ipt.pt arquivo.pt/api http://arquivo.pt/textsearch?versionHistory=http://www.ipt.pt &fields=originalURL,tstamp Parameters: • Fields: Filter answer to show only: Preserved page URL date (tstamp) in which it was preserved What is Information Retrieval? URL Extract text from a specific URL version arquivo.pt/api http://arquivo.pt/textextracted?m=http://www.ipt.pt/20161111160133 What is Information Retrieval? Text Search arquivo.pt/api http://arquivo.pt/textsearch?q=instituto politécnico de tomar Default Parameters: • offset =0 (first result) • itemsPerPage =50 (number of results) What is Information Retrieval? What is Information Retrieval? Response Item - Snippet What is Information Retrieval? Response Item – Link to Extracted Text What is Information Retrieval? Text Search Iterate every 50 results changing offset parameter (at a maximum of 1950) arquivo.pt/api https://arquivo.pt/textsearch?q=instituto politécnico de tomar &offset=0&maxItems=50 https://arquivo.pt/textsearch?q=instituto politécnico de tomar &offset=50&maxItems=50 ……….. https://arquivo.pt/textsearch?q=instituto politécnico de tomar &offset=1950&maxItems=50 What is Information Retrieval? Text Search arquivo.pt/api http://arquivo.pt/textsearch?q=instituto politécnico de tomar &maxItems=2000 Parameters: • maxItems =2000 (maximum number of results, but recommended only 200) What is Information Retrieval? Text Search arquivo.pt/api http://arquivo.pt/textsearch?q=“instituto politécnico de tomar” Parameters: • “”: search for an expression What is Information Retrieval? Text Search arquivo.pt/api http://arquivo.pt/textsearch?q=instituto politécnico de tomar&type=pdf Parameters: • File Format: pdf What is Information Retrieval? Images Search arquivo.pt/api What is Information Retrieval? Images Search arquivo.pt/api https://arquivo.pt/imagesearch?q=instituto politécnico de tomar What is Information Retrieval? What is Information Retrieval? Images Search arquivo.pt/api https://arquivo.pt/imagesearch?q=instituto politécnico de tomar&type=png Parameters: • File Format: png What is Information Retrieval? Images Search arquivo.pt/api https://arquivo.pt/imagesearch?q=instituto politécnico de tomar&size=sm Parameters (size): • sm (small image size) • md (medium image size); • lg (large image size); What is Information Retrieval? Images Search arquivo.pt/api https://arquivo.pt/imagesearch?q=instituto politécnico de tomar&&safeSearch=off Parameters (size): • safeSearch = off; Show all images including those that were automatically classified as Not Safe for Work What is Information Retrieval? https://www.youtube.com/watch?v=SibIAGU7sE0 Complete Video: https://www.youtube.com/watch?v=fNpUllm0dvcWhat is Information Retrieval? https://www.youtube.com/watch?v=rbCqcSpRjEc Complete Video: https://www.youtube.com/watch?v=FASxveLn6is What is Information Retrieval? Prémios Arquivo.pt 2018 What is Information Retrieval? Contamehistorias.pt What is Information Retrieval? https://www.youtube.com/watch?v=b4_ZBr2ijVw&feature=youtu.be When performing data science tasks, it's common to want to use data found on the internet. Some web pages make your life easier, they offer something called API, they offer an interface that you can use to download data. For instance, Twitter provides API to access data. However, there are times when the data you want can only be accessed as part of a web page. In cases like this, you'll want to use a technique called web scraping to get the data from the web page into a format you can work with in your analysis. What is Information Retrieval? “Web scraping is a computer software technique of extracting information from websites” It focuses on the transformation of unstructured data on the web, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. What is Information Retrieval? • When performing data science tasks, it's common that you want to use data found on the internet. You'll usually be able to access this data in csv format, or via an Application Programming Interface (API). “Web scraping is a computer • However, there are times when the data you want can only software technique of extracting be accessed as part of a web page. In cases like this, you'll information from websites” want to use a technique called web scraping to get the data from the web page into a format you can work with in your analysis. It focuses on the transformation of unstructured data on the web, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. What is Information Retrieval? • When we visit a web page, our web browser makes a request to a web server. This request is called a GET request, since we're getting files from the server. The server then sends back files that tell our browser how to render the Scarping is all about html tags. page for us. The files fall into a few main types: So you need to understand html inorder to scrape data.
Load more

Recommended publications
  • Cluster Setup Guide
    Frontera Documentation Release 0.8.0.1 ScrapingHub Jul 30, 2018 Contents 1 Introduction 3 1.1 Frontera at a glance...........................................3 1.2 Run modes................................................5 1.3 Quick start single process........................................6 1.4 Quick start distributed mode.......................................8 1.5 Cluster setup guide............................................ 10 2 Using Frontera 15 2.1 Installation Guide............................................ 15 2.2 Crawling strategies............................................ 16 2.3 Frontier objects.............................................. 16 2.4 Middlewares............................................... 19 2.5 Canonical URL Solver.......................................... 22 2.6 Backends................................................. 23 2.7 Message bus............................................... 27 2.8 Writing custom crawling strategy.................................... 30 2.9 Using the Frontier with Scrapy...................................... 35 2.10 Settings.................................................. 37 3 Advanced usage 51 3.1 What is a Crawl Frontier?........................................ 51 3.2 Graph Manager.............................................. 52 3.3 Recording a Scrapy crawl........................................ 58 3.4 Fine tuning of Frontera cluster...................................... 59 3.5 DNS Service............................................... 60 4 Developer documentation 63 4.1
    [Show full text]
  • Frontera Documentation Release 0.4.0
    Frontera Documentation Release 0.4.0 ScrapingHub December 30, 2015 Contents 1 Introduction 3 1.1 Frontera at a glance...........................................3 1.2 Run modes................................................5 1.3 Quick start single process........................................6 1.4 Quick start distributed mode.......................................8 2 Using Frontera 11 2.1 Installation Guide............................................ 11 2.2 Frontier objects.............................................. 11 2.3 Middlewares............................................... 12 2.4 Canonical URL Solver.......................................... 13 2.5 Backends................................................. 14 2.6 Message bus............................................... 16 2.7 Crawling strategy............................................. 17 2.8 Using the Frontier with Scrapy...................................... 17 2.9 Settings.................................................. 20 3 Advanced usage 31 3.1 What is a Crawl Frontier?........................................ 31 3.2 Graph Manager.............................................. 32 3.3 Recording a Scrapy crawl........................................ 38 3.4 Production broad crawling........................................ 40 4 Developer documentation 45 4.1 Architecture overview.......................................... 45 4.2 Frontera API............................................... 47 4.3 Using the Frontier with Requests.................................... 49 4.4
    [Show full text]
  • High Performance Distributed Web-Scraper
    High performance distributed web-scraper Denis Eyzenakh Anton Rameykov Igor Nikiforov Institute of Computer Science and Institute of Computer Science and Institute of Computer Science and Technology Technology Technology Peter the Great St.Petersburg Peter the Great St.Petersburg Peter the Great St.Petersburg Polytechnic University Polytechnic University Polytechnic University Saint – Petersburg, Russian Federation Saint – Petersburg, Russian Federation Saint – Petersburg, Russian Federation [email protected] [email protected] [email protected] Abstract—Over the past decade, the Internet has become the gigantic and richest source of data. The data is used for the II. EXISTING WEB SCRAPING TECHNIQUES extraction of knowledge by performing machine leaning analysis. Typically, web scraping applications imitate a regular web In order to perform data mining of the web-information, the data user. They follow the links and search for the information should be extracted from the source and placed on analytical storage. This is the ETL-process. Different web-sources have they need. The classic web scraper can be classified into two different ways to access their data: either API over HTTP protocol types: web-crawlers and data extractors “Fig. 1”. or HTML source code parsing. The article is devoted to the approach of high-performance data extraction from sources that do not provide an API to access the data. Distinctive features of the proposed approach are: load balancing, two levels of data storage, and separating the process of downloading files from the process of scraping. The approach is implemented in the solution with the following technologies: Docker, Kubernetes, Scrapy, Python, MongoDB, Redis Cluster, and СephFS.
    [Show full text]
  • Scrapy Cluster Documentation Release 1.2
    Scrapy Cluster Documentation Release 1.2 IST Research November 29, 2016 Contents 1 Introduction 3 1.1 Overview.................................................3 1.2 Quick Start................................................5 2 Kafka Monitor 13 2.1 Design.................................................. 13 2.2 Quick Start................................................ 14 2.3 API.................................................... 16 2.4 Plugins.................................................. 31 2.5 Settings.................................................. 34 3 Crawler 37 3.1 Design.................................................. 37 3.2 Quick Start................................................ 42 3.3 Controlling................................................ 44 3.4 Extension................................................. 49 3.5 Settings.................................................. 53 4 Redis Monitor 59 4.1 Design.................................................. 59 4.2 Quick Start................................................ 60 4.3 Plugins.................................................. 61 4.4 Settings.................................................. 64 5 Rest 69 5.1 Design.................................................. 69 5.2 Quick Start................................................ 69 5.3 API.................................................... 73 5.4 Settings.................................................. 77 6 Utilites 81 6.1 Argparse Helper............................................. 81 6.2 Log Factory..............................................
    [Show full text]
  • Frontera-Open Source Large Scale Web Crawling Framework
    Frontera: Large-Scale Open Source Web Crawling Framework Alexander Sibiryakov, 20 July 2015 [email protected] Hola los participantes! • Born in Yekaterinburg, RU • 5 years at Yandex, search quality department: social and QA search, snippets. • 2 years at Avast! antivirus, research team: automatic false positive solving, large scale prediction of malicious download attempts. 2 «A Web crawler starts with a list of URLs to visit, called the seeds. As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl frontier.». –Wikipedia: Web Crawler article, July 2015 3 Motivation • Client needed to crawl 1B+ pages/week, and identify frequently changing HUB pages. • Scrapy wasn’t suitable for broad crawling and had no crawl frontier capabilities* • People were tend to favor Apache Nutch instead of Scrapy. Hyperlink-Induced Topic Search, Jon Kleinberg, 1999 4 tophdart.com Frontera: single-threaded and distributed • Frontera is all about knowing what to crawl next and when to stop. • Single-Threaded mode can be used for up to 100 websites (parallel downloading), • for performance broad crawls there is a distributed mode. 6 Main features • Online operation: scheduling of new batch, updating of DB state. • Storage abstraction: write your own backend (sqlalchemy, HBase is included). • Canonical URLs resolution abstraction: each document has many URLs, which to use? • Scrapy ecosystem: good documentation, big community, ease of customization. 7 Single-threaded use cases • Need of URL metadata and content storage, • Need of isolation of URL ordering/queueing logic from the spider • Advanced URL ordering logic (big websites, or revisiting) 8 Single-threaded architecture 9 Frontera and Scrapy • Frontera is implemented as a set of custom scheduler and spider middleware for Scrapy.
    [Show full text]
  • Distributed Training
    Scalable and Distributed DNN Training on Modern HPC Systems: Challenges and Solutions Keynote Talk at SDAS ‘19 by Dhabaleswar K. (DK) Panda The Ohio State University E-mail: [email protected] http://www.cse.ohio-state.edu/~panda Understanding the Deep Learning Resurgence • Deep Learning is a sub-set of Machine Learning – But, it is perhaps the most radical and revolutionary subset – Automatic feature extraction vs. hand-crafted features • Deep Learning – A renewed interest and a lot of hype! – Key success: Deep Neural Networks (DNNs) – Everything was there since the late 80s except the “computability of DNNs” Courtesy: http://www.deeplearningbook.org/contents/intro.html Network Based Computing Laboratory SDAS (June ‘19) 2 Deep Learning Use Cases and Growth Trends Courtesy: https://www.top500.org/news/market-for-artificial-intelligence-projected-to-hit-36-billion-by-2025/ Network Based Computing Laboratory SDAS (June ‘19) 3 Increasing Usage of HPC, Big Data and Deep Learning Big Data HPC (Hadoop, Spark, (MPI, RDMA, HBase, Lustre, etc.) Memcached, etc.) Convergence of HPC, Big Deep Learning Data, and Deep Learning! (Caffe, TensorFlow, BigDL, etc.) Increasing Need to Run these applications on the Cloud!! Network Based Computing Laboratory SDAS (June ‘19) 4 Newer Workflows - Deep Learning over Big Data (DLoBD) • Deep Learning over Big Data (DLoBD) is one of the most efficient analyzing paradigms • More and more deep learning tools or libraries (e.g., Caffe, TensorFlow) start running over big data stacks, such as Apache Hadoop and Spark
    [Show full text]
  • HPC and AI Middleware for Exascale Systems and Clouds
    HPC and AI Middleware for Exascale Systems and Clouds Talk at HPC-AI Advisory Council UK Conference (October ‘20) by Dhabaleswar K. (DK) Panda The Ohio State University E-mail: [email protected] Follow us on http://www.cse.ohio-state.edu/~panda https://twitter.com/mvapich High-End Computing (HEC): PetaFlop to ExaFlop 100 PetaFlops in 415 Peta 2017 Flops in 2020 (Fugaku in Japan with 7.3M cores 1 ExaFlops Expected to have an ExaFlop system in 2021! Network Based Computing Laboratory HPC-AI-UK (Oct ‘20) 2 Increasing Usage of HPC, Big Data and Deep/Machine Learning Big Data (Hadoop, Spark, HPC HBase, (MPI, PGAS, etc.) Memcached, etc.) Convergence of HPC, Big Deep/Machine Data, and Deep/Machine Learning Learning! (TensorFlow, PyTorch, BigDL, cuML, etc.) Increasing Need to Run these applications on the Cloud!! Network Based Computing Laboratory HPC-AI-UK (Oct ‘20) 3 Converged Middleware for HPC, Big Data and Deep/Machine Learning? Physical Compute Network Based Computing Laboratory HPC-AI-UK (Oct ‘20) 4 Converged Middleware for HPC, Big Data and Deep/Machine Learning? Network Based Computing Laboratory HPC-AI-UK (Oct ‘20) 5 Converged Middleware for HPC, Big Data and Deep/Machine Learning? Network Based Computing Laboratory HPC-AI-UK (Oct ‘20) 6 Converged Middleware for HPC, Big Data and Deep/Machine Learning? Hadoop Job Deep/Machine Learning Job Spark Job Network Based Computing Laboratory HPC-AI-UK (Oct ‘20) 7 Presentation Overview • MVAPICH Project – MPI and PGAS (MVAPICH) Library with CUDA-Awareness • HiBD Project – High-Performance
    [Show full text]
  • Vergleich Aktueller Web-Crawling-Werkzeuge
    Hochschule Wismar University of Applied Sciences Technology, Business and Design Fakult¨at fur¨ Ingenieurwissenschaften, Bereich EuI Bachelor-Thesis Vergleich aktueller Web-Crawling-Werkzeuge Gedruckt am: 30. April 2021 Eingereicht am: von: Christoph Werner Betreuende Professorin: Prof. Dr.-Ing. Antje Raab-Dusterh¨ ¨oft Zweitbetreuer: Prof. Dr.-Ing. Matthias Kreuseler Aufgabenstellung Das Ziel der Bachelor Thesis ist ein Vergleich aktueller Web-Crawling-Werkzeuge. Die Crawler sind hinsichtlich ihrer Funktionsweise und Crawling-Ergebnisse zu ver- gleichen und zu bewerten. Des Weiteren ist ein Tool zu konzeptionieren, welches diese Werkzeuge fur¨ die Nutzung in Laborpraktika verfugbar¨ macht. 3 Kurzreferat Webcrawling im Verbund mit Webscraping ist eine der grundlegenden Technologien, um gezielt und automatisiert Daten aus dem Internet zu sammeln. Zun¨achst wird in dieser Thesis auf die Grundlagen des Crawlings und Scrapings eingegangen. Hierbei soll besonderes Augenmerk auf der Architektur und Funkti- onsweise eines Crawlers, dem Robots Exclusion Protocol (REP), zu bedenkende Sicherheitsaspekte, sowie Anti-Crawling/Scraping-Maßnahmen liegen. Darauf aufbauend werden verschiedenste Crawling Frameworks auf Grundlage ihrer Dokumentation bewertet und gegenubergestellt.¨ Abschließend wird ein Tool mit einer grafischen Benutzeroberfl¨ache (GUI) zum Ver- gleich von verschiedenen Crawling Frameworks entwickelt. 4 Abstract Web crawling in combination with web scraping is the key technology for the targeted and automated collection of data from the World Wide Web. First of all, this thesis deals with the basics of crawling and scraping. Special at- tention will be paid to the architecture and functionality of a crawler, the Ro- bots Exclusion Protocol (REP), security aspects to be considered as well as anti- crawling/scraping measures. Based on this, various crawling frameworks are evaluated and compared on the basis of their documentation.
    [Show full text]
  • Parsl Documentation Release 1.1.0
    Parsl Documentation Release 1.1.0 The Parsl Team Sep 24, 2021 CONTENTS 1 Quickstart 3 1.1 Installation................................................3 1.2 Getting started..............................................4 1.3 Tutorial..................................................4 1.4 Usage Tracking..............................................4 1.5 For Developers..............................................5 2 Parsl tutorial 7 2.1 Configuring Parsl.............................................7 2.2 Python Apps...............................................8 2.3 Bash Apps................................................8 2.4 Passing data...............................................9 2.5 AppFutures................................................9 2.6 DataFutures................................................ 10 2.7 Files................................................... 11 2.8 Remote Files............................................... 11 2.9 Sequential workflow........................................... 12 2.10 Parallel workflow............................................. 12 2.11 Parallel dataflow............................................. 13 2.12 Monte Carlo workflow.......................................... 14 2.13 Local execution with threads....................................... 15 2.14 Local execution with pilot jobs..................................... 15 3 User guide 17 3.1 Overview................................................. 17 3.2 Apps................................................... 23 3.3 Futures.................................................
    [Show full text]
  • Frontera Documentation Release 0.8.0
    Frontera Documentation Release 0.8.0 ScrapingHub Jul 30, 2018 Contents 1 Introduction 3 1.1 Frontera at a glance...........................................3 1.2 Run modes................................................5 1.3 Quick start single process........................................6 1.4 Quick start distributed mode.......................................8 1.5 Cluster setup guide............................................ 10 2 Using Frontera 15 2.1 Installation Guide............................................ 15 2.2 Crawling strategies............................................ 16 2.3 Frontier objects.............................................. 16 2.4 Middlewares............................................... 19 2.5 Canonical URL Solver.......................................... 22 2.6 Backends................................................. 23 2.7 Message bus............................................... 27 2.8 Writing custom crawling strategy.................................... 30 2.9 Using the Frontier with Scrapy...................................... 35 2.10 Settings.................................................. 37 3 Advanced usage 51 3.1 What is a Crawl Frontier?........................................ 51 3.2 Graph Manager.............................................. 52 3.3 Recording a Scrapy crawl........................................ 58 3.4 Fine tuning of Frontera cluster...................................... 59 3.5 DNS Service............................................... 60 4 Developer documentation 63 4.1
    [Show full text]
  • Frontera Documentation 0.6.0
    Frontera Documentation 0.6.0 ScrapingHub 2017 02 13 Contents 1 3 1.1 Frontera at a glance...........................................3 1.2 Run modes................................................5 1.3 Quick start single process........................................7 1.4 Quick start distributed mode.......................................8 1.5 Cluster setup guide............................................9 2 Frontera 13 2.1 Installation Guide............................................ 13 2.2 Frontier objects.............................................. 13 2.3 Middlewares............................................... 14 2.4 Canonical URL Solver.......................................... 15 2.5 Backends................................................. 16 2.6 Message bus............................................... 18 2.7 Crawling strategy............................................. 19 2.8 Using the Frontier with Scrapy...................................... 20 2.9 Settings.................................................. 23 3 33 3.1 What is a Crawl Frontier?........................................ 33 3.2 Graph Manager.............................................. 34 3.3 Recording a Scrapy crawl........................................ 40 3.4 Fine tuning of Frontera cluster...................................... 42 3.5 DNS Service............................................... 43 4 45 4.1 Architecture overview.......................................... 45 4.2 Frontera API............................................... 47 4.3 Using the
    [Show full text]
  • High Performance Distributed Web-Scraper
    Труды ИСП РАН, том 33, вып. 3, 2021 г. // Trudy ISP RAN/Proc. ISP RAS, vol. 33, issue 3, 2021 Eyzenakh D.S., Rameykov A.S., Nikiforov I.V. High performance distributed web-scraper. Trudy ISP RAN/Proc. ISP RAS, vol. 33, issue 3, 2021, pp. 87-100 Для цитирования: Эйзенах Д.С., Рамейков А.С., Никифоров И.В. Высокопроизводительный распределенный веб-скрапер. Труды ИСП РАН, том 33, вып. 3, 2021 г., стр. 87-100 (на английском языке). DOI: 10.15514/ISPRAS–2021–33(3)–7. DOI: 10.15514/ISPRAS-2021-33(3)-7 1. Introduction Due to the rapid development of the network, the World Wide Web has become a carrier of a large High Performance Distributed Web-Scraper amount of information. The data extraction and use of information has become a huge challenge nowadays. Traditional access to the information through browsers like Chrome, Firefox, etc. can provide a comfortable user experience with web pages. Web sites have a lot of information and sometimes haven’t got any instruments to access over the API and preserve it in analytical storage. D.S. Eyzenakh, ORCID: 0000-0003-1584-1745 <[email protected]> The manual collection of data for further analysis can take a lot of time and in the case of semi- A.S. Rameykov, ORCID: 0000-0001-7989-6732 <[email protected]> structured or unstructured data types the collection and analyzing of data can become even more I.V. Nikiforov, ORCID: 0000-0003-0198-1886 <[email protected]> difficult and time-consuming. The person who manually collects data can make mistakes Peter the Great St.Petersburg Polytechnic University (duplication, typos in the text, etc.) as far as the process is error-prone.
    [Show full text]