DOCSLIB.ORG

Data Scraping’ a Primer for Counseling Clients

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Data Scraping’ a Primer for Counseling Clients A NEW YORK LAW JOURNAL SPECIAL SECTION LitigationLitigation WWW. NYLJ.COM MONDAY, JULY 15, 2013 Internet ‘Data Scraping’ A Primer for Counseling Clients ways to avoid liabil- BY ANTHONY J. DREYER ity for scraping. The AND JAMIE STOCKTON common theories of he proliferation of Internet access liability arising from and mobile devices has led to an scraping are copyright T exponential explosion of content infringement, trespass on the Web, creating a vast repository to chattels, breach of of “publicly available” information. This contract, and viola- includes not only news, business, and tion of the Computer financial information, but also personal Fraud and Abuse Act data, movie and restaurant reviews, con- (CFAA). This article cert ticket sales, flight information, and discusses the lead- a virtually endless array of other cat- ing cases applying egories. This same technological explo- these legal theories sion, however, has made it far easier to website scraping, for third parties to extract this data for and concludes that commercial sale and use—and to do so the most effective for free and without authorization. This way to create potential data extraction, commonly referred to claims against scrap- as “scraping,” “crawling,” or “spidering” ers is through carefully (collectively “scraping”),1 creates legal drafted prohibitions issues and concerns for both sides of in a website’s terms this issue—those who want to scrape, of use. Conversely, the most effective infringement. However, such claims and those who want to protect against way to defend against a claim of unau- are often open to attack on several scraping of their websites. thorized scraping is to abide by such grounds. First, in order to have standing This article provides a primer on the terms of use, or to establish that scrap- to bring a claim for copyright infringe- legal framework surrounding scrap- ing constitutes a fair use and does not ment, the owner (or exclusive licensee) ing, addressing both the grounds for overburden the servers of the website of the website being scraped must also potential claims against scrapers, and being scraped. be the owner of the copyrightable con- tent that is the subject of the claim.2 Copyright Infringement This can pose a barrier to bringing a ANTHONY J. DREYER is a partner, and JAMIE STOCKTON is Scraping inherently involves copying, lawsuit if, for example, the content at an associate, with Skadden, Arps, Slate, Meagher & Flom. and therefore one of the most obvious issue is user-generated (such as vid- BRITTANY BETTMAN, a summer associate, assisted in the preparation of this article. claims against scrapers is copyright eos or reviews), and the rights in the MONDAY, JULY 15, 2013 content have not been transferred to if it taxes the plaintiff’s computer system tation of assent, so long as the terms the website owner. in such a way that would substantially do not violate other basic contract Second, copyright law does not pro- impair it, and, if so, an injunction may principles (e.g., unconscionability).17 tect ideas, but rather only tangible be granted.10 Specifically, the court held For example, in Bidder’s Edge, the expression.3 Thus, the scraping of gen- that there was a viable trespass cause court took note of the fact that the eral factual data does not give rise to of action due to the excessive scraping user agreement at the time, to which a viable claim for copyright infringe- of eBay’s website at the rate of 80,000- users were required to click “I Accept,” ment. For example, in Ticketmaster 100,000 times per day.11 expressly prohibited “any robot, spider, v. Tickets.com, the court rejected an other automatic device, or manual infringement claim because the mate- process to monitor or copy our web rial being extracted—factual informa- Scraping inherently in- pages or the content contained herein tion regarding concerts and URLs—was volves copying, and there- without our prior expressed written not copyrightable.4 permission.”18 The court stated that Third, even if the information copied fore one of the most obvious these terms of use constituted a limited by the scraper is protectable under claims against scrapers is license, and that actions not permitted copyright law, the defendant may be by this license were restricted.19 able to rely upon the “fair use” defense. copyright infringement. Browsewrap agreements, on the oth- Under the Copyright Act, courts are to er hand, involve the posting of a link consider the following factors to deter- Similarly, in Register.com v. Verio, the to terms and conditions on a website mine if a use is a fair use: (1) the pur- Court of Appeals for the Second Circuit for users to read, but do not require pose and character of the use; (2) the held that Verio’s use of search robots users to affirmatively manifest assent nature of the copyrighted work; (3) the consumed a significant portion of the to the terms and conditions—instead, amount and substantiality of the por- capacity of Register’s computer system, user consent is implied by continued tion used in relation to the work as a and that Verio was therefore engaged in use of the website.20 whole; and (4) the effect of the use upon a trespass.12 The court reasoned that if The enforceability of such agreements the potential market for or value of the it were to allow these queries, then it requires a fact-specific inquiry, and turns copyrighted work.5 For example, in Kelly was “highly probable” that other com- largely upon the location and accessibil- v. Arriba Soft, the court held that the panies would begin to do the same, ity of the terms of use.21 According to the use of scraping software by a search which would likely result in Register’s Specht court, “[r]easonably conspicu- engine to reproduce images in thumb- system being “overtaxed and [it] would ous notice of the existence of contract nail form was not a sustainable basis crash.”13 However, in Ticketmaster, the terms and unambiguous manifestation for a claim of copyright infringement, court held that the use of scrapers to of assent to those terms by consumers because the thumbnail images created extract data was not a trespass to chat- are essential if electronic bargaining is from the full-size scraped images were tels, because there was no evidence to have integrity and credibility.”22 “transformative” and qualified as a fair that the scraping caused any tangible For example, in Hines the court held use of the images.6 interference with the operation of Tick- that the browsewrap agreement was etmaster’s system.14 not enforceable, because in this case Trespass to Chattels the plaintiff had no actual or construc- Breach of Contract A trespass to chattels is defined as tive notice of the terms and conditions intentionally dispossessing another of Courts have held that a viable of use.23 However, in Southwest Airlines a chattel or using or intermeddling with method of preventing scraping is to v. BoardFirst, where there was evidence a chattel in the possession of another.7 include prohibitions against scraping that defendant had actual knowledge This legal theory applies to the Internet in the website’s terms of use.15 Such of Southwest’s terms and conditions, inasmuch as a website proprietor has a restrictions are generally conveyed to but nevertheless continued to use “fundamental property right to exclude website users through a “clickwrap” or Southwest’s website in violation of others from its computer system[.]”8 “browsewrap” agreement. those terms, the court held that the Moreover, even if a website is publicly A clickwrap agreement is an online browsewrap agreement was an enforce- accessible, its servers are private prop- agreement that requires the user to able contract.24 erty, and the proprietor may therefore consent to terms and conditions by Terms of use may also be binding grant conditional access to users, includ- affirmatively clicking a dialogue box where the terms are reasonably known ing prohibitions against scraping.9 agreeing to the terms before the user to the user—even in circumstances in For example, in Bidder’s Edge, the can proceed to use a website.16 Click- which the terms are not known to the court held that excessive scraping can wrap agreements are generally enforce- user before the first use of the website. support a claim for trespass to chattels able, due to the user’s clear manifes- For example, in Register.com, the user MONDAY, JULY 15, 2013 was made aware of the terms of use were prohibited from accessing and uti- or copy any of the material on this only after first accessing the informa- lizing the information on the website, the website; (iii) use any manual process tion provided on the website.25 The court held that there was no violation to monitor or copy any of the mate- court held that while the terms of use of the CFAA.33 The court concluded that rial on this website, or to engage in were technically neither a clickwrap the terms of use were not sufficiently any other unauthorized purpose nor a browsewrap agreement, because visible because the link was “buried” at without the express prior written they were only displayed after the user the bottom of the first page, in extremely consent of [CLIENT]; (iv) otherwise accessed the information on the web- fine print, and users had to scroll down use any device, software or routine site, the restrictions therein were nev- to see it, thereby rendering them insuf- that interferes with the proper work- ertheless enforceable, because the user ficient protection for the site.34 ing of this website; or (v) otherwise accessed the website repeatedly and Conclusion and Proposed Terms of Use attempt to interfere with the proper therefore was on notice during subse- working of this website.
Load more

Recommended publications
  • Automatic Retrieval of Updated Information Related to COVID-19 from Web Portals 1Prateek Raj, 2Chaman Kumar, 3Dr
    European Journal of Molecular & Clinical Medicine ISSN 2515-8260 Volume 07, Issue 3, 2020 Automatic Retrieval of Updated Information Related to COVID-19 from Web Portals 1Prateek Raj, 2Chaman Kumar, 3Dr. Mukesh Rawat 1,2,3 Department of Computer Science and Engineering, Meerut Institute of Engineering and Technology, Meerut 250005, U.P, India Abstract In the world of social media, we are subjected to a constant overload of information. Of all the information we get, not everything is correct. It is advisable to rely on only reliable sources. Even if we stick to only reliable sources, we are unable to understand or make head or tail of all the information we get. Data about the number of people infected, the number of active cases and the number of people dead vary from one source to another. People usually use up a lot of effort and time to navigate through different websites to get better and accurate results. However, it takes lots of time and still leaves people skeptical. This study is based on web- scraping & web-crawlingapproach to get better and accurate results from six COVID-19 data web sources.The scraping script is programmed with Python library & crawlers operate with HTML tags while application architecture is programmed using Cascading style sheet(CSS) &Hypertext markup language(HTML). The scraped data is stored in a PostgreSQL database on Heroku server and the processed data is provided on the dashboard. Keywords:Web-scraping, Web-crawling, HTML, Data collection. I. INTRODUCTION The internet is wildly loaded with data and contents that are informative as well as accessible to anyone around the globe in various shape, size and extension like video, audio, text and image and number etc.
    [Show full text]
  • Deconstructing Large-Scale Distributed Scraping Attacks
    September 2018 Radware Research Deconstructing Large-Scale Distributed Scraping Attacks A Stepwise Analysis of Real-time Sophisticated Attacks On E-commerce Businesses Deconstructing Large-Scale Distributed Scraping Attacks Table of Contents 02 Why Read This E-book 03 Key Findings 04 Real-world Case of A Large-Scale Scraping Attack On An E-tailer Snapshot Of The Scraping Attack Attack Overview Stages of Attack Stage 1: Fake Account Creation Stage 2: Scraping of Product Categories Stage 3: Price and Product Info. Scraping Topology of the Attack — How Three-stages Work in Unison 11 Recommendations: Action Plan for E-commerce Businesses to Combat Scraping 12 About Radware 02 Deconstructing Large-Scale Distributed Scraping Attacks Why Read This E-book Hypercompetitive online retail is a crucible of technical innovations to win today’s business wars. Tracking prices, deals, content and product listings of competitors is a well-known strategy, but the rapid pace at which the sophistication of such attacks is growing makes them difficult to keep up with. This e-book offers you an insider’s view of scrapers’ techniques and methodologies, with a few takeaways that will help you fortify your web security strategy. If you would like to learn more, email us at [email protected]. Business of Bots Companies like Amazon and Walmart have internal teams dedicated to scraping 03 Deconstructing Large-Scale Distributed Scraping Attacks Key Findings Scraping - A Tool Today, many online businesses either employ an in-house team or leverage the To Gain expertise of professional web scrapers to gain a competitive advantage over their competitors.
    [Show full text]
  • Legality and Ethics of Web Scraping
    Murray State's Digital Commons Faculty & Staff Research and Creative Activity 12-15-2020 Tutorial: Legality and Ethics of Web Scraping Vlad Krotov Leigh Johnson Murray State University Leiser Silva University of Houston Follow this and additional works at: https://digitalcommons.murraystate.edu/faculty Recommended Citation Krotov, V., Johnson, L., & Silva, L. (2020). Tutorial: Legality and Ethics of Web Scraping. Communications of the Association for Information Systems, 47, pp-pp. https://doi.org/10.17705/1CAIS.04724 This Peer Reviewed/Refereed Publication is brought to you for free and open access by Murray State's Digital Commons. It has been accepted for inclusion in Faculty & Staff Research and Creative Activity by an authorized administrator of Murray State's Digital Commons. For more information, please contact [email protected]. See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/343555462 Legality and Ethics of Web Scraping, Communications of the Association for Information Systems (forthcoming) Article in Communications of the Association for Information Systems · August 2020 CITATIONS READS 0 388 3 authors, including: Vlad Krotov Murray State University 42 PUBLICATIONS 374 CITATIONS SEE PROFILE Some of the authors of this publication are also working on these related projects: Addressing barriers to big data View project Web Scraping Framework: An Integrated Approach to Retrieving Big Qualitative Data from the Web View project All content following this
    [Show full text]
  • CSCI 452 (Data Mining) Dr. Schwartz HTML Web Scraping 150 Pts
    CSCI 452 (Data Mining) Dr. Schwartz HTML Web Scraping 150 pts Overview For this assignment, you'll be scraping the White House press briefings for President Obama's terms in the White House to see which countries have been mentioned and how often. We will use a mirrored image (originally so that we wouldn’t cause undue load on the White House servers, now because the data is historical). You will use Python 3 with the following libraries: • Beautiful Soup 4 (makes it easier to pull data out of HTML and XML documents) • Requests (for handling HTTP requests from python) • lxml (XML and HTML parser) We will use Wikipedia's list of sovereign states to identify the entities we will be counting, then we will download all of the press briefings and count how many times a country's name has been mentioned in the press briefings during President Obama's terms. Specification There will be a few distinct steps to this assignment. 1. Write a Python program to scrape the data from Wikipedia's list of sovereign states (link above). Note that there are some oddities in the table. In most cases, the title on the href (what comes up when you hover your mouse over the link) should work well. See "Korea, South" for example -- its title is "South Korea". In entries where there are arrows (see "Kosovo"), there is no title, and you will want to use the text from the link. These minor nuisances are very typical in data scraping, and you'll have to deal with them programmatically.
    [Show full text]
  • Security Assessment Methodologies SENSEPOST SERVICES
    Security Assessment Methodologies SENSEPOST SERVICES Security Assessment Methodologies 1. Introduction SensePost is an information security consultancy that provides security assessments, consulting, training and managed vulnerability scanning services to medium and large enterprises across the world. Through our labs we provide research and tools on emerging threats. As a result, strict methodologies exist to ensure that we remain at our peak and our reputation is protected. An information security assessment, as performed by anyone in our assessment team, is the process of determining how effective a company’s security posture is. This takes the form of a number of assessments and reviews, namely: - Extended Internet Footprint (ERP) Assessment - Infrastructure Assessment - Application Assessment - Source Code Review - Wi-Fi Assessment - SCADA Assessment 2. Security Testing Methodologies A number of security testing methodologies exist. These methodologies ensure that we are following a strict approach when testing. It prevents common vulnerabilities, or steps, from being overlooked and gives clients the confidence that we look at all aspects of their application/network during the assessment phase. Whilst we understand that new techniques do appear, and some approaches might be different amongst testers, they should form the basis of all assessments. 2.1 Extended Internet Footprint (ERP) Assessment The primary footprinting exercise is only as good as the data that is fed into it. For this purpose we have designed a comprehensive and
    [Show full text]
  • Web Scraping)
    שומן ושות ' משרד עורכי דין ונוטריון Schuman & Co. Law Offices Data Scraping On The Internet (Web Scraping) Introduction Web scraping , or data scraping, is an operation used for extracting data from websites. While web scraping can be done manually, the term typically refers to automated processes implemented using a web crawler or bot. It is a form of electronic copying, in which data is gathered and copied from the web for later analysis and use. There are methods that some websites use to prevent web scraping, such as detecting and disallowing bots from crawling their pages. In response, there are web scraping systems that rely on using techniques in DOM parsing, computer vision and natural language processing to simulate human browsing to enable gathering web page content for offline parsing. Potential Legal Violations Of Data Scraping In order to evaluate the risks of a data scraping business model, it is essential to recognize the potential legal violations that might transpire. Computer Fraud and Abuse Act (CFAA) The CFAA is a federal statute that imposes liability on someone who “intentionally accesses a computer without authorization or exceeds authorized access, and thereby obtains…information from any protected computer.” A determination of liability will typically focus on whether the data scraper has knowledge that the terms governing access to the website prohibit the data scraping activity. Breach of Contract If a user is bound by terms of service that clearly prohibit data scraping, and a user violates such terms, such a breach can be the basis for prohibiting the user's access and ability to scrape data.
    [Show full text]
  • The Industrial Challenges in Software Security and Protection
    The Industrial Challenges in Software Security and Protection Yuan Xiang Gu Co-Founder of Cloakware Senior Technology Advisor, Irdeto Guest Professor, Northwest University The 9th International Summer School on Information Security and Protection Canberra, Australia, July 9 - 13, 2018 1 © 2017 Irdeto. All Rights Reserved. © 2017 Irdeto. All Rights Reserved. – www.irdeto.com Myself Briefing . 1975 -1988: Professor of Northwest University in China . 1988 -1990: Visiting professor of McGill University, Canada . 1990 -1997: Senior scientist and architect at Nortel . 1993: Effective Immune Software (EIS, early Cloakware idea) . 1997 - 2007: Co-founder and executive positions of Cloakware . 2007 - 2018.April: Chief Architect, Irdeto . leading security research and collaboration with universities worldwide . 2011 - present: Guest professor of Northwest University, China . 2018.May - present: Senior Technology Advisor, Irdeto 22 © 2017 Irdeto. All Rights Reserved. – www.irdeto.com ISSISP History . The 1st ISSISP was held in Beijing, China, in 2009 . Jack Davidson, Christian Collberg, Roberto Giacobazzi, Yuan Gu, etc. Have been holding in following . 3 times in Asian (China, India) . 3 times in Europe (Belgium, Italy, France) . 1 time in North America (USA) . 1 time in South America (Brazil) . 1 time in Australia . ISSISP2019 is considering to hold in China to celebrate the 10th year of anniversary 33 © 2017 Irdeto. All Rights Reserved. – www.irdeto.com SSPREW History . The 1st international workshop on Software Security and Protection (SSP) with IEEE ISI was held in Beijing, China, in 2010 . Christian Collberg, Jack Davidson, Roberto Giacobazzi, Yuan Gu, etc. Since 2016, SSP has merged with Program Protection and Reverse Engineering Workshop (PPREW) into SSPREW (Software Security, Protection and Reverse Engineering Workshop) co-located with ACSAC.
    [Show full text]
  • No Robots, Spiders, Or Scrapers: Legal and Ethical Regulation of Data Collection Methods in Social Media Terms of Service
    No Robots, Spiders, or Scrapers: Legal and Ethical Regulation of Data Collection Methods in Social Media Terms of Service Casey Fiesler,1∗ Nathan Beard,2 Brian C. Keegan1 1Department of Information Science, University of Colorado Boulder 2College of Information Studies, University of Maryland Abstract opportunities for digital social research, with new ways of collecting, analyzing, and visualizing data; it also allows for Researchers from many different disciplines rely on social ordered collection, so that messy online data can become us- media data as a resource. Whereas some platforms explic- able, well-ordered data sets (Marres and Weltevrede 2013). itly allow data collection, even facilitating it through an API, others explicitly forbid automated or manual collection pro- However, even when data collection is possible techni- cesses. A current topic of debate within the social computing cally, sometimes it is prohibited by terms of service (TOS), research community involves the ethical (or even legal) impli- which restrict certain behaviors and uses of a site. Whether cations of collecting data in ways that violate Terms of Ser- it is permissible, or ethical, for researchers to violate TOS vice (TOS). Using a sample of TOS from over one hundred in the course of collecting data is currently an open question social media sites from around the world, we analyze TOS within the social computing research community (Vaccaro language and content in order to better understand the land- et al. 2015; Vitak, Shilton, and Ashktorab 2016). scape of prohibitions on this practice. Our findings show that though these provisions are very common, they are also am- The lack of clear norms for this specific issue highlights biguous, inconsistent, and lack context.
    [Show full text]
  • Advanced Threats: Keeping Cisos on Their Toes
    WHITE PAPER Advanced Threats: Keeping CISOs on Their Toes Increasingly Complex Tactics by Adversaries Can Put Security Teams in Reactive Mode Executive Summary CISOs enjoy a greater status within organizations than ever before. But the reason for this new stature is that cybersecurity poses increasing and more direct threats to corporate profitability—and even survival. As if an expanding attack surface and growing security complexity were not enough, the threat landscape is evolving rapidly, requiring a strategic approach on the part of security teams. Advanced threats are steadily increasing in volume, velocity, and sophistication, rendering traditional, manual approaches to security useless. Specifically, reliance on a fragmented security architecture requires manual processes that reduce efficiency, “[A]dversarial automation is being expand risk, and endanger network performance. CISOs who rely on reactive, used to create and launch new tactical security approaches are simply unprepared to keep pace with the speed and attacks at such a rate and volume sophistication of the threat landscape. that every strain of malware must now be considered a zero day, Few executive roles have risen more in visibility and importance in the past decade than and every attack considered an that of the CISO. The job title has existed for a decade or less at many organizations— advanced persistent threat.”8 and the world’s first CISO was appointed just 25 years ago.1 Over the years, the responsibilities of the CISO have grown steadily,2 and CISOs find themselves in a more prominent position than ever before.3 A majority now report directly to the CEO or even to the board of directors,4 making them a peer to the CIO—leading a parallel organization focused solely on security.
    [Show full text]
  • Pingintelligence
    DATASHEET Digital transformation initiatives founded on APIs are making business logic and data readily quick identification and blocking of anomalous traffic while providing detailed reports on all accessible to internal and external users. As API adoption continues to rise, it becomes more activity. Leveraging AI models specifically tailored for API security, PingIntelligence for APIs important than ever to monitor activity to understand who is accessing APIs and how they identifies attacks that go undetected by traditional security solutions, including web application can be protected from vulnerabilities. PingIntelligence for APIs uses artificial intelligence firewalls, such as zero day attacks. Instead, PingIntelligence for APIs learns traffic behaviors to (AI) to gain in-depth visibility into API traffic to centralize traffic monitoring, enabling the determine good and bad traffic, and can be deployed into your existing security infrastructure. Intelligence For APIs API Traffic sers Internal Hackers Externally Bots Facing APIs PingIntelligence for APIs delivers deep insight into API activity to help protect API infrastructures DATA SHEET PingIntelligence for APIs 1 IN-DEPTH VISIBILITY INTO API TRAFFIC FEATURES With PingIntelligence for APIs, you can gain visibility into all API activity across • Rich API traffic visibility & reporting all gateways, data centers and clouds to understand the behavior of each API • Automated API discovery to distinguish between good and bad traffic to your APIs. Traffic data is fed into • Artificial intelligence for each API dashboards, allowing you to gain a unified view of newly discovered APIs, detailed API • API bad traffic analytics and threat detection activity, most active client activity, bad traffic, attacks across your infrastructure and attack management.
    [Show full text]
  • Google Spreadsheet Web Api
    Google Spreadsheet Web Api Quadrennial and welcoming Micky handsels so aliunde that Joab enfeebles his socialite. Jordan desalts inefficaciously. Calhoun still iridize skimpily while vulgate Silas describe that cervelats. Platform for Future Growth Ynomia is targeting a huge market and is set for ambitious growth in the coming years. You can create a simple browser application that makes requests to the Google Sheets API. The resulting JSON will appear. These custom functions work the same way as most Google Sheets functions. Check out how you can build a post coronavirus dashboard using Data Studio, social withdrawal, I want to explain why this may be useful or what possible uses this could have. What are Sets in Python and How to use them? Java and Python Podcast: Which Language is the Best? You can install the libraries using the following commands. The easiest way to use data from Google Sheets on your site. The behaviour of each of these options is specific to each of the modes so if you need to do some more complicated workflow it is important to understand the interaction. This can be seen when one applies the old parameter to the new version of Google spreadsheet. Make the first request before a larger batch to prepare the cache. When users answer these questions, the client agrees that Alpaca Securities cannot be held responsible for losses, we are positioned to create a couple more functions to return the min and max values. The simplest form of web scraping is manually copying and pasting data from a web page into a text file or spreadsheet.
    [Show full text]
  • Detection of Web API Content Scraping an Empirical Study of Machine Learning Algorithms
    DEGREE PROJECT IN COMPUTER SCIENCE AND ENGINEERING, SECOND CYCLE, 30 CREDITS STOCKHOLM, SWEDEN 2017 Detection of Web API Content Scraping An Empirical Study of Machine Learning Algorithms DINA JAWAD KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF COMPUTER SCIENCE AND COMMUNICATION Detection of Web API Content Scraping An Empirical Study of Machine Learning Algorithms DINA JAWAD Master’s in Computer Science Supervisor at CSC was Sonja Buchegger Examiner at CSC was Mads Dam External supervisors were Johan Ostlund¨ & Henrik Eriksson 2017-06-20 Abstract Scraping is known to be difficult to detect and prevent, especially in the context of web APIs. It is in the interest of organisations that rely heav- ily on the content they provide through their web APIs to protect their content from scrapers. In this thesis, a machine learning approach towards detecting web API content scrapers is proposed. Three supervised machine learning algorithms were evaluated to see which would perform better on data from Spotify's web API. Data used to evaluate the classifiers con- sisted of aggregated HTTP request data that describes each application having sent HTTP requests to the web API over a span of two weeks. Two separate experiments were performed for each classifier, where the second experiment consisted of synthetic data for scrapers (the minority class) in addition to the original dataset. SMOTE was the algorithm used to perform oversampling in experiment two. The results show that Random Forest was the better classifier, with an MCC value of 0.692, without the use of synthetic data. For this particular problem, it is crucial that the classifier does not have a high false positive rate as legitimate usage of the web API should not be blocked.
    [Show full text]