BOTREPORT
The 2015 Bad Bot Landscape Report
www.distilnetworks.com [email protected] 1.866.423.0606 Table of Contents
Introduction and Methodology ...... 3 Key Findings ...... 3 Bad bots as a percentage of overall traffic ...... 3 Bad Bot, Good Bot and Human Traffic 2013 and 2014 ...... 3 Bad Bot, Good Bot and Human Traffic, 2014 ...... 4 Large sites’ staggering bot traffic; smaller sites struggling with bad bots ...... 4 Traffic by Size of Website (Alexa Score), 2014...... 4 Digital publishing, travel, directory and ecommerce sites hit hardest by bad bots..4 Traffic by Type of Website, 2014 ...... 4 Traffic to Digital Publishing Websites, 2014 ...... 5 Traffic to Travel Websites, 2014 ...... 5 Traffic to Directory Websites, 2014 ...... 5 Traffic to Ecommerce Websites, 2014 ...... 5 Mobile bots arrive in droves, no longer “emerging” threat ...... 5 Bad Bot Self-Reported Browsers, 2014 ...... 5 Mobile sites easier to scrape ...... 5 Amazon has a big bot problem; Verizon cleans up its act...... 6 Top 20 Bad Bot Originators, 2013 and 2014 ...... 6 China leads the world in bad bot mobile traffic, T-Mobile USA more of a concern...7 Percent Mobile Bad Bot Traffic for U.S., China and Rest of Word, 2014 ...... 7 Top 20 Mobile Carriers with Highest Percent of Bad Bot Traffic, 2014...... 7 Percent Mobile Bad Bot Traffic From Largest Chinese and US Mobile Carriers, 2014...... 7 USA worst bad bot originator, but Singapore, Israel, Slovenia and Maldives have highest “Bad Bot GDP”...... 8 China and Russia most blocked countries ...... 8 Top 10 Most Blocked Countries, 2014 ...... 8 Countries Originating the Most Bad Bots, and Most Bad Bots per Online User, 2014...... 8 Bad bots mimic human behavior and are very sophisticated ...... 9 Ways Distil Catches Bots ...... 9 Bad Bots Sophistication Levels, 2014...... 9 Bad Bots Mimicking Human Behavior, 2014 ...... 9 Unmasking Human Imposter Bots ...... 10 Unmasking Sophisticated Bots, 2014 ...... 10 A More Widely Dispersed Bad Bot Landscape ...... 10 Percent of Bad Bot Traffic by Hour Hitting the U.S. East Coast, 2013 and 2014...... 10 Countries Originating at least 1% of Worldwide Bot Traffic ...... 10 Conclusion ...... 11 About Distil Networks...... 11 If you have any questions regarding the report or Introduction and its findings, please contact us today at Methodology [email protected] or on Twitter at The 2015 Bad Bot Landscape Report is the @DISTIL. culmination of months of analysis by the Distil Networks’ Data Science Team. The dataset Key Findings covers the 23 billion bad bot threats we saw in Bad bots as a percentage of overall 2014 as well as good bot and human traffic. The traffic dataset resides in Distil’s Hadoop Cluster and includes data from 100s of customers as well as The amount of bad bot traffic we saw across our Distil’s global network of 17 data centers. network as a percentage of overall traffic dropped slightly from 24.22% in 2013 to 22.78% Similar to last year’s study, the goal of this report in 2014. The bigger move was in Good Bot is to help identify statistically significant data and traffic growing from 20.98% to 36.32%, which insights into the bad bot landscape. Bad bots might be due in part to more aggressive indexing continue to place a huge tax on IT security and by Bing and upstart search engines 2014. These web infrastructure teams across the globe. The results exclude traffic from our single largest variety, volume and sophistication of today’s bots customer of 2014 as the weight of that one wreak havoc across online operations big and customer would have distorted the results. small. They’re the key culprits behind web scraping, brute force attacks, competitive Bad Bot, Good Bot and Human Traffic data mining, brownouts, account hijacking, 2013 and 2014 unauthorized vulnerability scans, spam, man-in- the-middle attacks, and click fraud. By shining a 100 light on their origins, behavior, and obfuscation Bad Bot Bad Bot methods, we hope to fulfill our mission of making 24.22% 22.78% the web more secure. 80 Good Bot This study does not account for large Distributed Good Bot 60 20.98% Denial of Service (DDoS) attacks. Distil 36.32% Networks is not a DDoS mitigation service. Rather, our focus is on application-layer attacks. 40 It’s also important to note that Distil’s customers Human tend to care more about stopping bots, and thus 54.80% Human could be more heavily targeted by bots than an 20 40.90% average website.
0 2013 2014