Crawling Billions of Pages: Building Large Scale Crawling Cluster, part 1

Re:Crawling Billions of Pages: Building Large Scale Crawling... by exacube on Thursday June 11, 2015 @09:15PM (#9703459)

From part 2 of their article:

> Currently, more than 60 percent of global internet traffic consists of requests from crawlers or some type of automated Web discovery system.

Where is this number from and how accurate can you make it?

[ Parent | on HN ]

Re:Crawling Billions of Pages: Building Large Scale Crawling... by Axsuul on Friday June 12, 2015 @05:24AM (#9704805)

It currently feels like 60 percent of global internet traffic is actually crawling their blog right now. Still waiting for it to load.

[ Parent | on HN ]

Re:Crawling Billions of Pages: Building Large Scale Crawling... by wslh on Thursday June 11, 2015 @10:35PM (#9703702)

mmmmmmmm, on http://en.wikipedia.org/wiki/BitTorrent it says "BitTorrent is one of the most common protocols for transferring large files, and peer-to-peer networks have been estimated to collectively account for approximately 43% to 70% of all Internet traffic (depending on geographical location)" so... 60% of requests are done by crawlers?

Have a beer and read a more insightful blog about this subject: http://blog.databigbang.com

[ Parent | on HN ]