News for hackers. Stuff that matters.
Re:Crawling Billions of Pages: Building Large Scale Crawling... by wslh on Thursday June 11, 2015 @10:35PM (#9703702)
mmmmmmmm, on http://en.wikipedia.org/wiki/BitTorrent it says "BitTorrent is one of the most common protocols for transferring large files, and peer-to-peer networks have been estimated to collectively account for approximately 43% to 70% of all Internet traffic (depending on geographical location)" so... 60% of requests are done by crawlers?

Have a beer and read a more insightful blog about this subject: http://blog.databigbang.com

There are no comments on this comment yet.