Re:61.5% of Web Traffic Is Not Human
by kristopolous on Saturday December 14, 2013 @09:52PM
(#6908157)
I've got no investment either way. My nginx log has 19,333,928 entries ... when I do grep -Ei '(bot|spider|crawl|aggregator)' on it I get 2,609,874.
That means that bots who declare themselves make about 13.5% of requests to this site. The robots.txt has no restrictions.
I don't know if that's helpful.
That doesn't really tell us anything about whether or not they trigger javascript. Bots triggering javascript will inflate advertising and js-based analytics numbers, which would be more problematic.
Bots that don't trigger js, however, are more a drain on bandwidth than anything else.
I seem to remember from a job I used to have as an analyst that advertisers have a few ways to discern human and bot traffic. They don't pay for traffic that is known to be from bots. However there are clearly some bots that aren't accounted for this way, and some that are designed specifically to trigger ads.
Per spider.io, bots are ripping off advertisers left and right. That being said, scared advertisers are the target market for their products.
http://www.spider.io/blog/