News for hackers. Stuff that matters.
Re:61.5% of Web Traffic Is Not Human by jgalt212 on Saturday December 14, 2013 @09:11PM (#6908025)
that's good point. It would be nice to see someone who does not have a vested interest in under or over reporting the number of bots on the web take a good crack at this number.

Per spider.io, bots are ripping off advertisers left and right. That being said, scared advertisers are the target market for their products.

http://www.spider.io/blog/

Re:61.5% of Web Traffic Is Not Human by kristopolous on Saturday December 14, 2013 @09:52PM (#6908157)
I've got no investment either way. My nginx log has 19,333,928 entries ... when I do grep -Ei '(bot|spider|crawl|aggregator)' on it I get 2,609,874.

That means that bots who declare themselves make about 13.5% of requests to this site. The robots.txt has no restrictions.

I don't know if that's helpful.

Re:61.5% of Web Traffic Is Not Human by arn on Saturday December 14, 2013 @10:57PM (#6908312)
That doesn't really tell us anything about whether or not they trigger javascript. Bots triggering javascript will inflate advertising and js-based analytics numbers, which would be more problematic.

Bots that don't trigger js, however, are more a drain on bandwidth than anything else.

Re:61.5% of Web Traffic Is Not Human by ch4s3 on Sunday December 15, 2013 @01:00AM (#6908548)
I seem to remember from a job I used to have as an analyst that advertisers have a few ways to discern human and bot traffic. They don't pay for traffic that is known to be from bots. However there are clearly some bots that aren't accounted for this way, and some that are designed specifically to trigger ads.