News for hackers. Stuff that matters.
Re:61.5% of Web Traffic Is Not Human by kristopolous on Saturday December 14, 2013 @09:52PM (#6908157)
I've got no investment either way. My nginx log has 19,333,928 entries ... when I do grep -Ei '(bot|spider|crawl|aggregator)' on it I get 2,609,874.

That means that bots who declare themselves make about 13.5% of requests to this site. The robots.txt has no restrictions.

I don't know if that's helpful.

Re:61.5% of Web Traffic Is Not Human by arn on Saturday December 14, 2013 @10:57PM (#6908312)
That doesn't really tell us anything about whether or not they trigger javascript. Bots triggering javascript will inflate advertising and js-based analytics numbers, which would be more problematic.

Bots that don't trigger js, however, are more a drain on bandwidth than anything else.

Re:61.5% of Web Traffic Is Not Human by ch4s3 on Sunday December 15, 2013 @01:00AM (#6908548)
I seem to remember from a job I used to have as an analyst that advertisers have a few ways to discern human and bot traffic. They don't pay for traffic that is known to be from bots. However there are clearly some bots that aren't accounted for this way, and some that are designed specifically to trigger ads.