News for hackers. Stuff that matters.
Re:AVG disguises fake traffic as IE6 by ajross on Friday June 27, 2008 @11:43AM (#229835)
There aren't any. The User-Agent header has always lied, at least a little bit. What AVG wants from your site is the same thing that the users do: the content (in this case, so that they can scan it). If you insist on giving them a different page (like an error page) than a user would see, they're going to take steps to prevent you from doing that.

Remember that the desired outcome here is to see fewer AVG hits, not to "screen them out" just because they're not a human being (although they are a necessary condition for a human hit coming later). Try reducing the unique URL count on your site (basically: make it more RESTful and eliminate all the query strings on broadly identical content) so they have fewer URLs to crawl. Set your caching headers properly so that they don't have an excuse to recrawl you too often. That won't force them to hit you less, but it will "help them to help you" by reducing the total bandwidth. Remember they don't want to pull any more than they have to either.

Re:AVG disguises fake traffic as IE6 by BrandonM on Friday June 27, 2008 @01:40PM (#229967)
What AVG wants from your site is the same thing that the users do

Then shouldn't AVG report the same User-Agent that the user's browser is reporting?

Re:AVG disguises fake traffic as IE6 by aristus on Friday June 27, 2008 @04:04PM (#230122)
If AVG didn't have their head up their asses, yes. Though that would make the problem for webmasters even harder. What really chews my liver is that they didn't even do it right.