News for hackers. Stuff that matters.

A comment on Company withdrawing from Facebook as analytics show 80% of ad clicks from bots. Parent comment

Re:Company withdrawing from Facebook as analytics show 80% o... by don_draper on Monday July 30, 2012 @08:19PM (#4315398)
One quick way to detect bots is to look for a GET request of the robots.txt file
[ Parent | on HN ]
Re:Company withdrawing from Facebook as analytics show 80% o... by freeall on Tuesday July 31, 2012 @01:31AM (#4315869)
Sort of explained already, but "bad" bots would never go look for that file. And a good bot probably already identifies itself the the request, so no need to look through robots.txt.
[ Parent | on HN ]
Re:Company withdrawing from Facebook as analytics show 80% o... by fsniper on Tuesday July 31, 2012 @05:39AM (#4316417)
That's it. A "bad bot" would not check robots.txt, but a legitimate user would check it. So looking for the software not checking robots.txt combined with user agent matching for good bots, you would have a good matching ratio.
[ Parent | on HN ]