News for hackers. Stuff that matters.

A comment on Ask HN: Can I get in trouble for crawling using the Googlebot user agent?. Parent comment

Re:Ask HN: Can I get in trouble for crawling using the Googl... by mathrawka on Friday April 21, 2017 @01:22PM (#14167449)
Not to detract from your point, but blindly blocking a User Agent because of a bad actor and losing money is not a good solution.

A more robust solution can be coded using information from Google: https://support.google.com/webmasters/answer/80553?hl=en

[ Parent | on HN ]

There are no comments on this comment yet.