Re:Ask HN: Can I get in trouble for crawling using the Googl...
by mathrawka on Friday April 21, 2017 @01:22PM
(#14167449)
Not to detract from your point, but blindly blocking a User Agent because of a bad actor and losing money is not a good solution.
A more robust solution can be coded using information from Google: https://support.google.com/webmasters/answer/80553?hl=en
Say your bot misbehaves and effectively starts DOSing a site with a whole lot of pages, like a small Reddit clone or something. And say Reddit doesn't have another way to determine between your bot and the Googlebot. You have now put Reddit in a position where they have to either block the Googlebot (and possibly lose a huge pile of money in the process) or else buy up a lot more hardware and bandwidth to pay for your crawler as well. That's not cool, to put it bluntly.