Re:Breaking the Open Web Won't Stop the Bad Bots
by AnthonyMouse on Monday November 02, 2026 @11:55AM
(#50339728)
> No, I am proposing that if googlebot is not within that inaccurate query string, google is in violation.
"Googlebot" is their search indexer. That's generally not the thing anyone wants to block anyway because it would mean their pages don't show up in search results, but let's suppose you were actually willing to do that. You block anything with "Googlebot" in the user agent string.
Then Google is doing its AI training -- thing you actually wanted to prevent -- in some other business unit, and they have their own indexing operation, so they might use their own user agent and call it GeminiBot or whatever. This is in theory what you want, then you could distinguish between them, except that then your original rule to block "Googlebot" is having no effect and they've sucked up a copy of your entire site before you notice there is a new one.
Meanwhile it's still an arbitrary string and they can still have as many as they want or change them at any time. Google doesn't even need to index your site themselves, a third party could do it and provide them with the data, and if people were blocking "Googlebot" but not unknown user agents then that's what would happen. None of the third parties are "Googlebot" nor is there anything stopping new ones you've never seen from being created at any time. Blacklists don't work.
> Anyone pretending to be a common browser when they're not is in violation.
Then the common browsers are in violation because they're pretending to be each other. And what do you expect the uncommon browsers to do?
You can't get out of the consequences of your proposal. If you expect blacklists then scrapers will constantly appear with new user agents. There is nothing in that text that requires Google to always use "Googlebot" and even that wouldn't do any good when anyone else could create one called anything else. But if you expect whitelists while prohibiting anyone from using the same string as the incumbents then you're proposing something outrageous because you're saying that any innocent new entrant de facto gets blocked by default and there is nothing they can do about it.
"Googlebot" is their search indexer. That's generally not the thing anyone wants to block anyway because it would mean their pages don't show up in search results, but let's suppose you were actually willing to do that. You block anything with "Googlebot" in the user agent string.
Then Google is doing its AI training -- thing you actually wanted to prevent -- in some other business unit, and they have their own indexing operation, so they might use their own user agent and call it GeminiBot or whatever. This is in theory what you want, then you could distinguish between them, except that then your original rule to block "Googlebot" is having no effect and they've sucked up a copy of your entire site before you notice there is a new one.
Meanwhile it's still an arbitrary string and they can still have as many as they want or change them at any time. Google doesn't even need to index your site themselves, a third party could do it and provide them with the data, and if people were blocking "Googlebot" but not unknown user agents then that's what would happen. None of the third parties are "Googlebot" nor is there anything stopping new ones you've never seen from being created at any time. Blacklists don't work.
> Anyone pretending to be a common browser when they're not is in violation.
Then the common browsers are in violation because they're pretending to be each other. And what do you expect the uncommon browsers to do?
You can't get out of the consequences of your proposal. If you expect blacklists then scrapers will constantly appear with new user agents. There is nothing in that text that requires Google to always use "Googlebot" and even that wouldn't do any good when anyone else could create one called anything else. But if you expect whitelists while prohibiting anyone from using the same string as the incumbents then you're proposing something outrageous because you're saying that any innocent new entrant de facto gets blocked by default and there is nothing they can do about it.