News for hackers. Stuff that matters.
Re:Breaking the Open Web Won't Stop the Bad Bots by AnthonyMouse on Friday October 30, 2026 @05:31PM (#50310417)
> Someone who is using a residential proxy network, headless browsing, and/or TLS fingerprint manipulation to evade rate limits and other controls to issue large volumes of automated requests under a deceptive UserAgent string isn't difficult to describe and differentiate versus a normal user using a UA.

So then why isn't it trying to prohibit making multiple requests from different IP addresses in a residential proxy network, rather than the thing it actually does?

> The proposed text of the bill is rather straightforward. It turns more on the "stealth" bit than "bot".

Not really, look at the definition of "STEALTH BOT" -- the only thing that nominally distinguishes it from a browser is the word "bot" which isn't itself defined, because the phrase "without prior disclosure of its identity and purpose" is something that a normal web browser doesn't do either.

It can't even get it out of it via "a valid and accurate user-agent string", both because that clause says or and it still fails the second one, and because "a valid and accurate user-agent string" is a nonsense phrase. The Safari user agent strings start with "Mozilla" even though it's not. The Android ones do the same and then include "AppleWebKit" and "Mobile Safari" too. A browser extension uses the user agent string of the browser regardless of what the extension does. When Debian packages their version of Firefox, its user agent string says Firefox and not Debian.

User agent strings don't have a registry. "Valid" has no meaning and "accurate" is contrary to the default behavior of common browsers.

Re:Breaking the Open Web Won't Stop the Bad Bots by za_creature on Friday October 30, 2026 @05:52PM (#50310638)
User agent strings are a sad consequence of websites incorrectly parsing them to enable functionality, resulting in a "every bug becomes a contract" type of situation.

That being said, if you include GOOGLEBOT in your 1k long query string, alongside a very long page describing your crawlers, how to identify them, how to block them and how to set limits, no judge will find you guilty of concealing your purpose.

As for why the law is written as is instead of banning more specific functionality, it's so assholes don't get any ideas on how to bypass it.

Re:Breaking the Open Web Won't Stop the Bad Bots by TomatoCo on Friday October 30, 2026 @06:00PM (#50310705)
> User agent strings are a sad consequence of websites incorrectly parsing them to enable functionality, resulting in a "every bug becomes a contract" type of situation.

It's a bit old but https://webaim.org/blog/user-agent-string-history/ is always a good parable.

Re:Breaking the Open Web Won't Stop the Bad Bots by AnthonyMouse on Friday October 30, 2026 @11:55PM (#50313057)
> User agent strings are a sad consequence of websites incorrectly parsing them to enable functionality, resulting in a "every bug becomes a contract" type of situation.

Which is precisely why trying to legislate them is folly. Obviously the reason Android calls itself "Mobile Safari" is so that servers send it the mobile version of the site instead of the desktop one etc.

Which makes "accurate" an unreasonable requirement, since there is already a preexisting non-waivable requirement for inaccurate. When someone is making their "turn a web page into a PDF so you can view it later on the plane" software or whatever, it has to use a user agent string that will cause the site to send the version you need turned into the PDF.

> That being said, if you include GOOGLEBOT in your 1k long query string, alongside a very long page describing your crawlers, how to identify them, how to block them and how to set limits, no judge will find you guilty of concealing your purpose.

First, you're assuming that the person being prosecuted is the entities who are the nominal targets of the bill, when the real problem is when they're not. Because you're proposing a law that says anybody with an inaccurate user agent string is in violation, meanwhile everybody has an inaccurate user agent string.

Moreover, consider what you expect the outcome of legislating something like this is going to be. Sites could then try to either blacklist or whitelist user agents.

If they do the first one, that doesn't work, because the scraping service can change it whenever they want.

If they do the second one, that doesn't work, because now every third party has to start pretending to be a common browser even when they're doing something totally innocent or they get blocked by default for being something else. Which makes the user agent string even less useful than it is now, and subjects innocent parties to arbitrarily liability for doing something they have to do or their client is broken.

> As for why the law is written as is instead of banning more specific functionality, it's so assholes don't get any ideas on how to bypass it.

You're describing an autocracy. The way you make it so that you always prosecute someone is to make it so that you can always prosecute anyone, which is very bad.

Re:Breaking the Open Web Won't Stop the Bad Bots by za_creature on Saturday October 31, 2026 @05:23AM (#50314396)
> you're proposing a law that says anybody with an inaccurate user agent string is in violation

No, I am proposing that if googlebot is not within that inaccurate query string, google is in violation.

> Sites could then try to either blacklist or whitelist user agents.

Yes, that is a desired outcome.

> If they do the first one, that doesn't work, because the scraping service can change it whenever they want.

That would put them in violation, and make them prosecutable. Of course you need to start collecting data to prove that it's automatic, but this is already happening today for the purpose of temporarily blocking an IP. Having this law means that you also have some punitive measures at the end of the rainbow.

> If they do the second one ...

Anyone pretending to be a common browser when they're not is in violation. In addition, under this law there's no incentive for anyone to pretend to be googlebot since _that_ can get them blocked. You are intentionally and repeatedly trying to obfuscate these facts.

Re:Breaking the Open Web Won't Stop the Bad Bots by AnthonyMouse on Monday November 02, 2026 @11:55AM (#50339728)
> No, I am proposing that if googlebot is not within that inaccurate query string, google is in violation.

"Googlebot" is their search indexer. That's generally not the thing anyone wants to block anyway because it would mean their pages don't show up in search results, but let's suppose you were actually willing to do that. You block anything with "Googlebot" in the user agent string.

Then Google is doing its AI training -- thing you actually wanted to prevent -- in some other business unit, and they have their own indexing operation, so they might use their own user agent and call it GeminiBot or whatever. This is in theory what you want, then you could distinguish between them, except that then your original rule to block "Googlebot" is having no effect and they've sucked up a copy of your entire site before you notice there is a new one.

Meanwhile it's still an arbitrary string and they can still have as many as they want or change them at any time. Google doesn't even need to index your site themselves, a third party could do it and provide them with the data, and if people were blocking "Googlebot" but not unknown user agents then that's what would happen. None of the third parties are "Googlebot" nor is there anything stopping new ones you've never seen from being created at any time. Blacklists don't work.

> Anyone pretending to be a common browser when they're not is in violation.

Then the common browsers are in violation because they're pretending to be each other. And what do you expect the uncommon browsers to do?

You can't get out of the consequences of your proposal. If you expect blacklists then scrapers will constantly appear with new user agents. There is nothing in that text that requires Google to always use "Googlebot" and even that wouldn't do any good when anyone else could create one called anything else. But if you expect whitelists while prohibiting anyone from using the same string as the incumbents then you're proposing something outrageous because you're saying that any innocent new entrant de facto gets blocked by default and there is nothing they can do about it.

Re:Breaking the Open Web Won't Stop the Bad Bots by GoblinSlayer on Saturday October 31, 2026 @06:33AM (#50314714)
I don't think anybody does a decent ua check, webdevs test only in their favorite browser, i.e. Chrome, so only Chrome is allowed. Is it intended? The very purpose of webcompat is to masquerade as Chrome.

How do you even implement an ua check? How many browsers do you know? How many browsers you don't know? How do you implement an ua check for a browser that will appear tomorrow?

I wrote my own http downloader. Do you know about its existence? How do you implement an ua check for it? Can I call it "wget" to make ua check easier for you? Can I call it "Mozilla/5.0" so that it's compatible with your ua check so that you don't need to change anything?

Re:Breaking the Open Web Won't Stop the Bad Bots by za_creature on Saturday October 31, 2026 @09:35AM (#50316061)
> How do you even implement an ua check?

For this purpose, it looks like this:

  if 'googlebot' in request.headers.get('user-agent', 'googlebot'):
      return 403
Re:Breaking the Open Web Won't Stop the Bad Bots by oasisbob on Friday October 30, 2026 @10:25PM (#50312662)
You're slicing the meaning of UA strings too fine.

Bots can easily identify themselves through REP in RFC 9309. This isn't difficult or confusing.