AI scrapers request commented scripts

Re:AI scrapers request commented scripts by grayhatter on Saturday November 01, 2025 @11:00AM (#45782152)

Someone using a VPN is not lying. The intent of a user agent is to identify the software sending the request. The IP address isn't sent by the browser, and isn't part of the HTTP request. It's part of the routing information required to deliver the packet back to the client. If a client sent it's "real" IP address as an HTTP header, and I tried to respond to that IP instead of the IP address from the TCP packet. It would never arrive.

There's a difference between sending no data, and sending false data. I don't block requests without http referrers for that very reason.

[ Parent | on HN ]

Re:AI scrapers request commented scripts by wqaatwt on Saturday November 01, 2025 @11:08AM (#45782227)

IIRC Firefox (and I assume other browsers) when using privacy/no tracking mode does send fake data..

[ Parent | on HN ]

Re:AI scrapers request commented scripts by grayhatter on Saturday November 01, 2025 @11:19AM (#45782332)

You're incorrect. I've never seen any browser, on it's own lie about it's user agent. (I can set a custom string and lie with it, but that's not the agent doing it)

Do you have a specific / concrete example in mind? Or are you mistaking a feature from something other than a mainstream browser?

[ Parent | on HN ]

Re:AI scrapers request commented scripts by gkbrk on Saturday November 01, 2025 @05:54PM (#45785754)

Firefox sends an incorrect version and operating system on its User-Agent when the privacy settings are turned on.

IIRC it defaults to a Windows user agent even when you use it on other operating systems.

[ Parent | on HN ]

Re:AI scrapers request commented scripts by grayhatter on Sunday November 02, 2025 @01:30AM (#45788228)

You're incorrect. I have Firefox configured with the most strict privacy settings, and it returns `Mozilla/5.0 (X11; Linux x86_64; rv:142.0) Gecko/20100101 Firefox/142.0)` With the exception of it being Wayland instead of X11 it's entirely accurate. Would love to see whatever gaslit you of something so easy to test and validate.

[ Parent | on HN ]

Re:AI scrapers request commented scripts by gkbrk on Sunday November 02, 2025 @10:47AM (#45791160)

Nope, as it turns out this was actually a thing until 2025-01-24, where a commit removed this "pretend to be Windows even on Linux platforms" behavior.

https://github.com/mozilla-firefox/firefox/commit/eb2f90f870...

This OS spoofing behavior was added in 2019-01-09 with this commit:

https://github.com/mozilla-firefox/firefox/commit/264fe08c09...

So Firefox has spoofed the User-Agent as a Windows machine on Linux for around 6 years, and only stopped doing it early this year. Would love to see whatever gaslit you into forgetting this easy to test and validate behavior.

[ Parent | on HN ]

Re:AI scrapers request commented scripts by grayhatter on Sunday November 02, 2025 @11:03AM (#45791268)

https://support.mozilla.org/en-US/kb/resist-fingerprinting

This was part of the resist fingerprinting feature. Which is an advanced user configuration. I can alter the user agent directly myself too.

Sigh

I regret getting tricked into arguing over such a pedantic specific, So I'd like to redirect the actual point, which is that it's not meaningful if a Firefox browser pretends to be a slightly different Firefox browser, but instead the problem is when something that's not a browser, claims to be and behave like one.

Still, +1 for finding the commit, I'd forgotten about this feature. I thought only the tor browser was this foolish.

[ Parent | on HN ]