News for hackers. Stuff that matters.
Re:A special build of curl that can impersonate Chrome and F... by jchw on Thursday April 03, 2025 @12:40PM (#43572243)
I'm rooting for Ladybird to gain traction in the future. Currently, it is using cURL proper for networking. That is probably going to have some challenges (I think cURL is still limited in some ways, e.g. I don't think it can do WebSockets over h2 yet) but on the other hand, having a rising browser engine might eventually remove this avenue for fingerprinting since legitimate traffic will have the same fingerprint as stock cURL.
Re:A special build of curl that can impersonate Chrome and F... by rhdunn on Thursday April 03, 2025 @12:53PM (#43572413)
It would be good to see Ladybird's cURL usage improve cURL itself, such as the WebSocket over h2 example you mention. It is also a good test of cURL to see and identify what functionality cURL is missing w.r.t. real-world browser workflows.
Re:A special build of curl that can impersonate Chrome and F... by nonrandomstring on Thursday April 03, 2025 @03:25PM (#43574225)
When I spoke to these guys [0] we touched on those quirks and foibles that make a signature (including TCP stack stuff beyond control of any userspace app).

I love this curl, but I worry that if a component takes on the role of deception in order to "keep up" it accumulates a legacy of hard to maintain "compatibility" baggage.

Ideally it should just say... "hey I'm curl, let me in"

The problem of course lies with a server that is picky about dress codes, and that problem in turn is caused by crooks sneaking in disguise, so it's rather a circular chicken and egg thing.

[0] https://cybershow.uk/episodes.php?id=39

Re:A special build of curl that can impersonate Chrome and F... by immibis on Thursday April 03, 2025 @03:53PM (#43574560)
What should instead happen is that Chrome should stop sending as much of a fingerprint, so that sites won't be able to fingerprint. That won't happen, since it's against Google's interests.
Re:A special build of curl that can impersonate Chrome and F... by gruez on Thursday April 03, 2025 @04:22PM (#43574900)
This is a fundamental misunderstanding of how TLS fingerprinting works. The "fingerprint" isn't from chrome sending a "fingerprint: [random uuid]" attribute in every TLS negotiation. It's derived from various properties of the TLS stack, like what ciphers it can accept. You can't make "stop sending as much of a fingerprint", without every browser agreeing on the same TLS stack. It's already minimal as it is, because there's basically no aspect of the TLS stack that users can configure, and chrome bundles its own, so you'd expect every chrome user to have the same TLS fingerprint. It's only really useful to distinguish "fake" chrome users (eg. curl with custom header set, or firefox users with user agent spoofer) from "real" chrome users.
Re:A special build of curl that can impersonate Chrome and F... by dochtman on Thursday April 03, 2025 @04:30PM (#43574983)
Part of the fingerprint is stuff like the ordering of extensions, which Chrome could easily do but AFAIK doesn’t.

(AIUI Google’s Play Store is one of the biggest TLS fingerprinting culprits.)

Re:A special build of curl that can impersonate Chrome and F... by shiomiru on Thursday April 03, 2025 @04:37PM (#43575074)
Chrome has randomized its ClientHello extension order for two years now.[0]

The companies to blame here are solely the ones employing these fingerprinting techniques, and those relying on services of these companies (which is a worryingly large chunk of the web). For example, after the Chrome change, Cloudflare just switched to a fingerprinter that doesn't check the order.[1]

[0]: https://chromestatus.com/feature/5124606246518784

[1]: https://blog.cloudflare.com/ja4-signals/

Re:A special build of curl that can impersonate Chrome and F... by fc417fc802 on Thursday April 03, 2025 @06:12PM (#43576104)
> The companies to blame here are solely the ones employing these fingerprinting techniques,

Let's not go blaming vulnerabilities on those exploiting them. Exploitation is also bad but being exploitable is a problem in and of itself.

Re:A special build of curl that can impersonate Chrome and F... by shiomiru on Friday April 04, 2025 @05:18AM (#43579898)
> Let's not go blaming vulnerabilities on those exploiting them. Exploitation is also bad but being exploitable is a problem in and of itself.

There's "vulnerabilities" and there's "inherent properties of a complex protocol that is used to transfer data securely". One of the latter is that metadata may differ from client to client for various reasons, inside the bounds accepted in the standard. If you discriminate based on such metadata, you have effectively invented a new proprietary protocol that certain existing browsers just so happen to implement.

It's like the UA string, but instead of just copying a single HTTP header, new browsers now have to reverse engineer the network stack of existing ones to get an identical user experience.

Re:A special build of curl that can impersonate Chrome and F... by fc417fc802 on Friday April 04, 2025 @05:40AM (#43580006)
I get that. I don't condone the behavior of those doing the fingerprinting. But what I'm saying is that the fact that it is possible to fingerprint should in pretty much all cases be viewed as a sort of vulnerability.

It isn't necessarily a critical vulnerability. But it is a problem on some level nonetheless. To the extent possible you should not be leaking information that you did not intend to share.

A protocol that can be fingerprinted is similar to a water pipe with a pinhole leak. It still works, it isn't (necessarily) catastrophic, but it definitely would be better if it wasn't leaking.

Re:A special build of curl that can impersonate Chrome and F... by Jubijub on Friday April 04, 2025 @05:14PM (#43587654)
I’m sorry but you comment shows you never had to fight this problem a scale. The challenge is not small time crawlers, the challenge is blocking large / dedicated actors. The problem is simple : if there is more than X volume of traffic per <aggregation criteria >, block it. Problem : most aggregation criteria are trivially spoofable, or very cheap to change : - IP : with IPv6 this is not an issue to rotate your IP often - UA : changing this is scraping 101 - SSL fingerprint : easy to use the same as everyone - IP stack fingerprint : also easy to use a common one - request / session tokens : it’s cheap to create a new session You can force login, but then you have a spam account creation challenge, with the same issues as above, and depending on your infra this can become heavy

Add to this that the minute you use a signal for detection, you “burn” it as adversaries will avoid using it, and you lose measurement thus the ability to know if you are fixing the problem at all.

I worked on this kind of problem for a FAANG service, whoever claims it’s easy clearly never had to deal with motivated adversaries

Re:A special build of curl that can impersonate Chrome and F... by immibis on Saturday April 05, 2025 @03:01PM (#43595929)
Should be easy enough to create a DroneBL for residential proxy services. Since you work on residential proxy detection at a FAANG service, why haven't you done it yet?

If they're doing things the above-board way from their own ASN, block their ASN.

If they're doing things the above-board way from third-party hosting providers, send abuse reports. Late last year there was a commotion because someone was sending single spoofed SSH SYN packets, from the addresses of Tor nodes, to organizations with extremely sensitive security policies. Many people with Tor nodes got threats of being banned from their hosting provider, over a single packet they didn't even send. They're definitely going to ban people who are doing actual DDoSes from their servers.

DDoS is also a federal crime, so if you and they are in the USA, you might consider trying to get them put in prison.

Re:A special build of curl that can impersonate Chrome and F... by nonrandomstring on Thursday April 03, 2025 @05:05PM (#43575406)
> blame here are solely the ones employing these fingerprinting techniques,

Sure. And it's a tragedy. But when you look at the bot situation and the sheer magnitude of resource abuse out there, you have to see it from the other side.

FWIW the conversation mentioned above, we acknowledged that and moved on to talk about behavioural fingerprinting and why it makes sense not to focus on the browser/agent alone but what gets done with it.

Re:A special build of curl that can impersonate Chrome and F... by NavinF on Thursday April 03, 2025 @06:38PM (#43576317)
Last time I saw someone complaining about scrapers, they were talking about 100gib/month. That's 300kbps. Less than $1/month in IP transit and ~$0 in compute. Personally I've never noticed bots show up on a resource graph. As long as you don't block them, they won't bother using more than a few IPs and they'll backoff when they're throttled
Re:A special build of curl that can impersonate Chrome and F... by marcus0x62 on Thursday April 03, 2025 @06:55PM (#43576459)
For some sites, things are a lot worse. See, for example, Jonathan Corbet's report[0].

0 - https://social.kernel.org/notice/AqJkUigsjad3gQc664

Re:A special build of curl that can impersonate Chrome and F... by NavinF on Saturday April 05, 2025 @04:22PM (#43596528)
He provides no info. req/s? 95%ile mbps? How does he know the requests come from an "AI-scraper" as opposed to a normal L7 DDoS? LWN is a pretty simple site, it should be easy to saturate 10G ports
Re:A special build of curl that can impersonate Chrome and F... by lmz on Thursday April 03, 2025 @10:23PM (#43577683)
How can you say it's $0 in compute without knowing if the data returned required any computation?
Re:A special build of curl that can impersonate Chrome and F... by NavinF on Saturday April 05, 2025 @04:25PM (#43596554)
Look at the sibling replies. All the kvetching comes from blogs and simple websites, not the ones that consume compute per request
Re:A special build of curl that can impersonate Chrome and F... by nonrandomstring on Friday April 04, 2025 @03:39AM (#43579340)
Didn't rachelbytheebay post recently that her blog was being swamped? I've heard that from a few self-hosting bloggers now. And Wikipedia has recently said more than half of traffic is noe bots. ARe you claiming this isn't a real problem?
Re:A special build of curl that can impersonate Chrome and F... by NavinF on Saturday April 05, 2025 @04:24PM (#43596544)
How exactly can a blog get swamped? It takes ~0 compute per request. Yes I'm claiming this is a fake problem
Re:A special build of curl that can impersonate Chrome and F... by gruez on Thursday April 03, 2025 @04:32PM (#43575010)
What's the advantage of randomizing the order, when all chrome users already have the same order? Practically speaking there's a bazillion ways to fingerprint Chrome besides TLS cipher ordering, that it's not worth adding random mitigations like this.
Re:A special build of curl that can impersonate Chrome and F... by RKFADU_UOFCCLEL on Friday April 04, 2025 @11:51AM (#43584170)
What? Just fix the ciphers to a list of what's known to work + some safety margin. Each user needing some different specific cipher (like a cipher for horses, and one for dogs), is not a thing.
Re:A special build of curl that can impersonate Chrome and F... by gruez on Friday April 04, 2025 @12:52PM (#43585002)
>Just fix the ciphers to a list of what's known to work + some safety margin.

That's already the case. The trouble is that NSS (what firefox uses) doesn't support the same cipher suites as boringssl (what chrome uses?).

Re:A special build of curl that can impersonate Chrome and F... by thaumasiotes on Thursday April 03, 2025 @05:38PM (#43575789)
> Ideally it should just say... "hey I'm curl, let me in"

What? Ideally it should just say "GET /path/to/page".

Sending a user agent is a bad idea. That shouldn't be happening at all, from any source.

Re:A special build of curl that can impersonate Chrome and F... by Tor3 on Friday April 04, 2025 @12:17AM (#43578281)
Since the first browser appeared I've always meant that sending a user agent id was a really bad idea. It breaks with the fundamental idea of the web protocol, that it's the server's responsibility to provide data and it's the client's responsibility to present it to the user. The server does not need to know anything about the client. Including user agent in this whole thing was a huge mistake as it allowed web site designers to code for specific quirks in browsers. I can to some extent accept a capability list from the client, but I'm not so sure even that is necessary.
Re:A special build of curl that can impersonate Chrome and F... by nonrandomstring on Friday April 04, 2025 @03:35AM (#43579324)
Absolutely, yes! A protocol should not be tied to client details. Where did "User Agent" strings even come from?
Re:A special build of curl that can impersonate Chrome and F... by darrenf on Friday April 04, 2025 @04:47AM (#43579741)
They're in the HTTP/1.0 spec. https://www.rfc-editor.org/rfc/rfc1945#section-10.15

10.15 User-Agent

   The User-Agent request-header field contains information about the
   user agent originating the request. This is for statistical purposes,
   the tracing of protocol violations, and automated recognition of user
   agents for the sake of tailoring responses to avoid particular user
   agent limitations.
Re:A special build of curl that can impersonate Chrome and F... by eesmith on Thursday April 03, 2025 @01:40PM (#43573011)
I'm hoping this means Ladybird might support ftp URLs.
Re:A special build of curl that can impersonate Chrome and F... by navanchauhan on Thursday April 03, 2025 @03:13PM (#43574060)
and even the Gopher protocol!
Re:A special build of curl that can impersonate Chrome and F... by userbinator on Thursday April 03, 2025 @08:01PM (#43576912)
but on the other hand, having a rising browser engine might eventually remove this avenue for fingerprinting

If what I've seen from CloudFlare et.al. are any indication, it's the exact opposite --- the amount of fingerprinting and "exploitation" of implementation-defined behaviour has increased significantly in the past few months, likely in an attempt to kill off other browser engines; the incumbents do not like competition at all.

The enemy has been trying to spin it as "AI bots DDoSing" but one wonders how much of that was their own doing...

Re:A special build of curl that can impersonate Chrome and F... by hansvm on Thursday April 03, 2025 @09:02PM (#43577273)
Hold up, one of those things is not like the other. Are we really blaming webmasters for 100x increases in costs from a huge wave of poorly written and maliciously aggressive bots?
Re:A special build of curl that can impersonate Chrome and F... by refulgentis on Thursday April 03, 2025 @09:04PM (#43577293)
> Are we really blaming...

No, they're discussing increased fingerprinting / browser profiling recently and how it affects low-market-share browsers.

Re:A special build of curl that can impersonate Chrome and F... by hansvm on Thursday April 03, 2025 @10:47PM (#43577819)
I saw that, but I'm still not sure how this fits in:

> The enemy has been trying to spin it as "AI bots DDoSing" but one wonders how much of that was their own doing...

I'm reading that as `enemy == fingerprinters`, `that == AI bots DDoSing`, and `their own == webmasters, hosting providers, and CDNs (i.e., the fingerprinters)`, which sounds pretty straightforwardly like the fingerprinters are responsible for the DDoSing they're receiving.

That interpretation doesn't seem to match the rest of the post though. Do you happen to have a better one?

Re:A special build of curl that can impersonate Chrome and F... by userbinator on Thursday April 03, 2025 @11:48PM (#43578141)
"their own" = CloudFlare and/or those who have vested interests in closing up the Internet.
Re:A special build of curl that can impersonate Chrome and F... by jillyboel on Friday April 04, 2025 @12:16PM (#43584557)
Your costs only went up 100x if you built your site poorly
Re:A special build of curl that can impersonate Chrome and F... by hansvm on Friday April 04, 2025 @05:01PM (#43587562)
I'll bite. How do you serve 100x the traffic without 100x the costs? It costs something like 1e-10 dollars to serve a recipe page with a few photos, for example. If you serve it 100x more times, how does that not scale up?
Re:A special build of curl that can impersonate Chrome and F... by jillyboel on Friday April 04, 2025 @06:15PM (#43588274)
It might scale up but if you're anywhere near efficient you're way overprovisioned to begin with. The compute cost should be miniscule due to caching and bandwidth is cheap if you're not with one of the big clouds. As an example, according to dang HN runs on a single server and yet many websites that get posted to HN, and thus receive a fraction of the traffic, go down due to the load.
Re:A special build of curl that can impersonate Chrome and F... by immibis on Saturday April 05, 2025 @03:41AM (#43591692)
You got 100x the traffic if your traffic was near zero to begin with.
Re:A special build of curl that can impersonate Chrome and F... by SoftTalker on Friday April 04, 2025 @12:39AM (#43578375)
It's entirely deliberate. CloudFlare could certainly distinguish low-volume but legit web browsers from bots, as much as they can distinguish chrome/edge/safari/firefox from bots. That is if they cared to.
Re:A special build of curl that can impersonate Chrome and F... by cyanydeez on Friday April 04, 2025 @01:42AM (#43578710)
I dont think they're doing this to kill off browser engines; they're trying to sift browsers into "user" and "AI slop", so they can prioritize users.

This is entirely web crawler 2.0 apocolypse.

Re:A special build of curl that can impersonate Chrome and F... by nicman23 on Friday April 04, 2025 @02:07AM (#43578827)
man i just want a bot to buy groceries for me
Re:A special build of curl that can impersonate Chrome and F... by baq on Friday April 04, 2025 @03:39AM (#43579343)
That’s one of the few reasons to leave the house. I’d like dishes and laundry bots first, please.
Re:A special build of curl that can impersonate Chrome and F... by dodslaser on Friday April 04, 2025 @05:14AM (#43579880)
You mean dishwashers and washing machines?
Re:A special build of curl that can impersonate Chrome and F... by baq on Friday April 04, 2025 @07:02AM (#43580515)
Yes, but no. I want a robot to load and unload those.
Re:A special build of curl that can impersonate Chrome and F... by dec0dedab0de on Friday April 04, 2025 @10:01AM (#43582527)
I have been paying my local laundromat to do mu laundry for over a decade now, it’s probably cheaper than youre imagining and sooo worth it.
Re:A special build of curl that can impersonate Chrome and F... by baq on Friday April 04, 2025 @01:42PM (#43585596)
my household is 6 people, it isn't uncommon to run 3 washing machine loads in a day and days without at least one are rare. I can imagine the convenience, but at this scale it sounds a bit unreasonable.

dishwasher runs at least once a day, at least 80% full, every day, unless we're traveling.

Re:A special build of curl that can impersonate Chrome and F... by nicman23 on Monday April 07, 2025 @04:08AM (#43609056)
i mean it is called a dishwasher
Re:A special build of curl that can impersonate Chrome and F... by extraduder_ire on Friday April 04, 2025 @10:09AM (#43582666)
I think "slop" only refers to the output of generative AI systems. bot, crawler, scraper, or spider would be a more apt term for software making (excessive) requests to collect data.
Re:A special build of curl that can impersonate Chrome and F... by johnisgood on Friday April 04, 2025 @06:41AM (#43580376)
I used to call it "cURL", but apparently officially it is curl, correct?
Re:A special build of curl that can impersonate Chrome and F... by cruffle_duffle on Friday April 04, 2025 @09:37AM (#43582189)
As in “See-URL”? I’ve always called it curl but “see url” makes a hell of a lot of sense too! I’ve just never considered it and it’s one of those things you rarely say out loud.
Re:A special build of curl that can impersonate Chrome and F... by johnisgood on Friday April 04, 2025 @10:02AM (#43582551)
I prefer cURL as well, but according to official sources it is curl. :D Not sure how it is pronounced though, I pronounce it as "see-url" and/or "see-U-R-L". It might be pronounced as "curl" though.
Re:A special build of curl that can impersonate Chrome and F... by bdhcuidbebe on Friday April 04, 2025 @01:03PM (#43585134)
I’d guess Daniel pronounce it as ”kurl”, with a hard C like in ”crust”, since hes swedish.
Re:A special build of curl that can impersonate Chrome and F... by devwastaken on Friday April 04, 2025 @11:02AM (#43583469)
ladybird does not have the resources to be a contender to current browsers. its well marketed but has no benefits or reason to exist over chromium. its also a major security risk as it is designed yet again in demonstrably unsafe c++.