davidfischer

> Do you have a list of the addresses that were hammering your site?

I could get a list of IPs, but it wouldn't be particularly helpful. It was hundreds of millions of unique IPs. Most IPs made fewer than 100 requests and then disappeared.

> Have you tried any of the techniques I list here?

We were already doing essentially all of the techniques there and they were not helpful in this attack. This wasn't some spider run amok. This was a dedicated attack with intelligence behind it.

> Do sets of the IP's show up in here [2]?

If we're looking at individual IPs, we've already lost.

> Are the bots mostly residential, VPS, Tor? What is the HTTP protocol breakdown? HTTP/1.1, 2.0, 3.0? Are they missing any expected client headers?

The article outlines most of this. It was a mix of residential and major/minor hosting providers and some corporates. If I were to say which was the largest, it was minor hosting providers. However, the attack pivoted between coming from different sources. At first it was coming mostly from minor hosting providers/VPS. By the end, it was entirely residential and corporate IPs. Tor was not involved at all (RTD supports accessing through Tor, but you are more likely to get a challenge). All the expected headers were there and UAs cycled between a very large set of standard UAs for normal browsers/devices. The attack was 95%+ HTTP2/HTTP3.

> [Edit] appears you are behind Cloudflare so the blackhole would be up to them. One could still return a 429 or 525 to the attackers.

It's trivial to setup an IP list[1] in CF and then you can apply all sorts of rules (lower rate limits, outright challenges, etc.) to it. It can be managed through Terraform as well. You can also use CIDR notation. This along with more classifications to specific ASNs are something we are looking at.

[1] https://developers.cloudflare.com/waf/tools/lists/custom-lis...

Author here. This attack was extremely broad. I saw parts of this attack come from my own home ISP's ASN, though not my IP thankfully. If we just "blocked those places" there would be a lot of collateral damage. As it stood, we did temporarily bump up rate limiting for the biggest attack ASNs and we absolutely heard from real, regular users about it (Sorry to those affected).

> The description/approach seems static and limited? Why not maintain a leaky bucket that counts each request (tickets/points) with higher cost for expensive requests (404, redirects). As the IP's reputation deteriorates (IPv4/32), it begins to spill over to a broader subnet like IPv4/31 then /30 and so on. Fight adaptivity with adaptivity. // maybe I describe something totally obvious, I'm not involved in the web ddos protection side of things.

The penalty box strategy I described in the post is along these lines. It penalizes excessive expensive requests directly. Specifically, it does add those to a score and will rate limit more broadly as necessary.

It will probably be useless one day. In practice, it is still useful today though not for this attack.

Here's my take:

* JA3s are mostly useless. JA4s supersede them entirely.

* Using JA4s in rate limits is pretty useful and helps a lot against proxy scraping. It was not very helpful in this attack.

* Bot detections are somewhat helpful but they don't solve scrapers/attacks by themselves. They're useful as a 2nd/3rd data point (eg. low bot score + bot detection + something else)

My assumption would be that it would drastically reduce the attack to be borderline irrelevant. I've never turned on Under Attack so somebody else may have more insight and the docs[1] don't describe precisely what happens besides a JS interstitial.

I know that JS challenges, both interactive and non-interactive, can be solved by bots. I've seen it. However, I suspect that the challenges just get harder and harder until the attack levels drop.

[1] https://developers.cloudflare.com/fundamentals/reference/und...

Author here. This was not a misconfigured data scraper. We see those every week[1]. This attack wasn't scraping useful content. It was almost entirely 404s and 302s and pulled virtually zero real docs. It specifically looked for URLs not served by the CDN and when it found a pattern, did millions of variations of it. Whether built by an AI or not, it was designed to cause outages and financial damage from autoscaling. However, as others have suggested, we may have been a test run for a real target.

[1] https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse...

I've never seen the phrase "manual transmission user-agent". Using your own browser yourself is the new stick shift. Love it.

I talked about that directly in the post. We didn't want to just challenge everyone. We use JS challenges but we try to use them sparingly. The rest of the ops team and I were fighting to stay up but it never got so bad that it was a choice between complete outage and using the Under Attack mode.

It's terabytes of content and other than we're the host not really related to each other. However, for most projects, it's possible to download a zip file of all the HTML docs for that project. We have a lower rate limit to pull these, but a scraper can pull thousands of docs at once. We only host a few hundred thousand projects so pulling a zip of the latest docs for all of them could be done in a day or two at a very reasonable rate.

It's also possible to request the docs already processed into markdown[1]. Lastly, basically all of the docs come from Git. A smart scraper could just clone a project's repo.

[1] https://docs.readthedocs.com/platform/stable/reference/markd...

I'm the author of the blog. I don't know. Internally, we were half joking that we were going to get ransom notice, but we never did.

The only thing that sort of correlates with this attack is that before it started, we began rolling out some slightly more aggressive rate limits one by one. This was mostly because anytime any new "company" thinks they're going to catchup with Claude/OpenAI, they scrape us very aggressively (and they're not respectful about it). My guess is that the attackers behind this attack were already probing us (they were) and they thought the window of opportunity might be closing.

I use DDG on Android and it works pretty well there. I also saw this line in the article and it's somewhat of a stretch to call DDG either a "major" browser (as you point out) or a "non-Chromium browser". My understanding is it uses the browser engine bundled with the OS on Android/iOS/Mac/Windows and it isn't available for Linux.

The wikipedia article on it says:

> The core browser functionality is the WebView component provided by the operating system. This means the browser engine is Blink on Android and Windows, and WebKit on iOS and macOS.

Nowadays, somebody can just ask claude to build them a scraper/bot that hooks into a proxy network and all of a sudden they can easily send 20k+ reqs/min from hundreds or thousands of IPs cycling them as they get rate limited or banned. In my work, the scrapers have gotten way more aggressive in the last 2 years or so. Frankly, I'm happy there is a solution.

There may be things to criticize Cloudflare for, but the problem of bots and scrapers destroying the open web was getting worse no matter what.

I built EthicalAds (https://www.ethicalads.io/) for exactly this reason.

No tracking. No cookies. No behavioral targeting (targeting based on stuff you've previously done). Every website where our ads appear AND every advertiser is hand approved. No JS from advertisers: just a plain JPG/PNG and text.

We're small but on track to pay out $500k to publishers this year.

They absolutely do. Every sponsorship you see on a podcast or a youtube video or a streamer is a contextual ad. Many open source sponsorships are actually a form of marketing. You could argue that search ads are pretty contextual although there's more at work there. Every ad in a physical magazine is a contextual ad. Physical billboards take into account a lot of geographical context: the ads you see driving in LA are very different than the ones you see in the Bay Area. Ads on platforms like Amazon, HomeDepot, etc. are highly contextual and based on search terms.

Founder of EthicalAds here. In my view, this is only partially true and publishers (sites that show ads) have choices here but their power is dispersed. Advertisers will run advertising as long as it works and they will pay an amount commensurate with how well it works. If a publisher chooses to run ads without tracking, whether that's a network like ours or just buyout-the-site-this-month sponsorships, they have options as long as their audience generates value for advertisers.

That said, we 100% don't land some advertisers when they learn they can't run 3rd party tracking or even 3rd party verification.

My employer, Read the Docs, has a blog on the subject (https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse...) of how we got pounded by these bots to the tune of thousands of dollars. To be fair though, the AI company that hit us the hardest did end up compensating us for our bandwidth bill.

We've done a few things since then:

- We already had very generous rate limiting rules by IP (~4 hits/second sustained) but some of the crawlers used thousands of IPs. Cloudflare has a list that they update of AI crawler bots (https://developers.cloudflare.com/bots/additional-configurat...). We're using this list to block these bots and any new bots that get added to the list.

- We have more aggressive rate limiting rules by ASN on common hosting providers (eg. AWS, GCP, Azure) which also hits a lot of these bots.

- We are considering using the AI crawler list to rate limit by user agent in addition to rate limiting by IP. This will allow well behaved AI crawlers while blocking the badly behaved ones. We aren't against the crawlers generally.

- We now have alert rules that alert us when we get a certain amount of traffic (~50k uncached reqs/min sustained). This is basically always some new bot cranked to the max and usually an AI crawler. We get this ~monthly or so and we just ban them.

Auto-scaling made our infra good enough where we don't even notice big traffic spikes. However, the downside of that is that the AI crawlers were hammering us without causing anything noticeable. Being smart with rate limiting helps a lot.

I'm not the poster you're responding to but I'm one of the founding team of EthicalAds. We're a small team, focused exclusively on marketing to devs, and really trying to show high-quality ads without tracking people (ads are contextually targeted).

You can get a feel for what you'll earn here[1]. Basically you earn 70% of the gross of what we charge advertisers (see advertiser pricing[2]). Keep in mind these are ad views which aren't quite the same as pageviews. They're a subset.

Ads are a straight-forward path to monetization but not always the best. If you can make a project work as SaaS or really make sponsorships work for you (this requires effort), those will definitely earn A LOT more money per pageview than ads. Ads require a lot of traffic to make them work well. Usually you want high tens to hundreds of thousands of pageviews per month.

From a what we look at for publishers (sites that show ads) perspective, we're usually looking for high quality dev-focused sites or projects that don't want to just show Google ads. Per ad, publishers will earn much more with us than Google display ads but if you want to stick 4-5 Google ads on your site, video ads, or the like we can't compete with that and we don't want our ads on sites that do that. Devs hate them. My email is in my bio if you want to discuss further.

Regardless, good luck on the projects!

[1]: https://www.ethicalads.io/publishers/calculator/ [2]: https://www.ethicalads.io/advertisers/pricing/

My employer, Read the Docs is a heavy user of Cloudflare. It's actually hard to imagine serving as much traffic as we do as cheaply as we can without them.

That said, for publicly hosted open source documentation, we turn down the security settings almost all the way. Security level is set to "essentially off" (that's the actual setting name), no browser integrity check, TOR friendly (onion routing on), etc. We still have rate limits in place but they're pretty generous (~4 req/s sustained). For sites that don't require a login and don't accept inbound leads or something like that, that's probably around the right level. Our domains where doc authors manage their docs have higher security settings.

That said, being too generous can get you into trouble so I understand why people crank up the settings and just block some legitimate traffic. See our past post where AI scrapers scraped almost 100TB (https://news.ycombinator.com/item?id=41072549).

There's a few ways first party cookies can track you. Probably the biggest single way is Google Analytics which by default uses only first party cookies. Even without cookies at all, GA could track you across the web although first party cookies do make this a little easier and "better". However, first party cookies can help trackers in other ways like for CNAME cloaking[1] which basically makes a first-party cookie function similarly to a third-party one.

Disclosure: I work for a small privacy focused ad company.

[1] https://webkit.org/blog/11338/cname-cloaking-and-bounce-trac...

I edited to make my comment more clear but polyfill.io sends dynamic polyfills based on what features the identified browser needs. Since it changes, the SRI hash would need to change so that part won't work.

SRI generally won't work here because the served polyfill JS (and therefore the SRI hash) depends on the user agent/headers sent by the user's browser. If the browser says it's ancient, the resulting polyfill will fill in a bunch of missing JS modules and be a lot of JS. If the browser identifies as modern, it should return nothing at all.

Edit: In summary, SRI won't work with a dynamic polyfill which is part of the point of polyfill.io. You could serve a static polyfill but that defeats some of the advantages of this service. With that said, this whole thread is about what can happen with untrusted third parties so...

Keychain Access has not been "fine". It's had multiple unaddressed data loss bugs. For example, Keychain lost all passwords from all Keychains after the Catalina update[1] and this wasn't fixed in the next 3 Catalina minor updates. Multiple users reported the issue to Apple and the response was crickets. Even if you restored the passwords, it helpfully deleted them all again. I switched to 1Password and declared Keychain Access a lost cause. I don't think I'll be giving them a second chance here.

[1] https://discussions.apple.com/thread/250722178

I can't confirm if the price is the same or not, but their terms[1] specifically mention that the price can be different.

> Prices for Offerings you order for delivery or pickup through the Online Grocery Ordering Service may be higher than the prices for such Offerings in our physical stores.

Most outlets I've seen (eg. Target) are the same in that they just list a higher price on the website than it costs in-store and they're upfront about this. It takes me 20-25 minutes of in-store picking to shop including checkout for my weekly groceries. Even if that is done by a minimum wage worker (~$17/hr here) that's ~$6-8 of service on top of them bringing it out to the curb. In addition, eggs are usually specially packed in their own bag (frequently with a sticker on the bag labeled "eggs") when they're bought online and curbsided. It seems a bit naive to me to think that all this service would just be included/free.

[1] https://www.albertsonscompanies.com/policies-and-disclosures... (linked from safeway's footer)

While I agree with you to an extent, this is not a very good way to check actual in-store prices. Most grocery stores charge a different rate for products that are delivered or even curbsided than they do if you go to the store and buy it yourself. This is true even if you go directly vs. going through an intermediary like DoorDash.

I live in coastal CA and the cheapest eggs at my local market are $3.49/dozen. Trader Joes and Costco are closer to $2.20/dozen if you just want plain white eggs. The moment you go organic, the cheapest is Costco at about $4/dozen.

Edit: Just to be clear, at Costco you buy 2 dozen rather than 1 and I've divided the price to a single dozen.

I'd just like effective privacy laws in the US generally.