Seems like attacking Google would be a bad target for street cred as compared to govt websites.

Surely bringing down Google is a bigger technical achievement than some random government website maintained by someone who stumbled into their job after 20 years doing mid level government organizational work.

Heck, I'd imagine that making headlines and having Google benchmark your attack would bring some amazing street cred.

Darknet guerilla marketing. Definitely seems to have worked.

Now we need the SEO content side: "How we hit Google with 398M RPS".

"... you can do this manually, but our product makes it as easy as a sign up and API call. Talk to us about pricing. [Python API example].

Yes, but they are clearly going to fail to bring down Google.

It doesn't matter if you fail. The cred comes from how much bandwidth and resources you can soak up.

Right, so clearly the ability to bring down Google is not the point.

Well - they clearly were successful enough to get a thread on hacker news……

Aim high and go out with a bang choom

Nah it's even better because they're considered capable defenders so it's harder.

What I'm not sure of is why Google published this. I can't figure out what their strategy is here. We never published about the attacks we absorbed because we didn't want them to know our capabilities.

Unless this is marketing for Google Cloud?

This is certainly marketing. If they sell DDOS protection, then announcing that they stopped the largest attack ever is an ad.

Maybe Google is responsible for the attack, to be able to publish this blog post! <\tinfoil-hat>

If Google truly went rouge, they could turn all those Chrome installs and Android devices into one gargantuan botnet.

Google is already partly rouge, at least in their logo.

Sounds like a symbiotic relationship to me. The attackers get to advertise their capability for pulling off attacks, and Google gets to advertise their ability to stop them.

Almost all (but not all) of these attacks are based on some kind of problem that leads to amplification. Advertizing that people should fix these points of exploit help everyone on the internet.

False flag? :)

> Unless this is marketing for Google Cloud?

If you read the article, there are plenty of marketing remarks in there to get you to use Google Cloud

CDN is the ultimate solution for DDoS, so any report about DDoS finally become an ad for CDN

> Unless this is marketing for Google Cloud?

That seems likely here if they're claiming this is the largest DDOS ever.

> why Google published this

Besides publicity, there is also link to a list of advisories that may be of interest to other cloud operators and users.

https://nvd.nist.gov/vuln/detail/CVE-2023-44487

> We never published...

We? Netflix or Reddit? I know for a fact that Amazon doesn't.

Nowhere that I've ever worked published about attacks. We didn't want to validate the attackers.

At eBay/PayPal we filed patents on our DDOS shield, since it was as far as we knew the first one to exist, but that was about the only public information on it.

At reddit and Netflix we didn't actually have to deal with it because AWS just absorbed (or mitigated) it before it ever hit us. We only had to deal with L7 attacks, which we had shields in place for.

What capabilities did this post reveal the existence of? Not many, beyond it having been mitigated somehow and that it didn't cause an outage. The attackers knew that already, because they'd obviously be able to observe the system during the attack.

As for why to write about it, it's a new type of attack that resulted in almost an order of magnitude increase in attack size. That's interesting and newsworthy by itself, and publishing a concrete number gives people an idea of the size of the problem and the trendlines.

This is also something that needed a CVE, so it was going to be very public anyway. If nothing is written about it, at a minimum Cloud customers will be flooding their support reps with questions about whether the vulnerability applies to them.

They're not attacking Google, per se. Just the Google Cloud platform that hosts govt sites, Discord channels, gaming servers, etc.

Very useful, thanks. Do you know roughly what sort of resources, in time, money, and compromised machines, it takes to do something like this? (Order of magnitude.)

approx. 20,000 machines https://news.ycombinator.com/item?id=37831355

So a single machine can do ~ 20,000 rps?

Depends, but there seems to be a multiplier effect at play with this attack. A single client request may result in 100x the work for the server. More details here: https://cloud.google.com/blog/products/identity-security/how...

Yep: "a client can send a RST_STREAM frame for a single stream. This instructs the server to stop processing the request and to abort the response, which frees up server resources and avoids wasting bandwidth."

Pretty clever

> Another advantage the attacker gains is that the explicit cancellation of requests immediately after creation means that a reverse proxy server won't send a response to any of the requests. Canceling the requests before a response is written reduces downlink (server/proxy to attacker) bandwidth.

How is this an advantage? Can someone explain please?

The attacking system is shooting a firehose of requests at the target system, but doesn't have to deal with handling any responses being sent back to the requesting systems.

Makes sense, thank you!

It's an advantage because you as a botnet client have made the server side do extra work. You sent two packets, one to request a new connection, and a second to immediately cancel the request. The server on the other hand sees a connection request and does some work like allocating memory and fetching the resource you requested. Once the server starts sending the response back to the client via the reverse proxy, the reverse proxy notices the request is no longer current and just drops the response on the floor. As a result, you made the server do some amount of work and you don't have to worry about saturating your internet connection. They call this a magnification attack because for the cost of two requests you made the server do some multiple of work.

You could add some smarts to the server or reverse proxy that delays starting work in case a cancellation request quickly arrives. This is probably part of the mitigation work they refer to in the article.

This is sort of an aside based on something I read in the article but does anyone know why the RFC guidelines say that you should first send an informational GOAWAY that does not prevent opening new streams when gracefully closing a connection?

They point out in the article that it's a better practice to immediately limit stream creation when you detect abuse - not wait for a round trip to complete first. I'm sure there's a good reason for the original guidelines; I'm just trying to get it and haven't found anything clarifying through Google. Was it specified before the rise of modern attacks?

How do you act on a GOAWAY you haven't received and allowing the server to unilaterally stop supporting things can lead to weird edge cases.

After all you could say any client ignoring a GOAWAY is either bugged or malicious but certainly not until you get confirmation they go it.

Of course. Depending on the machine and on the network, a single modern machine can do even up to a few millions RPS. This is routinely used in benchmarking tools.

Here with the "attack", it's simply exploiting the ability of HTTP/2 to compress requests and reduce them to just a few bytes, meaning that within a few kilobytes of data you can easily have hundreds of requests. Again this is not new and was already being discussed in 2012 about SPDY's use of zlib to compress requests.

The extra stuff that seems to have made this attack "new" for such service providers is that attackers took care of closing their requests so as not to have to wait for a response and be able to fill the wire with a flow of request. Again this has been known from the inception of HTTP/2 and routinely met by those dealing with proxies which timeout and send headers followed by rst_stream.

Here it makes noise because new records were broken, and likely because the stacks in place were not properly designed so they omitted to check for the real number of streams and only focused on the protocol validity...

If this attack takes only 20,000 machines to orchestrate, how is it the "largest DDoS attack to date"? It was to my understanding that some botnets and organizations have placed far more than 20,000 machines under their control before or currently. Could you explain a bit better?

I think they're measuring "largest" by the number of requests per second.

But that's part of my question. If other botnets have previously been created with many more than 20,000 controlled machines, why has nobody else before orchestrated an attack with so many requests if it only requires 20,000 machines or so to pull off?

This is exploiting a new technique. This technique wasn't known previously, or wasn't viable when HTTP/2 wasn't widespread. This technique is a ~100x multiplier on the impact any size botnet can have. A previous botnet might have needed 20,000,000 machines to achieve the same impact.

Thanks for explaining in such a straightforward way, appreciated!

They compromise home internet users and/or their IOT devices mostly with scripts and malware. So the investment for the scammer is mostly in researching exploits and seeding their malware. Most of them just use exploits created by others, but the best ones with the biggest networks are actually very capable security researchers. Given different circumstances they could probably be highly paid engineers.

Some of their effort goes toward maintaining an exclusive hold on their botnet too. Patching them while maintaining control or blocking the vulnerability they used from being utilized by others.

>> Most of the people who do this have a lot of technical skill but not a lot of opportunity to get paid for it based on where they live or the circumstances of their upbringing.

LOL. No there are plenty of legitimate enterprises as well as opportunity to immigrate. Especially in tech. These guys are just criminals.

Have you tried that yourself? Especially as someone who has the skills but doesn't speak the language.

I know people who can't relocate because of communication issues and/or cultural differences.

No they aren't criminals, but they are definitely underpaid compared to those who managed to relocate.

If they operate botnets, I think it's fair to call them criminals

probably, or if they aren’t directly criminals they’re probably facilitating criminals. but if you were very particular about it you could in theory set up in a country which doesn’t have treaties with any of the countries in which the victims operate.

It is only criminal if the botnets are used to steal something. DDoS-in just for fun is at most an annoyance.

Yeah, hospitals can't stand that kind of thing...

They should have better IT.

Blaming it on the people that knock them off will not make improve the situation.

So your logic is: condemning people for criminal behavior is not useful because it de-incentivizes their victims from being vigilant against that criminal behavior. Am I getting that right?

No. My logic is that when it comes to computers, it's a battle of the brains: who is smarter than the other one. And the result is usually a novel way of attacking someone, and then finding novel ways to defend and on and on. So that's why I strongly believe in hiring smarter people to defend your castle, rather than swatting people smarter than you because you're not as smart as the other side.

Not to condone the DDoS activities in the least, but that's just ignorance. Which prosperous country accepts evrn remotely as many legal immigrants as apply / would want to move there? And a lot of people / political parties are constantly lobbying for less immigration :-/

Wow, I guess sometimes the real world is as cool as the movies

Not cool at all, just a waste of talent.

They should work to increase ad revenue by subliminal messaging and lootboxization instead!

i feel like this is 99% of capitalism

Crypto companies attacking eachothers' websites?! Color me surprised …