I've been working on anti-DDOS off and on for 20 years now. The answer is sometimes government actors, but oftentimes scammers in Eastern Europe. They do these big attacks for street cred amongst the botting community.

They then use their street cred to get paid by less scrupulous actors to attack their rivals. Sometimes the people paying are governments, sometimes just shady companies. For example last year there was a lot of crypto companies attacking each other's websites.

Most of the people who do this have a lot of technical skill but not a lot of opportunity to get paid for it based on where they live or the circumstances of their upbringing.

Seems like attacking Google would be a bad target for street cred as compared to govt websites.

Surely bringing down Google is a bigger technical achievement than some random government website maintained by someone who stumbled into their job after 20 years doing mid level government organizational work.

Heck, I'd imagine that making headlines and having Google benchmark your attack would bring some amazing street cred.

Darknet guerilla marketing. Definitely seems to have worked.

Now we need the SEO content side: "How we hit Google with 398M RPS".

"... you can do this manually, but our product makes it as easy as a sign up and API call. Talk to us about pricing. [Python API example].

Yes, but they are clearly going to fail to bring down Google.

It doesn't matter if you fail. The cred comes from how much bandwidth and resources you can soak up.

Right, so clearly the ability to bring down Google is not the point.

Well - they clearly were successful enough to get a thread on hacker news……

Aim high and go out with a bang choom

Nah it's even better because they're considered capable defenders so it's harder.

What I'm not sure of is why Google published this. I can't figure out what their strategy is here. We never published about the attacks we absorbed because we didn't want them to know our capabilities.

Unless this is marketing for Google Cloud?

This is certainly marketing. If they sell DDOS protection, then announcing that they stopped the largest attack ever is an ad.

Maybe Google is responsible for the attack, to be able to publish this blog post! <\tinfoil-hat>

If Google truly went rouge, they could turn all those Chrome installs and Android devices into one gargantuan botnet.

Google is already partly rouge, at least in their logo.

Sounds like a symbiotic relationship to me. The attackers get to advertise their capability for pulling off attacks, and Google gets to advertise their ability to stop them.

Almost all (but not all) of these attacks are based on some kind of problem that leads to amplification. Advertizing that people should fix these points of exploit help everyone on the internet.

False flag? :)

> Unless this is marketing for Google Cloud?

If you read the article, there are plenty of marketing remarks in there to get you to use Google Cloud

CDN is the ultimate solution for DDoS, so any report about DDoS finally become an ad for CDN

> Unless this is marketing for Google Cloud?

That seems likely here if they're claiming this is the largest DDOS ever.

> why Google published this

Besides publicity, there is also link to a list of advisories that may be of interest to other cloud operators and users.

https://nvd.nist.gov/vuln/detail/CVE-2023-44487

> We never published...

We? Netflix or Reddit? I know for a fact that Amazon doesn't.

Nowhere that I've ever worked published about attacks. We didn't want to validate the attackers.

At eBay/PayPal we filed patents on our DDOS shield, since it was as far as we knew the first one to exist, but that was about the only public information on it.

At reddit and Netflix we didn't actually have to deal with it because AWS just absorbed (or mitigated) it before it ever hit us. We only had to deal with L7 attacks, which we had shields in place for.

What capabilities did this post reveal the existence of? Not many, beyond it having been mitigated somehow and that it didn't cause an outage. The attackers knew that already, because they'd obviously be able to observe the system during the attack.

As for why to write about it, it's a new type of attack that resulted in almost an order of magnitude increase in attack size. That's interesting and newsworthy by itself, and publishing a concrete number gives people an idea of the size of the problem and the trendlines.

This is also something that needed a CVE, so it was going to be very public anyway. If nothing is written about it, at a minimum Cloud customers will be flooding their support reps with questions about whether the vulnerability applies to them.

They're not attacking Google, per se. Just the Google Cloud platform that hosts govt sites, Discord channels, gaming servers, etc.

Very useful, thanks. Do you know roughly what sort of resources, in time, money, and compromised machines, it takes to do something like this? (Order of magnitude.)

approx. 20,000 machines https://news.ycombinator.com/item?id=37831355

So a single machine can do ~ 20,000 rps?

Depends, but there seems to be a multiplier effect at play with this attack. A single client request may result in 100x the work for the server. More details here: https://cloud.google.com/blog/products/identity-security/how...

Yep: "a client can send a RST_STREAM frame for a single stream. This instructs the server to stop processing the request and to abort the response, which frees up server resources and avoids wasting bandwidth."

Pretty clever

> Another advantage the attacker gains is that the explicit cancellation of requests immediately after creation means that a reverse proxy server won't send a response to any of the requests. Canceling the requests before a response is written reduces downlink (server/proxy to attacker) bandwidth.

How is this an advantage? Can someone explain please?

The attacking system is shooting a firehose of requests at the target system, but doesn't have to deal with handling any responses being sent back to the requesting systems.

Makes sense, thank you!

It's an advantage because you as a botnet client have made the server side do extra work. You sent two packets, one to request a new connection, and a second to immediately cancel the request. The server on the other hand sees a connection request and does some work like allocating memory and fetching the resource you requested. Once the server starts sending the response back to the client via the reverse proxy, the reverse proxy notices the request is no longer current and just drops the response on the floor. As a result, you made the server do some amount of work and you don't have to worry about saturating your internet connection. They call this a magnification attack because for the cost of two requests you made the server do some multiple of work.

You could add some smarts to the server or reverse proxy that delays starting work in case a cancellation request quickly arrives. This is probably part of the mitigation work they refer to in the article.

This is sort of an aside based on something I read in the article but does anyone know why the RFC guidelines say that you should first send an informational GOAWAY that does not prevent opening new streams when gracefully closing a connection?

They point out in the article that it's a better practice to immediately limit stream creation when you detect abuse - not wait for a round trip to complete first. I'm sure there's a good reason for the original guidelines; I'm just trying to get it and haven't found anything clarifying through Google. Was it specified before the rise of modern attacks?

How do you act on a GOAWAY you haven't received and allowing the server to unilaterally stop supporting things can lead to weird edge cases.

After all you could say any client ignoring a GOAWAY is either bugged or malicious but certainly not until you get confirmation they go it.

Of course. Depending on the machine and on the network, a single modern machine can do even up to a few millions RPS. This is routinely used in benchmarking tools.

Here with the "attack", it's simply exploiting the ability of HTTP/2 to compress requests and reduce them to just a few bytes, meaning that within a few kilobytes of data you can easily have hundreds of requests. Again this is not new and was already being discussed in 2012 about SPDY's use of zlib to compress requests.

The extra stuff that seems to have made this attack "new" for such service providers is that attackers took care of closing their requests so as not to have to wait for a response and be able to fill the wire with a flow of request. Again this has been known from the inception of HTTP/2 and routinely met by those dealing with proxies which timeout and send headers followed by rst_stream.

Here it makes noise because new records were broken, and likely because the stacks in place were not properly designed so they omitted to check for the real number of streams and only focused on the protocol validity...

If this attack takes only 20,000 machines to orchestrate, how is it the "largest DDoS attack to date"? It was to my understanding that some botnets and organizations have placed far more than 20,000 machines under their control before or currently. Could you explain a bit better?

I think they're measuring "largest" by the number of requests per second.

But that's part of my question. If other botnets have previously been created with many more than 20,000 controlled machines, why has nobody else before orchestrated an attack with so many requests if it only requires 20,000 machines or so to pull off?

This is exploiting a new technique. This technique wasn't known previously, or wasn't viable when HTTP/2 wasn't widespread. This technique is a ~100x multiplier on the impact any size botnet can have. A previous botnet might have needed 20,000,000 machines to achieve the same impact.

Thanks for explaining in such a straightforward way, appreciated!

They compromise home internet users and/or their IOT devices mostly with scripts and malware. So the investment for the scammer is mostly in researching exploits and seeding their malware. Most of them just use exploits created by others, but the best ones with the biggest networks are actually very capable security researchers. Given different circumstances they could probably be highly paid engineers.

Some of their effort goes toward maintaining an exclusive hold on their botnet too. Patching them while maintaining control or blocking the vulnerability they used from being utilized by others.

>> Most of the people who do this have a lot of technical skill but not a lot of opportunity to get paid for it based on where they live or the circumstances of their upbringing.

LOL. No there are plenty of legitimate enterprises as well as opportunity to immigrate. Especially in tech. These guys are just criminals.

Have you tried that yourself? Especially as someone who has the skills but doesn't speak the language.

I know people who can't relocate because of communication issues and/or cultural differences.

No they aren't criminals, but they are definitely underpaid compared to those who managed to relocate.

If they operate botnets, I think it's fair to call them criminals

probably, or if they aren’t directly criminals they’re probably facilitating criminals. but if you were very particular about it you could in theory set up in a country which doesn’t have treaties with any of the countries in which the victims operate.

It is only criminal if the botnets are used to steal something. DDoS-in just for fun is at most an annoyance.

Yeah, hospitals can't stand that kind of thing...

They should have better IT.

Blaming it on the people that knock them off will not make improve the situation.

So your logic is: condemning people for criminal behavior is not useful because it de-incentivizes their victims from being vigilant against that criminal behavior. Am I getting that right?

No. My logic is that when it comes to computers, it's a battle of the brains: who is smarter than the other one. And the result is usually a novel way of attacking someone, and then finding novel ways to defend and on and on. So that's why I strongly believe in hiring smarter people to defend your castle, rather than swatting people smarter than you because you're not as smart as the other side.

Not to condone the DDoS activities in the least, but that's just ignorance. Which prosperous country accepts evrn remotely as many legal immigrants as apply / would want to move there? And a lot of people / political parties are constantly lobbying for less immigration :-/

Wow, I guess sometimes the real world is as cool as the movies

Not cool at all, just a waste of talent.

They should work to increase ad revenue by subliminal messaging and lootboxization instead!

i feel like this is 99% of capitalism

Crypto companies attacking eachothers' websites?! Color me surprised …

You don't need a lot of money or resources to pull one of these. Code is on Github: https://github.com/649/Memcrashed-DDoS-Exploit

Also the participants are sometimes innocently recruited victims for the attack. I blame app insecure defaults.

The trend since 2015 is to get worst as you will see in the bottom layer of this graph: https://www.digitalattackmap.com/

This is a novel ddos attack. Did you all even read the article?

I did. I replied to OP question.And that was about DDos attacks in general not "HTTP/2 Rapid Reset attacks"

> Who has an incentive to carry out these DDos attacks?

Did you read the comment I was replying to?

Yes, “these attacks” referring to the sophisticated novel attacks under discussion in the article. No need to be defensive, just read it next time.

Are there DDoS attacks that are not sophisticated in form or execution? :-)

ping

PR. Attack Google or cloudflare. Wait for them to publish a blog post about the biggest attack ever seen, then tell potential customers of your botnet that you can launch a bigger attack than anyone else and point to the above blog post.

The botnet is probably the critical thing. Even if the PR (or "avenge the global south", or whatever) value might not be enormous, the cost to a bad actor of having other peoples' computers do something is almost negligible.

Doesn't using your botnet expose your botnet IP addresses/devices?

Yes, but currently that has zero consequences. Say you infect 500.000 Windows XP machines or consumer routers, the owners of those devices isn't going to be informed, nor is their ISPs. In many cases the manufacturer of those devices also aren't going to provide security update, but those probably wasn't going to be applied anyway.

Are you positive that "tell nobody" is the mitigation strategy that Google used here? They could have easily asked router vendors to patch their devices, asked ISPs to blackhole those customers until they're patched, etc.

Patch what though? They know that they're getting hit with unprecedented traffic, not how those computers were infected.

It's mostly not infected computers, but rather poorly configured proxies that are open for anyone to bounce malicious traffic through. Convincing everyone to clean up their open proxies is a long-term, hard problem. But I plan to tackle it soon....

How? I suppose the most effective way is to have those proxies attack each other. But don’t, it’s likely illegal.

the most efficient way would be to write a script that gains root on those open proxies and then fixes the issue.

Effective or efficient? Would seem rather inefficient to spend time researching all the possible ways to gain route on x number of servers, finding an exploit, crafting some plan to execute it, keeping your prints clean etc etc

What way would be more efficient?

Get a few companies to agree that open proxies are a scourge that needs to be stopped. They each apply some action to open proxies (user-facing messaging, loss of functionality, captcha, or complete block), and the users of those proxies will get the problem fixed.

The hard part (and it truly is hard!) is convincing a few companies to do this. It risks user complaints in the short term, to solve a problem that may not be very acute for the largest companies (who can simply absorb these attacks).

[delayed]

So you're saying Google and Cloudflare, just as an example, should block consumers of other ISPs because they run "unpatched" software or they have malware running on their devices? Lol, this is a very absurd and narrow minded view how the internet works. You deal with the traffic, you don't randomly block eyeball networks because they're attacking you.

> you don't randomly block eyeball networks because they're attacking you.

ISPs do this literally all the time. They sell services that do this.

> the owners of those devices isn't going to be informed, nor is their ISPs

not necessarily true

But these ISPs that give something and inform and even isolate their infected customers are few and far between.

Shout out to Dutch ISP XS4ALL who was (is?) very very strict and active in this space.

Google should start using their ad network to silently update people’s security!

Uh, no thanks from this user.

Also, sounds illegal.

Definitely illegal in the US.

Anyone can claim that, there's no link to a specific actor

I'm guessing you would do this in advance - "pay attention to tech news next week - our botnet will unleash hell"

Step 1: Put message on blockchain beforehand with exact date/time and characteristics of DDoS

Step 2: Execute DDoS

Step 3: Prove to others you are responsible by using private key

There’s (as ever) no need for a blockchain, people do this with twitter and sha256 all the time. Hash the message, post the hash, wait for prediction to happen, post the full message.

Any examples of major predictions verified in this way?

And Google and Cloudflare also get good PR because how insanely good their are at deflecting those huge attacks. It's a win-win situation here... oh wait /s

(the /s is just on the "oh wait" part, not the whole post)

Why not attack a target that can actually be harmed? Are they afraid?

It's not obvious what's the value of having the largest ineffective attack.

Why not roll a couple defenseless grannies in the streets for pocket change, rather than throw rocks at the cops and then get away unscathed?

One gets you more money in the short term. The other one gets you more street cred - which gets you more money in the long term.

Well in this case it seems like they blew their "0-day" and Google worked with other providers to patch this type of attack.

If they're unsophisticated, it's for clout and "street cred" in hacking communities, no different than tagging a freeway overpass with graffiti.

If they're advanced, they are doing it to test capabilities and responses. The Taliban used to pay kids to light off firecrackers outside base to check defensive TTPs. It also had the effect of desensitizing the sound of gunfire.

Really good adversaries know how to accomplish the latter while appearing as the former.

Or... It's for fun. Are the 90s/00s that long ago? Not everything is about money and terrorism.

Bring back CotDC, MoD, even lulzsec. I miss the days of the internet being the open seas and everyone having their own fun on it, user beware.

For a recent and similar attack at scale, the authors of the botnet software were from an American security company who sold DDOS mitigation solutions (https://en.wikipedia.org/wiki/Mirai_(malware)).

Google keeps showing me a CAPTCHA and telling me I'm part of the Mirai botnet. I'm using Cloudflare WARP so I have an IP from Cloudflare.

My gut instinct is that this is a nation-state initiated.

For example a certain group decides to short on a share. The DDOS the company and "leak" it to the press. The bad press negatively impacts the share price. At least this was the way some time ago when I had to deal with such attacks.

for the lolz

I've never contemplated the cost of a DDos attack, I guess there's the upfront setup costs to secure the software and hardware that will execute the attack but are you speaking more about the costs on the day of the attack? Are those costs trivial, like the marginal costs I suppose it would be?

To be effective, you need to either be prepared to hide behind google, Cloudflare or AWS, OR you need some pretty expensive deal with you (large) ISP who can (quickly) filter on their edge.

Sitting at the end of whatever network, you will not be able to do anything against a sufficient volume attack.

You can (or could, my information is old) pay botnet owners a few hundred bucks to disrupt the servers of people you don't like. An example would be ruining a match for a competing game clan. There's a suprising amount of this kind of petty bullshit going on in the world.

With the Mirai botnet, some of the creators had a DDOS mitigation company as well: they'd sell one party the weapon, and sell another party the defense against that weapon.

Sometimes it's for the street cred, or the lulz, or just the challenge of building a botnet.

Disruption is part of it for sure, but often big, aggressive DDoS come hand-in-hand with other attacks.

Seen it happen with big DDoS on clients. Furaffinity, one of the larger furry webistes, and a constant drama magnet, was a client at a former job. They got DDoS'd hard, and in between scripted DDoS hits they slammed the hell out of their web applications to get vulns and do credential stuffing.

As in blast em, lighten it up just enough to get a ssh or nmap through a few times, blast em again, and repeat until they got in.

Is also why you want out-of-band solution that doesn't touch your infra much.

> in between scripted DDoS hits they slammed the hell out of their web applications

I've heard this before, but I still don't understand what purpose the DDoS serves here. Distraction, so the actual attack drowns in the noise?

Google?

If you analyze the situation from the perspective of "Who benefits from it?", then the answer is clearly: Google benefits from it (they are so good, they can mitigate gigantic DDoS attacks). So, I don't think it's that crazy to think this is all a publicity stunt .

I was thinking the same thing. Especially when the the bottom of the article is basically a sales pitch for their own services. Whilst what they are saying is true and correct even if they aren't behind it (which I really doubt that they are the culprit).

"With or without patches, organizations would need to make significant infrastructure investments to keep services running in the face of attacks of any moderate size and larger. Instead of bearing that expense themselves, organizations running services on Google Cloud can take advantage of our investment in capacity at global scale in our Cross-Cloud Network to deliver and protect their applications."

If I had the technical prowess to do this, I would do it just for the fun of it. I mean, why not? Anarchy is fun.

I'm pretty sure someone will find a way to take down GOOG/AWS/Azure/etc through a DDoS so large nothing will work for anyone.

Why do you think Finland or Spain might attack USA companies?

I don’t think either of those countries would attack US companies. Obviously I would suspect adversaries instead of allies

The USA sees all countries, even allies, as adversaries, and always has ... hence the deep paranoia the USA has, lack of trust, lies, coercion, and spying (the list is longer but time is limited).

Okay. In either case I doubt Spain or Finland is DDosing Google

I mean you can ask them https://t.me/s/noname05716eng chat of NoName group skids doing some DDOSing, they pool their bots by community of like-minded people (Russians supporting current government, murder rape etc. you know)

I'm just guessing here but it could easily be stock market manipulation

DDOS protection is a billion dollar a year industry.

few hours of inconvenience in corporate cloud infrastructure has a result also - large amounts of money lost. So most obvious incentive is simply ransom