News for hackers. Stuff that matters.
Re:Cloudflare Thanksgiving 2023 Security Incident by barkingcat on Thursday February 01, 2024 @04:08PM (#39221487)
I think they did have to do that far though.

Getting in at the "ground floor" of a new datacentre build is pretty much the ultimate exploit. Imagine getting in at the centre of a new Meet-Me room (https://en.wikipedia.org/wiki/Meet-me_room) and having persistent access to key switches there.

Cloudflare datacentres tend to be at the hub of insane amounts of data traffic. The fact that the attacker knew how valuable a "pre-production" data centre is means that cloudflare probably realized themselves that it would be a 100% game over if someone managed to get a foot hold there before the regular security systems are set up. It would be a company ending event if someone managed to install themselves inside a data centre while it was being built/brought up.

Also remember, at the beginning of data centre builds, all switches/equipment have default / blank root passwords (admin/admin), and all switch/equipment firmware are old and full of exploits (you either go into each one and update the firmware one by one or hook them up to automation for fleet wide patching) Imagine that this exploit is taking place before automation services had a chance to patch all the firmware ... that's a "return all devices to make sure the manufacturer ships us something new" event.

Re:Cloudflare Thanksgiving 2023 Security Incident by vasco on Thursday February 01, 2024 @04:12PM (#39221543)
What I think they meant is customers would keep paying them. And they are right, one just has to look at Okta, Solarwinds and other providers that have been owned, not done half of this and somehow are still in business. Everyone whistles to the side and pretends they shouldn't switch vendors, rotate all creds, cycle hardware, because it saves lots of work and this stuff falls under "reasonable oopsie" to the general public, when in fact there should be rules about what to do in the event of a breach that should be much stricter. So they do some partial actions to "show work" in case of lawsuits and keep going. The old engineers leave, new ones come in, and now you have systems who are potentially owned for years to come.

It takes some honesty and good values by someone in the decision-making to go ahead with such a comprehensive plan. This is sad because it should be tablestakes, as you say correctly, but having seen many other cases, I think although they did "the expected", it's definitely above and beyond what peers have done.

Re:Cloudflare Thanksgiving 2023 Security Incident by swyx on Thursday February 01, 2024 @04:32PM (#39221745)
do manufacturers share some of the cost of this kind of security related return or is this a straight up "pay twice for the same thing" financial hit?
Re:Cloudflare Thanksgiving 2023 Security Incident by eastdakota on Thursday February 01, 2024 @04:37PM (#39221821)
We have very good relations with our network vendors (in this case, Cisco, Juniper, and Arista). The CEOs of all of them 1) immediately got on a call with me late on a weekend; 2) happily RMAed the boxes at no cost; and 3) lent us their most senior forensics engineers to help with our investigation. Hat tip to all of them for first class customer service.
Re:Cloudflare Thanksgiving 2023 Security Incident by swyx on Thursday February 01, 2024 @04:46PM (#39221901)
shows how much they value you as a partner and i'm sure they appreciate your overall business.

thanks Matthew! love the transparency and dedication to security as always. really sucks to have this be continuing fallout from Okta's breach. wish large scale key rotation was more easily automatable (or at least as a fallback, there should be a way to track key age on clientside? so that old keys stick out like a sore thumb). i guess in the absence of industry standard key rotation apis someday you might be able to "throw AI at it".

Re:Cloudflare Thanksgiving 2023 Security Incident by tgsovlerkhgsel on Thursday February 01, 2024 @04:34PM (#39221781)
> It would be a company ending event if someone managed to install themselves inside a data centre while it was being built/brought up.

It wouldn't. Most people like to assume the impact of breaches to be what it should be, not what it actually is.

Look at the 1-year stock chart of Okta and, without looking up the actual date, tell me when the breach happened/was disclosed.

Re:Cloudflare Thanksgiving 2023 Security Incident by mschuster91 on Thursday February 01, 2024 @05:42PM (#39222508)
> Look at the 1-year stock chart of Okta and, without looking up the actual date, tell me when the breach happened/was disclosed.

The problem with this is that while security minded people know what Okta is and why to stay the fuck away from handing over your crown jewels to a SaaS company is warranted, C-level execs don't care. They only care about their golf course or backroom deal friends and about releasing PR statements full of buzzwords like "zero trust", "AI based monitoring" and whatever.

The stock markets don't care either, they only look at the financial data, and as long as there still are enough gullible fools signing up, they don't care and stonk goes up.

Re:Cloudflare Thanksgiving 2023 Security Incident by cqqxo4zV46cp on Thursday February 01, 2024 @06:35PM (#39223052)
Yes, that’s literally the point being made. The point is that it isn’t a company-ending event. You are going on an unrelated rant about how those darn dumb executives aren’t as smart as God’s gift to earth, engineers.
Re:Cloudflare Thanksgiving 2023 Security Incident by mschuster91 on Thursday February 01, 2024 @07:18PM (#39223384)
The thing is, some events should be company ending. Something like Okta shouldn't even exist in a halfway competent world in the first place - given how many Fortune 500 companies, even governments use it, it's just a too fucking juicy target for nation states both friendly and hostile.

Instead, even the "self correcting" mechanisms of the "free market" obviously didn't work out, as the free market doesn't value technical merit, it only values financial bullshittery.

And the end result will be that once the war with China or Russia inevitably breaks out, virtually all major Western companies and governments will be out cold for weeks once Okta and Azure's AD go down, because that is where any adversary will hit first to deal immense damage.

Re:Cloudflare Thanksgiving 2023 Security Incident by nolok on Thursday February 01, 2024 @04:47PM (#39221911)
> It would be a company ending event

Given they got out of cloudbleed without any real damage let alone lasting damage, I disagree.

(I don't disagree with your point about how bad of a problem this would be, I'm just insisting that security failure is not taken seriously at all by anyone)

Re:Cloudflare Thanksgiving 2023 Security Incident by chx on Thursday February 01, 2024 @05:20PM (#39222253)
Presuming taviso is not exaggerating and why would he CF's reply to cloudbleed was ... not quite nice.

https://twitter.com/taviso/status/1566077115992133634

> True story: After cloudbleed, cloudflare literally lobbied the FTC to investigate me and question the legality of openly discussing security research. How come they're not lobbying their DC friends to investigate the legality KF?

For those not familiar with the history , this tweet started the cloudbleed disclosure to cloudflare:

https://twitter.com/taviso/status/832744397800214528

> Could someone from cloudflare security urgently contact me.

This followed: https://blog.cloudflare.com/incident-report-on-memory-leak-c...

Re:Cloudflare Thanksgiving 2023 Security Incident by hughesjj on Thursday February 01, 2024 @09:06PM (#39224058)
Yeah cloudflare is pretty sketchy too imo. They present as transparent but they've had some actions over the years that signal otherwise. Heck, pretty much every performance blog post they hype up buries the caveats, kinda reminiscent of Intel always using their custom cpp compiler for benchmarks. Not technically lying, but definitely omitting some context.
Re:Cloudflare Thanksgiving 2023 Security Incident by cowsandmilk on Thursday February 01, 2024 @09:22PM (#39224168)
I love this quote:

> However, Server-Side Excludes are rarely used and only activated for malicious IP addresses.

So… you’re celebrating that you only had buffer overruns for malicious IP addresses?

Re:Cloudflare Thanksgiving 2023 Security Incident by eastdakota on Thursday February 01, 2024 @11:40PM (#39225115)
This came up before and it was super confusing to me because I had no idea what it was referring to but I also believe Tavis isn’t one to make something up. So I took some time to investigate.

Turned out, no one on our management, legal, communications, or public policy team had any idea what he was talking about. Eventually I figured out that a non-executive former member of our engineering team was dating someone who worked as a fairly junior staffer at the FTC. On the employee’s personal time they mentioned being frustrated by how the disclosure took place to the person they were dating. I believe the employee’s frustration was because we and Project Zero had agreed on a disclosure timeline and then they unilaterally shortened it because an embargo with a reporter got messed up.

There was never anything that Cloudflare or any executive raised with the FTC. And the FTC never took or even considered taking any action. The junior FTC staffer may have said something to Tavis or our employee may have said something about telling the staffer they were dating, but that was the extent of it.

I understand Tavis’s perspective, and agree it was inappropriate of the former Cloudflare employee, but this was two people not in any position of leadership at either Cloudflare or the FTC talking very much out of school.

Re:Cloudflare Thanksgiving 2023 Security Incident by nolok on Friday February 02, 2024 @03:53AM (#39226597)
> we and Project Zero had agreed on a disclosure timeline and then they unilaterally shortened it because an embargo with a reporter got messed up

This is not what happened at all. What happened is that after the initial discovery, the gzero team realized it was much worse than expected AND the cloudflare team who he synced with for the disclosure started ghosting him, and yet gzero still kept to the full timeline.

If you working there and having done research can get it this wrong while it's super easy to find the event log in the open, it doesn't give a very good vibe about the attitude inside cloudflare regarding what happened and fair disclosure.

Full even log on project zero is here : https://bugs.chromium.org/p/project-zero/issues/detail?id=11...

> The examples we're finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I've informed cloudflare what I'm working on. I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.

Meanwhile link with Cloudflare went from this

> I had a call with Cloudflare, they reassured me they're planning on complete transparency and believe they can have a customer notification ready this week.

> I'm satisfied cloudflare are committed to doing the right thing, they've explained their current plan for disclosure and their rationale.

To this

> Update from Cloudflare, they're confident they can get their notification ready by EOD Tuesday (Today) or early Wednesday.

> Cloudflare told me that they couldn't make Tuesday due to more data they found that needs to be purged.

> They then told me Wednesday, but in a later reply started saying Thursday.

> I asked for a draft of their announcement, but they seemed evasive about it and clearly didn't want to do that. I'm really hoping they're not planning to downplay this. If the date keeps extending, they'll reach our "7-day" policy for actively exploited attacks. https://security.googleblog.com/2013/05/disclosure-timeline-...

> If an acceptable notification is not released on Thursday, we'll decide how we want to proceed.

> I had a call with cloudflare, and explained that I was baffled why they were not sharing their notification with me.

> They gave several excuses that didn't make sense, then asked to speak to me on the phone to explain. They assured me it was on the way and they just needed my PGP key. I provided it to them, then heard no further response.

> Cloudflare did finally send me a draft. It contains an excellent postmortem, but severely downplays the risk to customers. They've left it too late to negotiate on the content of the notification.

So it was not project zero but cloudflare that moved the disclosure timeline around, and did so without keeping pzero in the loop, about an active in the wild exploit.

Re:Cloudflare Thanksgiving 2023 Security Incident by chx on Friday February 02, 2024 @04:40AM (#39226878)
> If you working there

For context: you are answering to the co-founder & CEO of Cloudflare.

Re:Cloudflare Thanksgiving 2023 Security Incident by patrickhogan1 on Tuesday February 13, 2024 @12:01AM (#39354599)
Yo. He's the CEO of CloudFlare. He also seems to have dug into it pretty deep. Not a standard PR response.
Re:Cloudflare Thanksgiving 2023 Security Incident by Hrundi on Thursday February 01, 2024 @04:55PM (#39222004)
I don't remember any companies that ended thanks to cloudbleed, but I'd be happy to be proven wrong
Re:Cloudflare Thanksgiving 2023 Security Incident by sneak on Thursday February 01, 2024 @04:56PM (#39222011)
> Imagine getting in at the centre of a new Meet-Me room and having persistent access to key switches there.

This wouldn't get you much. We already assume the network is insecure. This is why TLS is a thing (and mTLS for those who are serious).

Re:Cloudflare Thanksgiving 2023 Security Incident by noizejoy on Thursday February 01, 2024 @05:18PM (#39222231)
> We already assume the network is insecure.

Maybe naively, I wish this assumption became universal.

Re:Cloudflare Thanksgiving 2023 Security Incident by sophacles on Thursday February 01, 2024 @05:56PM (#39222702)
I suspect "we" is a much smaller group than you imagine. I've gotten pcaps from customers as recently as this year that include unencrypted financial transaction data. These were captured on a router, not an end host, so the traffic was going across the client's network raw.
Re:Cloudflare Thanksgiving 2023 Security Incident by meowface on Thursday February 01, 2024 @06:01PM (#39222752)
>Getting in at the "ground floor" of a new datacentre build is pretty much the ultimate exploit.

I can just imagine the attackers licking their lips when they first breached the data center.

Good reminder to use "Full (Strict)" SSL in Cloudflare. Then even if they do get compromised, your reverse-proxied traffic still won't be readable. (Of course other things you might use Cloudflare for could be vectors, though.)

Re:Cloudflare Thanksgiving 2023 Security Incident by ownagefool on Thursday February 01, 2024 @06:24PM (#39222961)
Cloudflare is essentially a massive mitm proxy. If you manage to pwn a key, you have access to traffic.

I'm sure they're better than this than me, but ipxe & tftp are plain text, so it wouldn't be shocking if something in the bootstrap process was plaintext.

At the very least you need to tell the server what to trust.

Re:Cloudflare Thanksgiving 2023 Security Incident by meowface on Thursday February 01, 2024 @06:27PM (#39222995)
>If you manage to pwn a key, you have access to traffic.

That's why I mentioned "Full (Strict)" SSL. If you configure this in Cloudflare then the entire user <-> Cloudflare <-> origin path is encrypted and attackers can't snoop on the plaintext even if they have access. They'll get some metadata, but every ISP in the world gets that at all times anyway.

Re:Cloudflare Thanksgiving 2023 Security Incident by is39 on Friday February 02, 2024 @04:19AM (#39226766)
While both client and origin network connections are encrypted with "Full (Strict)" SSL mode, Cloudflare proxy in the middle decrypts client traffic and then encrypts it towards the server (and vice versa). It does have access to plaintext, which is how various mitigations work. So it's indeed MITM proxy, by design.
Re:Cloudflare Thanksgiving 2023 Security Incident by meowface on Monday February 05, 2024 @10:05AM (#39261929)
Ah, yeah, you're right.