Getting in at the "ground floor" of a new datacentre build is pretty much the ultimate exploit. Imagine getting in at the centre of a new Meet-Me room (https://en.wikipedia.org/wiki/Meet-me_room) and having persistent access to key switches there.
Cloudflare datacentres tend to be at the hub of insane amounts of data traffic. The fact that the attacker knew how valuable a "pre-production" data centre is means that cloudflare probably realized themselves that it would be a 100% game over if someone managed to get a foot hold there before the regular security systems are set up. It would be a company ending event if someone managed to install themselves inside a data centre while it was being built/brought up.
Also remember, at the beginning of data centre builds, all switches/equipment have default / blank root passwords (admin/admin), and all switch/equipment firmware are old and full of exploits (you either go into each one and update the firmware one by one or hook them up to automation for fleet wide patching) Imagine that this exploit is taking place before automation services had a chance to patch all the firmware ... that's a "return all devices to make sure the manufacturer ships us something new" event.
It takes some honesty and good values by someone in the decision-making to go ahead with such a comprehensive plan. This is sad because it should be tablestakes, as you say correctly, but having seen many other cases, I think although they did "the expected", it's definitely above and beyond what peers have done.
thanks Matthew! love the transparency and dedication to security as always. really sucks to have this be continuing fallout from Okta's breach. wish large scale key rotation was more easily automatable (or at least as a fallback, there should be a way to track key age on clientside? so that old keys stick out like a sore thumb). i guess in the absence of industry standard key rotation apis someday you might be able to "throw AI at it".
It wouldn't. Most people like to assume the impact of breaches to be what it should be, not what it actually is.
Look at the 1-year stock chart of Okta and, without looking up the actual date, tell me when the breach happened/was disclosed.
The problem with this is that while security minded people know what Okta is and why to stay the fuck away from handing over your crown jewels to a SaaS company is warranted, C-level execs don't care. They only care about their golf course or backroom deal friends and about releasing PR statements full of buzzwords like "zero trust", "AI based monitoring" and whatever.
The stock markets don't care either, they only look at the financial data, and as long as there still are enough gullible fools signing up, they don't care and stonk goes up.
Instead, even the "self correcting" mechanisms of the "free market" obviously didn't work out, as the free market doesn't value technical merit, it only values financial bullshittery.
And the end result will be that once the war with China or Russia inevitably breaks out, virtually all major Western companies and governments will be out cold for weeks once Okta and Azure's AD go down, because that is where any adversary will hit first to deal immense damage.
Given they got out of cloudbleed without any real damage let alone lasting damage, I disagree.
(I don't disagree with your point about how bad of a problem this would be, I'm just insisting that security failure is not taken seriously at all by anyone)
https://twitter.com/taviso/status/1566077115992133634
> True story: After cloudbleed, cloudflare literally lobbied the FTC to investigate me and question the legality of openly discussing security research. How come they're not lobbying their DC friends to investigate the legality KF?
For those not familiar with the history , this tweet started the cloudbleed disclosure to cloudflare:
https://twitter.com/taviso/status/832744397800214528
> Could someone from cloudflare security urgently contact me.
This followed: https://blog.cloudflare.com/incident-report-on-memory-leak-c...
> However, Server-Side Excludes are rarely used and only activated for malicious IP addresses.
So… you’re celebrating that you only had buffer overruns for malicious IP addresses?
Turned out, no one on our management, legal, communications, or public policy team had any idea what he was talking about. Eventually I figured out that a non-executive former member of our engineering team was dating someone who worked as a fairly junior staffer at the FTC. On the employee’s personal time they mentioned being frustrated by how the disclosure took place to the person they were dating. I believe the employee’s frustration was because we and Project Zero had agreed on a disclosure timeline and then they unilaterally shortened it because an embargo with a reporter got messed up.
There was never anything that Cloudflare or any executive raised with the FTC. And the FTC never took or even considered taking any action. The junior FTC staffer may have said something to Tavis or our employee may have said something about telling the staffer they were dating, but that was the extent of it.
I understand Tavis’s perspective, and agree it was inappropriate of the former Cloudflare employee, but this was two people not in any position of leadership at either Cloudflare or the FTC talking very much out of school.
This is not what happened at all. What happened is that after the initial discovery, the gzero team realized it was much worse than expected AND the cloudflare team who he synced with for the disclosure started ghosting him, and yet gzero still kept to the full timeline.
If you working there and having done research can get it this wrong while it's super easy to find the event log in the open, it doesn't give a very good vibe about the attitude inside cloudflare regarding what happened and fair disclosure.
Full even log on project zero is here : https://bugs.chromium.org/p/project-zero/issues/detail?id=11...
> The examples we're finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup. I've informed cloudflare what I'm working on. I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.
Meanwhile link with Cloudflare went from this
> I had a call with Cloudflare, they reassured me they're planning on complete transparency and believe they can have a customer notification ready this week.
> I'm satisfied cloudflare are committed to doing the right thing, they've explained their current plan for disclosure and their rationale.
To this
> Update from Cloudflare, they're confident they can get their notification ready by EOD Tuesday (Today) or early Wednesday.
> Cloudflare told me that they couldn't make Tuesday due to more data they found that needs to be purged.
> They then told me Wednesday, but in a later reply started saying Thursday.
> I asked for a draft of their announcement, but they seemed evasive about it and clearly didn't want to do that. I'm really hoping they're not planning to downplay this. If the date keeps extending, they'll reach our "7-day" policy for actively exploited attacks. https://security.googleblog.com/2013/05/disclosure-timeline-...
> If an acceptable notification is not released on Thursday, we'll decide how we want to proceed.
> I had a call with cloudflare, and explained that I was baffled why they were not sharing their notification with me.
> They gave several excuses that didn't make sense, then asked to speak to me on the phone to explain. They assured me it was on the way and they just needed my PGP key. I provided it to them, then heard no further response.
> Cloudflare did finally send me a draft. It contains an excellent postmortem, but severely downplays the risk to customers. They've left it too late to negotiate on the content of the notification.
So it was not project zero but cloudflare that moved the disclosure timeline around, and did so without keeping pzero in the loop, about an active in the wild exploit.
For context: you are answering to the co-founder & CEO of Cloudflare.
This wouldn't get you much. We already assume the network is insecure. This is why TLS is a thing (and mTLS for those who are serious).
Maybe naively, I wish this assumption became universal.
I can just imagine the attackers licking their lips when they first breached the data center.
Good reminder to use "Full (Strict)" SSL in Cloudflare. Then even if they do get compromised, your reverse-proxied traffic still won't be readable. (Of course other things you might use Cloudflare for could be vectors, though.)
I'm sure they're better than this than me, but ipxe & tftp are plain text, so it wouldn't be shocking if something in the bootstrap process was plaintext.
At the very least you need to tell the server what to trust.
That's why I mentioned "Full (Strict)" SSL. If you configure this in Cloudflare then the entire user <-> Cloudflare <-> origin path is encrypted and attackers can't snoop on the plaintext even if they have access. They'll get some metadata, but every ISP in the world gets that at all times anyway.
Aha, the old replace-your-trusted-hardware trick.
Now imagine, instead of Steve from HR's laptop, it's one of Cloudflare's servers.
If Cloudflare is in a position where their security team can make a call to rotate every secret and reimage every machine, and then that happens in some reasonable amount of time, that's pretty impressive.
It's good that you think you can absorb a complicated security task, it's useless if you have no way to test or verify this action.
You find some scary things when you go looking for how exactly some written-by-greybeard script is authenticating against your started-in-1990s datastore.
If you assume that they only accessed what you can prove they accessed, you've left a hole for them to live in. It should require a quorum of people to say you DON'T need to do this.
Of course, this is ideal world. I'm glad my group is afforded the time to implement features with no direct monetary or user benefit.
> The threat actor also attempted to access a console server in our new, and not yet in production, data center in São Paulo. All attempts to gain access were unsuccessful. To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers. The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway.
They didn't have to go this far. It would have been really easy not to. But they did and I think that's worthy of kudos.