News for hackers. Stuff that matters.
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by NicoJuicy on Saturday November 04, 2023 @04:28AM (#38139123)
?

Tbh. As far as I can see, their data plane worked at the edge.

Cloudflare released a lot of new products and the ones that affected were: streams, new image upload and logpush.

Their control plane was bad though. But since most products worked, that's more redundancy than most products.

The proposed solution is simple:

- GA requires to be in the high availability cluster

- test entire DC outages

Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by throwaway6920 on Saturday November 04, 2023 @05:09AM (#38139293)
> Tbh. As far as I can see, their data plane worked at the edge.

Arguable, it's best to think of the edge as a buffering point in addition to processing. Aggregation has to happen somewhere, and that's where shit hit the fan.

Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by NicoJuicy on Saturday November 04, 2023 @05:26AM (#38139387)
? That would mean their data is at the core cluster. That's not true or I haven't seen any evidence to support that statement.

Cloudflare's data lives in the edge and is constantly moving.

The only thing not living in the edge ( as was noticed), is stream, logpush and new image resize requests ( existing ones worked fine) from the data plane

Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by throwaway6920 on Saturday November 04, 2023 @06:20AM (#38139658)
>That would mean their data is at the core cluster. That's not true or I haven't seen any evidence to support that statement.

You're being loose in your usage of 'data'. No one is talking about cached copies of an upstream, but you probably are.

Read the post mortem a bit more closely. They explicitly state that the control plane(s) source of truth lives in core, and that logs aggregate back to core for analytics and service ingestion. Think through the implications on that one.

Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by e1g on Saturday November 04, 2023 @08:35AM (#38140472)
That’s my interpretation as well. There is one central brain, and “the edge” is like the nervous system that collects signals, sends it to the brain, and is _eventually consistent_ with instructions/config generated by the brain.
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by yowai on Saturday November 04, 2023 @06:36AM (#38139736)
From what I was reading on the status page & customers here on HN, WARP + Zero Trust were also majorly affected, which would be quite impactful for a company using these products for their internal authentication.

It's not just streams, image upload & Logpush.

Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by NicoJuicy on Saturday November 04, 2023 @06:53AM (#38139823)
Those customers were impacted until the DC was back up ( 1-2 hours?) On the config plane.

The data plane ( which I mentioned) had no issues.

It's literally in the title what was affected: "Post Mortem on Cloudflare Control Plane and Analytics Outage"

Eg. The status page mentioned the healthchecks not working, while everything was fine with it. There were just no analytics at that time to confirm that.

Source: I watched it all happen in the cloudflare discord channel.

If you know anyone that is claiming to be affected on the data plane for the services you mentioned, that would be an interesting one.

Note: I remember emails were also more affected though.

Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by yowai on Saturday November 04, 2023 @07:07AM (#38139923)
> Those customers were impacted until the DC was back up ( 1-2 hours?) On the config plane.

Which was still like ~12+ hours, if we check the status page.

>Eg. The status page mentioned the healthchecks not working, while everything was fine with it. There were just no analytics at that time to confirm that.

What good is a status page that's lying to you? Especially since CF manually updates it, anyway?

>Source: I watched it all happen in the cloudflare discord channel.

Wow, as a business customer I definitely like watching some Discord channel for status updates.

Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by NicoJuicy on Saturday November 04, 2023 @09:07AM (#38140700)
?

This wasn't about status updates going to discord only. There is literally a discussion section on the discord.

Jeez

Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by yowai on Saturday November 04, 2023 @11:50AM (#38142154)
Oh, I just looked it up and I thought you mean that CF engineers were giving real time updates there. That's not the case.

However, I still fail to see your argument regarding Zero Trust and not being impacted. The status page literally mentioned that the service was recovered on Nov 3, so I don't understand what you mean by:

>The data plane ( which I mentioned) had no issues.

There's literally a section with "Data plane impact" on all over the status page, and ZT is definitely in the earlier ones. And this is given the fact that status updates on Nov 2 were very sparse until power was restored.

Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by NicoJuicy on Saturday November 04, 2023 @01:53PM (#38143472)
We don't use zero trust atm. So, I can't know for sure.

What I mentioned, was what I've seen passing by in the channel at the time.

I also saw no incoming help requests for zero trust tbh ( did some community help)

Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by kurok on Saturday November 04, 2023 @08:08AM (#38140293)
This was short downtime. But big companies must create own gateway but small just waiting and relying on CF
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage by jorams on Saturday November 04, 2023 @10:48AM (#38141510)
The combination of "newer products" and then having "our Stream service" as the only named service in the post-mortem is very odd, since Stream is hardly a "newer product". It was launched in 2017 and went GA in 2018[2]. If after 5 years it still didn't have a disaster recovery procedure I find it hard to believe they even considered it.

[1]: https://blog.cloudflare.com/introducing-cloudflare-stream/ [2]: https://www.cloudflare.com/press-releases/2018/cloudflare-st...