Sadly this reminds me of AWS outages too where the same applies. How is it that hundreds of developers know there's an issue before AWS do, or Cloudflare in this instance. See my blog post on similar AWS uptime reporting issues at https://www.ably.io/blog/honest-status-reporting-aws-service.

At Ably, our status site had an incident update about Cloudflare issues being worked on (by routing away from CF) before Cloudflare did: https://status.ably.io/incidents/647

We have machine generated incidents created automatically when error rates increase beyond a certain point stating "Our automated systems have detected a fault, we've been alerted and looking at it". See https://status.ably.io/incidents/569 for example. I think much larger companies like Cloudflare and Amazon could certainly invest a bit in similar systems to make it easier for their customers to know where the problem likely lies.

Heh, I am reminded of when the control plane at AWS went down... and we had a custom autoscaling config that would query for the number of instances running and scale appropriately... but when the AWS API died... we kept getting zero running instances...

So our system thought none were running and so it kept launching instances....

These were SPOT instances and thus only cost like .10 per hour...

But we launched like 2500 instances which all needed to slurp down their DB and config - so it overloaded all other control plane systems...

We had to reboot the entire system. Which took forever.

The only good things was this happened at 11am - so all team members were online and avail... and then AWS refunded all costs.

---

The other fun time was when a newbie dev checked in AWS creds to git - but he created the 201th repo (we had only paid for 200) -- and as it was the next repo which wasnt paid for, it was by default public - thus slurped up by bots asap - which then used the AWS creds to launch bitcoin mining bots in every single region around the globe. Like 1700 instances.

The thing that sucked about that was it happened at like 3am and we had to rally on that one pretty fast. AWS still refunded all costs...

> but he created the 201th repo (we had only paid for 200)

That's an odd choice of a failure mode.

> AWS still refunded all costs...

Yeah they should. It was their silly design choice that lead to disclosure of secrets after all.

What kind of failure mode is that even. Failing to create the repo would have led to a better user experience for sure.

Can you imagine if S3 charged more for private objects and once you reach your count, it just makes them public and posts them on Reddit?

>Can you imagine if S3 charged more for private objects and once you reach your count, it just makes them public and posts them on Reddit?

Exactly! What a stupid design UX.

>It was their silly design choice that lead to disclosure of secrets after all.

Wait, was the 200 private repos issue an AWS thing or a GitHub/GitLab/whatever thing?

What AWS product has a concept of private/public repos and limits on how many of the former you can get for a certain price?

It was a git thing.

Never post aws secrets to git.

>How is it that hundreds of developers know there's an issue before AWS do

Trust me, they know.

They know about problems we never find out about, too.

At least in the case with AWS, unfortunately there's business involved - because of their uptime guarantee, incidents that would be called downtime by a purely technical team are left as "operational" or "partly degraded". Otherwise, they might have to shell out millions or tens of millions.

You have to provide "your request logs that document the errors and corroborate your claimed outage" for the AWS Compute SLA https://aws.amazon.com/compute/sla/

slightly off-topic: Nice and clean, yet detailed enough status page. I like it. :)

Thanks :)

On the OT subject, the status logo is blurry (at least on a 4k monitor), whereas on the main website it's nice and crisp.

Ok, thanks for the heads up. Will raise a status website issue.

Are you sure it wasn't cached? For us it was showing correctly the incident...

If cloudflarestatus.com isn't serving text content as Cache-Control: no-cache, that's itself an egregious bug.

It serves Cache-Control: max-age=0, private, must-revalidate. Same thing as far as I get it, maybe wider support.

Their status page is hosted on Cloudfront, perhaps adding an unintended caching layer.

Is it? From my end it looks like a plain ec2 instance from statuspage.io.

Cloudfront supports Cache-Control: no-cache

Cloudfront itself can cache the content regardless. (if its headers are showing HIT)

It might have been cached on the server but was not cached on my clients: I did a forced refresh and used a private browser window but it only showed the “almost everything is good” view throughout the duration of the incident.

The actual incident page was correct.

Can confirm that we had the same view. Operational while not working.

All I saw was a vague message at the top, all services were marked green and operational throughout.

They should derive whether something is down from how much traffic the status page gets, and have alerts tied to that as well. I think it would be pretty accurate.

We saw an update within 5 minutes of our sites going down. Are you doing caching somewhere?

Interesting, I was command+shift+R refreshing and also tried from a VPN in another region. Perhaps our CF-hosted sites returned 502s in my region sooner than yours, causing me to check the status page sooner.

Seems unlikely, I don't use cloudflare have never visited the status page before. I noticied several 502 pages this morning and searched for "cloudflare status page" and saw the "all systems operational".

It isn't in their best interest to have a reliable status page. They want status page to contain information about failure only after people know about issues through other means. For the same reason cloud providers don't provide rebates on outages unless you ask for it explicitly.

I think the only way is to have status page operated by an independent 3rd party, but I don't think there's a viable business model for someone to provide such service. Perhaps there might be even a risk of lawsuits against you.

It was showing correctly for me the first time I checked it.

Why even have a status page if it doesn't reflect reality?