News for hackers. Stuff that matters.
Re:GCP Outage by ransom1538 on Thursday June 12, 2025 @02:19PM (#44260969)
Why can't companies be honest with being down. It helps us all out so we don't spend an hour internalizing.

We are truly in gods hands.

$ prod

Fetching cluster endpoint and auth data. ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=503, message=Visibility check was unavailable. Please retry the request and contact support if the problem persists

Re:GCP Outage by oxymoron on Thursday June 12, 2025 @02:28PM (#44261151)
Because a lot of the time, not everyone is impacted, as the systems are designed to contain the "blast radius" of failures using techniques such as cellular architecture and [shuffle sharding](https://aws.amazon.com/builders-library/workload-isolation-u...). So sometimes a service is completely down for some customers and fully unaffected for other customers.
Re:GCP Outage by hnuser123456 on Thursday June 12, 2025 @02:39PM (#44261361)
"there is a 5% chance your instance is down" is still a partial outage. A green check should only mean everything (about that service) is working for everyone (in that region) as intended.

Downdetector reports started spiking over an hour ago but there still isn't a single status that isn't a green checkmark on the status page.

Re:GCP Outage by spwa4 on Thursday June 12, 2025 @03:58PM (#44262485)
Just say it: they want to lie to 95% of customers.
Re:GCP Outage by deepsun on Thursday June 12, 2025 @05:12PM (#44263253)
With highly distributed services there's always something failing, some small percentage.
Re:GCP Outage by nijave on Thursday June 12, 2025 @07:18PM (#44264245)
Sure but you can still put a message up when it's some <numeric value> over some <threshold value> like errors are 50% higher than normal (maybe the SLO is 99.999% of requests are processed successfully)
Re:GCP Outage by deepsun on Thursday June 12, 2025 @08:01PM (#44264468)
Just note that aggregations like that might manifest as GCP didn't have any issues today actually.

E.g. it was mostly us-central1 region affected, and in there only some services (e.g. regular instances, and GKE kubernetes were not affected in any region). So if we ask "what the percentage of GCP is down", it might well be it's less than the threshold.

On the other hand, about a month ago, 2025-05-19 there was an 8-hour long incident with Spot VM instances affecting 5 regions, and which was way more important to our company, but it didn't make any headlines.

Re:GCP Outage by Eduard on Thursday June 12, 2025 @02:34PM (#44261268)
> Because a lot of the time, not everyone is impacted

then such pages should report a partial failure. Indeed the GCP outage page lists an orange "One or more regions affected" marker, but all services show the green "Available" marker, which apparently is not true.

Re:GCP Outage by deepsun on Thursday June 12, 2025 @05:13PM (#44263259)
There's always a partial outage in large systems, some very small percentage. All clouds should report all red then.
Re:GCP Outage by johannes1234321 on Thursday June 12, 2025 @02:55PM (#44261620)
They still could show that so.e.issues exist. Their monitoring must know.

The issue is that they don't want to. (For claiming good uptime, which may even be true for average user, if most outages affect only small groups)

Re:GCP Outage by jobs_throwaway on Thursday June 12, 2025 @02:43PM (#44261424)
That is still 100% an outage and should be displayed as such
Re:GCP Outage by nijave on Thursday June 12, 2025 @07:15PM (#44264230)
It's not rocket science. Put a message up "The service is currently degraded and some users may see errors"
Re:GCP Outage by kingstnap on Thursday June 12, 2025 @02:33PM (#44261234)
Because they have unrealistic targets so they make up fake uptime numbers. 99.999% would mean not even having an hour of downtime in 10 years.

I remember reddit being down for like a whole day or so and they claimed 99.5% in that month.

Re:GCP Outage by wbl on Thursday June 12, 2025 @03:33PM (#44262185)
Ma Bell hit that decently often.
Re:GCP Outage by Uehreka on Thursday June 12, 2025 @04:45PM (#44263006)
Is that even knowable? Like, I know they called it “The Astonishing, Unfailing, Bell System” but if they had an outage somewhere did they actually have an infrastructure of “canary phones” and such to tell in real time? (As in, they’d know even if service was restored in an hour)

Not trying to snark, I legit got nerdsniped by this comment.

Re:GCP Outage by wbl on Thursday June 12, 2025 @04:58PM (#44263126)
They absolutely did. Note that the reliability estimates exclude the last mine because trees falling and the like but they had a lot of self repair, reporting, and management facilities.

Engineering and Operations in the Bell System is pretty great for this.

Re:GCP Outage by Dylan16807 on Thursday June 12, 2025 @04:57PM (#44263121)
Running a much simpler system with much more independent nodes.

It's a lot easier to keep packets flowing than to keep non-self-contained servers serving.

Re:GCP Outage by jeanlucas on Thursday June 12, 2025 @02:20PM (#44260986)
Because there are contracts related to uptime :)
Re:GCP Outage by rixthefox on Thursday June 12, 2025 @02:30PM (#44261177)
Those contracts will be monitoring their service availability on their own. If Google can't be honest you can bet your bottom dollar the companies paying for that SLA are going to hold them accountable if they report the outage properly or not.
Re:GCP Outage by datadrivenangel on Thursday June 12, 2025 @02:31PM (#44261207)
The real point of SLAs is to give you a reason to break contracts. If a vendor doesn't meet their contractual promises, that gives you a lot of room to get out contracts
Re:GCP Outage by rustc on Thursday June 12, 2025 @02:27PM (#44261123)
Does any service even say they're "down" anymore? All I see is "elevated error rates".
Re:GCP Outage by colechristensen on Thursday June 12, 2025 @02:30PM (#44261181)
4 to 6 hours after the flames are visible from orbit and management has finally given up on the 37th quick fix you do get that red X

But really not until after it's been on CNN a while.

Re:GCP Outage by voytec on Thursday June 12, 2025 @02:34PM (#44261271)
> Why can't companies be honest with being down

SLA agreements.

Re:GCP Outage by organsnyder on Thursday June 12, 2025 @03:40PM (#44262262)
Any customer with enough leverage to negotiate meaningful SLA agreements will also have the leverage to insist that uptime is not derived from the absence of incidents on public-facing status pages.
Re:GCP Outage by remram on Thursday June 12, 2025 @08:27PM (#44264613)
Service level agreements agreements?
Re:GCP Outage by rapus95 on Thursday June 12, 2025 @02:40PM (#44261376)
if half the internet is down, which it apparently is, it's usually not the service in question, but some backbone service like cloudflare. And as internal health monitoring doesn't route to the outside through the backbone to get back in, it won't pick it up. Which is good in some sense, as it means that we can see if it's on the path TO the service or the service itself.
Re:GCP Outage by 9rx on Thursday June 12, 2025 @02:23PM (#44261061)
The program that updates the status page is hosted on Google Cloud.
Re:GCP Outage by tfsh on Thursday June 12, 2025 @02:41PM (#44261399)
It's not. You might be joking, but that comment still isn't helpful.

My understanding is this is part of Google's internal PSD offering (Public Status Board) which uses SCS (Static Content Service) behind GFE (Google Frontend) which is hosted on Borg, and deploys other large scale apps such as Search, Drive, YouTube, etc.

Re:GCP Outage by 9rx on Thursday June 12, 2025 @07:49PM (#44264411)
How could it not be helpful given that it gave you reason to provide more details that you wouldn't have otherwise shared? You may not have thought this through. There is nothing more helpful. Unless you think your own comment isn't helpful, but then...
Re:GCP Outage by refulgentis on Friday June 13, 2025 @01:43AM (#44265996)
Because "It's good to lie because it makes people correct me" is a joke about IRC, not a viable stable game-theoretic optimal position.
Re:GCP Outage by 9rx on Friday June 13, 2025 @08:19AM (#44267868)
Cunningham's Law emerged in the newsgroups era, well predating the existence of IRC.

Of course, I recognize that you purposefully pulled the Cunningham's Law trigger so that you, too, would gain additional knowledge that nobody would have told you about otherwise, as one logically would. And that you played it off as some kind of derision towards doing that all while doing it yourself made it especially funny. Well done!

Re:GCP Outage by refulgentis on Friday June 13, 2025 @02:08PM (#44270748)
I have 0 idea what Cunningham's Law is, so we can both agree that "recognizing purpose" was "mind-reading", in this case. I didn't really bother reading the rest after the first sentence because I saw something about how I joking and congratulating me in my peripheral vision.

It is what it says on the tin: choosing to lie doesn't mean you want the truth communicated.

I apologize that it comes across as aggro, its just that I'm not quite as giggly about this as you are. I think I can safely assume you're old enough to recognize some deleterious effects of lying

Re:GCP Outage by 9rx on Friday June 13, 2025 @04:04PM (#44271764)
> I have 0 idea what Cunningham's Law is

You had no idea what it is. Now you know thanks to you the lie you told.

> choosing to lie doesn't mean you want the truth communicated.

But you're going to get it either way, so if you do lie, expect it. If you don't want it – don't lie, I guess. Strange that someone wouldn't want to learn about the truth, though. Consider me skeptical that your claim is even a thing, especially as it is likely you're again fishing to learn more about the topic yourself.

> I apologize that it comes across as aggro

It doesn't. Attaching human attributes to software would be plain weird.

> I think I can safely assume you're old enough to recognize some deleterious effects of lying

Time and place. While it can be disastrous in the right context, context is significant. It makes no difference in a place of entertainment, as is the case here. Entertainment has always been rooted in tales that aren't true. No matter how old you are, even young children understand that.

Re:GCP Outage by mnordhoff on Sunday June 15, 2025 @08:19AM (#44281925)
Wellp. Incident report: "We posted our first incident report to Cloud Service Health about ~1h after the start of the crashes, due to the Cloud Service Health infrastructure being down due to this outage."
Re:GCP Outage by ashu1461 on Thursday June 12, 2025 @02:34PM (#44261249)
So even then, it should have been able to correctly report the status, it somehow shows that the status page is not automated and any change there needs to go through someone manual.
Re:GCP Outage by 9rx on Thursday June 12, 2025 @02:38PM (#44261347)
A program that updates the status page failing does not imply that the status page is manually edited. It is not like you would generate a status page on every request.
Re:GCP Outage by ashu1461 on Thursday June 12, 2025 @02:47PM (#44261500)
How do we know that the program is failing ?

How hard is it for the frontend to detect if the last update to the status page was made a while ago and that itself implies there is an error and should be reported ?

Re:GCP Outage by 9rx on Thursday June 12, 2025 @09:24PM (#44264887)
We don’t.

But why would the frontend have processing logic when all you need is to serve a static HTML document?

Even if it did, what would you do with that information? Throw up a screen with: Call us for service information at 1-HAHA-JUST-KIDDING

It’s not like it really matters if it’s accurate anyway.

Re:GCP Outage by rapus95 on Thursday June 12, 2025 @02:46PM (#44261481)
the services ARE healthy, status page is correct. The backbone which links YOU to the service isn't healthy. Take a look at cloudflare, they are already working on it
Re:GCP Outage by ikiris on Thursday June 12, 2025 @03:05PM (#44261774)
Not even close. The status page is manual and cloud flares outage is because of Google not the other way around.
Re:GCP Outage by supportengineer on Thursday June 12, 2025 @02:47PM (#44261503)
Nobody gets a promotion, that's why.
Re:GCP Outage by rozap on Thursday June 12, 2025 @02:32PM (#44261226)
Please, won't somebody think of the KPIs.