News for hackers. Stuff that matters.
Re:GCP Outage by fidotron on Thursday June 12, 2025 @02:45PM (#44261461)
It's completely nuts that Firebase has this: https://status.firebase.google.com/incidents/ZcF1YDUvpdixZ2e...

"Firebase Data Connect unavailable due to a known Google Cloud global outage"

While the Google Cloud status page https://status.cloud.google.com/ says "No major incidents" and everything is green. So Google Cloud know there is an outage but just deem it not major enough to show it.

Edit to add: within 10 minutes of this post Google updated their status page. More curiously the Firebase page I linked to has been edited to remove mention of Google Cloud in the status and now says "Firebase Data Connect is currently experiencing a service disruption. Please check back for status. ".

Re:GCP Outage by shmatt on Thursday June 12, 2025 @03:08PM (#44261828)
IIRC status pages drive customer compensation for downtime. Updating it is basically signing the check for their biggest customers, in most similar companies you need a very senior executive to approve the update

On the other side of this, Firebase probably doesn't have money at stake making the update

Re:GCP Outage by aiauthoritydev on Thursday June 12, 2025 @03:11PM (#44261879)
It is not the status page that drives customer compensation. It is downtime.
Re:GCP Outage by camdenreslink on Thursday June 12, 2025 @03:15PM (#44261943)
The status page is essentially an admission of guilt. It can require approval from the legal department and a high level official from the company to approve updating it and the verbiage used on the status page.
Re:GCP Outage by hodgesrm on Thursday June 12, 2025 @03:30PM (#44262150)
> It can require approval from the legal department and a high level official from the company to approve updating it and the verbiage used on the status page.

Is that true in this case or are you speculating? My company runs a cloud platform. Our strategy is to have outages happen as rarely as possible and to proactively offer rebates based on customer-measured downtime. I don't know why people would trust vendors that do otherwise.

Re:GCP Outage by camdenreslink on Thursday June 12, 2025 @09:59PM (#44265081)
I don't have any special knowledge about the companies involved in this outage. I do know most (all?) status pages for large companies have to be manually updated and not just anybody can do that. These things impact contracts, so you want to be really sure it is accurate and an actual outage (not just a monitor going off, possibly giving a false positive).
Re:GCP Outage by dpkirchner on Thursday June 12, 2025 @03:17PM (#44261978)
You are likely right, but it's still gross dishonesty. I'm not ready to let Google and their engineers off the hook for that.
Re:GCP Outage by refulgentis on Thursday June 12, 2025 @03:24PM (#44262078)
Inter alia, "is essentially", "it can", tell us this is just free-associating.

We should probably avoid punishing them based on free-associating made by a random not-anonymous not-Googler not-Xoogler account on HN. (disclaimer: xoogler)

Re:GCP Outage by dijit on Thursday June 12, 2025 @03:17PM (#44261975)
then it’s fucking useless. Let’s crowd source our own
Re:GCP Outage by Tijdreiziger on Thursday June 12, 2025 @03:23PM (#44262066)
That’s what Downdetector is.

https://downdetector.com/

Re:GCP Outage by shuntress on Thursday June 12, 2025 @03:25PM (#44262090)
We tried to do that. It didn't work. Too much spam, scams, and abuse.
Re:GCP Outage by baggy_trough on Thursday June 12, 2025 @03:26PM (#44262105)
You're in the crowdsourced version right now.
Re:GCP Outage by hugs on Thursday June 12, 2025 @03:23PM (#44262072)
working on it! (valet network)
Re:GCP Outage by refulgentis on Thursday June 12, 2025 @03:23PM (#44262073)
"It can", this is just free-associating, don't let it get to ya. (disclaimer: xoogler)
Re:GCP Outage by refulgentis on Thursday June 12, 2025 @03:13PM (#44261902)
Nah, its just some client side caching / JS stuff. Clicking the big refresh button fixed it for me, 15 minutes before OP noted it.

(n.b. as much as Google in aggregate is evil, they're smart evil. You can't avoid execs approving every outage because checks without some paper trail, and execs don't want to approve every outage, you'd have to rely on too many engineers and sales people, even as ex-employees, to keep it a secret. disclaimer: xoogler)

(EDIT: for posterity, we're discussing a "overall status" thing with a huge refresh button, right above a huge table chockful of orange triangles that indicate "One or more regions affected" - even when the "overall status" was green, the table was still full of orange and visible immediately underneath. My point being, you gotta suppose a wholeeee bunch of stuff to get to the point there was ever info suppressed, much less suppressed intentionally to avoid cutting checks)

Re:GCP Outage by kjuulh on Thursday June 12, 2025 @02:48PM (#44261508)
Something must be preventing them updating the status page at this point. Of course they could still deem it not enough, but just from my limited tests, docker, buf, etc (it may not be GCP that is down, but it is quite the coincidence). are outright down. I'd wager that this is much more widespread.
Re:GCP Outage by sss111 on Thursday June 12, 2025 @02:54PM (#44261613)
I'm actually on a bridge call with Google Cloud, we're a large customer -- I just learned today that their status page is not automated, instead someone actually manually updates it!
Re:GCP Outage by paxys on Thursday June 12, 2025 @02:57PM (#44261654)
That's the case with every status page. These pages are managed by business people not engineers, because their primary purpose is to show customers that the company is meeting contractually defied SLAs.
Re:GCP Outage by belter on Thursday June 12, 2025 @03:10PM (#44261854)
Surelly no SLA will be based on the display of the status page...
Re:GCP Outage by paxys on Thursday June 12, 2025 @03:54PM (#44262447)
They aren't automatically based on that page, but seeing a red status makes it too easy for customers to point to it and go "see you were down, give us a refund".
Re:GCP Outage by phatskat on Thursday June 12, 2025 @03:18PM (#44261998)
Maybe or maybe not, but someone with nothing better to do than monitor that page out of boredom might “get on the horn” with lots of people to complain if a green check mark turns to a red X.
Re:GCP Outage by Tostino on Thursday June 12, 2025 @03:16PM (#44261950)
should* be
Re:GCP Outage by redeux on Thursday June 12, 2025 @02:57PM (#44261662)
This is actually the norm for status pages. If you look at the various status page offerings you'll see that they're designed around manual updates.
Re:GCP Outage by quectophoton on Thursday June 12, 2025 @03:22PM (#44262050)
The best way to consistently having good "time to response" metrics, is to be the one deciding when an incident "actually" started happening, if at all :)
Re:GCP Outage by kjuulh on Thursday June 12, 2025 @02:57PM (#44261667)
This feels very much like when facebook, locked themselves out of their datacenters. ;)

* https://www.datacenterdynamics.com/en/news/facebook-blames-m...

Re:GCP Outage by dinvlad on Thursday June 12, 2025 @03:04PM (#44261765)
Except that AWS, CloudFlare and a bunch others are also down :-O
Re:GCP Outage by kjuulh on Thursday June 12, 2025 @03:19PM (#44262004)
Downdetector shows they've got issues as well, but it can be fairly unreliable, as people don't know which service is behind their apps.

I at least have no issues on their services across a few regions, and their console works fine.

Re:GCP Outage by cyberpunk on Thursday June 12, 2025 @03:15PM (#44261938)
AWS looks ok to me?

https://health.aws.amazon.com/health/status

Perhaps CF is dependant on some GCP services?

Re:GCP Outage by diggan on Thursday June 12, 2025 @03:18PM (#44261989)
> AWS looks ok to me?

> https://health.aws.amazon.com/health/status

Historically, the worst place to figure out if AWS is up/down is Amazons own status page.

Re:GCP Outage by peterjliu on Thursday June 12, 2025 @03:17PM (#44261967)
seems like misinformation for AWS. CloudFlare probably depends on GCP.
Re:GCP Outage by mpalmer on Thursday June 12, 2025 @03:04PM (#44261758)
The bigger you are, the more you want a human involved in the decision to publicly declare an incident.
Re:GCP Outage by spenczar5 on Thursday June 12, 2025 @02:57PM (#44261670)
That's fairly typical. You want a human in the loop for decisions like that.
Re:GCP Outage by paulddraper on Thursday June 12, 2025 @02:59PM (#44261693)
Most status pages are manual.

At least some of the information has to be.

The weird part is that it took them almost an full hour to update it.

Re:GCP Outage by alexcroox on Thursday June 12, 2025 @02:49PM (#44261532)
Re:GCP Outage by cherioo on Thursday June 12, 2025 @02:50PM (#44261543)
This extra funny that GCP status page even includes a “last updated” time, which is exactly built to convey possible failure to update in cases like this

No major incident as of “ Last updated time: 12 Jun 2025, 11:48 PDT”

Re:GCP Outage by Workaccount2 on Thursday June 12, 2025 @02:58PM (#44261676)
I asked testing to see if it was up, and it pointed out that Google shows nothing but Nest is showing an outage right now, lol

https://status.nest.com/posts/dashboard

Re:GCP Outage by codergautam on Thursday June 12, 2025 @02:48PM (#44261518)
Maybe the outage is preventing them from updating that specific page? Hmm

EDIT: Looks like it has been updated now (6:49 PM UTC)

Re:GCP Outage by artooro on Thursday June 12, 2025 @02:55PM (#44261625)
Anytime there is an outage that affects App Engine, Google can't seem to get their status page updated for an extended period of time.
Re:GCP Outage by alexcroox on Thursday June 12, 2025 @02:50PM (#44261550)
Almost an hour to update the page...
Re:GCP Outage by devMem on Thursday June 12, 2025 @02:49PM (#44261538)
I hope this is the case, or google is super unreliable for production grade work.
Re:GCP Outage by samdung on Thursday June 12, 2025 @02:49PM (#44261533)
:))))))
Re:GCP Outage by ike2792 on Thursday June 12, 2025 @02:52PM (#44261581)
Maybe their dashboard is hosted on GCP and they are displaying a cached version. :-)
Re:GCP Outage by aetherson on Thursday June 12, 2025 @02:49PM (#44261531)
More likely they are unable to update their own status page, but in either case not covering themselves in glory over at GCP right now.
Re:GCP Outage by cyberflame on Thursday June 12, 2025 @03:13PM (#44261919)
Services are recovering in some locations it seems - Discord is healing
Re:GCP Outage by octo888 on Thursday June 12, 2025 @02:59PM (#44261688)
Status pages are PR. It gets the same PR treatment as anything else
Re:GCP Outage by samdung on Thursday June 12, 2025 @02:49PM (#44261529)
GCP just updated their status
Re:GCP Outage by nailer on Thursday June 12, 2025 @03:14PM (#44261924)
AWS has this all the time. If you need to know if a service is down in a region, check for other engineers talking about it on X.
Re:GCP Outage by blibble on Thursday June 12, 2025 @02:54PM (#44261606)
lies, from big tech?

say it's not so!