And this is Railway, a big enough name to top the HN main page and presumably find someone from Google to intervene at some point. I would have zero recourse if it was some little product that I built.

Their account was restored in 10 / 19 minutes! It just took 4-6 hours to get everything fully healthy. I look forward to seeing the google response to this hopefully.

May 19, 22:10 UTC - Our automated monitoring detected API health check failures and paged our on-calls, who started investigating the issue. May 19, 22:11 UTC - Dashboard returning 503 errors. Users unable to log in. May 19, 22:19 UTC - Root cause identified: Google Cloud Platform has suspended Railway's production account. May 19, 22:22 UTC - P0 ticket filed with Google Cloud. Railway's GCP account manager engaged directly. May 19, 22:29 UTC - Incident declared. May 19, 22:29 UTC - GCP account access restored. All compute instances remained stopped and persistent disks inaccessible.

The timestamp inconsistency teraflop points out is interesting — but the bigger takeaway for me is that Railway's own automated API health checks caught the failure at 22:10, a full 10 minutes before the root cause was identified.

That's external dependency monitoring working exactly as it should. Most teams only monitor their own infrastructure. When a cloud provider, payment gateway, or third-party API fails — your own dashboards show green while users see failures.

The lesson isn't specific to GCP — it's that monitoring what you depend on but don't control is just as important as monitoring what you own.

100% agree, I've seen on Twitter and HN small players facing similar issues with no recourse and response from Google. I don't know what kind of place they are trying to build there.

They got TK to woo the enterprise customers who were forced to be hostage to OCI. But it seems they are still doing opposite of hostage here.

This is the bigger point of all of this. Scary.

> Don't expect Google quality from the name.

It sounds exactly like what I have experienced in terms of Google quality over the decades.

What/who is TK? What is OCI?

Thomas Kurian (GCP ceo)

OCI is oracle cloud infrastructure.

That Google poached from Oracle Cloud, fwiw.

The tired and disappointed angel on the right shoulder was still whispering “don’t be evil” while the devil on the left shoulder, leaning over, poked him in the ribs with the pitch fork and told him to shut his yapping. “Think about all money we can make, this is the guy we need !”

All google products work like this. Should never be used for anything critical.

Yeah, sadly know that from being burned from one of their depreciations. In fact, 2-3. But you live and you learn. And it is better to learn from other's mistakes always.

Hasn't theGoog acted this way of quick to suspend accounts well before Gemini? I like to bash on LLMs as much as the next guy, but this seems very much like the memory of a gold fish. Or, you are just too young to remember pre-LLMs???

Haha, no. I know Google bans anyone randomly with usually no recourse in sight. I just wanted to take a dig at how bad their LLM is too while we were at it and thinks like Google themselves which is not surprising.

They probably do use similar tech to make some of these decisions, though. And they always have done that as well.

You missed the humor part and focused on the tech part, it seems.

This feels like google applying the same anti-spam mindset everywhere: detect risk, ban first, ask questions later.

That seems to be the case. But as we see it backfires. Railway is very public, but we know at hacker news google has been doing this kind of thing for quite some time now.

yep, last year they deleted UniSuper private cloud too

It's pretty stupid that big customers like Railway are not somehow protected from this.

I think all customers should be protected by at least one CSR doing a quick approve before banning the account.

> It seems like they use Gemini 3.1 Pro to run their production decisions.

They said they are already using Gemini 3.5 Pro internally.

Then it's a bad endorsement for Gemini 3.5 pro too. But jokes aside, I think they need a customer centric thinking instead of a self-centred one they seem to harbour even before TK joined (not everything can be blamed on him although it should be his responsibility now).

Google? Customer-centric? The closest thing to that is their cloud division buttering up some big name clients.

Other than that, Google prefers to act like "customers" are some kind of unfortunate rash they can't quite seem to get rid of, but would love to do so.

Yup, updated with the article I mentioned by Steve Yegge. Still holds true today.

GCP was never known for their support and deprecation of services was always a huge risk. Its very sad because its actually a quality product. They should easily be the number 2 provider. Azure is extremely unreliable and their documentation is subpar. GCP being in 3rd place is more of their doing.

i wouldnt really call it a product without support, let alone a good product.

its a nicely design hobby, that somebody could make a good product out of, by following the same abstractions

> Don't expect Google quality

Google has an extremely poor reputation. Why are you thinking differently to that?

At least from the outside TK seems to be doing well given GCP's growth. My completely uninformed assessment is that he stepped in as the disciplined adult in the room to override Google's otherwise lackadaisical approach to enterprise. (Clearly still some ways to go, as this incident shows.) Now, that may have created a culture that is at odds with the rest of Google, but it was probably required to become a "serious" enterprise org.

That said, did OCI, being an Oracle division, have a culture worth destroying? On the other hand, I could see TK importing that culture into Google...

I think GCP has been giving out lucrative contracts with lots of discounts and credits which has led to this growth. I've seen many companies jump the ship cause they got 100/200k 2/3 year credits which is a lot for many of them at that stage. They also have their partners migrate them at their own expense. So it is essentially free money for those companies. I know a prominent entertainment content provider who went from AWS to Azure and then from Azure to GCP just because of this in 6-7 year. Plus the GPU hunt nowadays along with everyone producing code has definetely helped every cloud provider and even dedicated server providers/VPS (most of them dont have cheap options availaible). So how much of the growth has been organic vs TK led, hard to know.

If I think about it you might be right though some of this comes from Google's disdain for its customers be it hardware/software/cloud. So it is kinda inherited. One of the main reasons most good cloud consultants ask their clients to stay away from GCP is because of their depreciation policy. Plus stuff like this has been common in GCP.

So TK brought it to fix it instead of fixing it has brought the toxic OCI culture too which in the end brings them down even more.

I've been part of OCI for 10 years now. TK did nothing to the culture in OCI. Virtually everything that OCI was and has become has come from its own leadership (tone very strongly set by both Don and Clay, in their own ways). Mostly we lost that initial scrappiness of "do it now, ask for forgiveness afterwards" that is inevitable when you shift from a business with no customers, to one with a large and growing customer base. You can't exactly just fling stuff to production and see what happens any more.

opinion my own, yada yada yada.

This is just typical anti-Google FUD. Been on GCP for nearly a decade and as do my peers. Sure, you hear about a few stories like these and in this particular case with Railway, I would actually wait to see what caused the trigger for the suspension - both from Railway and from GCP. But, this sort of thing happens with every cloud provider including AWS (you can google for the same thing "AWS shut down our account with no warning") and you'll find tons of stories like these.

As a former GCP consultant, I can share that these sort of shut downs aren't random and it's usually due to the customer not being compliant - that breaks cloud compliance requirements for the big clouds, so automated systems flag it. Eg. Someone serving CP on their CDN, for instance.

The Railway incident report also doesn't directly address this at all other than:

    May 19, 22:22 UTC - P0 ticket filed with Google Cloud. Railway's GCP account manager engaged directly.

So, I would actually like to know more (What did the account manager say exactly?) before I just simply jump onto the Google hate train because it's cool to do so.

> (What did the account manager say exactly?)

I doubt we'll ever know, especially if it makes Google look negligent (ie. not reaching out to the customer first before restricting their production account)

Whatever the account manager said didn't inspire confidence that this wouldn't happen again.

Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover.

> Whatever the account manager said didn't inspire confidence that this wouldn't happen again.

Or..they couldn't remain compliant with one of the strictest cloud vendors' policies.

Yet their account was restored so clearly GCP themselves disagree with you.

Or they fixed the issue to remain compliant so GCP would restore access. Again, I know it's fashionable to hate on Google here, but there are always 2 sides to every story.

That’s not what you just claimed. I’ve quoted you below:

> Or..they couldn't remain compliant with one of the strictest cloud vendors' policies.

Which is it? They can comply or they can’t? Both sides can’t be true at the same time.

Just saying either way is possible unless both sides of the story are out in the open. You seem to start from a point of view of malice, I'm simply suggesting possible alternatives.

Except one way isn’t possible because the facts and logic tell you it’s not.

Being open minded is good but don’t keep it so open your brain leaks out.

> As a former GCP consultant, I can share that these sort of shut downs aren't random and it's usually due to the customer not being compliant - that breaks cloud compliance requirements for the big clouds, so automated systems flag it. Eg. Someone serving CP on their CDN, for instance.

If this was the case it would obviously be horrific. I did check to see, and I noticed that Railway is not listed as an ESP who sent any reports to NCMEC / CyberTipline in 2025, which seems like the wrong number for a hosting provider. Maybe they just have absurdly good customers.

https://www.missingkids.org/content/dam/missingkids/pdfs/202...

This is called "victim blaming" fwiw

> Don't expect Google quality from the name.

Google “quality”? We’re talking about the same company that has killed off dozens of useful apps? The company that’s made 6 different chat apps? The one who will kill user accounts with no recourse or person to call?

GCP literally had to spend the last 5 years trying to convince enterprises everywhere that they were nothing like Google proper. I’m not sure the last time you left Silicon Valley, but Google’s name across the test of the US was synonymous with flaky commercial products that might not exist a year from now.