News for hackers. Stuff that matters.
Re:Railway Is Having a Major Outage by fontain on Tuesday May 19, 2026 @09:24PM (#48201899)
I’m very sympathetic and understand that decisions are easy to criticize in hindsight but leaving your database in GCP while moving everything else to your own data centres seems so backwards I can’t even begin to imagine how that could happen. Was this really an intentional design decision?
Re:Railway Is Having a Major Outage by arjie on Tuesday May 19, 2026 @09:37PM (#48202006)
I have exactly the same architecture. You can easily administer a postgres/mysql on your own infrastructure, but it's also the one thing where backups and availability are super strict. I can easily support multi-region in Google Cloud or AWS and that's way harder to do on-prem, and it's also hard to handle the replication story as safely as with Google Cloud. The hope is that GCP et al. give you safety and availability for the control plane stuff and you can run your data plane on-prem.

At $2m/mo spend, this kind of thing is insane. GCP has never been the most reliable of clouds but this is pretty awful. I would never have expected this.

Re:Railway Is Having a Major Outage by ahofmann on Wednesday May 20, 2026 @12:42AM (#48203196)
I have kind of the same architecture. I host multiple dedicated servers and vps instances in the Hetzner "cloud", but all of these connect to a few hosted databases by Hetzners web hosting packages for like 20 bucks a month. It sounds insane, but the one thing that absolutely needs to stay online, is the database, so not hosting this myself makes sense. And since Hetzner is apparently tuned their dirt cheap databases pretty well, we can hammer them pretty hard without any problems.
Re:Railway Is Having a Major Outage by ndneighbor on Tuesday May 19, 2026 @09:33PM (#48201967)
> decisions are easy to criticize in hindsight

I mean, the pain we have caused our customer ultimately proves you correct. That said, we made our decisions with the information and constraints that we knew in that moment in time. Railway has hosts in AWS/GCP/and co-los, so coordinating those workloads in a fully distributed manner would be ideal but end of the day, we didn't forsee that would just have our project get deleted just like that.

(Even if we did get assurances from them in 2024, that it wouldn't happen again, although we just got auto-rate limited the last time.)

Re:Railway Is Having a Major Outage by csw-001 on Tuesday May 19, 2026 @10:04PM (#48202223)
Thanks for getting things back up (genuinely mean that, btw). Upon logging back in I was prompted to promise I'm not deploying naughty things (I'm not). Was this in response to GCP detecting illegal (prohibited) behavior from something deployed via railway?
Re:Railway Is Having a Major Outage by ndneighbor on Tuesday May 19, 2026 @10:20PM (#48202322)
Actually, when I made the TOS check, I put that in Redis. That + the feature flags got reset.
Re:Railway Is Having a Major Outage by r_lee on Tuesday May 19, 2026 @09:42PM (#48202043)
could you clarify, did an automated process by Google delete a GCP project/account/resource(s)? like, what exactly were you seeing when trying to get access or see what happened?
Re:Railway Is Having a Major Outage by ndneighbor on Tuesday May 19, 2026 @09:58PM (#48202166)
They deleted our GCP proj. sans warning. Still working the details, but that's how this whole thing began.
Re:Railway Is Having a Major Outage by yen223 on Tuesday May 19, 2026 @10:06PM (#48202229)
this is easily explained by "database migrations are incredibly difficult and very risky"