Which is better: 3 DCs for 2 DCs worth of traffic, running the same stack but isolated failure domains... or 4 DCs for 2 DCs worth of traffic, that work in completely different ways, where 2 are left cold until needed. I can see some advantages of the latter, but not a lot.
This also isn't about fault tolerance at the code level like in, say, avionics. In those scenarios having multiple distinct implementations makes sense, but that's very different, you run all paths at the same time and check they agree etc.
My guess is a little of both. I would bet that the 1% is the unused cost, but that for periods that they are running the stand-in system it's much closer to normal. But from what I've heard of their infrastructure I'd also bet that stand-in is significantly cheaper in part because it had the benefit of hindsight and could resolve issues in the first system.
None of this really changes my original feeling though. A cell based architecture spreads the risk, failover is essentially built-in without being a special mode you have to put the system into and then recover from again, on cloud you can scale those up and down with traffic and they also take peaks from each other, and so on. There's a reason why there are very few cloud outages that cross all regions for a provider, and if you're paranoid you can run multi-cloud (as they already do).
And yes you are completely right a completely distinct implementation is probably a no-go, but I'd think you could deploy a similar codebase with feature flags, with some frontend differences. I'm speculating but pretty sure they didn't reimplement backend functionality for this.
What I would say is that it's considered to be much better than CockroachDB, but that there's no getting around physics. Spanner is subject to the same speed of light... but in every other way it does what it can do to improve things at every level. Google's use of GPS hardware for better timing is public.
Add the instability of the average cloud control plane during an outage and things get rather nasty rather quickly in reality. We’ve been through that with AWS’ numerous outages.
My company is victim of this mindset. We sacrificed tested and assured reliability on physical hardware in multiple redundant data centres for cloud and microservices. Mostly driven by Monzo’s claims, semi religious thinking and me too.
Reliability is measurably down, cost is higher, latency is higher and we cannot fail over to another provider because someone went to an AWS conference and started using all the toys. Even if we did get rid of those dependencies, during an outage the kubernetes control plane is so tied into the vendor that we’d be fucked anyway. Last time the ingress controller couldn’t change ALBs for example.
Only winner here is Bezos and the decision makers CVs. The losers are the customers. We live in the stupidest of times.
I don't know how common this is in retail banking but I'm happy to see Monzo do it.
But the scaling problem is still there, when they switched over I assume they would start both horizontal and vertical scale of k8 pods and maybe even bump up DB instances as well which could result in a small congestion at the start of the switch over
The EU activated the Digital Online Resiliency Act (DORA) last year (passed in 2023) which, amongst other things, mandates that financial services providers who run their core business functions in the cloud (i.e. Monzo) must be able to demonstrate that they are not entirely reliant on a single cloud provider. Otherwise, big, big fines.
Like GDPR, there are auditors that are responsible for signing off on EU DORA compliance. Also like GDPR, there are ways to check those boxes without _really_ checking them. This is probably why Stand-In only has 18 services in its GCP failover "region" instead of the 1K+ production services in AWS; it's just enough to keep critical services (payments/card networks, balances, ledger, etc.) operational and, more importantly, keep DORA auditors at bay.
This article IMO is more for other FSIs who _want_ to run _entirely_ in the cloud (there are very, very few that do). Financial services is a very risk-averse industry. Most providers have biz-critical systems running on mainframes still. (Mainframes are practically indestructible and are much more technically-interesting than people give them credit for, but that's another post.) Being able to tell AWS "yeahhhhh we can run this business on GCP and/or Azure in literally 10 minutes" is a money line for procurement to use during price negotiations. Or at least it was until RAM prices mooned hard enough to make their existing datacenter inventory suddenly very attractive.
Cloud-aside, pretty much every bank in the US has some architecture like this for their critical banking systems, except they are usually an old-school hot-cold (sometimes hot-hot) datacenter failover strategy. Historically failover has been done manually with a runbook (with some automation) and tested during DR/BCP days (sometimes more frequently). They, of course, have zonal/regional replication in place.
I used to work in fintech where we’d face the same requirements and know a bunch of folks from Monzo. You can fight the status quo only for so long until it becomes much easier to just do what you’re being asked and produce a totally separate service that solidly answers any audit you may face.
It seems like it would be better to invest in a platform that is distributed across multiple failure domains. Some companies call this a cell architecture. For Monzo this might mean true multi cloud, but they already have bare metal services for bank integrations so they are already hybrid in some respects.
It's not the decision I would have made, and I'm not sure I agree with their reasoning, but I'll also admit that I have not run a bank, I don't know all their constraints, and there are some good tech folks at Monzo. That said, I am an SRE at Google so I'd like to think I'm not completely naïve here.