This might be an unpopular opinion but,

Going multi region on AWS should be safe enough.

If a multi region, multi service meltdown happens on AWS, it will feel like most of the internet has gone down to a lot of users. Being such a catastrophic failure, I bet the service will be restored pretty fast, not in 3 days.

You could go multi cloud though. But when half of the internet struggles to work correctly, I’d not feel too bad about my small business’ downtime.

> If a multi region, multi service meltdown happens on AWS, it will feel like most of the internet has gone down to a lot of users. Being such a catastrophic failure, I bet the service will be restored pretty fast, not in 3 days.

Additionally, from a "nobody ever got fired for buying IBM" perspective, you're unlikely to catch much blame from your users for going down when everyone else was down too.

Yeah, AWS has never had a global outage, Google has had 2 now.

I'm not sure where you get this idea, but AWS has definitely had global outages. 1.5 years ago there were massive global issues with S3, causing even their own status dashboard to malfunction.

edit: I stand corrected. Apparently the S3 outage wasn't global, though its effects were.

Meanwhile, this outage has only really been noticeable to ops teams, since it doesn't affect existing nodes or anything outside GKE. It's definitely concerning and the fix is taking far too long, but as far as global outages go the impact is relatively minor.

On February 28th, 2017, S3 had issues in us-east-1, not globally. It just happens that most customers create their buckets in us-east-1. (It’s the default bucket creation region.)

Buckets creation/deletion is a global operation, namespaces are all global for it. When us-east-1 goes down, so does bucket creation. All other operations proceed merrily, but yeah as we see over and over again, whenever us-east-1 has an outage everything seems to shit the bed because they've built everything out in us-east-1. Hopefully us-east-2 is starting to eat in to that.

This is correct (or at least was a couple years ago).

Correct, but quite a few AWS services are built on S3, and some apparently weren't using buckets per region.

That only affected buckets on us-east-1.

I worked at a company that was highly dependent on us-east at the time. We survived. This is very different

> You could go multi cloud though.

Multi cloud is almost always more pain than gain. You’d spend time and effort abstracting away the value that a cloud provider brings in canned services.

Hell, multi region is often more than many workloads need.

My entire infrastructure is on k8s which should make multi-cloud easy...

Nope.

So you’re not using any managed services?

Other than kubernetes and object storage, no.

It could work then. We had the same setup. Also, we store everything on S3 but there is a lambda function that pushes new objects to Google Cloud storage too. We figured, storage is cheap so we can duplicate things and it will make moving to GCP a lot easier.

This was a while back though. Now we depend on a lot more AWS stuff.

At one point, I was maintaining clusters in AWS, Azure, and GCP. It was a more work than I anticipated, for very little benefit.

Cluster administration and identity management are unique to each provider and fairly challenging to get right.

First, determine your tolerance. How much does downtime per min cost you? How long can you be down? A lot of the time it may be cheaper to apologize to your customers than build a truly reliable system.

Then start looking at points of failure and sort them based on severity and probability. Is your own software deployment going to generate more downtime per year than a regional aws outage?

There are formal academic ways to determine what your overall availability is, but don't have those on hand. Suffice to say, it takes significant research, planning, execution, and testing to ensure a target availability. (See Netflix https://medium.com/netflix-techblog/the-netflix-simian-army-... ) if someone says they have 99.9% or better up time, they had better have proof in my mind (or a fat SLA violation payout)

People outsource to cloud providers not because they are cheap, but because managing infra in house is hard. Also move fast and break things.

Read AWS docs about availability, there are availability zones in a region, spread across those to minimize impact. Then test when something goes down. Fix/repeat.

People outsource to cloud providers because building / hiring / maintaining a team of decent engineers that provide a baseline industry bar of SLAs, SLOs is much more expensive than the eye watering costs of most cloud providers at even a IaaS level. Opex is tough.

Most companies I’ve been at don’t offer multi region support for their services because it’s too expensive for the service provided even in so-called “price insensitive” enterprises (you can’t just make up a price that’s huge, they do have budgets still) and most of their customers are unwilling / unable to pay more for the extra availability. If your software is designed from the start better, multi region failovers should be fairly inexpensive though. But all the bolted on “multi region” software I’ve seen has been hideously expensive and oftentimes less reliable due to the design being soundly not able to tolerate failures well.

The baseline is that it takes 12 dedicated people across the world to run a 24/7 support operation.

Considering that even tech companies hardly manage to have a pair of DevOps or Sysadmin, running one own infrastructure is completely out of question.

Most small companies on AWS with revenue outsource support to an MSP.

Any stats / surveys on this? Most smaller (< 50 employees) b2c enterprise-style SaaS companies I'm anecdotally familiar with barely have enough funds to hire ops engineers let alone outsource to MSPs (even if the MSP practices labor rate arbitrage). A lot of the criteria I'd wager may be conditions around pre-revenue status and funding conditions moreso than headcount. I'm trying to understand just how biased my own experiences and indirect experiences are with the wider US market and appreciate data.

I don’t know about pre revenue companies, but, if you have to be secure and compliant with regulations from day one, you’re not going to take a chance with having a bunch of devs setting up your infrastructure. My bias probably comes from working mostly with companies in highly regulated fields.

Besides, I’m assuming that the cost savings a small company can get from being billed under a much larger organization account would make up for it. That and having cheap shared netops support.

Small companies put every developer on call and here comes the support coverage.

Of course, that doesn't make them knowledgeable to run stable infrastructure and they will move on as soon as they realize they are being abused to work overnight and week end.

I am developer and I have the knowledge to run stable infrastructure at least on a small company SAAS scale. But, I wouldn’t go near a company that expected developers to be on call for netops work.

A company that doesn’t want the overhead of an MSP which in my experience is less than the cost of a full time Dev is not a company I’m going to work for. It would tell me a lot about thier mentality.

What's the math/logic to get to 12 people?

Timezone coverage, vacations, training, sick days, etc.

Two people per 6-hour timezone chunk, with one overlapping person splitting each chunk?

2 at UTC 0-8

1 at UTC 4-12

2 at UTC 8-16

1 at UTC 12-20

2 at UTC 16-24

1 at UTC 20-04

(Repeat)

That does bring up an interesting point. In hindsight, we already have duplicate infrastructure - a dev account and a production account. Why in the world was it decided to put both accounts in the same region?

The separate account was setup partially on my insistence but it was set up in the same region.

If needed, we could have done VPC peerings across regions. (https://aws.amazon.com/about-aws/whats-new/2017/11/announcin...)

In most of the cases I've seen so far with multiple regions / POP (4 different companies of different sizes and verticals) the software never being deployed in another reason is the fundamental reason. Reasons ranged from "we hard-coded us-east-1 everywhere and don't know how to test region independence it turns out" to "we have 3 weeks to try to make our software distributed between different regions but have nowhere near the resources to test it well let alone develop anything new for it." Automation is completely remedial and non-repeatable in most of these companies that rarely provision clean environments from scratch. By keeping regions the same between different environments, you reduce the number of possible differences.

Some services in AWS are also not available in others (I'm quite familiar with AWS Data Pipeline not being available outside the "core" regions like us-east-1, eu-west-1) and having services in one region make usage of resources in another region is a huge change when most developers outside ones with technology literate customers are under the gun to push features out fast over sound design. The matrix of services and configurations necessary to mix and match regions and availability zones is non-trivial if you make extensive usage of AWS services above the IAAS layer.

Also, cross-region VPC peering has a TON of limitations that rather annoying depending upon how well your network has been architected (by default in most companies outside enterprises with a deep bench of network engineers, this would be rated at "complete crap barely better than a typical home wifi network"). Heck, even though I'm non-dumb at networks I have to keep reminding myself of various cross-region VPC limitations when working with refactoring cross-region VPCs like where you can reference security groups, how to propagate Route 53 records, etc.

There are some significant differences region to region in AWS. Different numbers of availability zones, different latencies based on datacenter location, different types and availabilities of EC2 instances, etc. I think it makes sense to develop in the same region that your production service runs in just so you don't shoot yourself in the foot by deploying something that runs fine in your development region but doesn't run as well in your production region.

If all your competitors are down too, the equation changes, no?

Yep - If all my competitors are down the equation changes such that I _definitely_ want to be up.

Any trade comes to me if it's urgent, and I appear more professional as I've got a functioning system.

I might be an chancer running my entire system off an shoestring but being up when everyone else has taken a dive looks good.

Basically, you have three options. You can go full milti cloud, with all the expenses and overhead that entails. You can have everything run in one place but have a plan to switch to a backup system. Or you can look at the overhead and cost associated with those, and decide that it's not worth it. If the business can handle the costs and risks, then any of them can be a valid option.

Assuming you have architected multi-region I'm not sure how realistic that scenario is. AWS regions are mostly standalone, I have seen services go down in a region but never globally.

The most recent example that I can think of when something went down globally was Route 53 - the one service that AWS promises 100% up time for.

Are you thinking of the event in April? That was not a Route 53 outage, that was a BGP hijack.

https://blog.thousandeyes.com/amazon-route-53-dns-and-bgp-hi...

Citation needed?

See the reply to my original post. It wasn’t an actually Route 53 “outage”.

Very little, unless that small business has a big enough budget to design something that spans multiple clouds.

Ultimately it’s a risk/return decision.

“Is going exclusively with AWS/azure/GCP etc a better decision in reliability, financial and mantainability terms than complicating the design to improve resiliency? And will this more complex solution actually improve reliability?”

You need to have a secondary location for backups, not AWS. Stores copie of customer data, orders, accounts, balances and anything that is critical to the business.

If AWS ever screws up, you will be able to continue running the business even if it might take weeks to start over.

For live redundancy, you should have a secondary datacenter on another provider, but realistically it's hard to do and most business never achieve that. Instead, just stick with AWS and if there is a problem the strategy is to sip coffee while waiting for them to resolve it. Much better this way than you having to fix it yourself.

> What would a small business do as a contingency plan?

Depends on your definition of small. If it's small enough not to have a dedicated infrastructure team designing multicloud solution, then the contingency plan may be: switch DNS to a static site saying "we're down until AWS fixes the issue, check back later".

Otherwise it depends on your specific scenario, your support contracts, and lots of other things. You need to decide what matters, how much the mitigation costs vs downtime, and go from there.

Using Kubernetes is a good start. It should be easy to migrate your server between EKS and GKE. However data is trickier to move around. So you won’t be immune from all global outages.

We invested early in being multi-region on GCP as well as multi-cloud with AWS as a fully redundant option if it ever became necessary to fail over to them.

Paying off big-time now :-)

It sure did. It took us about a minute to fully fail over to AWS, and we are running there now for the rest of the day. We will promote GCP to be the primary cloud again tonight.

Without our multi-cloud set up we would have been down for over an hour. In our business this is not an option.

Roll the dice? What are the consequences for you, especially if you can shift the blame? What are the odds of you having better uptime rolling your own tooling? Can you afford the complexity of multi cloud? Is the added complication worth it?

Unpopular opinion: very little perhaps. You have to make sure all your your 3rd party dependencies have the same contingency plan as you do, but I guess it is going to be difficult to even figure that out...

Multi-Cloud-Service-as-a-Service application redundancy wrappers.

I wish I was only being tongue-in-cheek.

My entire production infrastructure is in GCP. What happened here has caused approximately zero impact to the availability of my service.

Infrastructure as code.

Terraform using AMIs plus chef recipes that work in the cloud and bare metal. Dont use AWS specific services.

This would allow you to spin over to another cloud provider , vsphere or bare metal with minimal work

Totally impractical for any small business and of questionable usefulness for large ones. You'd be giving up the largest benefit of platforms like AWS--ready to use services for common tasks--to avoid the infinitesimally small chance of AWS having some doomsday global outage.

To answer the original question: It looks like this issue was just a UI bug that affected the console, the service itself wasn't impacted. Events that do impact the service will be contained to a region, meaning you can mitigate it with proper redundancy across regions, no zany multi-cloud solution required.

Evidently you dont know how to do this well

That’s not how Terraform works. Each provisioner has separate syntax depending on the cloud provider. The template for AWS wouldn’t work on GCP or Azure.

Correct. However written properly you get 90% of the way there.

Disaster recovery by switching to another provider is simple when minimal centos/rhel images are used.

If all you are doing with your cloud provider are a bunch of VMs, you’re wasting money. There are much cheaper and simpler ways just to host a bunch of VMs than to use one of the major cloud providers.

Are you not using any of thier managed services and are you maintaining your own on VMs? If so, you have the worse of both worlds. You’re spending more on hosting and you’re not saving money on letting someone else do the “undifferentiated heavy lifting”.

Going to the cloud thinking you are going to save money is silly. It is more expensive to run things in the cloud, what you are paying for is the ability to scale up and down without a logistics problem.

It’s only silly if you’re not changing your processes. If you’re keeping the same people doing everything manually instead of automating, or honestly if you’re keeping the same headcount and if you’re not developing and depending on services provided by your provider and instead you’re self hosting services on VMs, of course you’re going to spend more.

But I blame most of the cost overruns when using cloud providers on “consultants” who think they are “moving to the cloud” when all they really know is how to setup a little networking infrastructure and know nothing about how to use the developer, Devops, or other hosted solutions.

Most ”consultants” I’ve run across only know how to do a lift and shift and do a one to one mapping of the on prem VMs and networking infrastructure to the cloud. They know nothing about automation, transforming Devops practices, or transforming development practices and architecture.

A lift and shift should only be the first phase.

If your company culture isnt devops no consultant will fix that.

Thinking that is the reason for consultants is a problem

Note : I worked directly for companies as an employee when building these stacks

Assume I’m not fully informed here. What does “written properly” mean? Sure I can move over route53 to cloud DNS easily but Firehose to PubSub? Lambda to Cloud Functions? DynamoDB to BigTable and moving the data?

The syntax for provisioning these doesn’t work that well for some find and replace to work. Are you using a templater to generate cloud-specific HCL from a template or something? Sounds like a pretty big problem to solve to me and not just something where you can win via discipline.

Quit using vendor lock in resources then asking why its hard to leave that vendor.

You’re always “locked in” to your infrastructure. People have this brilliant strategy of thinking they have this brilliant strategy of avoiding “lock in” and again they get the worse of both worlds. They end up spending more to avoid lock-in and they aren’t taking advantage of what thier provider has to offer.

At the end of the day, changing your underlying infrastructure is so risky and usually not worth the cost benefit analysis, it’s rarely done.

Oh, I see. You intend to use purely EC2 or GCE instances and just run everything one would run in a colo on them. All right.

That's a pretty manpower intensive way of operating. I think the fact that you get cloud agnostic this way is probably not worth it.

The requirement was to be cloud agnostic. This is one of the few ways possible.

In that case, why be in the cloud at all? I still think it would be cheaper to have a colo in two geographically dispersed areas on bare metal.

Then struggle like hell for the other 90%

I think you are downplaying minimal

I don't think OP was intending "minimal" to mean it would be easy to get to the stage where it's possible, just that once you've got all your infrastructure-as-code stuff set up correctly, you ought to be able to just be pressing buttons / running scripts and have your infrastructure up and running in another cloud provider.

Even when working in small companies with small infrastructure, I've kept recreation of infrastructure as one of my high priorities (one reason it really bugged me in one job to have to depend on Oracle Databases that I couldn't automate to the same degree.)

In my mind, it's not different from the importance of having, and testing restoration of, backups. If your infrastructure gets compromised somehow, or you find yourself up the creek with your provider, you've got to be able to rebuild everything from scratch.

The infrastructure has always been the easy part, as long as the company is willing to pay for multiple datacenters.

Then you realize a lot of software and databases can only run from a single instance, zero support for multi regions, and you're not gonna to rewrite everything and resiliency just can't happen.

No, with over 20 years experience and having done exactly this for several different companies and startups I pretty much have this process down.

And this is the reason that people end up spending so much more in the cloud - treating AWS like a glorified colo.