Everything is built up and torn down via Terraform, and ZERO maintenance tasks are done via the console. Changes to infrastructure configuration are managed via CAB which approves PRs on Github which are automatically deployed via TF cloud.
We do monthly disaster simulations where we replace a _production_ environment.
I'll agree that AWS paints a very rosy picture of their own services, but there is substance underneath that PR.
It's really a matter of who within the company wants to know whats behind the curtain. I'd almost always vote for self hosting but its a hard argument internally when everyone thinks the cloud eliminates these issues magically.
Please compare apples to apples. If you are comparing to colo, I don't know of any colocation facilities which provide higher level services (say, Kinesis). They might provide some features like CDNs, but nothing very fancy.
You are left with a EC2 vs servers racked in some colocation facility comparison.
At that point, it's not even fair. I can run a terraform script and spin up a copy of the production environment in minutes. I don't have to create purchase orders and wait for hardware. I don't have to issue tickets. If any machine goes down it can be recovered in minutes, and it will spin up in another hypervisor, maybe even a different datacenter.
You don't have to spend time and resources with discussions on how the 'network topology' is going to look like, and then send people off to implement and wire up stuff.
You don't have to spend cycles diagnosing issues just to find out that there's a bad transceiver in a NIC somewhere. YOU DONT CARE. AWS cares and, even if they can't find an issue, stop/start, new hardware in minutes.
You cannot compare the two in good faith. Sure, it might make sense to host some workloads in a colo, but you are giving up a lot.
> I don't have to create purchase orders and wait for hardware.
> You don't have to spend time and resources with discussions on how the 'network topology' is going to look like, and then send people off to implement and wire up stuff.
This isn't that hard if you still have the expertise in house. Yes it is work and knowhow, but so is AWS networking/wiring, service interop.
> AWS cares
I think this is arguable and YMMV. Sure it depends on who you are and who you talk to.
Im not saying AWS isn't viable tech, and there are 30 ways to skin a cat at the end of the day if you end up with a skinned cat you like you're good to go. I believe most people don't need AWS to accomplish what they are looking for (subjective for sure). I also find it not the savings or experience that is in the marketing materials or sales pitches. This isn't a binary right or wrong decision, all we need is a skinned cat at the end of the day.
Anyone who thinks they planned everything correctly is just deluding themselves. Admitting you've fucked up, don't know how you fucked up yet but will one day need to fix the fuckup is very important in engineering.
On the other hand, it could be a very powerful combo.
They do not. We run many thousands of instances on AWS. Sure, given our scale, every week we get a couple of instance retirement emails. Usually they are issued many days in advance(sometimes, weeks). Per year, we may get a handful of instances that are suddenly unresponsive.
And we don't care. You know why? Because it's just a matter of issuing stop/start. Done! Server is back up, potentially even in a different datacenter, but it is none the wiser. It looks like a reboot. Even better, add an auto-recovery alert and AWS will do this for you, automatically. If part of an ASG, add health checks.
For the most part, we don't even notice when instances go down. Our workloads are engineered to be fault-tolerant. If a meteor destroys one AWS datacenter, it might temporarily take out some instances. So what? New ones will be back very shortly, all the while databases will fail-over, etc.
If these were physical instances, someone would have to do the maintenance work, purchase orders, wait for hardware to arrive, and so on and so forth. And, for most "co-location" scenarios, if your datacenter has issues, everything will go down. A single AZ in AWS has multiple datacenters, you might not even be affected if one goes up in flames.
But let's say you run a massive pet server farm and none of then can go down for any period of time. You have given names and everything, and you celebrate their birthdays. Cool. Run that on GCP then. They do auto-migration. I've never seen an instance go down.
> Do you understand that with AWS you still have to setup instances, manage security updates, permission and deal with disaster response
This is true. However, if you are running your own hardware, you have to do that IN ADDITION TO dealing with hardware and datacenter shenanigans, with either a specialized (and expensive) workforce, or a barely capable one that's shoehorned and doing double duty, with zero economies of scale, probably in a single data-center.
Anyone putting anything in "the cloud" and not designing for failure as a first principle is shooting themselves in the foot. I've encountered a lot of folks trying to do this and building systems that can't tolerate downtime and it starts long, difficult conversations always.
But I've never once needed to actually pay attention to those instance retirement emails or GCP auto-migrations. We've built to expect it and we let the provider handle that for us.
Time saved for everyone.
The advantage of AWS is that in theory you can script your infrastructure. Deploy to multiple zones, add hosts, remove hosts, deploy artifact, etc. Makes all kinds of things very easy. Dedicated SQL / Cassandra / DynamoDB offerings make that a few clicks to get going.
This saves people costs, but just so happens to be more expensive in terms of hardware costs.
You can replicate all of this in your own DC or whatever. You do need to invest in tooling, hardware, etc. which is worth it at the highest scales. Or worth it if you don't need everything and thus don't need to pay for it.
Personally, I think many large organizations can optimize by running with a mix of on-prem and cloud services.
Demand may be elastic but not totally elastic, so your base load could be cheaper in house. Storage is cheaper on premise for huge datasets.
You can afford enough staff to rack servers, but you aren't dominated by the lack of servers.
You buy more EC2 instances to scale your business, letting the product team deliver the product, and deploy on-prem to squeeze out more efficiency behind them. Yes, you probably don't use the full offerings of a cloud provider, but that also means less lock in at a cost of a little lower convenience and more staff.
Nothing is stopping you from spinning up ec2s for massive spikes in traffic just because your app is currently on-prem.
Nothing is stopping you from scaling first to AWS, realizing oh this is normal now, and then procuring hardware.
In terms of management burden, I will take the cloud infrastructure a thousand times out of ten.
Automation work is easily an order of magnitude of less effort in the cloud than on-prem. Labor dollars spent go exponentially further.
Setting up instances, managing updates, adding IAM perms is orders of magnitude faster/easier than dealing with rack-and-stack data centers.
Full downtime isn't that frequent, either. The Kinesis outage on Wednesday didn't affect any of our USE1 functionality, with the exception of maybe Cloudfront propagation.
A server is easily surviving over 1000 days and it's usually our side that needs to reboot.
"The cloud is just someone else's computer" - I used to say that with contempt, and now I say it with understanding.
We’ve had more issues with the collocation data center, like removing one of the good drives in a degraded raid-array and extremely expensive bandwidth.