Everything is built up and torn down via Terraform, and ZERO maintenance tasks are done via the console. Changes to infrastructure configuration are managed via CAB which approves PRs on Github which are automatically deployed via TF cloud.
We do monthly disaster simulations where we replace a _production_ environment.
I'll agree that AWS paints a very rosy picture of their own services, but there is substance underneath that PR.
It's really a matter of who within the company wants to know whats behind the curtain. I'd almost always vote for self hosting but its a hard argument internally when everyone thinks the cloud eliminates these issues magically.
Please compare apples to apples. If you are comparing to colo, I don't know of any colocation facilities which provide higher level services (say, Kinesis). They might provide some features like CDNs, but nothing very fancy.
You are left with a EC2 vs servers racked in some colocation facility comparison.
At that point, it's not even fair. I can run a terraform script and spin up a copy of the production environment in minutes. I don't have to create purchase orders and wait for hardware. I don't have to issue tickets. If any machine goes down it can be recovered in minutes, and it will spin up in another hypervisor, maybe even a different datacenter.
You don't have to spend time and resources with discussions on how the 'network topology' is going to look like, and then send people off to implement and wire up stuff.
You don't have to spend cycles diagnosing issues just to find out that there's a bad transceiver in a NIC somewhere. YOU DONT CARE. AWS cares and, even if they can't find an issue, stop/start, new hardware in minutes.
You cannot compare the two in good faith. Sure, it might make sense to host some workloads in a colo, but you are giving up a lot.
> I don't have to create purchase orders and wait for hardware.
> You don't have to spend time and resources with discussions on how the 'network topology' is going to look like, and then send people off to implement and wire up stuff.
This isn't that hard if you still have the expertise in house. Yes it is work and knowhow, but so is AWS networking/wiring, service interop.
> AWS cares
I think this is arguable and YMMV. Sure it depends on who you are and who you talk to.
Im not saying AWS isn't viable tech, and there are 30 ways to skin a cat at the end of the day if you end up with a skinned cat you like you're good to go. I believe most people don't need AWS to accomplish what they are looking for (subjective for sure). I also find it not the savings or experience that is in the marketing materials or sales pitches. This isn't a binary right or wrong decision, all we need is a skinned cat at the end of the day.
Anyone who thinks they planned everything correctly is just deluding themselves. Admitting you've fucked up, don't know how you fucked up yet but will one day need to fix the fuckup is very important in engineering.
On the other hand, it could be a very powerful combo.
They do not. We run many thousands of instances on AWS. Sure, given our scale, every week we get a couple of instance retirement emails. Usually they are issued many days in advance(sometimes, weeks). Per year, we may get a handful of instances that are suddenly unresponsive.
And we don't care. You know why? Because it's just a matter of issuing stop/start. Done! Server is back up, potentially even in a different datacenter, but it is none the wiser. It looks like a reboot. Even better, add an auto-recovery alert and AWS will do this for you, automatically. If part of an ASG, add health checks.
For the most part, we don't even notice when instances go down. Our workloads are engineered to be fault-tolerant. If a meteor destroys one AWS datacenter, it might temporarily take out some instances. So what? New ones will be back very shortly, all the while databases will fail-over, etc.
If these were physical instances, someone would have to do the maintenance work, purchase orders, wait for hardware to arrive, and so on and so forth. And, for most "co-location" scenarios, if your datacenter has issues, everything will go down. A single AZ in AWS has multiple datacenters, you might not even be affected if one goes up in flames.
But let's say you run a massive pet server farm and none of then can go down for any period of time. You have given names and everything, and you celebrate their birthdays. Cool. Run that on GCP then. They do auto-migration. I've never seen an instance go down.
> Do you understand that with AWS you still have to setup instances, manage security updates, permission and deal with disaster response
This is true. However, if you are running your own hardware, you have to do that IN ADDITION TO dealing with hardware and datacenter shenanigans, with either a specialized (and expensive) workforce, or a barely capable one that's shoehorned and doing double duty, with zero economies of scale, probably in a single data-center.
Anyone putting anything in "the cloud" and not designing for failure as a first principle is shooting themselves in the foot. I've encountered a lot of folks trying to do this and building systems that can't tolerate downtime and it starts long, difficult conversations always.
But I've never once needed to actually pay attention to those instance retirement emails or GCP auto-migrations. We've built to expect it and we let the provider handle that for us.
Time saved for everyone.
The advantage of AWS is that in theory you can script your infrastructure. Deploy to multiple zones, add hosts, remove hosts, deploy artifact, etc. Makes all kinds of things very easy. Dedicated SQL / Cassandra / DynamoDB offerings make that a few clicks to get going.
This saves people costs, but just so happens to be more expensive in terms of hardware costs.
You can replicate all of this in your own DC or whatever. You do need to invest in tooling, hardware, etc. which is worth it at the highest scales. Or worth it if you don't need everything and thus don't need to pay for it.
Personally, I think many large organizations can optimize by running with a mix of on-prem and cloud services.
Demand may be elastic but not totally elastic, so your base load could be cheaper in house. Storage is cheaper on premise for huge datasets.
You can afford enough staff to rack servers, but you aren't dominated by the lack of servers.
You buy more EC2 instances to scale your business, letting the product team deliver the product, and deploy on-prem to squeeze out more efficiency behind them. Yes, you probably don't use the full offerings of a cloud provider, but that also means less lock in at a cost of a little lower convenience and more staff.
Nothing is stopping you from spinning up ec2s for massive spikes in traffic just because your app is currently on-prem.
Nothing is stopping you from scaling first to AWS, realizing oh this is normal now, and then procuring hardware.
In terms of management burden, I will take the cloud infrastructure a thousand times out of ten.
Automation work is easily an order of magnitude of less effort in the cloud than on-prem. Labor dollars spent go exponentially further.
Setting up instances, managing updates, adding IAM perms is orders of magnitude faster/easier than dealing with rack-and-stack data centers.
Full downtime isn't that frequent, either. The Kinesis outage on Wednesday didn't affect any of our USE1 functionality, with the exception of maybe Cloudfront propagation.
A server is easily surviving over 1000 days and it's usually our side that needs to reboot.
"The cloud is just someone else's computer" - I used to say that with contempt, and now I say it with understanding.
We’ve had more issues with the collocation data center, like removing one of the good drives in a degraded raid-array and extremely expensive bandwidth.
I could hire four system administrators for that money to look after each of my four dedicated servers full time.
On top of that you first need to learn AWS. That thing is complicated - but I can set up a dedicated server in minutes because it's indistinguishable from my development environment.
I joke that dedicated servers are only expensive if your time and money are worthless. Because AWS is going to eat both.
Quote :
We are going to say we used four hours of labor. This includes drive time to the primary data center. Since it is far away (18-20 minute drive), we actually did not go there for several quarters. So over 32 months, we had budgeted $640 for remote hands. We effectively either paid $160/ hr or paid less than that even rounding up to four hours.
There is no way in the world that the only effort taken related to infrastructure is entirely encapsulated in ~ $200 per year.
Often you don’t need that much hardware.
If I am reading the post correctly, those are actuals, not estimates.
It isn't clear from this particular post, but considering Patrick refers to "overprovisioning" and having a "hot spare" I get the impression that they have designed their DC setup to minimize unexpected downtime. Sure, it can still happen, but with proper up-front planning what you call "perfect world" can become typical.
> or any ancillary costs outside of racking
As I see it from Patrick's post and my own colo experience, there are three groups of costs for colo that are baked in to the cost of AWS compute and storage: capital cost of hardware (Patrick has a specific line item for this); facility cost; and labor for maintaining the hardware (which is the cost under discussion. What do you think is significant, missing, and unique to colo?
You think bad things don't happen on AWS? You think that an experienced admin isn't going to spend significantly more time trying to communicate with an actual human at Amazon than fixing a problem on a physically controlled server?
This is incredibly naive.
I've never had AWS tell us that they can't find the server we're renting that suffered HDD failure. AWS has never cut us off from the Internet because a sysop typed a command in the wrong session.
And we find AWS support to be far quicker to respond, and far more effective, than our previous providers.
So, YMMV, but AWS is pretty damn good for our requirements.
Scalability is the thing and AWS its Viagra. Whoever has the biggest scalability get to pat itself in the back.
It's a sliding scale. What makes you money? What's your expected growth? What are your security needs? What experience does your team have?
In this thread, people at each end of the scale are 1. conflating different use cases (wordpress is not the same as a massive, scaled data ingest pipeline) and 2. are ignoring the hidden costs of both.
Is it only expensive if developer time is free or if it is, let's say, a quarter of a typical Bay Area senior engineer?
Salary levels are vary quite a lot across the world, even looking only at developed countries. At what $/hr level does managing your infrastructure become more cost-effective?
Which accounts for almost all hosting and startups. Most startups will never see more than a handful of users and yet microservices, lamdba, nosql etc must be deployed because ‘scaling and failover’, wasting their own and investers’ money for nothing. Only a small % will ever need any scaling and failover.
If aws were truly cheaper, we would be using it for everything; currently we mix which is far cheaper. Anything not fail critical that benefits from fixed price cpu/gpu, bandwidth and storage we host traditionally. We and our clients saved fortunes over the years. And for clients or projects that do not need such heavy lifting as aws, we simply host traditionally completely.
But ... most web applications are still IO bound, and labour is surprisingly expensive.
Running something on the open Internet is super annoying. If you properly subnet, or restrict to VPN, I can see it as useful but then you have a whole other host of problems to conquer.
If you're doing a static page or such, no issue, but the second you're hosting a webapp or need to have a database, yeah.
For sake, just spin up a single server somewhere cheaper and run it all in there and vertically scale up after you've put the appropriate monitoring to know when you need to scale up.
And put your money where it's worth it.
I want my team focused on building stuff that makes us money.
It's absolutely true that colo is cheaper than AWS for projects that can absorb the labour and failure related costs, but that usually is only true for hobby or very small business services.