> I think the more PaaS like services such as Heroku, Lambda, Fargate, Google Cloud Run do better realise the less maintenance story but not cloud generally.

Completely agree here. Running an EC2 Spot instance + ebs volumes for 730 hours a month is a total waste of money, but running RDS behind fargate and ECS with an ALB is likely to save you time and money by month 2.

I broadly agree with the statement that owning an OS in general is toil that applies any of these IaaS/VPS/BM scenarios.

Owning a BM server different toil - server parts fail, the network it attaches to needs control and it fails too, firmware needs updating more regularly than ever, DC space needs managing over time etc. For a small number of machines maybe this is NBD. For thousands of machines this is just grunt work which while automated, still needs change management and control - rebooting the whole fleet in the middle of the day definitely opens doors in your career.

Doing everything you did in the DC in the Cloud is absolutely the worst way to adopt Cloud. Owning an OS is a non-goal, you’ve gotta climb to a higher abstraction - workloads, and quit caring about machines. This is where most companies fail.

> A ec2 instance or other vps requires the exact same maintenance as a bare metal server. They are essentially the same except one is virtualised and the other isn’t.

Ani. If the hardware you’re running on is dying. In ec2 you stop it and start it. It’s on new hardware. If you run bare metal you’re screwed.

Disk is dying ? Don’t matter cos your data exists multiple times over in AWS elastic storage. With bare metal you got to shut down and replace.

The cost of managing hardware is gone with ec2.

Software on the other hand yes that is the same amount of effort.

The cost isn't gone. It's just included. Fact is, at scale, it's still cheaper to do it yourself. You just need to reach the scale it's worth paying people to do the managing.

I agree, with the caveat that at scale it's cheaper to do it well yourself. If that weren't true, then Amazon would not be making money on EC2.

If you don't have sysadmins and network admins with the right experience, you can easily find yourself in a bad spot with single points of failure, servers that can't be easily replaced, oversubscribed PDUs, misconfigured switches/routers... and any number of other problems that aren't occurring to me right now.

If you don't have sysadmins with the right experience, there is a vast number of companies that can do this for you on a fractional basis on retainer. I was one of them.

The crossover point where cloud is more expensive is really low even if you have zero in-house experience. Exactly where it is depends on your amount of egress, as that is where AWS in particular really takes advantage of you.

This is very true. You also probably need the scale having multiple install locations makes sense.

We went dedicated early and while it didn't make sense at the time we now run at way lower cost than the competition could dream of.

> In ec2 you stop it and start it. It’s on new hardware. If you run bare metal you’re screwed.

You mean, you reimage? That is the slow step, you reimage, and plug the new server. Wait a bit, and your service has one more server.

> With bare metal you got to shut down and replace.

You take the disk out and plug a new one. You don't turn things off because of a disk.

No doubt, those are costly. They are also rare (disk failure is less rare, but still rare).

> You mean, you reimage? That is the slow step, you reimage, and plug the new server. Wait a bit, and your service has one more server.

No. When you /stop/ an EC2 instance, and /start/ it again, it moves. You do not need to reimage. This is even requested from AWS when they are having hardware failures and need to move customers off so they can decomission the hardware. They request you stop / start the instance, if you do not do it by the due date they do it for you.

> You take the disk out and plug a new one. You don't turn things off because of a disk.

If you have a storage array sure. But if you're getting bare metal hosting from a provider, you're not always getting hot swappable storage arrays.

> No doubt, those are costly. They are also rare (disk failure is less rare, but still rare).

It was 1 example, obviously there's many different hardware issues that can go wrong.

> If you have a storage array sure. But if you're getting bare metal hosting from a provider, you're not always getting hot swappable storage arrays.

If you have any server-level hardware bought in the last 20 years or so, it will have the drives in hot-swappable bays. If you then choose to not set it up in RAID, it's just incompetence.

> If you have a storage array sure. But if you're getting bare metal hosting from a provider, you're not always getting hot swappable storage arrays.

If you're getting bare metal hosting from anywhere including your own colo, you have failover and the ability to order replacements while your system is still running. This is only an issue if you're architecture is fundamentally flawed, in which case you're likely to mess things up whether you're on bare metal or in a cloud.

Not really If you running bare metal and have a San you can easily change what it is pointing to. Also most bare metal servers have redundancy and disks can be changed with 0 down time.

So now you're throwing a ton more money at something, costing you in purchasing the bare metal, but also the cost to maintain it. If you want to build compariable redundancy as you get with EC2 it will cost you. It wont be cheaper.

I have no idea what you think the costs of this is. I have managed setups like that. Every year we priced out what a cost to EC2 would cost us, and every year it was about 3x the cost of running our own, with my time - accounted for to the hour - of running the system added in. Every year we also priced out Hetzner and a few other options. After a years Hetzner eventually won out (colo space in Germany was cheaper than where we were in London). So we tied Hetzner servers into our private cloud layer, and migrated containers and shut down servers as it fit into our schedule. Not having to physically go to the colo's to swap drives now and again saved me an average of maybe 2 days a month to deal with several racks worth of hardware.

> Every year we priced out what a cost to EC2 would cost us, and every year it was about 3x the cost of running our own

> we tied Hetzner servers into our private cloud layer, and migrated containers and shut down servers as it fit

Building your own services on top of AWS is always going to come out more expensive. EC2 + EBS volumes alone are going to be more expensive than going with hetzner (particularly if you're not looking at reserved instances, and not utilising spot for burst). You mentioned that you are building your own private cloud layer and migrated containers; the cost of building that out in the first place is likely enormous compared to building and running on top of fargate.

The cost of building out our private setup was my time for about a month.

At the time we didn't have a choice, as nothing like Fargate existed, but today it's also easier to do setups like the one we did. It mostly involved rsyncing base images over, rsync and a super simple storage service for backups, a LDAP based directory service, and and a thin layer over vzctl (first) and docker when that became an option, coupled with a VPN setup to tie our locations together, and a reverse proxy setup that did dynamic lookups in our private DNS fed from LDAP.

It is hard to do as a multitenant public service, it's trivially easy to do as an internal tool that needs to support only exactly what you need.

I've built out setups like this for a number of clients since, and it's typically 1-3 months of work to automate pretty much everything depending on complexity, and so it pays for itself quickly from a very low scale.

The first company I did this at wouldn't have been profitable at all if we'd relied on AWS

>Disk is dying ? Don’t matter cos your data exists multiple times over in AWS elastic storage. With bare metal you got to shut down and replace.

Not really, this is what RAID is for

Sure, raids some drives together, you still have downtime when you switch it off replace the drive switch it on wait for it to rebuild.

Or you can fork out the extra cost to have hot swappable.

If you're saying: "I dont want to pay $300 / month for EC2 instances at AWS when I can get the same hardware specs for $150 at X"

Chances are you're getting a shitty old desktop or barebones rack that lacks features like hot swap and raids.

Do you believe you're getting redundancy when you go with hetzner and a cheap desktop grade processor, ddr4 non ecc memory, and a consumer grade SSD?

There is no extra cost to have hot swappable unless you're considering buying consumer grade hardware, but consumer grade hardware is more expensive to host because it won't fit in 1U bays.

> Do you believe you're getting redundancy when you go with hetzner and a cheap desktop grade processor, ddr4 non ecc memory, and a consumer grade SSD?

Irrespective of how much I'd skimp on the hardware, I always have a HA setup, including on EC2, so it doesn't matter in any case.

Depending somewhat on the quality of the hardware in question. Hot swap capable isn't rare, but it's not going to happen at the bottom of the price continuum.

> A ec2 instance or other vps requires the exact same maintenance as a bare metal server. They are essentially the same except one is virtualised and the other isn’t.

Definitely not true. With a dedicated server, you need to handle backup and security yourself.

With ec2 you still need to backup. You still need to validate the backups. Security it still something you need to do since an instance is still just a VM. Same with s3 buckets etc. Google for public s3 bucket "breaches." You still need to apply patches, to configure access, to expand volumes, to configure VPCs and security groups.

As far as I understand, when you use an EC2 volume, it is already backed up for you. It is not the case for an OVH dedicated server. For instance, backing up your OS image is a lot more work with a dedicated server than with an EC2 instance.

That addresses one possible set of failure scenarios. If you think that means you have a backup that is guaranteed to be accessible to you whenever you need it, you don't have a backup. If your only backup is in the same cloud provider where your primary system is, you don't have a backup, you only think you do.

Any reasonable dedicated setup will involve imaging your server, and so the OS image is not something you need to back up - if it fails you reimage. If you even store the OS image on the server at all rather than network boot.

That's still a lot better than what you have with a dedicated server.

For you to lose your OS image with a dedicated server only takes your HDD to die.

For you to lose your OS image on EC2 (where you made a snapshot of your volume in one-click) would take a lot of shitstorm to happen at AWS -- as I presume that they backup across sites.

> For you to lose your OS image with a dedicated server only takes your HDD to die.

Only if you don't have a backup.

Why in the world do you think anyone would store their only copy of an OS image on a single server?

For systems I set up, to start with, the OS is mostly immutable, booted and updated transparently to match a master image. If it gets destroyed, we just image a new server. The applications all run in containers, based on images stored on replicated file servers. If they get destroyed, we just re-deploy on a different server (in fact, automatically redeploying is trivial).

Only the application data is unique to running servers, and that needs to be backed up just as much whether those containers run in a cloud environment or locally, and again it's trivial to have automation in place for the backup and re-deployment of that. Been there, done that many times.

> For you to lose your OS image on EC2 (where you made a snapshot of your volume in one-click) would take a lot of shitstorm to happen at AWS -- as I presume that they backup across sites.

For me to lose my data on any bare metal system I've run, multiple servers in at least two different data centres operated by at different companies would need to fail at the same time. This is not hard to set up, and it's a one of setup. You then need to test your backups, just as you need to with AWS - an untested backup is not a backup.

But your assumptions of failure scenarios is also flawed. You need to protect against e.g. disgruntled employees, hackers, bugs as well. If you rely on the same security to protect your backups as your main setup, you don't have a backup.

EC2 is great when you can justify the cost, but it does not remove the need for a proper backup policy and processes to test them.