I'd say if you're not spending one day a month you're doing something wrong. Namely, you're not testing your back ups and disaster recovery often enough.

You're right, but given the context of the thread, it's worth pointing out that's not extra work that one doesn't also need to do with cloud hosting. Ought to test your backups and disaster recovery regularly, either way.

That's not managing the server. That's managing your application setup and not something that takes additional time for managed servers over cloud setups, because you still need all of those things for a cloud setup.

It also should not take anywhere near a day per server per month - if it does, then in a disaster scenario it means you're unable to recover at a reasonable pace.

>That's not managing the server. That's managing your application setup and not something that takes additional time for managed servers over cloud setups, because you still need all of those things for a cloud setup.

It is additional time because the cloud handles the whole class of "your hardware died" problems.

Most decent bare metal setups abstracts away the same class of "your hardware died" problems. E.g. my first rule on this is everything runs in containers whether I run in a cloud environment or on bare metal. The OS image is identical, and include tying into service discovery and a suitable orchestration mechanism (which can range from something trivially simple to, say, Kubernetes). Any modern server hardware has IPMI or something equivalent, which means you plug it in, configure the IPMI, configure network boot from a tftp/bootp or similar server holding your installation image, and from there on out you're deploying containers the same way as you would in a cloud environment, and back them up and arrange for failover the same way as in a cloud environment.

I've set up more than one hybrid setup where you didn't need to know if your workload was running in AWS or Hetzner or somewhere else, so we could use AWS for elasticity and Hetzner to keep cost down.

This is not a hard problem. And if people don't have the right skills in house, it's easy to outsource this (I for one used to make my living of automating setups like this and operating them on a retainer basis).

Containers running some work load dying is a lot different from a HDD going kaput on your production DB server. Or someone compromising your file system and encrypting all of your user uploads.

I don't think anyone is claiming it's a particularly hard problem. It's the opposite. We're talking about 1 hour a month. Even that little effort still ~$200 a month and a $200 a month managed DB instance is pretty beefy.

None of this is any different I a cloud setup. You still need to check the integrity of your backups. You still need to have appropriate failover. The effort is pretty much the same. In a competent setup it's automated, so you're not spending an hour a month per server. Not even for the production DB server, because you have replicas and setups to automate the switch, and setups to bring back snapshots from backups.

Been there, done that, many times, and I know exactly what it costs people, because I billed by the hour for it, and your cost assumptions are just way off.

Broadly speaking, managing 100 "cloud" servers is roughly the same amount of work as managing 2. Moving from managing 2 cloud servers to 100 is trivial. Moving from 1 server to 2 is an architecture problem. That architecture problem sometimes comes up when it comes to scaling too. it's the cattle vs pets problem. The last thing you want to find out is that someone ssh'ed in and installed a package that's required for your service to run, _during_ vertical resizing, and that sort of thing is far more common in systems with 1 server instead of many. If you're running containers, you would likely have even less overhead (and save money) by running on ECS/DigitalOcean/Azure App Service, and if you're running a big old monolith, you likely need > 1 instance for some sort of redundancy anyway.

> Broadly speaking, managing 100 "cloud" servers is roughly the same amount of work as managing 2

If this was the case, my billable hours when I was doing contracting would be about 1/10th of what they were. I lived very comfortably of troubleshooting for teams who had gotten themselves into a thorough mess with this attitude. In fact, I earned more from the teams who insisted on cloud setups because they rarely understood the operational issues with it, whereas teams who chose dedicated servers generally thought about operational concerns more.

> Moving from 1 server to 2 is an architecture problem.

Moving from 1 container to 2 is an architecture problem.

Moving from 1 server to 2 running those containers is an architecture problem with well established known solutions.

E.g. for starters you're assuming no containers. But putting the application in containers and leaving the host OS only for basic infrastructure setup is basic practice today if you're running your own servers.

Once you've set up a network boot source (tftp etc.) to network boot of an installer with a suitable setup script that you can trigger via IPMI, and a directory service and a basic orchestrator (be it Kubernetes or something else) on your network, it doesn't matter much if you have 1 server or a 100 - they come up and you put containers on them, and they look no different than a cloud service to the devs.

You're right that it's easier to take shortcuts if you have just a few servers, but it's just as easy to take shortcuts with just a few cloud instances - the number of pet containers I've seen over the years is terrifying.

> The last thing you want to find out is that someone ssh'ed in and installed a package that's required for your service to run

Which is why you don't provide ssh access to the host servers to anyone without an understanding of ops concerns, log everything, and do all updates via an automated setup of your preference, and why you regularly recycle the containers whether you run a cloud environment or dedicated servers.

The reality is that cloud systems do not at all make you immune to this - I've done year long projects to regularise AWS setups that were full of undocumented manual changes to bring everything into a terraform config for example. Often they're worse, because there are a whole lot of unobvious places to look for extra bits and pieces.

> and if you're running a big old monolith, you likely need > 1 instance for some sort of redundancy anyway.

Nobody here suggested a monolith. Nobody is suggesting you forgo redundancy. The. The point of this is that often you can put clear upper limits on the computation you will need to do for either your system as a whole or for a given subsystem, and you can guarantee that you will never need more than one server for a given part of the system.

E.g. a real example: I've worked on a system that did some processing of data about companies. We know this will always fit on a single system because the population growth of humanity is slower than the performance growth of a server and the total number of companies worldwide fits on a single system with several magnitudes to spare today, and the type of companies we were interested in is just a subset. At this point, if you architect a system like that on the basis of assuming you will need to resize, you will risk making choices that makes you far more likely to have to resize. E.g. all of the data I'm talking about can easily fit in RAM on a relatively moderate server now and forever, but the moment you start planning for partitioning the data you have added orders of magnitude of performance overhead for communication.

Properly assessing which parts of a system needs to be able to scale is at the core of architecture, and architects (or devs; it's terrifying how many places lack anyone with architecture experience) who are just planning for infinite scalability for everything is huge red flag to me. It usually means they don't understand their system. For a startup, especially, this is an existential question - preparing for unnecessary scaling has killed many startups.