> "Making servers run as reliable as in cloud datacenters is really hard work and imposes additional cost."

Not really. It's easy enough to have dedicated server hardware that runs just as smoothly as cloud hosted hardware. There are many options for doing so. The key is to pay for support.

For example, if you go with Microsoft or Red Hat or Ubuntu servers, you can pay for support contracts to get help from experts in configuring systems. Such options can work out cheaper than cloud hosting (depending on the IT infrastructure requirements), with the added benefit of having hardware more directly under your control.

What happens when you run into trouble with one of the servers? Like it's suddenly stopped responding and you can't SSH?

For my systems built on EC2, I might not need to do anything at all for typical hardware failures, if I've set up EC2 auto recovery. It transparently relaunches my instance on another server. If I'm not using auto recovery, then I might just stop and restart the instance, in which case I'm also migrated to another physical server if the first one wasn't working. It has the same identity, by the way: same IP address, same disk contents, and all that, and all the machine perceived was an OS reboot (or maybe kernel panic followed by reboot, depending on the failure).

For stateless systems I don't even need to bother with that. I'll just configure it to spin up a replacement instance if the desired number of servers I want to have online is not met because one of them failed. When I have 100+ servers in a fleet, this is really convenient. I don't need to keep track of them individually, I just say: "This is the image I'd like to run, and I want to have 100 of them". Hardware failures that would bring down a normal physical machine don't need to involve me at all.

The old server hardware that's broken is now in the hands of EC2 to repair or swap out, and my server's back up and running in minutes, possibly with no action on my part.

Is something like that easy to achieve with colocation and support offerings with those vendors? Genuine question - I've never operated in colo or worked with those vendors. I realize that colo facilities can take care of repairing and swapping servers for me, but the benefit I get from the cloud is that I can perform those actions in moments with an API, or they can happen automatically, and I don't need to engage other humans. I can't imagine spending my time coordinating with vendors, getting on the phone, opening support tickets, waiting for people to do things, etc... right now I can do everything at the speed of computing with APIs and automation. It'd be tough to give that up.

> "What happens when you run into trouble with one of the servers? Like it's suddenly stopped responding and you can't SSH?"

You've got plenty of options. Reboot the server (using a local login, as the server is hosted where employees/support staff are physically close to it), restore the server from a backup, etc...

Let's put it like this, how do you think companies managed before cloud servers? Do you think they just had the attitude of 'Oh damn the server has gone down, nobody can do any work today'? I'd put it to you that contingency plans existed long before cloud hosts did.

If you start running EC2 clusters, you're more in the "cloud storage" dominion and not really in the "self hosting" area anymore.

Not everybody needs that and not everybody who does need it also realizes it before a major failure strikes out one of their machines and there is no working failover and their one admin is not reachable and when they finally reach him he has to be flown in, then they need to wait for replacement parts, which all in all delays recovery by several days.

Out of band management gets you remote console access and fallback to a local technician if there's a hardware problem.

Do you need to store data?

If yes, how is it replicated over multiple datacenters?

If yes, does failover work flawlessly?

> "If yes, how is it replicated over multiple datacenters?"

Depends on the database engine used. For example...

https://msdn.microsoft.com/en-gb/library/ms151799.aspx

>"If yes, does failover work flawlessly?"

Again, depends on the database engine used. To use SQL Server as an example again...

https://technet.microsoft.com/en-us/library/ms189590(v=sql.1...

Traditional clustering with SQL Server is a huge pain in the ass. When it works (most of the time) it works well but failures can get hairy quick. There's tradeoffs but if you can make it work AlwaysOn is so much easier to deal with from an ops perspective.

Hardware fails? It will start on other node automagically. May be we will need to switch psql instance. What kind of additional cost? Do you ever hit something really awful with k8s on self-hosted servers? What can go wrong except losing node?

So, you have redundancy and failover at the application level. You don't need reliable servers then.

Reliability is with respect to some SLA; it could very well be that one would _like_ to timeout more quickly than can be reasonably expected of cloud infrastructure, but well within the capabilities of hardware itself.

> You don't always need a fleet of tanks to deliver a box of milk bottles.

A bottle of milk filled with Gold is worth about $800k.

By extension the 6-pack is $4.8M

If tanks can fit seamlessly in the budget, we shall give tanks a serious thoughts! :D