Nonsense. It's very easy to have that kind of setup fail - remove one from the load balancer, take it down for maintenance, whoops we removed the wrong one from the load balancer. I've been in similar-sized companies using dedicated servers or AWS, so I've seen both sides. It's like how people feel safer when driving themselves than when being driven by a professional, even though a professional is overwhelmingly likely to be a better driver - everyone thinks "oh, there's no way I'd make that kind of mistake".
It is very easy to overengineer "high availability" systems - I certainly think things like STONITH and dedicated control planes are more trouble than they're worth, and running a system that you don't understand is a recipe for failure. But I'll take FreeBSD on a basic AWS setup - four EC2 nodes (multiple AZs), and an ELB - over bare metal any day. Physical server maintenance is not where I have a competitive advantage, and would mean more complexity to understand, not less.
I'm sorry - you're already missing the point.
There is no load balancer. There's no firewall. There's no services running except for sshd.
I guess I should qualify what I mean by "better", though. What I mean is, "I know exactly how this will fail and it won't be interesting or surprising. Or take any thought or time to fix."
It will be comprehensible. "FreeBSD on a basic AWS setup - four EC2 nodes (multiple AZs), and an ELB" is not. You have no idea (nor do I, nor do probably most folks at Amazon) the different and fascinating ways that will fail.
(disclosure: we use EC2 instances for backup DNS. We are not anti cloud or anti amazon)
This is from you and this is from the wikipedia article on NIH(not invented here) syndrome ...
>Not invented here (NIH) is the philosophical principle of not using third party solutions to a problem because of their external origins. False pride often drives an enterprise to use less-than-perfect invention in order to save face by ignoring, boycotting, or otherwise refusing to use or incorporate obviously superior solutions by others.
I am not saying don't have a copy, I am saying the if you have several copies the safe one is in the cloud.
I think there is an easy rebuttal that should be considered...
First, the "nines" rating of any service or resiliency is just gibberish. Go find the statistical likelihood of money market funds "breaking the buck" or of CDS blowing up - both in 2007/2008. Those had a lot of nines too and a lot of very smart , well qualified people attesting to those nines (in venues even more serious than IT).
A highly complex system becomes incomprehensible, even to the people that built it. Those nines mean nothing.
Second, you absolutely can build something more stable and predictable than Amazon precisely because you're the one that built it - which means that it is more comprehensible and fails more predictably and gracefully.
I don't care who does the calculation and how many nines they come up with - if you load FreeBSD on two bare metal servers and put them in two different datacenters and run them with any kind of conservative and cautious sysadminning you'll have a better solution. Yes, it will be more expensive.[1][2]
The standard closure to a comment like this is to refer to Talebs Black Swan and Antifragile books ... which you certainly should read ... but even more important is "Normal Accidents" by Charles Perrow[3] which I hope will convince you to stop looking for complex things that never fail, and instead look for simple things that fail gracefully.
[1] ... but we have a HN-Readers discount - just ask!
[2] You know who we are.
[3] https://en.wikipedia.org/wiki/Normal_Accidents