Early in my career when in a discussion with my manager at the time, he threw out the "planes with two engines have twice as many failures as single engine planes" line.

So in my now close to 30 years of experience later, working on systems expected to run 24/7/365, I can't tell you what percentage of failures and outages were caused by the redundancy/fail-over/etc software and hardware layered into a system, but its definitely a very significant part of the problem. Frequently because its just that a "layer" someone bolted on, even if that layer was some software engineer designing for "web scale". All the edge cases are frequently not completely thought out, and when you hit one its a lot harder to recover, than simply restarting a single service running on a single server.

I once managed a thoroughly non-critical server that lived in a closet. In the couple years I managed it, we had quite a few failures due to the UPS tripping and no unscheduled downtime at all from any other reason.

I’ve had massive problems with fancy (HPE, IIRC, but it wasn’t called HPE then) raid arrays caused by the controller being a POS. Using plain mdraid would have been far more reliable.

Sometimes I wonder if all the effort people put into protocols that run Paxos and Raft could be better spent running a small number of coordinator servers with manual failover. I’m suspicious that, in many use cases, leader elections cause failures more often than the leaders themselves fail.

>> "planes with two engines have twice as many failures as single engine planes"

Sounds like someone didn't understand redundancy...

> "planes with two engines have twice as many failures as single engine planes"

There has to be a smart reply to that, like: 100% of pilots of single engine planes don’t make it home after a single engine failure. 100% of pilots of dual engine planes do make it home after a single engine failure.

Planes are natural gliders and you can recover from engine failure. It's not like they just fall out of the sky.

say for the zoomy lawn darts we call fighters.

pushes glasses

If you lose an engine on a twin engine plane, you have to decide if you have an emergency or not. If you lose an engine on a single engine plane, you have to decide where you can make an emergency landing.

This is a big lesson to heed. Sure, sometimes a startup really needs extravagantly scalable infrastructure. For the 99.9% of startup, no, you don't.

So the odds are you don't. Keep it very simple, very small. I can't highlight this enough. A few boxes will get you very far. If you ever actually hit the limits, throw a big party to celebrate success and only then start to think about making your infrastructure a bit (just a bit) more complex.

We have 10 petabytes of storage and a ~terabyte scale Postgres instance. It runs on a few racks and we get a respectable amount of traffic. It’s not “WeBScALe” but it’s not peanuts. Better uptime than AWS and our operational expenses are negligible. People forget how insanely powerful and cheap hardware is nowadays. Now, you can buy a 64 core chip for $7k. That alone could handle some serious traffic.