It was overly complicated for robustness. I tested multiple database crashes, servers going down in one zone, and it would always fail over just fine, because application code would connect to a slave if it could not connect to the master, so forwards would work, unless there were multiple outages in multiple data centers.
Now switching the floating IP to the backup forwarder did take about 5 seconds, and I could have made that better by putting a proxy in front, but I'm just moving the point of failure up a level, and it was never needed anyway.
It sounds like you’ve tested the simple failure modes, machines dropping off altogether, which is good; but I’d still be sceptical about the system’s robustness, because those aren’t the interesting failure modes, and the interesting failure modes are the ones that give the most trouble. Systems are easy when a node is there or not there, but when a node becomes unreliable (e.g. overloaded or network troubles of a variety of kinds), that’s when the real trouble happens, and when it’s most common that measures that were supposed to make things available actually make them unavailable. (This stuff is hard.) That’s why avoiding a distributed architecture until you actually need it can be well worthwhile.
I would use something like Caddy which manages it's own config and has an API for live updates.
SQLite for the user management and administration.
I would possibly have 2 Caddy nodes though, with a LB between them (instead of my potentially flawed heartbeat floating IP mechanism). The application would do an atomic update to both Caddy servers, if either failed, it would roll it back.
But you are still looking at around $30-40/mo for hosting, not saving much money, but making it much simpler and saving headaches.
It’s well documented that making systems distributed to achieve high availability very often actually reduces availability, because distributed systems failure modes are gnarly, and so there’s just a lot more complexity in the system as a whole, a lot more moving pieces to go wrong. (Relatedly, it is also well understood that error-handling code is about the buggiest code out there.)
For starters, I can’t think of any reason to separate web application server from forwarding servers, so that’d lop it by one.
Then I’d lop the three database servers and put them on the forwarding servers as well, with something like a simple streaming replica arrangement if I was keeping both of the forwarding servers rather than reducing it to one.
This is pretty much guaranteed to be more robust, because now requests are depending on one machine only, perhaps with a second just hanging out in the background paying attention to what’s going on; whereas in your distributed system, there are at least two machines that can take things down by failing (if the DB server you’re talking to goes down, it’ll still be a short time before the system sorts itself out and switches to another one).
I would love kernel live patching to be better understood, tooled and used; then you could safely keep your single machine up indefinitely, just gracefully restarting your HTTP servers and databases as needed. As it is, if you go down to a single box, you’ll want to do the occasional quick restart. Ten seconds of downtime every so often doesn’t feel too terrible to me, and if you happened to have an HTTP proxy in front of you it might even turn that into just a slow request. But yeah, the second server might be warranted so you can avoid this altogether: just switch which server is the primary and direct all traffic to it, and then freely reboot what is now the standby machine.