Re:Ask HN: How do you make sure your servers are up as a sin...
by jmstfv on Wednesday November 06, 2019 @04:15PM
(#21467389)
From time to time I ponder about all the terrible things that could happen: what can go wrong will probably go wrong so it helps to be (kinda) prepared.
Re:Ask HN: How do you make sure your servers are up as a sin...
by o-__-o on Thursday November 07, 2019 @12:04AM
(#21470566)
As a single owner (of also a monitoring SaaS) I am currently putting out a fire where my primary datacenter died. I have suffered data loss even with precautions I had in place, one server lost all filesystems which took out my git repos.
I have backups, I have clones. I've still been in partial outage for 4 days and will be fully up tomorrow when I literally drive my servers to a new DC. Surprisingly I have slept 8 hours every night and I'm not worried. I've been in contact with my customers and provided solutions to keep them alive. If they leave they're going to leave, nothing I can do. I am looking to make sure everything is built uniformly (the server that died was the last of an old build process) and invest in scaling to the cloud in a bit more efficient and orderly manner.
Re:Ask HN: How do you make sure your servers are up as a sin...
by tnolet on Wednesday November 06, 2019 @05:02PM
(#21467962)
Yes, however you are extremely and acutely aware of this. Your business depends on it.
This is not a long term solution though. You need to add people and take a break at some stage.
This is exactly how I run my (also monitoring) SaaS and it shows that the OP has learned how to minimize risk and prepare for the worst.
You are married to your mobile & laptop.
I also shows there is no free lunch: you need to invest in redundancy.