Re:The SalesForce Outage: A Nail In The Coffin For SaaS?
by devishard on Wednesday June 01, 2016 @08:47PM
(#11819553)
I won't deny the role of luck, but I find that the more careful I am to keep my systems redundant, flexible, and rapidly deployable, the luckier I am.
I did have an issue once where a very large power outage that took out a data center. Total downtime was about 50 minutes, which consisted of:
1. Calling the hosting provider and ascertaining the problem.
2. Pulling the latest code onto the fallback server (from a different hosting provider).
3. Restoring the database from the backup server into a newly-created database server.
4. Changing the configs.
2 and 4 were able to happen while 3 was running.
As an aside, this is largely why specialized cloud services like AWS or AppEngine that promote vendor lock-in seem crazy to me. AWS can and does go down, and if your infrastructure is built around their tooling, you can't just provision a new box on a new provider.
Through luck or careful planning and good resources?
Once you have enough hardware to handle load, it all really comes down to risk mitigation.
Assuming it handles load, your service could probably run on a single server, with a single drive, on a DSL connection for years without any issue. I would say if that happens, it has less to do with your skill as an operator and more to do with the luck of not having a power outage, drive failure or backhoe operator dig up a line.
Can your service withstand your office/datacenter burning down? Or your datacenter being shutdown with all assets seized by the SEC (true story)? Or a massive earthquake or flood or hurricane that disrupts the entire city?