Always be proving that your infrastructure is resilient. Pain should be felt by stakeholders, not engineering.
When 24/7 means millions of dollars, hire a four-shift redundant geo-distributed team.
Re:Goodbye Heroku
by unethical_ban on Monday June 23, 2014 @11:10PM
(#7936016)
Thing is, a lot of places are doing that. The downside is the geo-distribution gives us Indian contractors with strange accents, inability to communicate and assert, and overall a huge communications issue by default.
Australia needs to step up its subsidies to IT outsourcing firms.
Hope for the best, and plan for the worst: Build resilient systems when possible, but why risk (or guarantee) outages during the day? OS upgrades, router swapouts, and so on can NOT take place during the day. That would be foolish in most industries.
Also realize that "just make systems resilient" is not an easy thing when multi-million dollar transactions are occurring on 30 year old code. If everything were greenfield, it'd be different. But it's not.