News for hackers. Stuff that matters.

A comment on A terrible, horrible, no-good, very bad day at Slack. Parent comment

Re:A terrible, horrible, no-good, very bad day at Slack by input_sh on Tuesday July 07, 2020 @03:52AM (#23756726)
To add to that, Netflix has open sourced a tool called Chaos Monkey which randomly terminates production instances so that you can be sure that your overall system is resilient to instance failures.
[ Parent | on HN ]
Re:A terrible, horrible, no-good, very bad day at Slack by brazzy on Tuesday July 07, 2020 @03:56AM (#23756747)
I don't think that would have ever provoked this error mode, though.
[ Parent | on HN ]
Re:A terrible, horrible, no-good, very bad day at Slack by harpratap on Tuesday July 07, 2020 @04:12AM (#23756837)
It would have. Their haproxy-server-state-management also comes under the scope of Chaos testing. Which was failing but they never set up alerts for it. So they could have seen that their chaos engine killed this process but they never got notified.
[ Parent | on HN ]
Re:A terrible, horrible, no-good, very bad day at Slack by adrianN on Tuesday July 07, 2020 @03:59AM (#23756765)
It might have prevented this failure by killing stale HAProxy instances...
[ Parent | on HN ]