News for hackers. Stuff that matters.

A comment on A terrible, horrible, no-good, very bad day at Slack. Parent comment

Re:A terrible, horrible, no-good, very bad day at Slack by brazzy on Tuesday July 07, 2020 @03:56AM (#23756747)
I don't think that would have ever provoked this error mode, though.
[ Parent | on HN ]
Re:A terrible, horrible, no-good, very bad day at Slack by harpratap on Tuesday July 07, 2020 @04:12AM (#23756837)
It would have. Their haproxy-server-state-management also comes under the scope of Chaos testing. Which was failing but they never set up alerts for it. So they could have seen that their chaos engine killed this process but they never got notified.
[ Parent | on HN ]
Re:A terrible, horrible, no-good, very bad day at Slack by adrianN on Tuesday July 07, 2020 @03:59AM (#23756765)
It might have prevented this failure by killing stale HAProxy instances...
[ Parent | on HN ]