Sure, the chaos monkey could kill haproxy-server-state-management but that wouldn't uncover the bug in question — it'd just demonstrate that without it running HA proxy's view of the world goes stale, which anyone would expect. Triggering the bug would require reducing the number of HA proxy slots below the number of webapps running for many hours. This is clearly something chaos engineering could do but IMO it's highly unlikely anyone would think to do this. If they thought of this they would also have thought about adding tests that caught such as issue long before the code went into production.
In my experience chaos engineering is often only as good as the amount of thought put into the things it does. Killing processes here and there can be useful but it often won't expose the kind of when-the-stars-align issues that take down infrastructures.
It looks like a classic lack of monitoring, as the article says. Alerting on webapps > slots, early exits, or differing views of the number of webapps up would have likely caught this.
No it won't. But it would uncover their missing alerts for a critical platform component. Their issue was exacerbated by the fact that state-management kept failing for nearly 12 hours and no one noticed.
I'm not totally against chaos testing. I just haven't seen it done well and think it's actually pretty hard to pull off (particularly the non-technical aspect of convincing people it's okay to let this thing go mad). I'd love to see how effective it was within Netflix.
...or just "availability testing" or simply "testing".
Yet, testing systemic failure modes at scale is way more tricky than shutting down some VMs or some network devices.
For example: saturating the uplink bandwidth on a whole datacenter.
It seems like it would take a superhuman level of foresight to catch this scenario for chaos engineering.