Re:Chaos Monkey Guide for Engineers
by tayo42 on Saturday February 16, 2019 @06:16PM
(#19181561)
something that i dont think gets talked about or addressed enough are partial failures. a whole instance dying is pretty much a solved problem and i think most people are aware of it and how to deal with it. if its not addressed people are just taking their chances at this point. a server being removed is an on and off kind of thing. something sporadic latency or corrupted data can cause some surprising and unpredictable issues.
[ Parent
| on HN ]
Chaos Monkey does that too. We used to have a separate monkey called Latency Monkey that induced network latency, because as you aptly point out, detecting if something is down is a lot easier than detecting if it is slow or intermittently down. Netflix has since incorporated those failure modes into chaos monkey: https://github.com/Netflix/SimianArmy/wiki/The-Chaos-Monkey-...