Netflix Chaos Monkey Upgraded

Re:Netflix Chaos Monkey Upgraded by saulrh on Wednesday October 19, 2016 @11:38AM (#12744726)

If you can handle a server failing and you have good reporting, you can handle many of those random issues by simply rebooting the affected server. So I can see them taking out things like "load up the memory" or "load up the CPU". Logic errors (bad RAM, corrupted packets, high packet loss) are another story, but I don't know if V1 did those.

[ Parent | on HN ]

Re:Netflix Chaos Monkey Upgraded by fyshotshot on Wednesday October 19, 2016 @12:23PM (#12745183)

Seems to me those kind of issues would be a good way to test your monitoring. For instance verifying that the appropriate people are notified or automatic action is taken when within a reasonable amount of time after the CPU on the box starts spiking.

[ Parent | on HN ]

Re:Netflix Chaos Monkey Upgraded by drdrey on Wednesday October 19, 2016 @02:18PM (#12746264)

We have another system for outlier detection that can kill instances that start behaving badly in terms of CPU, response time, error rates, etc.

http://techblog.netflix.com/2015/07/tracking-down-villains-o...

[ Parent | on HN ]