News for hackers. Stuff that matters.
Re:Principles of Chaos Engineering (2018) by KenanSulayman on Friday June 14, 2019 @03:00PM (#20185716)
We built a system called „friendly fire“ that nukes a server every 10 minutes. It has changed the mindset of all engineers and made our infrastructure missile-proof.

Funnily enough it also improved our latencies a lot (which I guess is mostly due to memory leaks et al.)

Re:Principles of Chaos Engineering (2018) by jaggederest on Friday June 14, 2019 @03:26PM (#20185939)
We used to boot ~3x the servers we needed, run a hard load on them for a while, performance test them, and kill the "weakest" 2/3rds. You can get a bunch of nodes on iffy hardware or greedy neighbors (or all on the same physical box) and see significant performance improvements that way.

Of course this was a decade ago, but I think the fundamentals are still sound, as far as being skeptical about the quality and longevity of your nodes in a virtual environment.

Re:Principles of Chaos Engineering (2018) by isbvhodnvemrwvn on Friday June 14, 2019 @05:44PM (#20186909)
At my previous job we did that for DB nodes on AWS. Still definitely a known technique.
Re:Principles of Chaos Engineering (2018) by brianpayne2 on Friday June 14, 2019 @03:25PM (#20185928)
That's awesome. Probably has helped your overall uptime numbers too. We couldn't get SLT approval to implement this, we had to settle for war games in dev/stage
Re:Principles of Chaos Engineering (2018) by agumonkey on Saturday June 15, 2019 @05:03PM (#20192129)
It reminds me about the feeling of competition. You might fail during a tournament, but you know perfectly where you stand now.