Re:Principles of Chaos Engineering (2018)
by jaggederest on Friday June 14, 2019 @03:26PM
(#20185939)
We used to boot ~3x the servers we needed, run a hard load on them for a while, performance test them, and kill the "weakest" 2/3rds. You can get a bunch of nodes on iffy hardware or greedy neighbors (or all on the same physical box) and see significant performance improvements that way.
Of course this was a decade ago, but I think the fundamentals are still sound, as far as being skeptical about the quality and longevity of your nodes in a virtual environment.
Re:Principles of Chaos Engineering (2018)
by isbvhodnvemrwvn on Friday June 14, 2019 @05:44PM
(#20186909)
At my previous job we did that for DB nodes on AWS. Still definitely a known technique.
Re:Principles of Chaos Engineering (2018)
by brianpayne2 on Friday June 14, 2019 @03:25PM
(#20185928)
That's awesome. Probably has helped your overall uptime numbers too. We couldn't get SLT approval to implement this, we had to settle for war games in dev/stage
Funnily enough it also improved our latencies a lot (which I guess is mostly due to memory leaks et al.)