Curious Case of a 99.9% Latency Hike

Re:Curious Case of a 99.9% Latency Hike by tayo42 on Monday January 14, 2019 @11:56AM (#18904396)

There's that and you could be monitoring swap. Though this reads like the writer might not the potential impact swapping has on performance. They seem to have their own datacenters? Id imagine the servers would have some sort of monitoring?

It would be nice to language and os your talking about.

Also, to the writer, I think you want to turn off swap?

[ Parent | on HN ]

Re:Curious Case of a 99.9% Latency Hike by devereaux on Monday January 14, 2019 @01:57PM (#18905383)

Better than turning it off, on Linux try to use vm.overcommit_memory

It will give you the option of swap when OOM would otherwise kill process, but you can decide exactly how much, to prevent swapping from hurting performance -- say have a 10% extra.

Assuming you have a significant number of identical servers, turning off swap should be reserved for after you have done your tuning of the daemons and their memory use, and you are confident nothing will run out (still, you should monitor dmesg) to maximize bang for bugs.

If however you need swap a good old trick- don't put the swap on LVM or RAID. Just have separate partition on each physical device. The kernel will always know to use them in parallel, and you get to squeeze out a bit of extra performance

[ Parent | on HN ]

Re:Curious Case of a 99.9% Latency Hike by aybassiouny on Monday January 14, 2019 @02:19PM (#18905561)

Lucy, even in a healthy state, already have many page faults. They are just not on the hot path, e.g. logging. They don't affect latency much and nobody cares for them.

Turning off swapping has been a heated discussion, and I am on the negative side of it. Swapping is good; not all memory is important and some of it should not occupy precious physical space. Not all processes are important though. What I would rather have is a per-process swapping flag, allowing me to disable swapping for one or two essential processes, but swap the rest.

[ Parent | on HN ]

Re:Curious Case of a 99.9% Latency Hike by sitkack on Monday January 14, 2019 @02:02PM (#18905419)

> Also, to the writer, I think you want to turn off swap?

Using kernel panics as an alerting mechanism is a brutal solution.

Why not track page faults? Most orgs that I interact with track paging across the whole fleet.

[ Parent | on HN ]