Re:Do Not Use Task Manager for Memory Info
by quotemstr on Sunday January 06, 2019 @03:10PM
(#18840323)
The bigger problem is that there's no good number that captures the memory impact of a process on modern unified-memory page-cache-ful architectures. Practically nobody gets this right, and the author of the article is himself glosses over some important details. Every byte of address space is either allocated and backed by some memory object ("reserved" memory) or it's unallocated. Commit charge is a measure of the amount of memory that the system has guaranteed will be available in the worst case, should all faultable address ranges be faulted, but that's not the same thing as memory actually being used. For example, if you MapViewOfFile a 1GB file PAGE_READONLY, you burn very little commit --- just enough for the page metadata --- but if you change page protections to PAGE_WRITECOPY, then you incur an extra 1GB of commit charge, even though you still haven't faulted anything into memory or reserved more address space --- and that's because you could legally COW-fault every page in that file, and the kernel has to commit (thus the word) to providing that memory in the worst case.
I work on Linux these days, which is even more confusing, because thanks to overcommit, most people don't distinguish these different kinds of memory allocation, even though the distinction between commit and reserved memory exists on Linux too. (The kernel just lies about satisfying commit charges unless you tell it not to lie to you. Most people are happy with overcommit's optimism.)
Anyway, the key thing to realize about modern virtual memory subsystems is that "memory consumption" is an incoherent concept. You can derive lots of different numbers from memory management statistics, but each of these numbers is useful for a specific purpose. There is no one number that will give you an accurate measure of the impact of a particular process in all scenarios. People constantly say, "Look: just give me a number that I can plot on a dashboard and drive down over time". No such thing exists.
Task manager has to pick one of these numbers to show users by default, and its choice, roughly equivalent to Linux Private_Dirty, probably isn't terrible, since it's a decent proxy for how much RAM you get back if you kill the process. I don't think total commit charge is as good a choice, since with a large pagefile (which everyone should have) total commit can be much larger than total resident memory. Linux PSS is another popular choice, since (unlike Private_Dirty) it reflects the impact of a program's use of shared memory, but PSS behaves in perverse ways --- e.g., starting an instance of memory-hungry process can make PSS decrease because some pages in this program are distributed across more processes, increasing the denominator in the PSS calculation.
Are you worried about running out of page file space? Yes, you want to look at commit. Are you wondering why you're seeing a large number of page faults starting a game? Commit won't help you, but RSS might. It really depends on the situation.
I wouldn't take the advice in the article at face value. If you want to understand the impact a particular program has on the system's memory behavior, you need to understand how the virtual memory system actually behaves, and that's non-trivial.
Re:Do Not Use Task Manager for Memory Info
by edf825 on Tuesday January 08, 2019 @02:08AM
(#18853881)
Drepper's What every programmer should know about memory is a little old and goes into perhaps unnecessary detail at times, but is a great place to start.
I work on Linux these days, which is even more confusing, because thanks to overcommit, most people don't distinguish these different kinds of memory allocation, even though the distinction between commit and reserved memory exists on Linux too. (The kernel just lies about satisfying commit charges unless you tell it not to lie to you. Most people are happy with overcommit's optimism.)
Anyway, the key thing to realize about modern virtual memory subsystems is that "memory consumption" is an incoherent concept. You can derive lots of different numbers from memory management statistics, but each of these numbers is useful for a specific purpose. There is no one number that will give you an accurate measure of the impact of a particular process in all scenarios. People constantly say, "Look: just give me a number that I can plot on a dashboard and drive down over time". No such thing exists.
Task manager has to pick one of these numbers to show users by default, and its choice, roughly equivalent to Linux Private_Dirty, probably isn't terrible, since it's a decent proxy for how much RAM you get back if you kill the process. I don't think total commit charge is as good a choice, since with a large pagefile (which everyone should have) total commit can be much larger than total resident memory. Linux PSS is another popular choice, since (unlike Private_Dirty) it reflects the impact of a program's use of shared memory, but PSS behaves in perverse ways --- e.g., starting an instance of memory-hungry process can make PSS decrease because some pages in this program are distributed across more processes, increasing the denominator in the PSS calculation.
Are you worried about running out of page file space? Yes, you want to look at commit. Are you wondering why you're seeing a large number of page faults starting a game? Commit won't help you, but RSS might. It really depends on the situation.
I wouldn't take the advice in the article at face value. If you want to understand the impact a particular program has on the system's memory behavior, you need to understand how the virtual memory system actually behaves, and that's non-trivial.