OOOOOOO, something I actually know!

> Now why would you want to know the committed memory over the actual physical RAM in use?

Because in Windows, committed size is relative to physical size. You can commit a lot more than RAM, but watch your page-file grow.

malloc() can fail on Windows for this reason. This is not the same on Linux or any of the BSD's I've tried. :)

I experienced/discovered this August last year.. sometimes understanding a lot about Linux can make you blind to the architectural differences Windows has.

I wrote some cross-platform CPP to show it[0]

[0]: https://gist.github.com/dijit/cb2caa1a40d48e03613f5af0e518d6...

> This is not the same on Linux

Linux lets you choose an overcommit policy. https://www.kernel.org/doc/Documentation/vm/overcommit-accou...

You can choose an overcommit policy on Linux, but most library developers on Linux have chosen the default and regularly allocate wide swaths they don't intend to use.

This is a real pain when moving an application from FreeBSD to Linux, as effective limits on memory are lost (ulimit set at ~90% of ram results in a malloc failure and a clean crashdump rather than death by thrashing, or an untrapable oom kill).

There could maybe be a middle ground where malloc would allocate large chunks of address space for ease of administration, and then ask the OS to commit those pages in smaller chunks as needed. Often, there's not much a lot you can do when allocation fails, but it's way more actionable if the failure is returned from a syscall vs failing when you write to an unbacked page, which could happen basically anywhere in your program.

This doesn't occur if you memory map a file though (barring certain flags that you can set as stated by quotemstr below)

You can legitimately have Windows stating many GB's of committed RAM without actually using that RAM and it's not using the systems pagefile/swap. It's also common for this to occur. Pretty much every program capable of opening large files (GB+) in a non-sequential fashion does this.

Hm, maybe I wasn't clear. It's not actually /using/ the memory when it's committed.

But the sum of your committed memory across all applications must exist in some form on the host system.

So, for example it's a common performance optimisation to double the amount of allocated space when you grow anything in C++, what this means is that you're not actually using the space yet but malloc() and zeroing is kinda slow.

So, you have 128MB of ram which is your programs address space and you just doubled your array from 75MB to 150MB, well, that extra 22MB must exist. Even though you're only using 76MB.. even if the OS shows it as free. (which, it will)

Thems the rules, and I promise you that I have thoroughly tested this; as it was causing a really nice crash on my servers even though we had more than 50% of memory "free"

Memory Mapped files do exist in some form or another though. As the file itself. That's the point of memory mapping files. You can right now memory map every file on your computer. That's TB of files. There will be no physical RAM usage and no swap file usage unless you start to actually work with those files (at which point they will be paged in). This will show as 'committed memory' in task manager.

Your example above isn't memory mapping files. It's just allocating RAM. That does have to exist in RAM or the swap file. But that's not what 'committed memory' above shows. Which is the whole point. The column the article is telling people to use is misleading.

They're talking about mmap'ed files, not memory.

>Task Manager is doing the right thing here

I had a problem on my windows 10 pc for a long time, where I would clearly be running out of ram, task manager would show 100% usage, everything getting slow. However if I added up all of the processes using ram, it was nowhere close.

So something was using up ram that task manager gave me no visibility into. I had to download obscure tools and do some guesswork to figure it out. It would have been really nice if Task Manager could just report it in the first place (it turned out to be a network card driver with a memory leak in it)

Yep, this is particularly evident if you use Virtualbox a lot - the memory use isn't in the actual process but in one of the drivers. Has also happened to me with a VPN drivers.

Best tool I've seen to start finding where it's going is RAMmap [1].

1. https://docs.microsoft.com/en-us/sysinternals/downloads/ramm...

> committed memory is that in use + memory that's being paged out

That's not actually true though. You can see an increase in a process's commit charge without anything new being written to the pagefile. Commit is a check that the kernel writes to applications; you're confusing that check for cash in a wallet. You can also have commit without any virtual address space to blame for it through section handles or other tricks. It's complicated.

I'm not going to dispute that but just want to highlight it doesn't change the fact that committed RAM can show as extremely high just by working with memory mapped files.

Memory mapping a GB log file for example will absolutely show GB's of committed memory but in reality you'll only have the last page in actual physical RAM.

Right. When you memory-map a file, what you've essentially done is add a temporary new pagefile to the system (your mapped file), and when you work with memory backed by that file, it's no different from working with "anonymous" memory backed by the system-wide pagefile.

Aside: I have to say this is a genius perspective on mmap, and I now finally understand why the same syscall is used for both. I never understood the link.

Thank you.

I understand commit can happen without physical page file being written to. But you say commit can happen even without any increase in virtual address space usage? That seems strange, could you explain how it can happen?

Change a PAGE_READONLY mapping to a PAGE_WRITECOPY one. The commit charge is billed at VirtualProtect time, and the protection can fail if you run out of commit. The kernel doesn't have to commit anything for a PAGE_READONLY mapping because all pages in such a mapping are guaranteed to be clean and trivially evictable. Not so once you introduce the possibility of COW faults. Reserving a big range of address space and committing it a little bit at a time is a very common pattern.

Another thing: create a 1GB section. Map it, and fill it up. Unmap it. Map it again. What you wrote is still there. Between the map and remap, you have commit without corresponding address space.