Re:When the memory allocator works against you
by deathanatos on Sunday March 12, 2017 @05:45PM
(#13853309)
You can still allocate more virtual memory; it just has to be backed by a file, e.g., through mmap(). So, for example, what mongod does — mmap() in its entire DB, which can be terabytes of RAM — would still work.
You can almost think of file-backed mmaps as swap, in fact.
The downside is memory allocations that would oversubscribe memory will immediately fail. That means processes won't be able to allocate tens of gigabytes of virtual memory anymore.
If I understand the limitation correctly, this also means that all process's allocated virtual memory can no longer exceed the amount of physical+swap memory in the machine. I think this probably interacts badly with COW memory semantics for child processes: even though children and parent reuse address space, the kernel can't know whether the child will write to the memory it inherited from its parent's memory map. I think if the kernel is being too careful, all "shared" memory would have to be accounted separately, which would severely limit the number of processes you could have open.
(Is that really what happens? That seems awfully limiting. I hope I'm wrong.)