That explains why Windows, which has a process model based on spawning and not forking, never had overcommit.

However Windows has stacks that commit on-demand, which is kinda similar to overcommit. It can result in programs terminating on any random function call (if that call requires allocating a new page of stack). https://learn.microsoft.com/en-us/windows/win32/procthread/t...

However this is very rare to happen in practice -- stack growth isn't very frequent in typical applications; so usually a malloc() will fail first in low-memory situations.

A Windows program can avoid the risk of getting terminated on stack growth by specifying equal commit and reserve sizes for the stack. So in least in theory, it's possible to write a windows program that is reliable in low memory situations.

There are also legitimate uses of overcommit for reserving huge areas of virtual memory and relying on demand paging to allocate physical memory, although strictly speaking enabling overcommit isn’t necessary if you separate the “reserve” (mmap(PROT_NONE)) and “commit” (mmap(MAP_FIXED, PROT_READ | PROT_WRITE)) stages (which is what Windows forces you to do).

I’m not sure how this works on Linux exactly but in principle you don’t need to support overcommit to avoid that problem. On illumos for example anonymous memory allocations come from swap, which is a virtual resource that includes both disk and memory. You do still have this problem if something is using more than half of swap, but (1) that’s much larger, and (2) you can augment it with more disk space, which won’t actually be used unless stuff actually uses all the physical memory. This has its own tradeoffs but it’s useful to keep in mind that there are other ways to do things.

If available memory means physical + swap then you're saying the same thing as pcwalton.

Right, available memory includes swap. (That's why I didn't say "available RAM".)

I thought fork was copy on write nowadays.

Fork is copy-on-write. The problem is that there's no way for fork() to know how much memory the child process needs. It might call exec() immediately, using very little memory. It might keep running, mostly sharing pages with the parent. It might touch every memory page, requiring a lot of memory.

It is, but “write” doesn’t have an error code it can return on out-of-memory.

The whole crux of overcommit is that the affected process doesn’t get a chance to gracefully fall back or shutdown on OOM conditions at defined places in the code (like at invocations of malloc).

This article explains the problems with fork pretty well: https://www.microsoft.com/en-us/research/uploads/prod/2019/0...

Also I would suggest using a Windows machine which seems to run out of memory despite hundreds of gigabytes of swap and yet still ~70% of RAM full. I don't know what's causing the bug but it affects multiple systems I use and it's a right nuisance with a nearly full disk to have >10% of it sitting there idle just in case whatever's over-allocating memory decides to actually use it (I mean, I guess I could also direct my ire at whatever broken accounting fails to actually tell me what's causing it as well, but I can be mad at two things at once).

yup. i recall first being exposed to this while trying to understand why a memory-hog backend service was being oom-killed in production while attempting to spawn a resource-light subprocess via fork+exec, while there was still a good portion of free memory to use.

the more you read about linux oom killer, overcommit, linux memory accounting heuristics, copy-on-write, fork+exec as a mechanism to spawn an unrelated process, the more surprising it is that anything works at all even some of the time.