Re:When the memory allocator works against you
by tedunangst on Sunday March 12, 2017 @01:26PM
(#13851791)
> It's also worth noting that most programs don't tolerate malloc failure well at all, and will just crash anyway.
But at least then it's the program that doesn't handle failure that dies, not one that does!
Re:When the memory allocator works against you
by paulddraper on Sunday March 12, 2017 @01:45PM
(#13851894)
> Some programs will malloc a huge area and only use a small portion of it, and in this scheme, that memory is "in use." It's also worth noting that most programs don't tolerate malloc failure well at all, and will just crash anyway.
...and it would be absurd for bad programs to eat memory and crash? We must have other programs pay for its sins?
Re:When the memory allocator works against you
by mikeash on Sunday March 12, 2017 @01:59PM
(#13851992)
Tha program that hits a malloc failure will not necessarily be the program that ate up all the RAM.
Re:When the memory allocator works against you
by tedunangst on Sunday March 12, 2017 @02:09PM
(#13852069)
Malloc failure is at least synchronous with the out of memory condition. And if your program has reached a fixed state, it may not even be calling malloc.
Re:When the memory allocator works against you
by cturner on Sunday March 12, 2017 @03:30PM
(#13852550)
Ot: your flak blog could benefit from a page showing links to all articles. Currently, hard to read beyond 'recent'.
Re:When the memory allocator works against you
by tedunangst on Sunday March 12, 2017 @03:36PM
(#13852593)
Don't waste your time with the old crap. :)
Re:When the memory allocator works against you
by paulddraper on Sunday March 12, 2017 @03:01PM
(#13852383)
Sure, but (a) it can handle the malloc failure and (b) it's pretty easy for the user to find the bad guy.
Re:When the memory allocator works against you
by deathanatos on Sunday March 12, 2017 @05:50PM
(#13853337)
It might handle the malloc failure by crashing. Some languages (e.g., Rust) and runtimes (e.g., glib) presume that small allocations always succeed; a failure in such an allocation immediately crashes the program (as the underlying API provides no means to report the failure).
Re:When the memory allocator works against you
by steveklabnik on Sunday March 12, 2017 @10:26PM
(#13854777)
Rust the language doesn't know anything about allocation; it's entirely a standard library decision.
Re:When the memory allocator works against you
by mikeash on Sunday March 12, 2017 @09:19PM
(#13854494)
(a) it probably won't therefore (b) the "bad guy" is going to be most/all processes.
The OOM killer doesn't pick the process by an entirely random method, it seeks to kill the problem
Re:When the memory allocator works against you
by charleslmunger on Sunday March 12, 2017 @02:11PM
(#13852078)
Android handles this by having a total ordering of the importance of every application process, and building its APIs such that process death of something the user isn't interacting with doesn't lose state.
Re:When the memory allocator works against you
by mikeash on Sunday March 12, 2017 @02:14PM
(#13852098)
iOS is like that too. It keeps all programs running by default, and then when memory gets low it'll start killing them. It starts with backgrounded apps that aren't doing anything, and kills the current foreground app as a last resort.
Apps are supposed to be built to tolerate being killed while in the background and restoring their state when relaunched, so in theory this will mostly be invisible to the user other than apps taking longer to reload. In practice, lots of apps don't get this right.
After a fork, does the amount of committed memory increase by the size of the child's allocated virtual memory space?
I'm curious how committed memory interacts with COW fork semantics. After all, a child can just overwrite the COW/copy of the memory it inherited from its parent without allocating anything else.
Re:When the memory allocator works against you
by haldean on Sunday March 12, 2017 @01:42PM
(#13851878)
No, all of the child's memory mappings are shared with its parent. Upon write, though, that becomes a new committed mapping; I believe the entire page is copied, so you could end up committing another 4k with a one-byte write from the child.
Hm. Then suppose I turn off the OOM killer. What happens to a process that writes to its parent's page if the system has no memory left to commit?
Re:When the memory allocator works against you
by tedunangst on Sunday March 12, 2017 @02:11PM
(#13852076)
COW is usually a kind of loophole. Most systems only count the pages once (otherwise you'd have to reject the fork, but that becomes inconvenient fast).
One is to commit memory on request. If all of your RAM and swap is committed, then malloc will fail. This is probably how we expect it to work, but it can be really wasteful. Some programs will malloc a huge area and only use a small portion of it, and in this scheme, that memory is "in use." It's also worth noting that most programs don't tolerate malloc failure well at all, and will just crash anyway.
Another way is to overcommit, where malloc doesn't actually reserve memory initially but only on first use, but only pause processes when memory fills up. If a process needs more memory and there isn't any more available, that process blocks until more memory becomes available. This means nobody gets killed, but it's not uncommon for the system to just deadlock like this forever once memory fills up.
And the third way is as you describe, to overcommit and kill processes when you run out of memory.
Note that there's no right answer, just different tradeoffs. Linux lets you configure this to an extent as well, so if you don't like the default, you can change it!