Linux memory overcommit (compared to airlines selling tickets)

Re:Linux memory overcommit (compared to airlines selling tic... by ajross on Friday May 13, 2011 @12:37PM (#2545192)

Your null-checking has always been insufficient. Nothing you can do in the context of a single app can fix things anyway. Actually try it some time: hack up a malloc() that "fails" beyond a certain heap size and start testing all your "handling" code. I guarantee you it won't do what you think it does.

And this isn't about "malloc" anyway. It's about mmap. Mappings are overcommited because mappings are routinely undercommitted. How much stack do you use for each of your threads? How much stack is mapped? If you don't know the answers to these questions (or worse, don't know how to find out), then you need to do a little more research before pontificating on kernel memory allocation strategies.

Sorry, pet peeve. This is one of those areas where too many people think they understand something that they simply don't.

[ Parent | on HN ]

Re:Linux memory overcommit (compared to airlines selling tic... by pc1234 on Friday May 13, 2011 @03:40PM (#2545886)

The thing that bugs me about this is that I'm pretty sure there is some way to make a damned good effort at at least saving results out to disk or dying gracefully even when malloc starts asploding.

Also, this means that I can screw up other processes for no good reason, even though only my proc should get killed for going over quota.

"Mappings are overcommited because mappings are routinely undercommitted"

Could you please explain this again? I'm not quite sure I parsed that out correctly. Thanks!

[ Parent | on HN ]

Re:Linux memory overcommit (compared to airlines selling tic... by ajross on Friday May 13, 2011 @06:42PM (#2546493)

Saving results to disk requires disk buffers, which can require the kernel to allocate, which won't work on an OOM system. This is how you get deadlocks: process trying to "handle" OOM ends up trying to allocate, and the system freezes. Which is why the OOM killer exists.

I should have said "underutilized" above. But the point is very real: applications map stuff all over the place. It's quite rare that they actually fault in those pages.

The stack thing is the obvious example: measure the stack size of your application (easy trick is to subtract the address of a local variable "deep" in the tree from the address of one of the arguments to main, or pthread_create, or wherever the top is). Now check the size of the mapping that contains the stack. And tell me if you want to actually reserve all that memory.

[ Parent | on HN ]