News for hackers. Stuff that matters.
Re:In C, how do you know if the dynamic allocation succeeded? by dekhn on Friday November 05, 2021 @04:36PM (#29124460)
The biggest reason overcommit exists is because it allows the system to operate more efficiently. The reality is most applications touch only some of the pages they allocate, and it's silly for the system to fail a malloc. Often times other expensive cleanup activities can be deferred (you don't really want to drop a handy directory entry cache just so an app can be sure it got physically backed memory for its request, 99.99% of the time).

IIUC Linux was really the first OS to make overcommit so prominent. Most systems were a lot more conservative.

Re:In C, how do you know if the dynamic allocation succeeded? by toast0 on Friday November 05, 2021 @09:06PM (#29126501)
Replying up here instead of way down in the branches.

I don't think disabling of overcommit implies that physical pages are mapped immediately. If caches are instantly droppable, you can use a page that's allocated but unused for cache, and drop the cache page (and zero it) when the allocated page is written to.

You'd still have all of your caches until you have memory pressure with actual data written (but of course, with overcommit, you'd drop caches then too), but if you attempt to allocate more than you have (including through fork attempts as discussed elsewhere), you get a system call failure rather than an OOM kill.

Re:In C, how do you know if the dynamic allocation succeeded? by joosters on Friday November 05, 2021 @04:49PM (#29124592)
In theory, a system without overcommit can run just as efficiently, IF you have reserved huge amounts of swap space. As long as swap is available, the OS can do all the same COW and efficiency tricks. It’s not the physically backed memory is the limiting factor, it’s RAM+swap
Re:In C, how do you know if the dynamic allocation succeeded? by dekhn on Friday November 05, 2021 @04:56PM (#29124655)
no, the kernel maintains other allocated objects (dentry, inode caches) that it can't swap out. Under memory pressure, those get dropped before application pages. See https://unix.stackexchange.com/questions/17936/setting-proc-... and https://unix.stackexchange.com/questions/111893/how-long-do-...

I've found the linux memory system to be far too complicated to understand for quite some time, compared to what's documented in, for example, The Design and Implementation of The FreeBSD Operating System, for a more comprehensible system.

Re:In C, how do you know if the dynamic allocation succeeded? by joosters on Friday November 05, 2021 @05:41PM (#29125013)
Those objects must exist with and without overcommit, I don’t understand why they must make one less efficient than the other.

(I’m talking in general here - not Linux specifically)

Re:In C, how do you know if the dynamic allocation succeeded? by dekhn on Friday November 05, 2021 @05:52PM (#29125100)
dentry caches exist with and without overcommit, but you get higher cache hit rates with overcommit, because you flush them less recently. Depending on workload, this can matter a lot. It mattered more in the time of hard drives.
Re:In C, how do you know if the dynamic allocation succeeded? by joosters on Friday November 05, 2021 @05:59PM (#29125160)
I’m sorry, but I’m still not following… why must a cache be (in theory) flushed more often without overcommit?
Re:In C, how do you know if the dynamic allocation succeeded? by dekhn on Friday November 05, 2021 @06:14PM (#29125275)
if overcommit is disabled, then the system drops its internal caches (dentry cache, pagecache) to satisfy a memory allocation. Since most applications don't touch pages they allocate, that means the OS could have avoided dropping the caches. Since it did drop the caches, other parts of the system will then have to do more work to reconstruct the caches (looking up an dentry explicitly, or loading a page from disk instead of RAM).

Everything I'm describing is about a busy server with heterogenous workloads of specific types.

Re:In C, how do you know if the dynamic allocation succeeded? by wongarsu on Friday November 05, 2021 @07:55PM (#29125986)
I think what joosters is trying to say that instead of overcommitting a system could just allocate memory directly in swap. Both creating the page in swap and swapping it in on first write can be mostly optimized away apart from the bookkeeping, but you guarantee that you always have enough RAM+Swap to provide what you have committed to.
Re:In C, how do you know if the dynamic allocation succeeded? by josefx on Saturday November 06, 2021 @05:41AM (#29128812)
> if overcommit is disabled, then the system drops its internal caches (dentry cache, pagecache) to satisfy a memory allocation.

Couldn't the system just reserve the pages for future use by the application but still use them for caching until the application actually tries to use them?

Re:In C, how do you know if the dynamic allocation succeeded? by yc12340 on Sunday November 07, 2021 @09:39PM (#29145690)
This sounds like it would cause LRU caches to lose their LRU properties...
Re:In C, how do you know if the dynamic allocation succeeded? by josefx on Monday November 08, 2021 @12:37AM (#29146376)
It doesn't need to reserve a specific page during allocation, just ensure that the number of reserved pages is smaller than the total amount of pages. I would expect at least read only caches to mostly behave like they currently do on a system with over commit enabled.