News for hackers. Stuff that matters.
Re:In C, how do you know if the dynamic allocation succeeded? by dekhn on Friday November 05, 2021 @04:56PM (#29124655)
no, the kernel maintains other allocated objects (dentry, inode caches) that it can't swap out. Under memory pressure, those get dropped before application pages. See https://unix.stackexchange.com/questions/17936/setting-proc-... and https://unix.stackexchange.com/questions/111893/how-long-do-...

I've found the linux memory system to be far too complicated to understand for quite some time, compared to what's documented in, for example, The Design and Implementation of The FreeBSD Operating System, for a more comprehensible system.

Re:In C, how do you know if the dynamic allocation succeeded? by joosters on Friday November 05, 2021 @05:41PM (#29125013)
Those objects must exist with and without overcommit, I don’t understand why they must make one less efficient than the other.

(I’m talking in general here - not Linux specifically)

Re:In C, how do you know if the dynamic allocation succeeded? by dekhn on Friday November 05, 2021 @05:52PM (#29125100)
dentry caches exist with and without overcommit, but you get higher cache hit rates with overcommit, because you flush them less recently. Depending on workload, this can matter a lot. It mattered more in the time of hard drives.
Re:In C, how do you know if the dynamic allocation succeeded? by joosters on Friday November 05, 2021 @05:59PM (#29125160)
I’m sorry, but I’m still not following… why must a cache be (in theory) flushed more often without overcommit?
Re:In C, how do you know if the dynamic allocation succeeded? by dekhn on Friday November 05, 2021 @06:14PM (#29125275)
if overcommit is disabled, then the system drops its internal caches (dentry cache, pagecache) to satisfy a memory allocation. Since most applications don't touch pages they allocate, that means the OS could have avoided dropping the caches. Since it did drop the caches, other parts of the system will then have to do more work to reconstruct the caches (looking up an dentry explicitly, or loading a page from disk instead of RAM).

Everything I'm describing is about a busy server with heterogenous workloads of specific types.

Re:In C, how do you know if the dynamic allocation succeeded? by wongarsu on Friday November 05, 2021 @07:55PM (#29125986)
I think what joosters is trying to say that instead of overcommitting a system could just allocate memory directly in swap. Both creating the page in swap and swapping it in on first write can be mostly optimized away apart from the bookkeeping, but you guarantee that you always have enough RAM+Swap to provide what you have committed to.
Re:In C, how do you know if the dynamic allocation succeeded? by josefx on Saturday November 06, 2021 @05:41AM (#29128812)
> if overcommit is disabled, then the system drops its internal caches (dentry cache, pagecache) to satisfy a memory allocation.

Couldn't the system just reserve the pages for future use by the application but still use them for caching until the application actually tries to use them?

Re:In C, how do you know if the dynamic allocation succeeded? by yc12340 on Sunday November 07, 2021 @09:39PM (#29145690)
This sounds like it would cause LRU caches to lose their LRU properties...
Re:In C, how do you know if the dynamic allocation succeeded? by josefx on Monday November 08, 2021 @12:37AM (#29146376)
It doesn't need to reserve a specific page during allocation, just ensure that the number of reserved pages is smaller than the total amount of pages. I would expect at least read only caches to mostly behave like they currently do on a system with over commit enabled.