I wonder if part of the solution is smarter caching at the OS level. Do all modern operating systems give the application a way to tell the OS, "This chunk of memory is a cache, so free it if it's needed for something else"? Of course, when the OS frees that memory, the app would need to know about that, so it can re-fetch or re-compute that cached data the next time it's needed.