Now I'm suddenly wondering if this isn't really easy after all (or rather: should be really easy if all was well). If Linux behaves better when it has some slack space past the normal RAM that it doesn't like to put stuff in but will when necessary (i.e. swap), let it do the exact same thing to the last portion of your RAM. In other words: make a swap file of, say, 10% of your memory size, and put it in a tmpfs. Linux probably doesn't want that I believe, but it would, if working, give the results you want.