This behavior is the reason Java had such a bad time with sysops.
Just make use of proper libraries that will invalidate caches if needed, or simply refute to spawn new threads/actors/processes.
I’d rather have some application halt/pause or simply crash if it ran out of memory.
This is absolutely not why Java had/has a bad rap memory wise--that has more to do with a combination of code that allocates irresponsibly on the happy (not memory-error-anticipating) path, and the JVM's preference to preallocate as much as possible for every purpose, not just OOM handling.
> Just make use of proper libraries that will invalidate caches if needed
What if I've dumped every cache I can and I'm still getting allocation (or spawn, or whatever) errors because the system is out of memory? It's useful to have a contingency in place to turn out the lights room-by-room rather than cutting power to the whole building, as it were.
This is more useful than OOM'd programs silently disappearing, but isn't foolproof (that's why I said "increase the probability" rather than "guarantee"): if the OOM killer gets invoked before your telemetry, if you've forked, or if whatever your language does in order to even detect an allocator error is itself memory-costly (looking at you, Python--why on earth would you allocate in order to construct a MemoryError object?!), you will still go down hard.
Other than going back in time and reversing Linux's original sin (allocation success is a lie), swap is the "solution" to these situations, but for many people that cure is worse than the disease.
I often wish there was a generally-available way to map memory to files (rather than mmaping files to memory) selectively in my programs. For cases like this--handling OOM conditions and doing cleanup/reporting before turning out the lights, when the OOM condition was caused either by code in my program outside of my control or other programs on the system outside of my control--having a "all allocations after this instant should occur in a file on disk, not memory" bit to flip would be nice. I don't need my cleanup to be fast; if this is happening I'm already on a sad enough path that I'm happy to trade shutdown/crash performance for fidelity of diagnostic data and increased probability of successful orderly cleanup.
Of course, not being an OS developer, I assume this is impossible for reasons outside of my understanding; the few complicating factors I can think of (copy-on-write pages, for example) are hard enough, and I'm sure there are others.