Globally

For a process you can use setrlimit() (ulimit command)

I'd want the OOM killer to kill processes with more over-commit first.

Why? There are many innocent applications that consume lots of virtual memory with no intention of using it. Sometimes sparse algorithms call for that sort of thing. It can be a nice crutch when you're in a pinch for time. A good example of a tool that does this is TSAN. It gobbles up virtual address space, which is practically free, thanks to the magic of modern MMUs. That's too powerful and good a thing to need to give up completely.

Another example would be the JVM’s new low-latency GC, ZGC, that does some smart addressing tricks to store some information in pointer addresses themselves, effectively tripling the actual memory usage when naively looked at through virt memory.

Such applications could handle this explicitly, e.g. with mmap and mprotect. In principle there is no reason for such use-cases to rely on implicit overcommit behavior. Which also isn't present on Windows.

That's because Windows forces you to use special userspace libraries that account for memory and let you create "reservations" for large regions, without actually populating entries in the MMU (i.e. PML4t). Linux doesn't have that. You create reservations for large memory regions by allocating virtual memory, which of course doesn't become real (resident) until you use it. That's overcommit. But since there's no userspace libraries like WIN32 forced upon you on Linux, there's really no way for the kernel to know your intent when it comes to memory, because the abstractions for memory on Linux are so simple. Windows has a ridiculous byzantine system, which some people think is better. But people are really missing the point when they try to make an apples-to-apples comparison between the two systems of memory management. It's not a question of (a) overcommit good, vs. (b) overcommit bad. Anyone who thinks either is a fool. Linux is simply a system that requires a smart system administrator who understands your use cases and can tune sysctl accordingly when you want to push beyond the normal boundaries.

It seems to me that the prot parameter and/or MAP_NORESERVE would indicate the intent.

It might actually be using that, I’m not intimately familiar with the topic.

Yes, I would expect that as well.

Implicit overcommit is really only necessary for fork and general backwards compatibility.

BTW the newer Generational ZGC is not using multi-mapped memory any more.