Well, malloc() is very much doing the right thing in a world where (virtual) memory is plentiful. The performance of the program improves substantially, without impacting memory usage that much.

I never understood why the system doesn't do the accounting strictly, even if it waits to actually hand out individual pages. Then, it could give an error on brk() or mmap() when the memory would be "overbooked". Memory "saved" this way would be unusable to other processes, but it would be usable as buffer cache, and fork() would still be fast.

edit: Another possible solution would be a special "out-of-memory" signal. The program could trap that signal and do the right thing with it. The problem with that is that it's difficult to program for out-of-memory conditions. Something as simple as printf() won't work at all, and could cause a signal loop depending on implementation. Given that, almost no programs would do it, so there would be little benefit.

Why would you deny allocation requests larger than immediately available physical memory when they could be serviced perfectly well by paging?

On modern systems malloc and friends manage address space, not physical memory. Failure should only be reported if that resource is exhausted.