> I wonder why the system can't just stop giving the process memory when it's starting to become critical

If you turn swap off the allocator will error out when you are out of memory. I don't think most software handles that error but all the wiring is there to do that.