ulimit on Linux doesn’t support limiting RSS, only VSZ. I’d argue that limiting RSS — i.e., how much of the process in main memory — is more aligned with what someone wants than how big its virtual address space is, which can easily be distorted by e.g. mmap()ing a huge file in.

IME trying to restrict by VSZ just leads to surprises when malloc() fails at surprising times. cgroups are a much better way to go.