Cgroups

For example, systemd integrates with cgroups and you can limit resource use for services started with it

  man ulimit

Not sure what constitutes a regular Linux system but apart from the shell command the underlying ulimit syscall getrlimit(2) is in the kernel itself. One can set hard and soft limits on a per process granularity.

Normally each service has a dedicated uid and limits as well as nice levels are set in /etc/security/limits.conf which is read on login sessions by pam_limits.

This is in every bsd- and posix-like system and some of the soft limits have standardised signals assigned to them. The man page has all the details and is easy enough to understand.

ulimit on Linux doesn’t support limiting RSS, only VSZ. I’d argue that limiting RSS — i.e., how much of the process in main memory — is more aligned with what someone wants than how big its virtual address space is, which can easily be distorted by e.g. mmap()ing a huge file in.

IME trying to restrict by VSZ just leads to surprises when malloc() fails at surprising times. cgroups are a much better way to go.