If you're using systemd, drop docker and start your browser through something like this:

  #!/bin/sh

  systemd-run \
    --user \
    --scope \
    --property=MemoryHigh=4G \
    --slice=background \
    /usr/bin/firefox "$@"

Theoretically this does not prevent Firefox from using more than 4 GBs of memory (you need MemoryMax for that), but practically it does.

  MemoryHigh=bytes
    Specify the throttling limit on memory usage of the executed processes in this unit. Memory usage may go above the limit if unavoidable, but the processes are heavily slowed down and memory is taken away aggressively in such cases. This is the main mechanism to control memory usage of a unit.

  MemoryMax=bytes
    Specify the absolute limit on memory usage of the executed processes in this unit. If memory usage cannot be contained under the limit, out-of-memory killer is invoked inside the unit. It is recommended to use MemoryHigh= as the main control mechanism and use MemoryMax= as the last line of defense.

I use a similar script to prevent my torrent client from pushing more useful data out of the page cache.

You can put resource limits on basically anything (see systemd.resource-control(5)).

Docker is pretty overkill for this; you can just create a new cgroup with a maximum memory, CPU, etc.

OpenBSD has login.conf

https://man.openbsd.org/login.conf.5

Speaking of which, Firefox behaviour has improved on OpenBSD significantly recently. It used to either crash or act weird when it hit the memory limit in login.conf (which it would eventually do). Now it just works. No idea what changed.

This is what ulimit is for.