If you're running X session though systemd, as LimitCORE=... line to that service's file.

As an extreme option you can always replace the binary with a shell script setting ulimit and exec-ing the original Xorg.