If make -j4 makes your UI unresponsive, it is probably due to swapping. The CPU scheduler lowers the priority of CPU intensive tasks over time.

It could be that you need to “renice -5” X11 or something (I think some distributions used to tweak the UI runtime priorities. Not sure if they still do.)