Interestingly I actually ran gentoo for a while on that machine precisely for that reason, but the overall latency improvement was marginal.

Certainly not worth recompiling things all the time in my opinion, even with a beefy machine like mine.