I have mitigations disabled on my laptop. I mainly work offline and only run trusted code. I also use noscript and only enable javascript on trusted sites. I also use auto tab suspend to stop javascript from running if I am not using the tab.

Realistically, there is very little threat to turning off the mitigations with enough precautions in place. Obviously not recommended on a vps or other shared servers etc.