Pipewire (which is being released in mainstream distros since this summer) has mostly/entirely fixed these issues. On my particular end (Arch, for gaming) it's 100% perfect with no latency

Edit: Pipewire is a new system AND a "drop-in" replacement for any of PulseAudio/Alsa/JACK (just add whichever package you want and it will replace that particular audio stack with Pipewire's replacement)