Check out the Bela project[1], which has back-ends for a few different audio packages (e.g. PD and SuperCollider) that totally bypass the kernel and handle the audio callback directly (maybe from a bare-metal ISR, not exactly sure). That’s how they get sub-ms round-trip latencies.

[1]: http://bela.io