Whenever I went for low latency, I noticed 2 things: 1. Actual latency was not what I set in the encoder because of frame sizes, chunk sizes, tcp Windows, decided buffering 2. Forcing super low latency caused drop-outs, which is worse for music than delays.

Overall I just found flac to be rock solid and lower latency without much tweak