Most of the latency comes from internal buffering inside ffmpeg or vlc. Most go away if you tune them down.