For minimum latency you want a recurrent model that works in the time domain. A Mamba-like model could do it.