It makes sense to me that you should send more bits at first to describe the speaker. If I load a theoretically perfect audio GAN of myself, I should be able to send many fewer bits later to reproduce my speech effectively.

For most practical current applications, I'd like to see more work on reducing latency. 20ms seems to be the range on imperceivable, and the "processing latency" of 90ms. Light takes 3.34ms to travel 1000km, so for most calls transmission is not a theoretical limitation to long range low latency communication.

Anyway, this is great work, but I would be curious to see what could be done with <10ms of processing time as well.

I believe the 90ms is the window over which the bitrate is constant... If you want to reduce the latency further, you might need to have variable bitrate (eg. perhaps more bits to encode the start of each syllable, and fewer to encode the end of the same syllable).

That in turn adds complexity to other subsystems - typically variable bitrate leads to loss (bad), packet queuing and therefore variable latency (bad), or requires feedback into the video stream (ie. if the audio uses more data, encode the video with fewer bytes, which in turn means encoding must be serial rather than parallel, increasing latency).