Whoa, I had never thought of monitoring VBR as an attack vector for recovering audio.

Do you have a link discussing this?

Sure, here are a couple papers on the topic:

https://www.cs.jhu.edu/~cwright/oakland08.pdf

https://www.cs.jhu.edu/~cwright/voip-vbr.pdf

It's fundamentally very similar to the sorts of issues you end up with if you compress then encrypt. If the attacker can make some educated guesses about the plaintext prior to the compression, the compression ratio can be a very powerful tool in their arsenal.

Wire implemented CBR for their encrypted calls, upstreamed it to WebRTC and submitted a patch to Signal, https://medium.com/wire-news/call-security-constant-bit-rate...

Silent Phone has used CBR since day 1.

Correct, last article I recall reading about deciphering VBR from packet size alone was something in the neighborhood of 50% success rate.

Then why not compress really efficiently by just transmitting packet sizes?

Because 50% of them won't be understood?

On the other hand, if you could do it, you'd probably have invented a convoluted speech-to-text (where text is a index into a dictionary of words). Note that you would also likely lose things like inflection, voice, accent etc - so while it might work as a texting system with voice input - it would be a poor substitute for voice chat..