One of the biggest limits is the processing I think. Good algorithms try not to omit anything, but those also begin to blur vowels in to consonants. Fine-tuning of the window size helps (not sure if this was on the FFT or resynthesis part).

Close to the speed limit for understanding two things are difficult:

- personal names (especially of companies) and aberrations.

- a change of the speaker to someone who has a slightly higher speed or worse audio quality

Unavoidable and super annoying:

how back in the "real world" everything appears to be in slow-motion. It takes ages for someone to just finish a single word...