When it all comes together it's kind of a nightmare for an ASR model. I'm not sure which contributes most but I know from my prior experiences with ASR for telephony even clean speech on pristine connections does much worse with models trained on 16 kHz being fed native 8 kHz audio that gets resampled. There were plenty of times in reviewing the recordings and ASR output where I'd listen to the audio and have no idea what they said.

I've done some early work with Whisper in the telephony domain (transcribing voicemails on Asterisk and Freeswitch) and the accuracy already seems to be quite a bit worse.