Reading the paper the table is showing performance on out-of-ditribution test sets.

I see: what does that mean, exactly? :) if it means "data they dont usually test on", 10% does still sound pretty high

Yes, it's not that clear to me either what test sets get a 10% error rate. Because in my use (native English dictation or native English podcast transcription) the small or medium original whisper models have what I'll call a "discrepancy" rate of say 1-2% which is mostly punctuation and "umms/errs" inclusion or not. The actual "error" rate is below 1% in my experience, and excluding surnames, brands and place names that I don't know how to spell either the remaining errors tend to be minor (missed plural etc.).

So I infer that these data sets are some deliberately difficult audio: call centre recordings with lots of background noise, phoneline quality audio etc. Maybe non-native speakers. If I only heard that sort of audio once I also might have an error rate of 10%.

I agree. When using the small or medium en models either for real-time speech recognition of a native English speaker or for transcribing podcasts of native English speakers the error rate is nowhere near 10%. I might say it's something like 1% of which the majority of errors are possibly subjective decisions about punctuation. But I have found the error rates are much higher on the tiny model and higher on the base model.

I assume therefore that the 10% word error rate is on very difficult audio such as pilots speaking to Air Traffic Control (distorted or clipped microphones with significant background noise), which I personally find can be difficult to 100% understand even though I'm a native English speaker and when both pilots and air traffic control are native English speakers.

WER is a pretty strict metric. IIRC it can penalize missing repeats, disfluencies, and the like that an ASR model may reasonably decide to drop. Additionally it will penalize for incorrect pluralization, unique proper nouns that aren't in the model, etc. 10% is still very readable.

I built a tool in the mid-201Xs on an ASR engine with 20%+ WER, and even that was good enough for what we were trying to do.