I mean, Microsoft claimed in 2016 to surpass human performance and google subsequently claimed in 2017, and yet anyone that regularly uses voice dictation knows that <5% WER is far from realistic due to lack of generalization to new audio/noise regimes and domain-specific vocabulary.

5% WER or less isn't that unrealistic. It's just not very good. It means that there's an error every 20 words on average, which with a slow rate of speech of 2 words per second amounts to an error every ten seconds.

That humans can still communicate effectively with each other despite a higher transcription error rate is just due to the the way humans do transcriptions: listen to a section of audio, then write down what they heard. If they understood the meaning, they'll likely write down a transcription that retains this meaning, but they might drop an article or duplicate it or switch the order of two words. It's easier for an algorithm to avoid those mistakes of inattention, but harder to prevent errors that change the meaning.

So WER isn't a perfect indicator of transcription quality. It also matters which words are affected. I wouldn't be surprised if people preferred a human-written transcription that reads well despite a high WER over a machine-generated one with fewer errors, but more obvious ones.

Bonus: find the word duplication in this comment.

>That humans can still communicate effectively with each other despite a higher transcription error rate is just due to the the way humans do transcriptions: listen to a section of audio, then write down what they heard.

Part of it is that human communication has something similar to error correction built in. We recognize patterns of words and can guess what the next word is based on that. This allows us to discern the meaning without always understanding every word. We effectively fill in the blanks.

That would be easy to duplicate, wouldn't it? Train GPT-2 (because no API key required) to just repeat the sentence correctly, feed the output of the transcription through.

I'm not sure if these models do that.

Yeah have a model predict the misheard word in a sentence where you replace one word with a speech recognition error

> Bonus: find the word duplication in this comment.

Gur ercrngrq jbeq vf gur. V unq gb hfr n cebtenz gb svaq vg: crey -yar 'juvyr (/\o(\j+)\o\f+\o\1\o/t) {cevag $1}'

This (https://arxiv.org/abs/2010.03449) is by far not the first publication which claims super human speech recognition, or human parity. For example https://arxiv.org/abs/1610.05256 or https://arxiv.org/abs/1703.02136. And since then, the WER has dropped much more on the same dataset (Switchboard), for example https://arxiv.org/abs/2001.07263.

They highlight here the aspect of low latency, so this work might be current SOTA under this condition.

Anyway, yes, we have reached human parity, or even surpassed it, for this dataset, under the same conditions. The conditions are:

* English telephone speech, mostly no overlapping speech (also 2 speakers max), mostly no background noise (although quality is not great either), speakers are directly next to the microphone (telephone).

* You don't know the speakers beforehand, specifically you have never heard their voice or accent beforehand!

Both are crucial points.

Automatic speech recognition (ASR) models are quite good when there is not too much noise, and when there is not too much speech overlap. In this case, we usually have super human performance. But it's very different when you are somewhere in a room not directly next to the microphone, when there is some background noise, or when multiple people speak at the same time. Then we have not really reached human performance yet.

Also, the condition about not having heard the speaker beforehand is special, and maybe artificial. Test that for yourself. You will see that your recognition rate gets quite bad. But once you adapt to it, you will get much better. ASR also can do adaptation, but I think it's not as good at it, i.e. it will get some improvement, but not as much as a human.

The key word is “recognition”, not comprehension. Humans do much better with a higher error rate since they can fill in gaps through comprehension and context.

This should be the top rated comment. Moreover, once you unpack the definition of "recognition," you see that it's largely an artificial exercise. A 0.5% improvement over a human baseline is not particularly meaningful, because we're measuring a pointless task that the humans weren't optimized for in the first place.

But aren't these results saying that "AI" outperforms humans in recognition (converting to text) - even if humans can use comprehension to improve their recognition?

Humans can cope with errors in transcription/recognition by using their comprehension

What I'm curious about is the ability for the AI to cope with transcribing technical vocabulary. Even on the MIT OpenCourseWare lectures the transcription mistakes seem to be highest with unusual technical words (I usually watch them sped up with subtitles on), such as "Van-Emde-Boas tree" being transcribed as "Venom de Boss tree". Or "prologue" instead of "Prolog" in a SICP video. The paper[0] this article refers to doesn't seem to talk about this aspect.

[0] https://arxiv.org/pdf/2010.03449.pdf

For me, the common failure mode of browser-based text translation is that they try to translate names. Mark Sugar Mountain, Albert One Stone, that sort of thing.

DeepL doesn’t when entering “Mark Zuckerberg” or “Albert Einstein” alone, FWIW

I bet it doesn’t work with a Glaswegian accent, but then I guess everyone can struggle with regional dialects. I guess humans can adapt and figure out terms or call a local and ask...

Famously, voice recognition doesn't work with Scottish accents:

https://www.youtube.com/watch?v=NMS2VnDveP8

Well of course it wouldn't work if the input is vastly different from what it's been trained on. Why would it?

Because humans can do it

No we can't.

In so far as we can understand people with different accents it's because we have been trained on them. Even if they are not common around us we've had some exposure, from occasional visitors, travels, or media. When we hear an accent we've really never been exposed to we aren't likely to understand it. A good example is foreign speakers trying to speak our native language... even if they've learned our language in school for years, their even slightly off pronunciation can make it very difficult to understand what they are saying.

Indeed. Or even native speakers speaking our native language. A fond memory of mine is having to translate a Glaswegian colleague when we were on business in Kentucky. By which I mean I basically just repeated what he said in my "standard" English accent for the benefit of a couple of locals we were talking to, because their reaction when he first said something was along the lines of "Hell, I didn't understand a word he just said!"

That is a well-known effect that has nothing to do with speech recognition:

https://www.youtube.com/watch?v=oLt5qSm9U80

But computers must have access to far more than a single (fluent) person's exposure to Glaswegian/Scots.

Yes, BUT that's not taken into account in these tests. It is given beforehand what you get to train on, and shall way say "there are known problems" on this front.

So the benchmarks say how well model X does on this exact transcription taks given this exact training data, and no other knowledge.

Even basic things, like female/male voices in train vs test set don't match.

I listened to some Glaswegian on YouTube and didn't understand much.

> On conversational speech, the human error rate amounts to about 5.5%. Our system now reaches 5.0%.

I’m surprised that the human error rate is that high; is this the error rate from native to second language, or from second language to native? And what is the average second language skill level of the translators?

(My translation accuracy feels like it would be about 50-75% of words in freeform outdoors conversation at normal speed, but because of this I wouldn’t even consider a course taught in German at this time).

It looks like they used NIST's switchboard corpus, so it's an error rate for native language, and it's well-documented that it typically gets you around 5-6% errors on human transcriptions, yep.

However, it's worth remembering that this refers only to the transcription accuracy, i.e. mapping sound to a particular word in a permanent record (e.g. a written word, in a human transcriber's case). Humans can cope with error rates around 5% (and even higher) when recognizing words because we can use context to deduce the intended meaning, or ask for clarification when the word doesn't match the context. AI has some way to go until there.

This isn't to say this is useless, quite the contrary. Being able to transcribe words in conversational language is hugely useful (for both nefarious and non-nefarious purposes). It's just that it refers to understanding the words, not the sentences :).

I can absolutely believe that. Even in lab conditions, one's own personal thoughts can lead to all kinds of distractions. Even perfectly focussed minds might be biased by certain conditioning, or a susceptibility to certain words.

But I would like to see where the 5.5% figure comes from.

I'm surprised the word error rate is that low (in native speakers) - have you tried transcribing a natural conversation in your native language?

A lot of communication happens non-verbally. A lot of redundancy in language means that parsing the exact word that was said doesn't matter: the meaning is still understood.

50% might mean you are only picking up common functional words, 75% means you are mostly getting the gist of the conversation, but 95% is really plenty to follow everything that's going on.

It's strange that I've seen Google take an absolute nosedive over the past 24 months with regards to speech recognition accuracy. Particularly irritating is its new Habit of randomly Capitalizing Words that are not proper nouns. Also its refusal to voice type the word "o'clock" under any circumstances.

I have noticed this as well.

Also "Covid", for example, doesn't seem to be in the vocab.

A key problem yet to be solved is to create a standardised way to process multi-lingual speech.