Researchers reach human parity in conversational speech recognition

Re:Researchers reach human parity in conversational speech r... by Eridrus on Tuesday October 18, 2016 @01:44PM (#12736913)

The actual paper has a section on error analysis that is particularly enlightening: https://arxiv.org/abs/1610.05256

On the CallHome dataset humans confuse words 4.1% of the time, but delete 6.5% of words, most commonly deleting the word "I".

Their ASR system confuses 6.5% of words on this dataset, but only deletes 3.3% of words, so depending on how you view this their claim about being better than humans isn't definitely true, if you consider the task to be speech recognition, rather than transcription.

Also, while the overall "word error rate" is lower than humans it's not clear if this is because the transcription service they used is not seeking perfect output, but rather good enough output and the errors the transcription service makes may not be as bad as the errors the ASR system makes in terms of how well you can recover the original meaning from the transcription.

It's clearly great work, but reaching human parity is marketing fluff.

[ Parent | on HN ]

Re:Researchers reach human parity in conversational speech r... by xdh168 on Tuesday October 18, 2016 @10:24PM (#12740514)

The conversational speech has been very difficult. The switchboard task has been around for over 20 years. We used the exact same published data and evaluation protocols. The Human Parity claim on the switchboard task is as scientific as we can be based on the best knowledge of ours.

[ Parent | on HN ]

Re:Researchers reach human parity in conversational speech r... by derekja on Tuesday October 18, 2016 @10:45PM (#12740610)

Ah, you finally made a hackernews account! Congrats, XD, that's a great milestone you guys hit.

[ Parent | on HN ]

Re:Researchers reach human parity in conversational speech r... by joe_the_user on Tuesday October 18, 2016 @04:02PM (#12738132)

I was recently at a bar where they showed a movie with incomprehensible subtitles (English to English). I assume this was because they skimped and bought automatic subtitling.

I think one important aspect is while humans miss words, they often get the sentence meaning correct. When computers miss words, they tend to substitute words that sound similar. That's readable if you have time but not necessarily as a stream of text going by...

[ Parent | on HN ]

Re:Researchers reach human parity in conversational speech r... by ChrisClark on Tuesday October 18, 2016 @05:57PM (#12739095)

It might have also been a 'bootleg' DVD from China or downloaded a version from there. I've had quite a few where they had done a horrible job translating into Chinese for the subtitles already, and then did a literal machine translation back to English.

The character on the screen said "Hello", the English subtitle said "You good." Which would be the literal translation of nihao.

[ Parent | on HN ]

Re:Researchers reach human parity in conversational speech r... by Arkaad on Tuesday October 18, 2016 @09:14PM (#12740183)

see also the infamous Star Wars Episode 3 Chinese bootleg "Do no want".

[ Parent | on HN ]

Re:Researchers reach human parity in conversational speech r... by joe_the_user on Wednesday October 19, 2016 @02:14AM (#12741506)

It English subtitling on an movie in English, as they often have at bars.

[ Parent | on HN ]

Re:Researchers reach human parity in conversational speech r... by codekansas on Tuesday October 18, 2016 @08:40PM (#12740014)

From an ML perspective, recently there's been a lot more work done on language modeling (e.g. billion word corpus). Good language models can usually fix those types of perspective. So hopefully you can expect to see that change in the future!

[ Parent | on HN ]

Re:Researchers reach human parity in conversational speech r... by aswanson on Tuesday October 18, 2016 @09:05PM (#12740131)

Hilarious that the grandparent states performance gained by deleting "I"from the training corpus and the child comment starts with that pronoun.

[ Parent | on HN ]

Re:Researchers reach human parity in conversational speech r... by cloudjacker on Tuesday October 18, 2016 @04:36PM (#12738428)

> I was recently at a bar where they showed a movie with incomprehensible subtitles (English to English). I assume this was because they skimped and bought automatic subtitling.

I don't think that is possible. Like, worse subtitling isn't even an option.

[ Parent | on HN ]

Re:Researchers reach human parity in conversational speech r... by john_oshea on Tuesday October 18, 2016 @05:00PM (#12738666)

"Trainspotting" would be an interesting challenge: <https://www.theguardian.com/books/2008/may/31/irvinewelsh>

[ Parent | on HN ]

Re:Researchers reach human parity in conversational speech r... by WildUtah on Wednesday October 19, 2016 @11:11AM (#12744473)

Don' You Go Rounin' Roun To Re Ro:

http://www.nbc.com/saturday-night-live/video/british-movie/n...

[ Parent | on HN ]

Re:Researchers reach human parity in conversational speech r... by AnimalMuppet on Tuesday October 18, 2016 @04:01PM (#12738123)

I'm not an expert on this at all, but my suspicion is that humans are more tolerant of missing words than we are of mistaken words. Especially a word like "I", we just assume it if it's missing.

So if this speech recognition is for creating a transcript for humans, this (in my uneducated opinion) isn't as good as humans do, at least not yet.

[ Parent | on HN ]

Re:Researchers reach human parity in conversational speech r... by rhizome on Tuesday October 18, 2016 @05:18PM (#12738808)

If nothing's changed in the past 15 years, anything under 95% accuracy is not accurate enough for automation without human intervention (copy editing).

[ Parent | on HN ]

Re:Researchers reach human parity in conversational speech r... by nshm on Tuesday October 18, 2016 @05:12PM (#12738754)

From the paper eight different recognizers combined together approached results of a single person reviewed once. Not really a parity.

[ Parent | on HN ]

Re:Researchers reach human parity in conversational speech r... by rattray on Tuesday October 18, 2016 @05:28PM (#12738894)

Isn't it substantially simpler + cheaper to combine 8 computer programs than to combine >1 people?

[ Parent | on HN ]

Re:Researchers reach human parity in conversational speech r... by djsumdog on Tuesday October 18, 2016 @05:41PM (#12738994)

Looks like they used various neural networks and trained based on data they currently already transcribe professionally. So it had to be trained and only represents a subset of people who speak and have their works transcribed at Microsoft events.

Being Microsoft, I'm sure the place is diverse, but there's no mention in the paper on accents or dialects that I can see (might just be missing it).

[ Parent | on HN ]