News for hackers. Stuff that matters.
Re:Whisper – open source speech recognition by OpenAI by gok on Wednesday September 21, 2022 @12:57PM (#32927955)
Comparing this model's word error rates to the state of the art [1] on a few common test sets:

                           Whisper    SoTA
  LibriSpeech test-clean      2.7%     1.8%
  LibriSpeech test-other      5.6%     2.9%
  Switchboard                13.1%     4.9%
  CallHome                   15.8%     9.5%
The authors do explicitly state that they're trying to do a lot of fancy new stuff here, like be multilingual, rather than pursuing just accuracy.

[1] https://github.com/syhw/wer_are_we

Re:Whisper – open source speech recognition by OpenAI by lunixbochs on Wednesday September 21, 2022 @02:16PM (#32929029)
I suspect Whisper is more robust than other "SOTA" models, but this release is likely leaving a fair bit of accuracy on the table considering the amount of resources OpenAI is capable of throwing at training it.

Comparing the readily available test sets from the paper to some of my personal robust models (for the Talon models, this is greedy decoding, no language model):

                       Talon  Talon  Talon  Whisper  wav2vec 2.0
                       28M    300M   1B     Large    960h
    librispeech clean   3.21   2.52   2.40   2.7      2.7
    librispeech other   8.21   6.56   5.63   5.6      6.2
    common voice       13.88  11.65   8.86   9.5     29.9
    tedlium             7.51   6.55   5.47   4.0     10.5
I have a battery of more difficult tests on hand (including adversarial tests, and diverse accent-specific metrics). I'll look at running these tests on each of the Whisper model sizes and following up with a larger comparison.
Re:Whisper – open source speech recognition by OpenAI by ma2rten on Wednesday September 21, 2022 @03:44PM (#32930514)
I'm looking forward to your comparison. It's really hard to make sense of how good this model actually is without being an expert in the area.
Re:Whisper – open source speech recognition by OpenAI by lunixbochs on Tuesday September 27, 2022 @03:52PM (#33000277)
Re:Whisper – open source speech recognition by OpenAI by allanrbo on Wednesday September 21, 2022 @05:47PM (#32931962)
Talon was the first thing that came to my mind when I saw this news. Would be nice if it could benefit from Whisper. (Big fan of your work on Talon!)
Re:Whisper – open source speech recognition by OpenAI by nshm on Wednesday September 21, 2022 @03:14PM (#32930031)
It is interesting how they compare with wav2vec2 instead of nemo conformer (which is more accurate) in Table 2.
Re:Whisper – open source speech recognition by OpenAI by sjnair96 on Friday September 23, 2022 @04:49AM (#32949461)
Indeed interesting.

On that note, a core Nvidia NeMo developer I follow posted this: https://twitter.com/HaseoX94/status/1572748653189791745

He calls it a "T5 for ASR" paper :) More insights in there, have a look! Curious to see what your blog would put up as well!

Re:Whisper – open source speech recognition by OpenAI by StevenWaterman on Wednesday September 21, 2022 @01:14PM (#32928205)
One of the things they point out is that the SoTA on e.g. LibriSpeech is only good at LibriSpeech, and doesn't generalise as well.

> Because Whisper was trained on a large and diverse dataset and was not fine-tuned to any specific one, it does not beat models that specialize in LibriSpeech performance, a famously competitive benchmark in speech recognition. However, when we measure Whisper’s zero-shot performance across many diverse datasets we find it is much more robust and makes 50% fewer errors than those models.

Re:Whisper – open source speech recognition by OpenAI by lunixbochs on Wednesday September 21, 2022 @02:12PM (#32928971)
My own experience agrees: the generally available "SOTA" models are not especially robust, and can be _extremely_ bad (>50% absolute error rate) at some tasks. I'll post some preliminary numbers in a sibling comment and look into running my full set of tests on Whisper.

It looks like Whisper is probably leaving a lot of accuracy on the table, but initially it does seem to be a lot more robust than general "SOTA" models.

For a quick comparison, Silero's accuracy charts are kind of nice because they post results for a large variety of datasets. Scroll down to the EN V6 xlarge EE model (not the xlarge CE) [1]

[1] https://github.com/snakers4/silero-models/wiki/Quality-Bench...