News for hackers. Stuff that matters.

A comment on Universal Speech Model. Parent comment

Re:Universal Speech Model by mmcwilliams on Wednesday March 29, 2023 @09:16PM (#35366422)
It's interesting because while evaluating Whisper for an ASR task I found it to have some entertaining hallucinations when provided with silent or garbled audio.

For instance, this was added to the transcription of a silent section of audio:

> Hello everyone welcome to my channel. Today Im going to show you how to make a very simple and easy recipe. I hope you enjoy the video. If you like my recipe dont forget to subscribe to my channel

It makes me wonder how much of Whisper is trained on audio from Youtube, which was transcribed by this model.

[ Parent | on HN ]
Re:Universal Speech Model by coffeebeqn on Wednesday March 29, 2023 @10:54PM (#35367297)
Don’t forget to smash that subscribe button
[ Parent | on HN ]
Re:Universal Speech Model by resoluteteeth on Wednesday March 29, 2023 @11:22PM (#35367490)
I wonder if that's specifically a result of training it on youtube videos where the audio/subtitles don't actually match (recipe videos with no speech that have additional text added in the subtitles)
[ Parent | on HN ]
Re:Universal Speech Model by nodja on Thursday March 30, 2023 @12:37AM (#35368040)
Similar experience, mine would turn background noises when no one's talking into random japanese words repeated. I was using the large model. I ended up fixing it by using the medium.en model and setting condition_on_previous_text to false.

Now if only the timestamp timings were correct for noisy audio... I've tried stable whisper and another fork I forget the name, but I need to run the audio through RTX voice if I want consistent timestamps...

[ Parent | on HN ]