News for hackers. Stuff that matters.
Re:Show HN: Text-to-speech and speech-to-text open-source so... by grizzles on Sunday January 26, 2020 @05:40PM (#22155344)
Facebook has released wav2letter++. I'd wager that will outperform kaldi by a wide margin.
Re:Show HN: Text-to-speech and speech-to-text open-source so... by ftreml on Sunday January 26, 2020 @05:44PM (#22155367)
in our tests the performance was comparable, so we had no reason to switch from kaldi to something else (german data only). after all, what matters the most is the available training data. are there ready trained models available for wav2letter ?
Re:Show HN: Text-to-speech and speech-to-text open-source so... by grizzles on Sunday January 26, 2020 @05:55PM (#22155428)
Oops, there I go being Anglocentric again. The datasets are out there; but idk if there are German trained models. Fortunately wav2letter++ looks like it's been built to be very friendly to train a more extensive one if there isn't. My understanding is they are still releasing parts of it.
Re:Show HN: Text-to-speech and speech-to-text open-source so... by Nimitz14 on Sunday January 26, 2020 @06:51PM (#22155732)
It is very unlikely a E2E toolkit can outperform kaldi if they're trained on the same data.

Also I suspect these guys aren't using the latest kaldi architectures.

Re:Show HN: Text-to-speech and speech-to-text open-source so... by grizzles on Sunday January 26, 2020 @10:04PM (#22156587)
What's your reasoning? Google's latest published research is analogously parsimonious with this approach. They are doing decoding using a simple beam search through a single neural network. See https://ai.googleblog.com/2019/03/an-all-neural-on-device-sp...
Re:Show HN: Text-to-speech and speech-to-text open-source so... by ftreml on Monday January 27, 2020 @12:26AM (#22157022)
very very interesting, didnt know this one. as soon as i am having some hours left will try to run some evaluation on this. after all, for my project only performance in german language counts
Re:Show HN: Text-to-speech and speech-to-text open-source so... by ftreml on Sunday January 26, 2020 @07:04PM (#22155789)
we used existing recipies for training (tuda german), slightly adapted and with additional training data. i am sure with more knowledge we would have made some extra percent ... performance is not really good, but good enough for our purposes, and with a satisfying cost vs quality trade off
Re:Show HN: Text-to-speech and speech-to-text open-source so... by Nimitz14 on Sunday January 26, 2020 @07:30PM (#22155917)
You should look at the latest recipe's for tedlium, librispeech, swbd etc you'll get significantly better results with those models.
Re:Show HN: Text-to-speech and speech-to-text open-source so... by lunixbochs on Monday January 27, 2020 @10:11AM (#22159967)
Did you miss wav2letter achieving a new 2019 SOTA on librispeech? Both projects have a _lot_ of architectures and papers in play, I know it can be hard to keep up.

Aren’t some of the peak Kaldi numbers also with _huge_ language models that aren’t so practical to deploy? The best kaldi results I’ve found here both use “fglarge” LMs, but a quick search didn’t tell me how big that actually is.

The best I can find for the test set are (from main RESULTS):

    test-clean: 4.31% WER
    test-other: 10.62% WER
From here (not sure why buried): https://github.com/kaldi-asr/kaldi/blob/master/egs/librispee...

    test-clean: 3.80% WER
    test-other: 8.76% WER
This is the 2019 wav2letter SOTA: https://github.com/facebookresearch/wav2letter/blob/master/r...

Sure, one of their techniques is to semi supervised train on 60k hours of audio, so I’ll post their best numbers both with and without that step (as you said “if kaldi is trained on the same data”)

Without 60k unsupervised audio:

    test-clean: 2.31% WER
    test-other: 5.18% WER
With 60k audio:

    test-clean: 2.03% WER
    test-other: 4.11% WER
That’s so far ahead of kaldi’s librispeech RESULTS, let’s go up the list and look at their acoustic model tests that don’t even bother with a language model:

Without 60k or language model:

    (test-clean / test-other WER)
    3.05 / 7.01
With 60k without language model:

    2.30 / 5.29
Am I missing a kaldi whitepaper here?

I found “State-of-the-Art Speech Recognition Using Multi-Stream Self-Attention With Dilated 1D Convolutions” on wer_are_we claiming 2.2 / 5.8 using Kaldi as a base, which is impressive, but they changed the network architecture and I didn’t find this in any of the open-source kaldi librispeech subdirectories. (If arbitrary model changes count, at some point you might as well treat both wav2letter and kaldi as “tensorflow with some speech helpers” because I’m sure you can implement either framework’s best neural architecture in the other framework.)

If there’s a better paper on kaldi than 2.2 / 5.8 you should comment with it and PR it to wer_are_we, because wav2letter is currently beating every other LS result posted there.

Re:Show HN: Text-to-speech and speech-to-text open-source so... by Nimitz14 on Monday January 27, 2020 @11:57AM (#22161169)
The reason those kaldi results are noticeably worse is because they're not using RNN rescoring. The E2E models also take far more compute to train (facebook and google train their models for hundreds of epochs, kaldi for... 4!). All the papers you read game results by adding unpruned ngram LMs or massive NNLMs.

Point being if you put some more effort I believe you can get better results with kaldi. Remember we're discussing with which toolkit one can achieve better performance (given a certain amount of effort), which is not the same as just taking the results from different teams some of which have spent an order of magnitude more compute in gaming the results.

For SOTA results with a hybrid approach check out RWTH's stuff. The kaldi people have recently been busy rewriting their backend to work with pytorch.

In general using librispeech to gauge performance is a bad idea. It's read speech. I prefer to base my opinions on real world datasets.

Have to say I am impressed by how good their (FB) results are without using an LM though.

Re:Show HN: Text-to-speech and speech-to-text open-source so... by lunixbochs on Monday January 27, 2020 @02:02PM (#22162602)
I agree that librispeech isn’t the best benchmark, but it’s the benchmark we have. If you have a significantly better benchmark that both wav2letter SOTA and Kaldi SOTA have participated in, I’d love to see it. Without some kind of numbers cited, it’s easier to assume you’re speculating. Especially because, as I think we both acknowledge, wav2letter requires a lot of training power to actually use, so I doubt you’ve done the training yourself in the same way you’ve worked with Kaldi to be able to talk about it authoritatively. (One of my goals with releasing w2l models is that people will be able to do e.g. transfer learning or distillation and not repeat the expensive up front training themselves)

RWTH has an impressive entry on the wer_are_we list for librispeech (2.3 / 5.0), but unfortunately I couldn’t find a way to actually use that research without a lot of work. Facebook has easy recipes posted that reproduce their results, which have been quite nice to work with. RWTH, Google, and Capio held an impressive top spot in the results for a while, which might be where you’re basing your judgement.

I don’t think the biggest benefit of wav2letter was the architecture (until their SOTA paper), I think it was the speed and simplicity. I managed to port the original wav2letter inference to numpy/pytorch in two small files (same model weights, same inference output) because it was such a simple convolutional architecture. (The only fiddly part was matching the output of their featurization code)

I agree the data and training overhead is a big aspect, which is why I’ve been training and releasing freely available wav2letter models on far more data than what Facebook has published using librispeech.

My upcoming goal is to reproduce the SOTA results but with stronger training data. My baseline data is around 4000h currently and is far more diverse than librispeech. Unfortunately I’m independent and can’t afford the LDC datasets, though I have some neat ideas for extending FB’s semi supervised work to even more data.

Re:Show HN: Text-to-speech and speech-to-text open-source so... by Nimitz14 on Monday January 27, 2020 @03:35PM (#22163502)
You are right I've never tried out wav2letter on a large amount of data.

Cool project! I'll try out your models.

Re:Show HN: Text-to-speech and speech-to-text open-source so... by ftreml on Monday January 27, 2020 @02:11PM (#22162678)
are there any wav2letter models ready for download ?
Re:Show HN: Text-to-speech and speech-to-text open-source so... by lunixbochs on Monday January 27, 2020 @02:56PM (#22163137)
Facebook posted some of the models listed in their SOTA paper already. I haven’t tried these yet.

https://github.com/facebookresearch/wav2letter/blob/master/r...

My models are here: https://talonvoice.com/research/ I haven’t yet posted the model I’ve been working on most recently. I’m 120 epochs into a large size model trained on all of my datasets. I also have another 1000-1500h of audio I haven’t finished prepping to train on.

Here is a web demo. It’s currently running a slightly older checkpoint of my WIP large model, and the deepspeech LM: https://web2letter-west-1.talonvoice.com/

Re:Show HN: Text-to-speech and speech-to-text open-source so... by ftreml on Tuesday January 28, 2020 @04:29AM (#22167798)
so this is a real cool project. as soon as you finished your training I will be happy to add it as option (or maybe default setup) in the botium speech processing setup (if you want that).

Do you have any experience with online decoding in wav2letter ? Is there something like a Websocket API available somewhere ?

Re:Show HN: Text-to-speech and speech-to-text open-source so... by lunixbochs on Tuesday January 28, 2020 @10:22AM (#22170173)
How does accuracy compare to what you’re using now?
Re:Show HN: Text-to-speech and speech-to-text open-source so... by ftreml on Tuesday January 28, 2020 @10:52AM (#22170455)
the german model is from the kaldi tuda recipe with WER of 15%.

the english is from the tedlium recipe with WER of 7%.

room for improvement, but for our original purpose it was sufficient.

Re:Show HN: Text-to-speech and speech-to-text open-source so... by dsteinman on Sunday January 26, 2020 @06:46PM (#22155709)
Any idea how it compares to Mozilla's DeepSpeech?
Re:Show HN: Text-to-speech and speech-to-text open-source so... by ftreml on Sunday January 26, 2020 @06:52PM (#22155738)
from my experience, when trained with the same data, kaldi is slightly better and with custom recipes adaptable to changing conditions. deepspeech has way better documentation and is more developer friendly. wav2letter seems to be the quickest. i guess there is no real winner here ... depends what criteria are applied.

just an example, kaldi is a weird mixture of c++, python2, python3, shell scripts, java, perl. hard to oversee. deepspeech is python. wav2letter is an exe file.

Re:Show HN: Text-to-speech and speech-to-text open-source so... by magicalhippo on Sunday January 26, 2020 @07:07PM (#22155798)
> deepspeech is python

I was under the impression DeepSpeech was native (C++), with bindings for Python and others. Personally I've used it with Node.js so far, and I couldn't see any dependencies on Python.

edit: I was talking client, you're talking training I guess.