Shershah1024

To answer, words are not extracted in isolation. The model sees a context of about 20 words around it always.

He he. Thanks for the suggestion. Just wanted to show that an embedding model can transcribe. Maybe we should call it emscribe :D

The reason it generalizes to english now is because the reader is trained to identify English words, and it obviously understands Portuguese, so it just gave the text in English. Once I train it on multiligual vocabulary, it should be able to handle Portuguese

You can just embed those new words and they get added to vocabulary. It is fairly simple. One risk is - When a new word is very close to a known one, it tends to fall to it.

It can already classify decently well, and I tried training it as a decision model, It worked well on domains I trained it for. It was beating Liquid's small decision model, but somehow, it never generalized. Out of the trained domains, it was just above chance.

I am experimenting with the reader with more varied audio. Plus, possibly an ultra fast post processing layer.

Yes. Granite is really fast. IBM has actually been releasing some great models that are flying under the radar

That's close to what this is: the reader is a CTC head on the audio encoder, but its output words are EmbeddingGemma's own text embeddings for ~229k words, instead of a separately learned vocabulary. I also tried a plain letter-level CTC head on the encoder. It did a bit better on clean audiobooks (7.0 vs 8.2 WER at the time) but much worse on harder audio like a lecture (16.5 vs 10.8), mostly from misspellings.

Only English for now. I dont see any roadblocks for at least the major European Languages(Not sure about German though with their compound words)

Yes. Planning to train a multilingual reader. Have not done a prototype yet.Will do one once I have the right dataset

Hi all. I am Shahir. This was my experiment. First of all - Thanks to whoever posted this on HN. It came as a surprise.

Just to clarify - This started as just an experiment to see if we can use the fact that text and audio share a vector space in Embedding Gemma 2 to convert audio to text. Had to hack around a few things, but, it worked. Speed was just a side benefit. Was not aiming for it.

The accuracy is good, but not in the range of cloud models, but decent enough for everyday use, and this is just a couple of days work. The WER can improve a lot with more training on varied data. Will try that next.

I will opensource the whole thing soon. Might need a few days though. Also, releasing the TTS version for this soon.

Happy to answer any questions you have.