My girlfriend asked me if I could transcribe some audio files for her with my "programming stuff". I immediately thought of Whisper from OpenAI.

I first used the official CLI tool. With the largest model it took long 8 hours to transcribe a 30 min long file. I noticed it was running on the CPU - tried switching it to use the GPU instead with no luck. Running it on WSL was probably not helping.

Then I found this gem: https://github.com/Const-me/Whisper A C++ Windows implementation of Whisper. I opened the program, fed it with the largest model and the file. The transcript was done in 4 minutes, instead of 8 hours... Downside? The program has a GUI, lol.

Of course, I could probably get the CLI tool to run on the GPU with some tinkering and installing some Nvidia packages for Whisper to use. But frankly, I have so little experience with that kind of stuff, that installing the Windows implementation was a much easier choice.

40GB for the big llama.

Maybe not first of the class but there is a lot value to be found in the middle ground and it's good for the market as a whole.

We have almost closed the gap in no time whereas on ChatGPT day 1 it seemed they were years ahead. Hopefully there will be more models and developments.

[1] https://dev.l1x.be/posts/2023/03/12/using-llama-with-m1-mac

There are only a few hundred folks who have actual experience training at this scale.

llama is both not oss and not as good as da-vinci-003/etc. Prob best actual oss model is like flan-t5-xxl.

My biggest hope is that meta/google/anthropic actually decide to release training code + data + checkpoint. Meta is most likely to do it but still unlikely

Without that it would take eutherai/carperai to really exceed expectations for OSS to be competitive.