News for hackers. Stuff that matters.
Re:Stanford Alpaca, and the acceleration of on-device LLM de... by karmasimida on Monday March 13, 2023 @06:16PM (#35143791)
They currently only supports a single input/response format of input right? Multi turns will be more challenging to handle.

I am optimistic for 30B or 66B to catch up with OpenAI, but 7B is unlikely to have the same quality.

Re:Stanford Alpaca, and the acceleration of on-device LLM de... by enlyth on Monday March 13, 2023 @07:06PM (#35144459)
Here is a sample chatbot style conversation I had locally with LLAMA 30B 4bit quantization on a 3090 RTX (generation speed 5 tokens per second):

https://i.imgur.com/V4lzLz7.png

It does quite well at simple reasoning. More complex stuff, it does struggle sometimes, but I am impressed by the output and the fact that it even runs on my computer.

I used the oobabooga repository on Ubuntu 22.