Re:Stanford Alpaca, and the acceleration of on-device LLM de...
by yunyu on Monday March 13, 2023 @05:50PM
(#35143393)
I assume you haven't tried Alpaca (which hasn't been released), only Llama. See the instruction fine tuning section in the article.
Re:Stanford Alpaca, and the acceleration of on-device LLM de...
by karmasimida on Monday March 13, 2023 @06:16PM
(#35143791)
They currently only supports a single input/response format of input right? Multi turns will be more challenging to handle.
I am optimistic for 30B or 66B to catch up with OpenAI, but 7B is unlikely to have the same quality.
Re:Stanford Alpaca, and the acceleration of on-device LLM de...
by enlyth on Monday March 13, 2023 @07:06PM
(#35144459)
Here is a sample chatbot style conversation I had locally with LLAMA 30B 4bit quantization on a 3090 RTX (generation speed 5 tokens per second):
https://i.imgur.com/V4lzLz7.png
It does quite well at simple reasoning. More complex stuff, it does struggle sometimes, but I am impressed by the output and the fact that it even runs on my computer.
I used the oobabooga repository on Ubuntu 22.
Re:Stanford Alpaca, and the acceleration of on-device LLM de...
by plaba417 on Monday March 13, 2023 @07:06PM
(#35144465)
I have had a few questions where it gave a completely nonsensical answer, but by fiddling with the parameters, I've gotten more usable output. Have you tried lowering the noise and increasing top_p?
Edit: I just tried it with a single task of my own (that I've successfully used with ChatGPT and Bing) and it flubbed it horribly, so this model at least is noticeably inferior to the SOTA, which is not surprising given how small it is.