News for hackers. Stuff that matters.

A comment on MLC-LLM: GPT/Llama on consumer-class GPUs and phones. Parent comment

Re:MLC-LLM: GPT/Llama on consumer-class GPUs and phones by yieldcrv on Sunday April 30, 2023 @02:40PM (#35765354)
yeah agreed, its sad to me that we're 2 months after llama and "nobody" is seemingly doing any advances of fine tuning on models with more than 7B or 13B parameters.

I have 64gb RAM (not gpu just normal)

[ Parent | on HN ]
Re:MLC-LLM: GPT/Llama on consumer-class GPUs and phones by UncleEntity on Sunday April 30, 2023 @05:54PM (#35766942)
Probably because it’s extremely affordable to train the smaller models.

If I had the gumption (and a data set) I could afford to spend a few hundred bucks to fine tune a model for shits and giggles and I’m just a Random Internet Dude.

I’m all for it, Any Day Now™ I have this idea I want to try and having these people do all this optimization work will probably make it affordable to attempt given I don’t actually know what I’m doing so there will be a whole lot of “yeah, that doesn’t work” going on.

[ Parent | on HN ]
Re:MLC-LLM: GPT/Llama on consumer-class GPUs and phones by nullsense on Sunday April 30, 2023 @06:16PM (#35767083)
I find, since 30B models are actually quite usable locally if you have good hardware, that I really want something like a Vicuna 30B. That would be amazing. I can only run 65B locally at a speed of 1 token per second, which is too slow to be usable unfortunately.
[ Parent | on HN ]