Re:The Coming of Local LLMs
by synergy20 on Tuesday April 11, 2023 @11:59AM
(#35527036)
are we talking about training or inference for local LLM here? it's hard to do any meaningful training on the edge unless we all carry a heavy gaming pc, even that, the training quality will be subpar?
[ Parent
| on HN ]
Inference, even fine-tuning a few layers would be difficult since one needs to use non-quantized model, I'd imagine
Checkout LoRA and Alpaca LoRA and the whole huge group of people who have already figured this out. I think there was another breakthrough (yesterday?) which is a further adaption of LoRA to touch even less parameters at runtime.
DyLoRA https://news.ycombinator.com/item?id=35514228 is what you meant
Re:The Coming of Local LLMs
by whimsicalism on Tuesday April 11, 2023 @12:55PM
(#35527837)
Disagree that this is a major breakthrough :) but it is probably a marginal improvement.
Re:The Coming of Local LLMs
by whimsicalism on Tuesday April 11, 2023 @12:55PM
(#35527838)
They're not training on edge :)