Re:I'm making LLMs work on small and old devices
by mrashad74 on Thursday November 12, 2026 @12:48PM
(#50471223)
This is an interesting question, so the idea is, based on experience, fine tuned small models can do narrow tasks very well (my experience running models fine-tuned on customer data as low as 1-2B is very promising) also the goal is to find a breakthrough were instead of the line of work (which I respect) that tries to squeeze much bigger models into smaller hardware like strata and maya, I'm working backward to discover/search for custom kernels and modifications till we exhaust the limit of what a smaller machine is capable of, so instead of people doing 64gb machines and old gpus to run a deepseek, I want to see what is the biggest usable qwen small model can run with reasonable speed on smaller devices, and by scaling as we discover more tricks, normal laptops can unlock more compute - for example a qwen 3.8 27B can do pretty serious work when coupled with a specialized harness - but we are yet to figure out what is the smallest machine that can run it with usable output and usable speed, and this can unlock interesting business.
[ Parent
| on HN ]
Re:I'm making LLMs work on small and old devices
by yjftsjthsd-h on Thursday November 12, 2026 @02:56PM
(#50473269)
I'm told that qwen 3.8 27B is good enough for practical programming work, so if that's doable then I could easily see the utility. Or... the biggest qwen I see here is Qwen3.5-2B; is that actually capable of programming or agentic work with code?
Re:I'm making LLMs work on small and old devices
by Miraste on Thursday November 12, 2026 @03:34PM
(#50473828)
[delayed]
Re:I'm making LLMs work on small and old devices
by NooneAtAll5 on Thursday November 12, 2026 @07:45PM
(#50476629)
> In the last month we've gotten 4B models that can do small tasks.
where can I read up on that?
Re:I'm making LLMs work on small and old devices
by mrashad74 on Friday November 13, 2026 @01:01AM
(#50478956)
Check LFM as a starting pointer : https://www.liquid.ai/models
Re:I'm making LLMs work on small and old devices
by Miraste on Friday November 13, 2026 @10:17AM
(#50484773)
The best one I know of at the moment is XHToken's Spark-x2.6 4B.
Re:I'm making LLMs work on small and old devices
by rybosome on Thursday November 12, 2026 @07:10PM
(#50476340)
I find this compelling “merely” as an engineering challenge, but I’m curious if you’ve observed an actual practical benefit?
By that I mean, have you found a case where a fine-tuned, small model is outperformed by stuffing a heavily tuned larger model in its place?
Re:I'm making LLMs work on small and old devices
by mrashad74 on Friday November 13, 2026 @06:42AM
(#50482121)
I had this case in fine tuned small models (up to 27B) in finance and legal, over bigger models (cost-wise, on-prem for a regulated institution, and better perofrmance because we owned the data pipeline)