I'm making LLMs work on small and old devices

Re:I'm making LLMs work on small and old devices by cjdell on Thursday November 12, 2026 @09:13PM (#50477208)

I have a few Maxwell and Pascal series cards lying around, anywhere from 2GB to 11GB of VRAM. Have experimented with small/tiny models for agentic stuff with mixed results, however they are getting better at tool calling (before they'd just produce junk output that the harness couldn't parse).

I'm excited to see what we can squeeze out of older hardware, in the same spirit of demoscene or "will it run DOOM?".

Especially timely given current hardware scarcity. Will be watching this space.

[ Parent | on HN ]

Re:I'm making LLMs work on small and old devices by mrashad74 on Friday November 13, 2026 @01:09AM (#50479030)

if you're having some cards already try to explore what people are doing with weight streaming and model compression, I got good results on older cards with far better models than the 16B

and the new thing will be "will it run qwen" :D

[ Parent | on HN ]