I'm making LLMs work on small and old devices

Re:I'm making LLMs work on small and old devices by bArray on Friday November 13, 2026 @04:33AM (#50480724)

So this work appears to work on inference speed and getting large models to work on simpler RAM constrained hardware, whereas my (private) work has been on getting simpler RAM constrained hardware to give agentic behaviour with small models. I've already started the work of using some of the learnings described in this repository. Looking forward to seeing what can be achieved.

[ Parent | on HN ]

Re:I'm making LLMs work on small and old devices by mrashad74 on Friday November 13, 2026 @04:48AM (#50480877)

I'm also working on this thread in parallel because it has interesting business aspects, would be happy to see your progress when you release it

[ Parent | on HN ]

Re:I'm making LLMs work on small and old devices by bArray on Friday November 13, 2026 @10:21AM (#50484856)

For me it's just about empowering people to jump on this LLM bandwagon with minimal resources. I'm not expecting top tier work, but I can already sent it off to check the weather on a certain day, or build a small Python GUI. There's still cases where it struggles and cannot recover - which is where I'm actively trying to improve it - without implementing some kind of case-by-case solution.

[ Parent | on HN ]