I'm making LLMs work on small and old devices

Re:I'm making LLMs work on small and old devices by saturn8601 on Thursday November 12, 2026 @05:01PM (#50474944)

I have a basement full of Pentium 4-Core 2 Quad machines lying around. Has anyone strung together machines of this era to make something work? I always imagined that even if I were able to provide something like 4-8GB of ram per box the token speed would be measured in tok/hr or something.

[ Parent | on HN ]

Re:I'm making LLMs work on small and old devices by gabucino on Thursday November 12, 2026 @10:34PM (#50477898)

the thing that kills them is the complete lack of any kind of matmul instructions. With my PicoLM fork it's possible to "play" GPT-2-AIdungeon on an iBook PowerPC 1.2Ghz with 1GB RAM (scalar instructions, as I said), but it's only for men of the more... patient variety.

Even a Pi 3 is a no-go for any model of consequence. At the very least get a Pi 4 or old ARM phones with NEON.

[ Parent | on HN ]

Re:I'm making LLMs work on small and old devices by mrashad74 on Friday November 13, 2026 @02:37AM (#50479697)

ipad Air I tried doesn't have matmul, but there are tricks you can do to work around it actually.

[ Parent | on HN ]

Re:I'm making LLMs work on small and old devices by mrashad74 on Friday November 13, 2026 @12:55AM (#50478913)

this kind of old processors are smth I want to explore honestly, I don't know if the processor capabilities will allow smth interesting to happen or not, but they're for sure on my target list.

I think realistically smth around 5tok/sec is possible with them for 2B models or so if you provided enough ram for streaming. worth a shot

[ Parent | on HN ]