News for hackers. Stuff that matters.

A comment on I'm making LLMs work on small and old devices. Parent comment

Re:I'm making LLMs work on small and old devices by gabucino on Thursday November 12, 2026 @10:34PM (#50477898)
the thing that kills them is the complete lack of any kind of matmul instructions. With my PicoLM fork it's possible to "play" GPT-2-AIdungeon on an iBook PowerPC 1.2Ghz with 1GB RAM (scalar instructions, as I said), but it's only for men of the more... patient variety.

Even a Pi 3 is a no-go for any model of consequence. At the very least get a Pi 4 or old ARM phones with NEON.

[ Parent | on HN ]
Re:I'm making LLMs work on small and old devices by mrashad74 on Friday November 13, 2026 @02:37AM (#50479697)
ipad Air I tried doesn't have matmul, but there are tricks you can do to work around it actually.
[ Parent | on HN ]