I'm making LLMs work on small and old devices

Re:I'm making LLMs work on small and old devices by datadrivenangel on Thursday November 12, 2026 @01:07PM (#50471513)

So does it make sense to try and run models on older 4gb GPUs? I assume the energy efficiency isn't worth it?

[ Parent | on HN ]

Re:I'm making LLMs work on small and old devices by mrashad74 on Thursday November 12, 2026 @01:11PM (#50471567)

well, the whole idea honestly is curiousity about limits of hardware more than actual application today plus the idea that in the future people can run more capable models (in terms of capability @ small parameters) on such devices.

It makes more practical sense today to just go into 4gb GPUs and many people are working on this already.. on my side of things, it's more on the exploration side for now, until we see what good models get released that can use this capability.

There are many people on github and x running decent models especally MoE models on older GPUs and it's doing nice, I'm personally tinkering with this at home.

[ Parent | on HN ]

Re:I'm making LLMs work on small and old devices by zardo on Thursday November 12, 2026 @03:17PM (#50473550)

Unless you have electric resistance heat, and it's cold.

[ Parent | on HN ]

Re:I'm making LLMs work on small and old devices by ThatPlayer on Thursday November 12, 2026 @10:16PM (#50477723)

How old is old? I keep a Gemma 4 e4b running on a Plex server's GTX 1660 (so Turing hardware). It's got 6GB of RAM though, so maybe try e2b on 4GB.

[ Parent | on HN ]