Re:I'm making LLMs work on small and old devices
by mrashad74 on Thursday November 12, 2026 @02:27PM
(#50472845)
I would actually like to see the Wii one, I don't have one unfortunately so I can't directly try on it but this esp is interesting I would like to replicate and/or improve the results on it, this technique is similar to the streaming i did from flash but probably more efficient, I need to look it up and understand it more. Thanks for sharing it
Re:I'm making LLMs work on small and old devices
by saturn8601 on Thursday November 12, 2026 @05:09PM
(#50475028)
There is a lot of random e-waste being tossed. Think TV's, Wifi routers, IOT devices. I mean many of these should run right? WRT54G was an extremely common router so something like that? Or the old Laserjet printers that were everywhere.
Re:I'm making LLMs work on small and old devices
by mrashad74 on Friday November 13, 2026 @10:19AM
(#50484822)
Imagine your printer has it's driver failing every 2 days but thankfully it can generate YouTube videos
Re:I'm making LLMs work on small and old devices
by saturn8601 on Friday November 13, 2026 @01:20PM
(#50487554)
Well the driver issue is just begging to be fixed by AI as well and I imagine given all the posts we see of people freeing their devices, eventually that printer will have excellent rock solid drivers. I was thinking more about the times the printer is not printing which is ~99% of the time for most printers. During downtime or in the mornings, it can generate a motivational quote or poem to brighten your day and send it off via text. It could be programmed to know your habits so it can print off directions or a schedule ahead of time without being asked. Essentially whatever that compute can do to aid your life (and it can be entirely personalized to you) makes its value more than a box that sits there taking up space and not earning its keep 99% of the time. The majority of the electricity is used by the print engine and if the world is transitioning to renewable energy one day the energy cost to run its computational load could be negligible as well.
https://github.com/slvDev/esp32-ai
It shows using Per Layer Embeddings to run models that normally take 100s of MB of ram run efficiently in just a few KB of ram and keeps most of the model on disk. It’s somewhat similar to MoE so it doesn’t stream in the whole model from flash for every token, but takes it even farther. If some of these devices have 1GB of ram and a few GB of disk there might be some interesting options here.