I'm making LLMs work on small and old devices

Re:I'm making LLMs work on small and old devices by mikeayles on Thursday November 12, 2026 @06:19PM (#50475819)

I was looking for the lowest claim in this thread, as I managed to get a nano LLM running in the ≈3mb of SRAM of an FPGA with zero DDR in the loop at 20k tok/s, but 2mb on a 80386, great job.

Have you got a write up for it?

[ Parent | on HN ]

Re:I'm making LLMs work on small and old devices by mrashad74 on Friday November 13, 2026 @02:47AM (#50479798)

I ran YOLO on an FPGA for my bachlor's project degree project like 7 years ago, but this is even more interesting, and I think we can push bigger models with weight streaming. I still have an FPGA lying around.

[ Parent | on HN ]

Re:I'm making LLMs work on small and old devices by mikeayles on Friday November 13, 2026 @04:49AM (#50480892)

I haven't written a new blog post for it yet but there's currently a demo live of my FPGA running bonsai 8b on the same FPGA here

https://www.mikeayles.com/blog/on-chip-llm-kv260/

Its right up against the wall of DDR bandwidth, so I'm trying every trick in the book to get the speed up

[ Parent | on HN ]

Re:I'm making LLMs work on small and old devices by mrashad74 on Friday November 13, 2026 @06:40AM (#50482100)

Decent!! Also I love the idea of the live chat, would you mind if I put a reference to it on my repo?

[ Parent | on HN ]