Hey Christian. I've been building something similar for the last 3 months (yanahchat.com). I run 80B A3B Qwen3 Next and am able to get 5-9 tok/s on an 8gb mac. I originally downloaded 72 AIs, abliterated, think, instruct models and lots of others. I played with LMStudio first, then WebUI, then OpenClaw, then Letta. WebUI was decently fast but I wanted something more agentic but OpenClaw and Letta were crazy slow with local models. I have a MacBook m1 64gb and the biggest model I could get to run decently was a 24b. even then it was like emailing instead of chatting. I found LangChain/Smith and used that and found ways to make latency sub 1 sec. LangSMith/Chain was actually adding latency so I changed to a custom Daemon with an Sqlite memory system that the AI helped design by me asking the ai if you could design your own memory system what would you do. We calibrated it by tweaking it together. Then I started seeing SSD Streaming and I got excited. I immediately downloaded the 125B model and got it working well using Ollama.cpp. I set out like a madman to get this to work on an 8gb MacBook and came up with my own streaming architecture called MOE_Prefetch. That wasn't enough, so I took the largest model that could possibly work (the 80b A3B)and modified it to what other people are now calling 3bit (I thought I invented it but then found other people doing similar things), where we made the routers at 8 bit, the dense backbone and shared expert at 4 bit, then 2 bit on the rest of the experts. Claude and Antigravity have both tested the modification and show it retains 90-92 percent intelligence/ability of the full fp16. So that's how we got 5-9 tok/s on an 8gb MacBook.