News for hackers. Stuff that matters.
Re:Flash-Moe: Running a 397B Parameter Model on a Mac with 4... by shubhamintech on Monday March 23, 2026 @01:12AM (#47485701)
4.4 tok/s with reliable structured output is a solid local benchmark altho the question is whether SSD streaming introduces per-token latency variance that messes up tool call parsing downstream. The gap between 400 GB/s unified memory bandwidth and 17.5 GB/s SSD reads means you're in the hot path pretty much every time an expert isn't cached.

There are no comments on this comment yet.